Ask your content (RAG)

Kiln serves GET /api/ask?q=… — a retrieval-augmented "ask your content" endpoint over published content (issue #339). It finds the passages most relevant to a question and returns them as cited sources, and — when a generator is configured — a synthesized answer grounded in those sources.

Response

{
  "question": "world",
  "answer": null,
  "generated": false,
  "generation": "disabled",
  "retry_after": null,
  "sources": [
    { "type": "page", "title": "Welcome to KilnCMS", "url": "/welcome",
      "excerpt": "Welcome to KilnCMS. A world-class, Elixir-native headless CMS built on Phoenix and Ash …",
      "score": 0.0328, "legs": ["keyword", "semantic"] },
    { "type": "post", "title": "Hello, World", "url": "/blog/hello-world",
      "excerpt": "The first post on a KilnCMS-powered site. Posts live under /blog and …",
      "score": 0.0164, "legs": ["keyword"] }
  ]
}

Parameters: q (the question), optional locale and limit (max sources, clamped to 12).

sources are in relevance order across every content type: each hit's score is the fused Reciprocal Rank Fusion score it was ranked by (the reranker's score instead when reranking is enabled, or the blend of the two ranks under rrf — see Replace or blend), and the scores are comparable across types because every section of the sweep shares the same k and leg weights. legs names which of keyword, keyword_any, semantic, block, title, alias, fuzzy and tag returned the hit — a keyword-and-semantic hit is a stronger claim than a fuzzy-only one, and a tag hit (a record carrying a tag whose name the question is near) is a topic match, the weakest claim of all; block is the embedding leg at block grain, a record reached through its nearest section rather than its opening (what a question about a section deep in a long monograph needs); an alias hit is a record the question names by one of its other names — a custom field an admin flagged "names the record", such as a Latin binomial or a pinyin spelling; a title hit is a record the question names outright (its title appears in the question, stemmed and at word boundaries); keyword (every query term matched) is a stronger claim than keyword_any (the relaxed leg that joins when the every-term match comes up short on a multi-word query: some of the terms matched). Both are additive; a client reading only title/url/excerpt needs no update. They exist so a client can threshold, debug a ranking, or build an evaluation set against the public API rather than the internals.

excerpt is a grounding passage, not the search page's snippet: up to three fragments of 40 words around the matches, with no <mark> tags, and — when the matches are only in the title and headings, which is exactly what a question naming the record produces — the document's opening ~300 characters instead. The search page's 18-word highlight stripped of its tags used to be cited here, and on a question about two ingredients it grounded the generator on "Star Anise Flavour Profile Illicium Verum"; a well-behaved generator then truthfully answers that its sources say nothing.

Why there is no generated answer

The endpoint always answers 200: when generation does not run you still get the cited sources, which are a useful answer on their own, and a 429 for a partial result would be worse. generated: false therefore covers several situations that a client should treat differently, so generation names which one it was:

retry_after is a whole number of seconds, rounded up (never 0), and is set only for "rate_limited". It is a body field rather than a Retry-After header because a header describes the whole response, and this response succeeded — only the generation part of it was throttled. Telling a client to retry the request would throw away the sources it just got. Kiln reserves Retry-After for genuine refusals (see api.md), where retrying is the instruction.

These fields are additive. question, answer, generated and sources are unchanged, so a client reading only those needs no update. generated stays for compatibility and is exactly generation == null; new clients should branch on generation, which says why, rather than on generated, which only says whether.

Anonymous callers are told about "rate_limited" too, and that is a decision rather than an oversight. The per-caller budget keys on the caller's address — mostly their own traffic reflected back, though everyone behind one NAT or proxy egress shares a key. The per-org budget is shared by construction, and retry_after gives away which of the two it was: the windows differ (a minute per caller, an hour per org by default), so a value above the per-caller window can only have come from the shared one.

Reported anyway, because the pipeline's own per-IP limiter already answers 429 to the same anonymous caller and discloses more, and because withholding it would leave an operator unable to tell "off by configuration" from "off because the shared bucket is spent" — the case they most need to diagnose. Clamping retry_after to the per-caller window would hide the distinction at the cost of telling a client to retry in a minute when the real wait is most of an hour, which is worse than the disclosure.

How it works

• Retrieval reuses KilnCMS.Search.global/2 — the same keyword + semantic RRF hybrid search behind /api/search, reranked when a switch says so (see Reranking ask sources). It degrades to keyword when semantic search is disabled, so /api/ask works with no model stack; turning on semantic search (config :kiln_cms, KilnCMS.Search, semantic: true) improves retrieval quality automatically.
• Long records are seen whole only with chunking on. A record's vector covers the first 512 tokens of its text, so a question about a passage deep in a long record retrieves it only through a keyword match or one of its blocks. chunk_long_documents: true embeds the rest in overlapping windows (about 2× the embedding work for long records; backfill with mix kiln.embed_all) — see semantic-search-plan.md.
• Selection is by score, across types. Every section comes back ranked within its type and every hit carries its fused score (KilnCMS.Search.hit_score/1); the sources are the top limit of all sections sorted together. They used to be the sections flattened in registry order — sorted by type label — so every hit of an alphabetically earlier type outranked every hit of a later one, however weak. Ties keep the registry order.
• Policy-scoped: an anonymous request only ever sees published, world-readable content (the same read policies as every headless surface), so drafts and gated content can never leak into an answer or a citation. A bearer token widens visibility like other headless endpoints.
• Generation is off by default. With no model configured /api/ask returns retrieval-only (answer: null, generated: false, generation: "disabled") and nothing leaves the deployment.

Measuring retrieval

mix kiln.search.eval golden.json --ask scores the order sources come in against a golden set of questions and the slugs that should answer them — recall@k and MRR per query class, and the rank every expected record landed at. Run it before and after any change to the search stack (a floor, a leg, a reranker, a model), and with --url https://your-site --ask against the deployment itself, where it reads the score/legs fields above. The format, the classes and how to write a set for your own corpus are in search-roadmap.md, §11 Ranking eval harness.

A number you will compare against another run wants --ef-search 200 or --exact (or KILN_HNSW_EF_SEARCH=200 / KILN_VECTOR_SEARCH_EXACT=true on the server a --url run reads): at pgvector's default ef_search the semantic legs' recall is approximate, and the report's header says which mode it ran under. See semantic-search-plan.md.

Reranking ask sources

Kiln ships a cross-encoder reranker — BAAI/bge-reranker-base behind a Bumblebee Nx.Serving, see Phase 3 — Hybrid fusion (+ optional rerank) in semantic-search-plan.md — that rescores every fused candidate against the query and reorders each section by that score. It is off by default, and until now it had a single switch, config :kiln_cms, KilnCMS.Search, rerank: true, which reranks every search surface on every query: the public /search page, the editor palette on each keystroke, /api/search, and /api/ask. That is CPU inference a modest host cannot afford, which is how a deployment came to have the reranker wired and never running.

/api/ask is different. A question is one bounded call, and since the sources are already sorted by score across types, reranking each section's candidates with the same model for the same question makes that sort a rerank of the whole union — at most limit candidates per registered content type through the model, per question (or the merged union in one call, see Rerank the union once). So the ask path has its own switch:

ASK_RERANK=true

or

config :kiln_cms, KilnCMS.Ask, rerank: true

KilnCMS.Ask.rerank?() reports the effective setting — it is also true under the global switch, which covers this surface with the rest. Either switch loads the model at boot; a default install loads nothing. With it on, each source's score is the reranker's rather than the fused one, so the number still agrees with the order it came in. That score is the cross-encoder's relevance as a sigmoid between 0 and 1, so it compares across content types. (Before this was fixed, softmax over the model's single logit made every score exactly 1.0. Reranking then reordered nothing, and the flat sort fell back to registry order.)

The model has to be loadable at boot: in the image's Bumblebee cache, or downloadable. If it is not (an offline image that never baked it, or a misspelt rerank_model), Kiln starts without a reranker rather than failing to boot. It reports the problem once through KilnCMS.Config.Report (the log and Sentry), and every surface keeps its fused order, exactly as when reranking is off. KilnCMS.Search.RerankerServing.status/0 says which state a node is in (:off, :running or {:unavailable, reason}), and the adapter warns at most once a minute while reranking requests fail.

Two caveats before setting it, both from the report that asked for this (a production deployment's search-ranking report, 2026-09-04), quoted as written:

Two honest caveats: reranking fixes ordering, not the recall defects (D2/D3 must be fixed first or there is nothing right to reorder), and CPU inference cost needs documenting for modest hosts — ours currently runs on hardware without AVX2, so this stays default-off with a clear knob.

In Kiln's terms: a reranker can only promote a record the fused legs returned. If the keyword leg's implicit AND or the semantic distance floor dropped the right record before fusion, no reranker will see it — fix recall first (the exact-title leg, the re-measured floor and the OR-relaxation fallback proposed alongside this), then turn this on. And the cost is yours to measure: the cross-encoder runs on the CPU through EXLA, whose kernels assume AVX2, and the per-question price on a host without it is what kept the report's deployment from enabling reranking at all. Try a question against a staging copy with the switch on before exposing it on a public endpoint.

What the reranker reads

The cross-encoder scores the question against the record's title and its search_text — the plain text the keyword leg matches and the embedding is computed over — capped at eight characters per token of the compiled window (4096 at 512); the tokenizer cuts the pair at the window. It falls back to the passage calc, then the excerpt, then the title alone. It used to read the title and the excerpt, and a content type whose substance lives in searchable custom fields has no excerpt: the model judged "intensely bitter and cold" against the title "Huang Lian" alone (logit −8.1; +4.7 with its text). On the golden set behind #1938, reranking on title + excerpt cut r@10 from the fused 0.851 to 0.667; title + search_text held 0.808 and lifted MRR above fused. A caller that narrows hybrid/3's :select (GET /api/search does) has the select widened to these fields whenever it reranks.

The compiled shape

Every rerank call pads to the serving's compiled shape, so that shape — not the number of candidates — is what a call costs. It is configurable, like the embedder's:

config :kiln_cms, KilnCMS.Search,
  rerank_batch_size: 8,             # the default
  rerank_sequence_length: 512       # the default; a list compiles buckets, e.g. [128, 256, 512]

Eight pairs, measured on an M5 laptop CPU (torch, 6 threads):

A licence note before choosing a smaller model. The ms-marco and mmarco MiniLM cross-encoders carry Apache-2.0 model cards, but they are trained on MS MARCO, whose terms restrict it to non-commercial research use. Whether that reaches the weights is a question for each deployment to answer for itself. The default, bge-reranker-base, is MIT-licensed and does not name its training data.

Replace or blend

rerank_mode (SEARCH_RERANK_MODE; KilnCMS.Ask's own rerank_mode / ASK_RERANK_MODE for this path) says how the reranker's order meets the fused one:

• :replace (the default) — the reranker's order outright; each hit's score is the reranker's 0–1 relevance.
• {:rrf, k} (rrf, rrf:<k>) — Reciprocal Rank Fusion of the two orders: 1/(k + fused rank) + 1/(k + rerank rank), ranks 1-based, an exact tie broken by fused order. Replacing outright regresses what fusion already gets right: on the golden set, question-form MRR fell from 0.672 to 0.388, because the cross-encoder prefers a record that answers the question over the record the question names.

A blended hit's score is its blend — the number its order came from, between 2/(k + n) and 2/(k + 1) (at most 0.0328 at k = 60). It is built from ranks within the list that was reranked, so it compares across that list and nothing else. In the union scope below that list is every content type at once, so sources compare exactly. Per section, a section's first hit carries nearly the same blend as every other section's first hit, however different their relevance, so the flat sort interleaves the sections rank by rank. The reranker's own score would compare across sections, but then the number would no longer be the one the order came from, and a client sorting or thresholding on it would quietly undo the blend.

Rerank the union once

ASK_RERANK_SCOPE=union, or rerank_scope: :union on KilnCMS.Ask. By default (rerank_scope: :section) each content type's candidates are reranked on their own, one call per non-empty section, and the flat sort above merges them. With rerank_scope: :union, no section reranks. The sections are merged by fused score, the top rerank_union (default 30) of that merge are reranked in one call, and the sources are taken from the result in its order. That costs one padded call per question instead of one per section, and makes a blend meaningful: the fused rank {:rrf, k} reads is the candidate's place in the merge across every type.

If that one call fails, the union keeps its fused order and fused scores. This removes the per-section path's rough edge. There, if one section's call failed while another's succeeded, the failed section kept its RRF scores (around 0.01–0.05) beside the others' 0–1 reranker scores, and the flat sort ranked every reranked source above every fused one. That edge remains in the section scope.

rerank_pin: true (ASK_RERANK_PIN, union scope only) adds the named-entity pin preregistered in #1937: a source whose legs include title or alias, one the question names, keeps its fused position. The pinned sources are placed first, at their fused ranks, and every other source fills the remaining positions in reranked (or blended) order. Under the pin the scores need not descend. A pinned source holds its slot whatever its own score says, so read the order, not the numbers.

The default stays section with replace: adopting another variant waits on the held-out real-traffic measurement #1937 preregistered.

Measuring each variant

mix kiln.search.eval golden.json --ask runs /api/ask's retrieval in-process, under whatever KilnCMS.Ask and KilnCMS.Search config the run has. Each variant is therefore a config change, not a flag. Use --limit 6 for /api/ask's own per-section depth, which is the depth #1937 measures at. Reranking must be on (rerank: true on KilnCMS.Ask) with the model loadable.

The environment variables are read by the production runtime config. To score a deployment running a variant, use mix kiln.search.eval golden.json --ask --url https://your-site. For an in-process run in dev, set the keys in config/dev.exs or a project overlay.

Turning on generated answers

Set one environment variable to a req_llm model spec:

ASK_MODEL=ollama:llama3.1

or configure it directly:

config :kiln_cms, KilnCMS.Ask,
  generator: KilnCMS.Ask.Generator.ReqLLM,
  model: "ollama:llama3.1"

KilnCMS.Ask.enabled?() tells you whether it is on (KilnCMS.Ask.rerank?() is the same question for the reranker above).

On-prem (strongly recommended here)

req_llm carries ollama and vllm providers, and every provider's base_url is overridable, so pointing Kiln at a model running inside your own network needs no Kiln code. KilnCMS.Ask.egress?/0 reports whether content actually leaves the deployment, resolved from the endpoint host rather than the provider name — naming a provider ollama while pointing its base_url at a rented GPU box is egress, and reporting it as local would be a lie. The same classifier (KilnCMS.LLM) backs SEO drafting and block assist.

A hosted provider

ASK_MODEL=anthropic:claude-sonnet-5
ANTHROPIC_API_KEY=...

Kiln never reads or stores the operator's provider API keys; req_llm resolves them from its own environment. A hosted provider logs a warning at boot.

A site admin can give the site its own provider, key and answer model at /editor/site-ai. A site whose own provider is set but unusable gets "generation": "failed", never an answer from the operator's provider. See Per-site provider.

ASK_GENERATOR overrides the adapter module if you have written your own.

Why this switch deserves more thought than the other two

SEO_MODEL and ASSIST_MODEL are triggered by an authenticated editor clicking a control. ASK_MODEL is triggered by strangers: /api/ask is public and anonymous, so enabling it against a hosted provider means anyone on the internet can cause published passages to be sent there. Two things bound that:

• Only published, world-readable content is ever retrieved, so no draft can reach the model whatever the setting — the policy scoping above is upstream of the generator, not a check the generator performs.

  This holds for every caller. Unlike the other headless read surfaces, /api/ask does not widen for a bearer token: retrieval runs with no actor at all, so an editor's or admin's credential retrieves exactly what a stranger retrieves. It used to forward the caller as the :actor, and because Content's read policy sits behind an OrgAdmin bypass, that shipped drafts and member-gated records to the provider (#916). Searching drafts is what /editor/search and the editor palette are for.

• Generation carries its own budget (KilnCMS.LLM.Budget) on top of the pipeline's per-IP limiter, which allows 120 requests a minute — a fine ceiling for a search query and an absurd one for model inference. The per-caller bucket keys on the signed-in user when there is one and on the client address otherwise — a rate-limiting identity only, which never widens what is retrieved. Exhausting a bucket degrades to retrieval-only; it never fails the request, because the cited sources are still a useful answer.

Every other failure degrades the same way: an unset model, an unreachable endpoint, a generator that raises or exits, an empty or unparsable response. The shape of the response is identical to a default install's — same 200, same answer: null, same sources — but it is no longer indistinguishable: a configured-but-broken generator reports generation: "failed" where a default install reports "disabled". That difference is the point of the field, and it is what tells an operator a transient outage from a switch they never flipped.

Writing your own generator

Implement KilnCMS.Ask.Generator and point generator: at it. c:KilnCMS.Ask.Generator.generate/2 is the required callback and receives the question plus the retrieved sources; the optional c:KilnCMS.Ask.Generator.generate/3 is preferred when exported and additionally carries :locale — the content locale the answer should be written in. Whatever it returns is trimmed and length-capped by KilnCMS.Ask before it reaches a caller.

Content intelligence (#339 phase 2, shipped)

KilnCMS.Search.Related, built purely on the existing block embeddings (D16) — no new model, no egress; every function degrades to empty results when semantic search is off:

• Related content — related_documents/2, and public delivery at GET /api/content/:type/:slug/related (published-only on both ends, org-scoped, cacheable): nearest foreign block embeddings to the document's centroid, aggregated per document by minimum cosine distance.

• Near-duplicates — near_duplicates/2: documents within a cosine distance threshold (:near_duplicate_threshold, default 0.1), any workflow state — catches a draft duplicating live content.

• Tag suggestions — suggest_tags/2: existing tags ranked by similarity to the document's centroid, minus the ones already applied, and minus anything past a cosine-distance ceiling (:suggest_tags_threshold, default 0.35). The ceiling is not optional decoration: the candidate set is the site's whole tag list, so ranking alone meant a five-tag site suggested all five tags for every document (#851). Returning nothing is the right answer when nothing is close. Tune it for your embedder — measure with suggest_tags(record, threshold: 2.0) (the ceiling of cosine distance, so nothing is filtered) and read the distances off the result.

  Tag-name vectors are persisted (KilnCMS.Search.TagEmbedding, tag_embeddings, #1085): a name is stable and its vector is a pure function of it, so it is embedded once — lazily, by the first suggest_tags/2 call that needs it, which is also where the embedding budget is charged — and re-embedded only if the tag is renamed (the row stores the name it was computed for). From then on the ceiling and the ranking are one exact pgvector query (ORDER BY embedding <=> centroid … WHERE distance <= ceiling LIMIT n) rather than a vector lookup and a cosine computation per unapplied tag per call — the same shape BlockEmbedding.nearest_to_vector has for blocks, minus the HNSW index: a taxonomy is hundreds of rows, and an exact scan sidesteps the post-filter recall trap (#998).

The two thresholds are measured, and one of them is a judgement call

Both defaults above are properties of BAAI/bge-small-en-v1.5, and #1086 measured them rather than deriving them. Change the model and you should re-measure; the harness is in test/kiln_cms/search/tag_suggestion_calibration_test.exs, behind mix test --include calibration, and the corpus and recorded numbers are KilnCMS.TagSuggestionCorpus.

Method. Eight documents on unrelated subjects, thirty-five tags, and a human label on every one of the 280 pairs: would a person tick this tag for this document? Each document's vector is the mean of its per-block embeddings, which is what Related.centroid/1 computes from stored BlockEmbedding rows; each tag's is its name embedded as a document. Every document is scored against the whole vocabulary, because that is the real regime. The vocabulary includes near-misses on purpose — fermentation beside a cold-brew article, sql beside a maths one — since "carburetors" on a cold-brew article is not the failure an editor actually meets.

Tag suggestions (:suggest_tags_threshold).

The bands overlap, and that is the finding: no ceiling keeps every wanted tag and admits no unwanted one.

0.35, because the two errors are not symmetric. A panel with one odd suggestion beside three good ones is a picker an editor uses; an empty panel reads as a broken feature, and suggest_tags/2 takes only the five closest, so a stray admission is bounded while a missing good one is simply gone. 0.40 and up hands the filtering to that limit: 5 and leaves the ceiling doing nothing.

This also corrects the reasoning behind the number #851 shipped. That one came from bge-small's published behaviour on sentence pairs — unrelated around 0.6–0.8 similarity, so 0.2–0.4 in distance — and #1086 predicted the band might not transfer to a one-word tag label against a whole-document centroid. It does not: measured, an unrelated tag sits at 0.35 and up. Reasoning from the wrong band produced 0.25, which keeps 3 of 27 wanted tags — the feature shipped inert.

Near-duplicates (:near_duplicate_threshold). Measured on the axis this one actually compares, document centroid against document centroid, which behaves nothing like the above:

0.1 sits in the gap with room on both sides, which is exactly what this feature needs: "the same article rewritten" is a duplicate worth flagging, "another article about sourdough" is not. Unchanged in value, but it is a config key now rather than a literal — the number is a property of the model, and an operator who changes the model had no way to change it.

• Content gaps — content_gaps/2: recorded zero-result search queries (the search-analytics log), most-searched first — what readers looked for and didn't get. The one function here that needs no embeddings, so it works on a keyword-only deployment too.

The vector primitive behind them is BlockEmbedding's :nearest_to_vector read (nearest neighbours of an already-computed vector, self-excluded).

Where an editor sees them

• "Similar content" — a section in the content editor's Settings rail. Loaded on an explicit click, never on mount: it costs a pgvector query, a record read per neighbour, and one embedding per unapplied tag name. It shows near-duplicates (each linking to the other document, in a new tab, so unsaved edits survive) and suggested tags, which attach on one click. A suggested tag is ticked in the tag picker, not written to the record — it saves with everything else the author is editing, and unticking the checkbox undoes it.
• "Content gaps" — a section on /editor/analytics, above "Most viewed". Not windowed by the dashboard's ?range=: a gap is a standing absence, and the counter table keeps a running total per query rather than per-day buckets. Its own retention window (SearchQuery's nightly purge) bounds it instead.
• Automation rules — the :flag_duplicates, :suggest_tags, :suggest_links and :suggest_metadata reactions email findings on an event such as "moved to in review" (#377). All four suggest and never write; see Editorial automation for why that boundary is the point.

Both editor surfaces need this document's own block embeddings, which are written by firing, and firing runs on publish. A never-published draft therefore has nothing to compare; the panel says so rather than reporting "nothing similar".