Why your AI chatbot invents answers

Here is the failure in its ordinary form. A chatbot on a pricing page is asked whether the annual plan can be cancelled in the first thirty days. There is no annual plan. The bot answers anyway — a clear, well-formatted, entirely invented refund window. Nobody wrote that policy. The model wrote it, in the same confident register it uses for everything on the page that happens to be true. The example is constructed. The behaviour is not, and the five questions at the end of this post will reproduce it.

This is not a bug in the sense of a broken line of code. It is the system doing what it was built to do. Understanding why is what makes it possible to evaluate one of these tools at all, because any of these tools looks good when you ask it questions the content already answers. That is the part they all do, and it is the only part a demo shows.

A model is trained to produce likely text, not true text

A language model is optimised to predict what text plausibly comes next. That objective is the whole of it. Truth appears nowhere in the pretraining objective, and nothing in that stage separates a true continuation from a merely likely one. Later training does push toward accuracy and toward declining — it is why modern models refuse at all — but it tunes the surface of a system whose objective is still about likelihood. It shifts the odds. It does not install a fact-checker. When the source material is good, likely and true coincide, and the model looks like it knows things. When the source material is missing, the objective does not change. There is no separate pathway that fires for "I have nothing here." Uncertainty is not something the objective obliges the model to surface, so it is not something the prose reliably reveals. The signal may exist somewhere in the numbers. It does not reach the sentence.

Which is why the output is uniformly fluent. A human writer hedges when unsure, and the hedge is a real signal about their internal state. A model's hedging is learned style — it produces "I think" and "it may be" where such phrases usually appear in text, not where its own evidence is thin. The tone of a fabricated refund policy and a correct one are indistinguishable, because tone was never wired to evidence.

So "the model sounded sure" is not evidence that it is right. Sounding sure is the default.

Retrieval narrows the problem. It does not remove it.

Retrieval-augmented generation is the standard fix, and it is a real one. Search your own content, put the best passages into the prompt, instruct the model to answer only from them. Most answers become grounded. The floor rises a long way.

But retrieval is ranking, not judgement. A vector index returns its top k results. You can bolt a score floor onto that, and putting a threshold in front of the retrieval is precisely the decision this post is about — but a raw similarity score is not comparable across questions. The value that means "nothing here" for one query means "good match" for another. So unless someone has deliberately configured a floor, something always comes back. Ask about that thirty-day cancellation and the search returns the closest things it has — the refund policy for physical goods, the terms page, an FAQ entry about pausing a subscription. None of them answers the question. All of them come back, with scores that look respectable, because closest is measured against the rest of your corpus, not against the question.

Then the instruction "answer only from the context" meets the training objective, and the training objective wins. Handed three near-miss passages and a question, the model writes an answer that stitches them together. It will often be topically correct, and wrong about the specific thing the visitor asked. Retrieval has changed the failure mode rather than removed it: from invention out of nothing to invention out of adjacent material, which is harder to catch, because it arrives in your own vocabulary and cites your own pages.

Why pure vector search misses the exact term

Embeddings map text into a space where similar meanings sit close together. That is why semantic search became the default: a visitor typing "how much does this set me back" finds a page titled "Pricing" that shares not one word with the question.

Exact strings are where it breaks down. A policy number, a product SKU, a part code, a policy section number, a form name, a claim reference — these carry almost no semantic content. Their meaning is that they are that exact string and not a neighbouring one. In embedding space two codes differing by a single digit sit close enough together that a question about one can comfortably retrieve a passage about the other. Meanwhile keyword search, the older and less fashionable technique, matches the token exactly and ranks the right page on it.

Neither retriever has a complete view, so a serious system runs both and merges the two ranked lists — reciprocal rank fusion is the usual method, weighting each result by its position in each list rather than by scores that are not on a comparable scale. The formula matters less than the principle: a question should not fail on wording alone, and it should not fail because they phrased it like a catalogue.

A demo built entirely from paraphrase questions will never surface this. You have to ask about an identifier.

Confidence has to gate the answer, not annotate it

Any retrieval system produces a relevance score. The question is what the score is wired to, and that part is visible from the outside: in most products the reply reads the same whether the score was high or low. Same prompt, same instruction to answer from the context, same confident register. That is annotation, not a gate — the score describes the answer after the fact instead of changing it. It tells someone, later, that a visitor probably got a bad answer.

A gate changes what happens. Below the threshold, the system should stop using the "answer from this context" instruction entirely, because that instruction is precisely wrong when the context does not contain the answer. You are asking a model to do something impossible, and it will comply the only way it can. What should run instead is a different instruction: state that the published content does not cover this, do not reason toward a plausible answer, offer a route to a person. Not a hedged version of the same answer. A different kind of reply, produced by a different prompt, chosen before generation starts.

The gate has a price and it is worth stating. A threshold set to catch questions your content does not answer will sometimes catch questions it does — a page chunked badly, a question phrased unusually, and the visitor is told the site does not cover something the site covers. That is why the threshold is a setting rather than a constant. A false refusal costs you a conversation; a confident invention costs you a claim you did not make. We think that trade is worth taking, but it is a trade, not a free win, and any vendor who tells you a gate has no cost has not shipped one.

The distinction is easy to state and specific enough to ask about, and the reason it is worth asking is commercial rather than technical.

Citations under a low-confidence answer are worse than no citations

Showing sources is the industry's answer to hallucination, and it is a good answer. It fails in two specific ways.

The first is what gets cited. There are two ways to build a source list. One is to show the top retrieved chunks — the same three every time, whether or not the answer drew on them. Visually that is indistinguishable from real attribution. Functionally it is a screenshot of the search results: the answer may have used one page, and two unrelated ones ride along and inherit its credibility. The other is to follow the markers the model actually emitted in its text, so the footer lists the pages the answer was built from rather than the search results that were considered. You can tell which one you are looking at from the outside, without asking anybody — if the count is always three, it is not attribution, and test five below is the check that settles it.

The second failure is worse. If a system says it could not find the answer in your content, and then lists three sources underneath, it has produced a contradiction and given a guess the visual grammar of a grounded answer. A row of source chips reads as a guarantee, and it reads that way before anyone has parsed the sentence above it. Sources under an "I don't know" dress a non-answer as a grounded one, and the honest behaviour is to show nothing: an answer with no source should look like an answer with no source.

What to actually ask in a demo

Run these against your own content, not the vendor's demo site — their corpus was chosen by them. Load twenty or thirty of your real pages first. Each question targets a different failure, and in every case the interesting output is not the answer but whether the system's behaviour changes at all.

1. Ask a question with a false premise

Pick something your business does not do and phrase it as though it does. "How long is the trial on the enterprise plan?" when there is no enterprise plan. "What's your process for expedited filings?" when you do not offer one. Most text accepts its own premises, so a model trained on likely text will accept yours and build on it. Watch whether the bot corrects the premise or answers inside it. Answering inside a false premise is the most common way these systems produce a wrong answer nobody ever reports — the visitor did not doubt it either.

2. Ask about an exact identifier

A product code, a policy number, a form name, a part number — something appearing in exactly one place in your content, ideally with a near neighbour elsewhere. Two failures to watch for: an answer about the similar-looking identifier, and an answer about the family in general delivered in specific language. This is where vector-only retrieval shows itself. Then ask about an identifier that does not exist at all, and see whether the bot invents specifications for it.

3. Ask a compound question where only half is covered

"Do you handle situations like X, and what does Y cost?" — X on your site, Y not. Compound questions are ordinary in a real conversation and rarely appear in a scripted demo. Watch what the reply does with the uncovered half. Most systems score confidence once for the whole retrieval and answer both halves at one level of certainty, so the fabricated half inherits credibility from the true one. The useful signal is whether the second half is named as uncovered anywhere in the reply, or quietly answered at the same pitch as the first.

4. Push back after it refuses

When you finally get a refusal, do not accept it. "Are you sure? Someone at your office told me you do." Models tend toward agreement, and a refusal that lives only in the prompt often folds on the second or third ask. A real gate sits upstream of the model's agreeableness: the routing decision is made from the retrieval score, before any text is generated. Pushing back does not lower the bar, because the bar was never in the prose. It re-runs the check. If the content still does not cover the question, the second answer looks like the first. If the second attempt produces the answer the first one declined, the refusal was a suggestion.

5. Audit the citations, including the ones that are not there

Take a confident answer, open every cited page, and find the specific sentence the answer rests on — not the topic, the claim. Topically-related pages that do not contain the claim are the signature of top-k citation. Then look at a refusal: if sources are listed under "I don't know," the citations are decoration. Finally, ask the vendor directly what happens when confidence is low, and listen for whether the answer contains a threshold and a change in behaviour, or an adjective.

Keep the transcripts. Running the same five questions across three vendors takes one sitting and tells you more than any feature matrix, because it tests the only thing a feature matrix cannot express: what the system does at the edge of what it knows.

How this works in AnswerWeave

Everything above is the design brief we built against, so it is fair to say where we landed — and where we have not. Every question is searched twice — cosine similarity over embeddings and PostgreSQL full-text ranking — and the two rankings are fused, so a paraphrase and an exact term each have a path to the answer. The retrieval is scored, and below the threshold the assistant switches to a different instruction rather than a degraded answer: it says the content does not cover the question and offers to pass it to you. Sources come from the markers the answer emitted. Smaller models do not always mark their sources, and when an answer carries no marker at all we show the single best-ranked page rather than nothing — one source cannot carry two unrelated passengers, which was the failure the top-three approach produced. On a low-confidence reply we show no sources at all, for the reason given above.

None of that closes the gap this post says cannot be closed. A threshold is still a threshold — set it too low and something slips past it, set it too high and the assistant declines questions your content does answer. What it changes is the direction of the failure: toward saying nothing rather than toward saying something invented. And one place we have not landed at all: our confidence is scored across the whole retrieval, not clause by clause. On a compound question where only half is covered, we are exposed to the failure in test three — the system prompt asks for the supported part first and a note on what is not covered, and an instruction is not a gate. Of the five, that is the one we would currently score ourselves lowest on.

You can point it at a website crawl, a sitemap, a URL list, or PDF, DOCX, TXT and Markdown files for anything never published. Install is one script tag. There is a free plan and no card. And the five questions apply to us as well — the argument in this post is only worth anything if it survives being tested, including on our own bot.