Why support bots make things up, and how to make one say “I don’t know”

What making things up looks like

Say a visitor asks whether refunds on Apple Pay orders take the usual fourteen days. Your help pages say two true things. You accept Apple Pay, and refunds arrive within fourteen days. A bot that is allowed to reason across its material will happily join those two facts into a third one, and tell the visitor that yes, Apple Pay refunds take fourteen days. It will say it with the same confidence as the two facts it was actually given, even though nobody on your team ever wrote that sentence. If your payment provider handles Apple Pay refunds differently, your own website has just told a customer something false.

The word for that third sentence is a hallucination, and the tricky part is that it sounds exactly like a real answer. The same process produced both, so nothing in the tone gives it away.

Why it happens

A language model predicts the next word given everything that came before it, and that is more or less all it does. It has no separate sense of whether it was actually given the information, so when the material runs out it carries on with whatever fits best.

A bot that answers from its training data has never seen your product. A bot that reads your documentation first is a big improvement, but nothing stops it filling the gaps from what it already knows. What fixes this is a much stricter brief for the model, plus a proper way out for when the brief cannot be met. We built four rules into Deacon.

Rule one. Answer only from what was found for this question

When a visitor asks Deacon something, the question goes to your knowledge base first. The best few passages come back, numbered, each with the name of the page or file it came from. The model is then told to answer using only those passages, and to say which one it used. Grounding is the name for tying an answer to retrieved material that way.

If the passage it used is a page on your site, the answer comes with a link to that page, so the visitor can read the source for themselves.

Rule two. Only the facts the passages state

The brief lists prices, numbers, dates and policy details as the things a model is most tempted to invent. If the passages talk about the topic but do not give the fact, Deacon does not give it either. The brief also says that two passages must never be stitched together into a claim that neither of them makes on its own, which is exactly the Apple Pay case above.

Rule three. Ask the clarifying question first

Often the honest answer is really “it depends”. Someone asks how to export their data and the answer is different on each plan. Saying it does not know would help nobody, because the answer is in the passages. So when one detail would settle it, Deacon asks for that detail. It asks at most three clarifying questions in a conversation. After the third it hands the conversation over to you.

When no passage covers the question and no detail would change that, Deacon tells the visitor it does not have the answer, in the language they wrote in, and invites them to leave their email. The transcript and their address land in your dashboard marked as unanswered. You reply from there when you are next at your desk, and the visitor gets your reply by email.

Rule four. Write the missing answer yourself

Every question Deacon could not answer is listed in your dashboard with a box beside it. Write the answer once and it joins your knowledge base as a team-verified answer, and the next visitor who asks gets it. If an older page says something different, the brief tells the model to prefer yours.

What you give up

Some questions will get “I don’t know” where a looser bot would have guessed. We think that is the right trade, because the guess would have gone to a real customer from your own website. We test for it as well, with a set of questions in Deacon’s own test suite that the documentation cannot answer. The suite fails if any of them gets an answer.

How to check the bot you already have

None of this needs our product to try. Ask your bot something your documentation genuinely does not cover and see whether it admits that. Ask it for a price or a date you have never written down anywhere. Then give it two facts that are true on their own and see whether it joins them into a third one nobody wrote. If it answers all three confidently and never says where the answer came from, you have the problem this article is about.