Skip to content

Does an AI chatbot make up answers when it doesn’t know?

Short answer

Yes, it can, and that’s called a hallucination. The model writes an answer that sounds believable because it’s built to produce convincing text, not to admit it’s missing a fact. No system is right every time, so what matters is how it fails. "I don’t know, let me get a person" is acceptable; inventing a billing figure is not.

By Adrià Castany, updated

In detail

Why it happens

A language model doesn’t distinguish between what it knows and what it’s filling in. Ask it about a feature that doesn’t exist, with nothing written about it, and the most likely output is a plausible description of that feature. It isn’t lying. It’s doing what it always does, writing the most likely text.

What reduces it

Answers that come from your sources instead of the model’s memory, instructions on what to do when a fact is missing, and documentation of what your product doesn’t do. Almost nobody writes that last part, and it prevents the most hallucinations. An article that says "XML export isn’t available" closes that door.

The test before you buy

Ask it about a feature your product doesn’t have and see what it says. A well-built system says it doesn’t exist or that it will check with someone. A badly built one describes the feature in detail, and it will do the same to a customer.

What isn’t enough

Asking in the instructions and stopping there. "Don’t make up prices" is a request, and a model can ignore it in a strange enough conversation. The limit has to live in the system. The fact comes from a lookup in your systems, or it doesn’t come at all.

Terms used here

How Intake handles it

This question comes from The Intake agent


Questions

Related questions

See every question
  • Can an AI agent act without me reviewing it?

    Yes, and you decide how far it goes. The sensible split is by what can be undone. Lookups change nothing and can run on their own; changes you can reverse can run on their own if they’re logged; and anything you can’t undo, like canceling, deleting or refunding, should be confirmed by a person, because if the agent gets it wrong there, nobody will notice in time.

  • Can I see which actions an AI agent has run?

    Yes, and you should be able to. If a vendor won’t show you, that’s reason enough to rule them out. Every action the agent takes has to be logged with what it did, on which account, when and in which conversation. Without that log you can’t review a mistake, you can’t answer a customer who complains, and you won’t pass a mid-sized customer’s security questionnaire.

  • Can an AI agent change a customer’s plan?

    Yes, if you give it that action, although it isn’t always a good idea. An upgrade can be undone, and some companies automate it. A downgrade or a cancellation moves money and doesn’t undo itself, so the sensible setup is for the agent to prepare it and a person to confirm it.

Add it without dropping what you use

15 minutes, no commitment