Does an AI chatbot make up answers when it does not know?
It can, and that is called hallucination: the model builds an answer that sounds plausible because its job is producing convincing text, not recognising that it lacks the fact. What separates a useful system from a dangerous one is not never failing, it is how it fails: "I do not know, let me get a person" is acceptable; inventing a billing figure is not.
Short answer
It can, and that is called hallucination: the model builds an answer that sounds plausible because its job is producing convincing text, not recognising that it lacks the fact. What separates a useful system from a dangerous one is not never failing, it is how it fails: "I do not know, let me get a person" is acceptable; inventing a billing figure is not.
In detail
Why it happens
A language model does not distinguish between what it knows and what it is completing. Ask it about a feature that does not exist with nothing written on the subject, and the statistically likeliest output is a reasonable description of that feature. It is not lying: it is doing exactly what it is for.
What actually reduces it
The answer coming from your sources rather than the model memory, instructions about what to do when a fact is missing, and explicitly documenting what **cannot** be done. That last point is the one nobody writes and the one that prevents the most hallucinations: an article saying "XML export is not available" closes that door.
The test before you sign
Ask it about a feature your product does not have and watch what it says. A well-built system says it does not have it or that it will confirm with somebody. A badly built one describes the feature in detail, and that is exactly what it will do to a customer.
What does not fix it
Asking in the instructions and stopping there. "Do not invent prices" is a request, and a model can ignore it given a strange enough conversation. The real limit lives in the system: either the fact comes from a real lookup or it does not come at all.
Terms that come up here
Other questions in this thread
Can an AI agent act without me reviewing it?
It can, and how far is your decision, not the vendor. The sensible split is by reversibility: what only reads breaks nothing and can run alone; what writes something recoverable can run alone with a log; and anything irreversible — cancelling, deleting, refunding — should pass through a person, not because the agent is wrong more often, but because when it is wrong nobody will notice in time.
Can I see which actions an AI agent has run?
You should, and a vendor who will not show you is reason enough to rule them out. Every action an agent runs has to be logged with what it did, on which account, when, and off the back of which conversation. Without that log you cannot audit a mistake, you cannot answer a customer who complains, and you cannot pass any mid-sized customer security questionnaire.
Can an AI agent change a customer plan?
Technically yes, if you give it that tool. The useful question is a different one: whether it should. An upgrade is recoverable and some teams leave it automated; a downgrade or a cancellation touches money and does not undo itself, so the sensible arrangement is the agent preparing it and a person confirming.
Where this question comes from
The Intake agent →