Predicting text is not knowing
A model does not query a database or verify anything.
It produces the most plausible continuation. That explains both why it writes so well and why it invents things with total confidence. Those are the same property, and no amount of training separates them.
Why it matters
It is the piece that writes, and it is almost never where the problem is.
When an AI support system answers badly, the cause is usually what information it was given, not which model wrote it. Swapping models is the first reaction and rarely the fix.
Context window
How much text it can hold at once.
The question, the conversation history and the retrieved documents. When it fills up something has to be dropped, and that choice affects the answer more than the brand of the model.
Temperature
A setting that controls how far it strays from the most likely continuation.
High gives more varied text; low, more predictable. Support wants low. Nobody wants creativity in the answer about an invoice.
Where it runs matters
If your customers’ conversations pass through a model hosted outside the European Union, that has data protection consequences you need to be able to explain.
It is a question worth asking any vendor before signing.
What it does well and what it does not
Worth being clear on before deciding what to hand it.
- It writes, summarizes and translates fluently.
- It classifies a message by topic, language or urgency.
- It does not know the company’s or its customers’ data unless that data is passed in.
- It does not remember earlier conversations unless the system sends them again.
- It does not check what it states, so it can present something false as true.
