What you are actually talking to

When you first use a modern AI assistant, it answers like an expert: full sentences, structured reasoning, confident tone, no hesitation. Most of the time, the output is accurate and useful. That usefulness builds a habit of trust, and the habit is reasonable, up to a point.

The limit of that trust becomes visible the moment you ask about something you know in detail. The answer comes back equally fluent, equally confident, and wrong in a specific, plausible way that only fails under close inspection: a named case that did not happen, a date that is off by years, a process described the way an outsider would guess it works.

In 2023, in the US federal case Mata v. Avianca, attorneys submitted a legal brief containing six citations to judicial opinions. The opinions did not exist. An AI chatbot had generated them, including the case names, courts, dates, and quoted passages, complete with docket numbers. The attorneys did not verify the citations against any legal database before filing. A federal judge discovered the fabrications and the court sanctioned the attorneys. The output had been fluent, formatted, and specific. None of that was evidence it was true.

To understand why this happens, you need to understand what a large language model (LLM) actually does. It does not look anything up. It has been trained on a very large body of text, and from that training it learned one skill to a very high level: predicting what the next words should be. When you ask it a question, it is not retrieving an answer from a database of facts. It is generating what a plausible answer to your question would look like, one token at a time. Most of the time, the most plausible answer and the true answer are the same thing. That is why these systems are so useful. But when the truth is rare, specific, recent, or simply not well-represented in the training data, the most plausible-sounding answer can be entirely fabricated. And it will be delivered in the same confident voice, because the voice is part of the prediction.

There is also a hard limit on how much the system can hold in view at once. This is called the context window: a sliding frame over your conversation and any documents you share. Everything inside the frame is visible to the model. Everything outside it has, in effect, never been said. In a short conversation this is invisible. In a long one, or when you share a large document, earlier parts of the exchange fall outside the window. The system is not forgetful in the way people are forgetful. It simply cannot see what is no longer inside the frame.

The practical consequence is this: you cannot grade an AI answer by how it sounds. Fluency and confidence are always part of the output, whether the content is accurate or fabricated. Knowing the difference requires checking against something outside the conversation.

Why it lies (it isn't lying)

When a large language model produces something false, it is not deceiving anyone. Deception requires knowing the truth and choosing to hide it. The model has no such knowledge. It is doing the only thing it does: predicting plausible text. Sometimes plausible text describes a world that does not exist. This failure mode has a name: hallucination. Fabrication is the failure mode of plausibility. The same engine that produces an accurate summary also invents a citation with identical confidence, because both are just likely-sounding text.

The model's confidence is not evidence. Its fluency is not evidence. The only evidence is something you can check outside the conversation. This one idea will save you more pain than anything else in this program. Lesson 3 is where it gets expensive.