What you are actually talking to

In my first months with AI, I treated it like a very fast, very patient expert. I asked it about visa rules, about contracts, about cities I was traveling to. The answers came back instantly, fluent, organized, confident. I was hooked.

Then one day I asked it about something I happened to know deeply, a corner of my own industry I had lived in for years. The answer came back instantly, fluent, organized, confident. And wrong. Not wildly wrong. Wrong in the specific, plausible way that only sounds right if you don't know better. It named real-sounding things that didn't exist. It described how the business worked the way an outsider would guess it works.

That shook me more than a wild error would have. Because I realized: every answer it had ever given me sounded exactly like this one. The visa rules, the contract advice, the travel tips. Same fluency, same confidence. I had been grading them all on tone, and tone was the one thing it could always produce.

Around the same time, something else kept happening. In long conversations, it would forget things I had told it half an hour earlier. My name's spelling. A budget I had given it. A decision we had already made together. It would just drift, like a person who hadn't been listening. I took it personally until I learned what was actually going on.

Here is what nobody had told me. The machine is not looking anything up when it answers you. It is not consulting a database of facts. It has read a staggering amount of text, and from all that reading it learned one skill to an almost supernatural level: predicting what the next words should be. When you ask it a question, it is not retrieving the answer. It is generating what an answer to your question would plausibly sound like, one word at a time.

Most of the time, the most plausible answer and the true answer are the same thing. That is why it is so often right, and why it is so useful. But when the truth is rare, or specific, or recent, or just unlucky, the most plausible-sounding answer can be completely invented. And it will be delivered in the same confident voice, because the voice is part of the prediction too.

And the forgetting? The machine has a working memory called a context window. Think of it as a sliding window over your conversation. Everything inside the window, it can see. Everything that slides out, it has never heard of. Not forgotten the way you forget. Gone, as if it was never said. Long conversation, big documents, lots of back and forth: the window slides, and the early stuff falls off the edge.

Why it lies (it isn't lying)

When the machine makes something up, it is not deceiving you. Deception requires knowing the truth and hiding it. The machine doesn't know there is a truth to hide. It is doing the only thing it ever does, predicting plausible words, and sometimes plausible words describe a world that doesn't exist. People call this hallucination. I think of it more simply: fabrication is the failure mode of plausibility. The same engine that writes you a beautiful summary also invents a citation, with identical confidence, because both are just likely-sounding text.

This one idea will save you more pain than anything else in this library. The machine's confidence is not evidence. Its fluency is not evidence. The only evidence is something you can check outside the conversation. Hold onto that. Chapter 3 is where it gets expensive.