In 2024, a Canadian tribunal issued a ruling that shows precisely how this failure works. A passenger named Moffatt had consulted Air Canada's website chatbot about bereavement fares after a family death. The chatbot told him he could purchase a full-price ticket and then apply for a bereavement discount after traveling. That was not Air Canada's actual policy. The real policy required that bereavement fares be requested before travel. The correct information was available on the same website, one check away. The tribunal held Air Canada responsible for what its chatbot had stated and ordered the airline to pay compensation. The airline's argument that its chatbot was a separate entity whose statements should not bind the company was rejected.
The chatbot had not done anything unusual. It answered in the same authoritative, helpful tone these systems use for every response. The claim it made was plausible. A customer reading it had no obvious reason to doubt it. What was missing was a verification: a check of the actual policy, the actual source. The real answer existed. The cost of building on an unchecked claim landed in a legal ruling.
This is the pattern that matters for anyone working with AI systems. When an assistant states a constraint, "that data is not available," "the system does not support that," "the policy does not allow it," the statement arrives in a confident voice. It feels like retrieved information. It is more accurately described as a prediction: the most plausible-sounding answer, generated from patterns, not looked up from the actual source. The discipline of evidence over reasoning means treating every stated constraint as a hypothesis until you have verified it against the actual thing.
Consider a team designing a system around a constraint their AI assistant stated: a particular piece of data, the assistant said, was not available from the service they were using. The team built around it. Extra storage to reconstruct the data. Extra logic to keep it consistent. Extra handling for when their reconstruction disagreed with reality. Every piece of the design was a rational response to the constraint. When someone finally checked the actual API documentation, the data was available. It had been available the whole time. The wall the team had spent significant effort designing around did not exist.
Reasoning is capable of producing elaborate, internally consistent structures on top of an unchecked foundation. It takes very little additional effort to verify the foundation before building. The question to put to any AI claim about what is or is not possible: is this retrieved from the actual source, or predicted from memory? Then go and check the actual source yourself as well. Read the documentation. Open the settings. Try the action. Before building around any wall, touch it.
Complexity is usually a tax paid on an unverified assumption. When a plan starts growing in scope, the right move is to locate the constraint forcing the growth and verify that it is real. Most constraints are real. Some are not. The ones that are not will cost far more than a simple check would have.