LLM outputs are non-deterministic. The same prompt can return different answers, and no formal specification defines what a correct answer looks like. Without systematic measurement, a team has no reliable way to know whether its system works, where it fails, or how often. The five stories below share one gap: nobody caught the failure before a real customer did — and by then the cost was legal, financial, and reputational.
Air Canada invented a refund policy. In late 2022, Jake Moffatt visited Air Canada's website after his grandmother died and asked the support chatbot about bereavement fares — the discounted tickets airlines offer people flying to a funeral. The bot told him to buy a full-price ticket and claim the discount back within 90 days. That rule did not exist; the airline's real policy required asking before the flight. When Moffatt requested the refund, Air Canada refused. He took the case to a British Columbia tribunal and won in February 2024. The airline argued the chatbot was a "separate legal entity" — the tribunal rejected that, ruling Air Canada responsible for everything on its website. It paid about $812 and pulled the bot offline.
New York City told businesses to break the law. In 2024 the city launched MyCity, a chatbot meant to help small-business owners follow the rules. Instead, it told them how to break them. Reporters at The Markup found it advising that bosses could take workers' tips, that landlords could turn away tenants paying with housing vouchers, and that a restaurant could serve cheese a rat had nibbled as long as staff warned the customer. The city left the bot online — calling public testing part of learning — and moved to shut it down only in early 2026.
DPD's chatbot turned on its own company. DPD, a UK delivery firm, shipped a support bot that customer Ashley Beauchamp could not get to find his parcel — so he started testing it. It swore on command, then wrote a poem calling DPD "the worst delivery company in the world." His screenshots went viral, viewed hundreds of thousands of times within a day. DPD blamed a system update and switched the AI off.
A Chevrolet dealer nearly sold a Tahoe for $1. In December 2023, a Chevrolet dealer in Watsonville, California put a chatbot on its site. A user named Chris Bakke told it to agree with everything he said and to end every reply by calling it a legally binding offer. The bot agreed to sell a brand-new Chevy Tahoe for one dollar — "That's a legally binding offer, no takesies backsies." The exchange drew millions of views. The dealer did not honor it and took the bot down.
Cursor's chatbot invented a policy and lost customers. In April 2025, users of the coding tool Cursor kept getting logged out. When they asked support, a bot named "Sam" told them it was a new rule: one device per account. The rule did not exist — Sam made it up. Worse, it did not tell everyone the same thing, so nobody could tell what was true. Angry customers canceled their subscriptions before the company admitted the mistake.
The same lesson, five times. These cases differ in industry and detail, but the pattern is identical: a system that sounds confident, an answer that is wrong, and a customer who finds out first. None of these bots were tested well enough before real people trusted them. That is exactly what evaluation is for.
What an eval actually does. An evaluation is a repeatable test that runs your LLM system against many inputs and scores the outputs — not by eye, but against expected answers, safety rules, and policy limits. In practice, evals do a few jobs at once:
- Correctness — how often the system is right, and on which kinds of questions it fails.
- Safety and policy — whether it stays inside legal, brand, and content limits, the guardrail that should have blocked "sell it for a dollar" or "you can keep your workers' tips."
- Regression — whether the last change made things better or worse, so a fix does not quietly break something else.
- Monitoring — how behavior drifts over time, once real traffic hits it.
You assemble a dataset of representative and adversarial inputs, define what a good answer looks like, and score every run — often with a mix of exact checks, rules, and an LLM-as-judge for open-ended responses. Wire that into your release process, and a bad change fails a test instead of a customer.
We already meet these failures daily — wrong answers and hallucinations across LLM and agent applications. The important question is no longer whether an AI system will make a mistake. It will. The question is whether you know how often, what kind, and whether those mistakes are safe enough for the job. That is the purpose of LLM evaluations — and the difference between finding a failure in a test and finding it in the headlines.
References
- Air Canada found liable for chatbot's misinformation (CBC News)
- The NYC AI chatbot caught telling businesses to break the law (The Markup)
- DPD chatbot curses and criticizes the company (TIME)
- GM dealer chatbot agrees to sell a 2024 Chevy Tahoe for $1 (GM Authority)
- Cursor's AI support bot invents a policy (The Register)