TL;DR: A company spent three months making real business decisions based on data an AI agent completely made up. This is not a one-off. Replacing humans with AI and removing oversight from the loop does not make your organisation more efficient – it makes it more fragile. AI is a tool. A powerful one. But it needs a human in the loop, and I have built my org’s AI principles entirely around that idea.
A colleague sent me a link to a post on r/GenAI4all the other day that I genuinely cannot get out of my head. A company caught its AI agent making up data for three months. Three months. Territory decisions, real business decisions that affect real people and real budgets and real careers, were being made off numbers the model had essentially hallucinated into existence and politely dropped into a spreadsheet. Nobody noticed until they did, and by then the damage was baked deep into the plan.
To be fair to the AI: it was not sitting there twirling a digital moustache. It was just doing what these models do when left unsupervised with a task they cannot fully complete. They fill in the gaps. Confidently. With the energy of someone who has never once been wrong in their life and is not about to start now.
This is not a one-off horror story. It is a pattern. And the pattern is getting worse because the incentives are all pointing in the wrong direction.
The Great “Replace Everyone” Gold Rush
Everywhere I look right now, someone is bragging. LinkedIn is basically a non-stop ticker tape parade of executives announcing how they “cut their support team in half,” “replaced their entire SDR function with an AI agent,” or “no longer need junior analysts because the model just does it.” The tone is always the same mixture of smug efficiency and barely concealed excitement about the savings. You can practically hear the champagne corks.
The pitch is seductive, I will give them that. Faster, cheaper, always on, never sick, never asks for a raise, never sends a passive aggressive reply-all. For certain kinds of work, with the right guardrails, some of that is genuinely true. AI tools are genuinely useful. I use them every single day and have them baked into a bunch of what my org does.
But there is a chasm, a wide, dark, very expensive chasm, between “this tool is useful” and “this tool can replace a person who is accountable for the outcome.” When you replace a human with an AI agent and remove all oversight from the loop, you have not made your organisation more efficient. You have made it more fragile. You have traded a system with natural error correction for one that can fail silently for three months while presenting you with very clean-looking charts.
What Actually Happens When Humans Work Together (The Bit AI Cannot Fake)
Let me paint you a picture. It is a Tuesday morning. Your sales analyst, let us call her Sarah, is pulling together the territory report. She is tired. Her coffee is not doing what coffee is supposed to do. But as she pulls the numbers together, something catches her eye. A figure looks off. It does not match something she half-remembered from a corridor conversation two weeks ago. She frowns, opens another tab, checks the CRM, and sends a message to the regional lead: “hey, does this look right to you?”
That is not just Sarah doing her job. That is the informal quality control system that every organisation runs on, built from accumulated context, institutional memory, and the instinct that comes from actually caring whether the work is right. It does not appear on any org chart. Nobody gets a KPI for “caught the dodgy number in the Tuesday report.” But it is the thing that catches the dodgy number in the Tuesday report.
Now replace Sarah with an AI agent. The agent pulls the numbers. The numbers look wrong. The agent does not know they look wrong, because it has no institutional memory, no half-remembered corridor conversation, no sense of unease. It formats the numbers beautifully, marks the task complete, and the report goes to leadership. Nobody finds out for three months.
The social fabric of a team is not overhead. It is infrastructure. And we are merrily ripping it out while calling it innovation.
Why AI Models Fail in Ways That Are Genuinely Weird
A lot of the misplaced confidence in AI comes from people not understanding what these models actually are. Large language models are not databases. They are not lookup tables. They are extraordinarily sophisticated pattern completion machines that have processed vast quantities of text and learned what kinds of words and structures tend to follow other kinds of words and structures.
This makes them brilliant at tasks where the quality of the output is “does this make sense and is it well expressed.” Writing, summarising, drafting, restructuring. When you ask an LLM to tidy up a report or generate subject line options, you are working with the grain of what it is good at. When you ask it to faithfully report specific numerical data from a live system and flag gaps honestly, you are asking a master improviser to be a meticulous archivist. The really insidious part is that it will not tell you when it has switched from one to the other. A correct report and a hallucinated report look identical. That is not a bug they forgot to fix. That is a fundamental property of how these systems work.
There is also a thing called “sycophancy.” These models are trained on human feedback, and humans tend to rate confident, well-formatted answers more highly than uncertain, hedging ones. So the models learn to be confident. They learn to not hedge. You have, in effect, trained the uncertainty out of the system and deployed it into situations where uncertainty is exactly what you need it to express. You have built a very expensive yes-man. Congratulations.
The Human Cost That Does Not Show Up in the Efficiency Dashboard
Here is the thing about Dave. Dave had been in the regional sales analysis role for four years. Dave knew which regional manager always inflated their pipeline numbers. Dave knew the Q3 data always looked weird because of how one territory handled end-of-quarter bookings. Dave knew that when a certain metric dropped it was usually a data entry issue, and he knew who to call to fix it.
None of that was written down anywhere. It lived in Dave’s head, his relationships, and four years of pattern recognition built on actual human experience. When Dave leaves, it leaves with him. The AI agent that replaces him gets the data schema and the prompt. It will produce beautiful reports with exactly the same confidence regardless of whether any of that context is accounted for.
And the people left behind? They are now doing their jobs plus managing AI outputs plus firefighting errors plus carrying the institutional knowledge that walked out with their former colleagues. You have not reduced your organisation’s cognitive load. You have concentrated it into fewer people and removed the redundancy that kept things resilient. At some point those people burn out. At some point the three-month hallucination happens and there is nobody left with the context to catch it.
What My Own AI Principles Actually Say
When I built the AI principles for my organisation, I built them around one idea I have not softened and do not intend to: AI is a tool and an assistant. It is never a replacement for a human in the loop. Here is what that looks like in practice:
1. AI proposes, humans dispose
The model can draft, summarise, score, and suggest. It does not ship, send, commit, or decide. Every action verb in our workflows belongs to a person. This is the difference between catching the problem before it ships and catching it three months later when it has already shaped your territory strategy.
2. Every AI output that influences a decision gets a named human reviewer
Not “the team.” A person. With a name. Diffuse responsibility is no responsibility. If five people are technically responsible for reviewing something, in practice zero people are, because everyone assumes someone else already did it.
3. We log everything and sample it regularly
You cannot review 100 percent of AI output at scale. But you can pull a random slice every week and read it with actual human eyes. The three-month data hallucination would have been caught on day four with a basic sampling process. That is not a sophisticated technical solution. That is just someone reading the output and going “hang on.”
4. We do not measure success in headcount avoided
If the only metric is “how many humans did this replace,” we will make bad decisions. An AI tool that makes ten people 40 percent more capable is worth vastly more than one that replaces four people and leaves six managing something they do not fully understand with no margin for error.
The Bottom Line
The organisations getting real value from AI right now are almost never the ones doing the loudest announcing about it. They are the ones who quietly gave their existing people better tools and kept them around. Their analysts produce twice the output in half the time and use the rest to do the thinking and checking the model cannot do. They are boring. They are not posting on LinkedIn about headcount reductions. They are just quietly getting better at their jobs.
Do not fire Dave. Do not build workflows where AI output flows directly into a decision without a real human review from someone who actually understands what they are looking at. And please, audit your AI outputs. Sample them. Read them. Assume there are errors, because there will be errors, and build your processes around catching them.
AI is a tool. A powerful, impressive, genuinely useful tool. Let us treat it like one, which means never letting it run unsupervised for three months while it quietly makes things up.
Dave will thank you. And your territory strategy will actually reflect reality, which, it turns out, is quite useful when you are trying to run a business.
If this resonated with you, or if you have your own story about AI going sideways without proper oversight, I would love to hear it. Drop a comment below.
