Five hypotheses about an organisation. Every one wrong.
Not wrong in the way AI is usually wrong. There was no hallucinated statistic, no invented source, no confident nonsense that a quick search would catch. Each hypothesis was well argued, properly sourced from what I’d given it, and coherent from the first sentence to the last. Each one was better written than the one before it. And each one was killed by the same single question, asked five separate times before I noticed it was the same question.
Was this thing what it appears to be?
I usually use AI to build an outline before I write. Not because I can’t structure my own thinking, but because I often arrive with too many points and no order to them, and a blank page is a bad place to sort that out. I used to hand that job to a team. It never needed one. They were what I had at the time to get those messy thoughts onto a page.
This time was different. Two hours in, having poked a hole in every hypothesis it produced, the AI told me to write the rest myself. That’s worth considering. Not because it’s a funny anecdote about a tool admitting defeat, but because of what had actually happened in those two hours, and what it tells you about where the real human-work in AI-enabled work sits.
The shape it kept reaching for
Every hypothesis about this organisation assumed something had drifted or accumulated over time until it became visible as a problem. That’s a reasonable assumption to default to, because it’s usually correct when you are looking at an organisation over a certain size. It’s also the most canonical idea in operating model work: the model that got you here will not get you there. Organisations outgrow their own architecture constantly, and most of what looks broken is exactly that. Something that used to fit no longer does.
Five times, the AI started from that assumption and worked backwards, fitting evidence to a frame it had already selected. That’s not a flaw specific to this tool or this session. That’s what pattern matching looks like when it’s very good at its job. The frame arrives first because the frame is the most statistically likely explanation, and the evidence gets read through it rather than against it. It’s efficient. It’s also exactly backwards for diagnosis, because diagnosis has to start with the possibility that the frame is wrong before it can trust any evidence.
Each time, the answer to the question, was this thing what it appears to be, turned out to be no. Because those things hadn’t drifted, they had been deliberate from the start. Someone had built them that way, for reasons that made sense at the time, and no amount of evidence about the current state would provide insight into an accidental occurrence when it was something that had been made by design.
What one conversation can and can’t tell you
I want to be careful here. One conversation is an illustration. It doesn’t prove anything on its own. I’m not arguing that AI defaults to decay narratives as some kind of general law, and I’d be suspicious of anyone who drew that conclusion from a single afternoon with one tool on one problem. What I can say with more confidence is narrower: the hypotheses that survived those two hours were better than either of us would have produced alone, and they only survived because someone in the room knew enough to keep asking whether the pattern itself was the thing to distrust.
That’s a different claim from “AI got it wrong.” AI didn’t get it wrong in the way that would have been easy to catch. Nothing here would have shown up as an error if I’d been reading for errors. Every hypothesis was internally consistent. The problem wasn’t in the reasoning inside each hypothesis. It was in the unexamined choice of which hypothesis to reason inside.
What this actually requires
This is the part I think gets skipped in most conversations about AI and judgement. The advice tends to land on one of three things: trust the output, watch for hallucinations and errors, or don’t use the tool for anything that matters. None describes what happened here. I used the tool. I trusted it enough to let it propose five confident, well-built answers. I didn’t trust it enough to stop there, because the assumption underneath all five answers was one I recognised, and recognising it was the only reason to go looking for what it had missed.
That recognition isn’t a prompting technique. You can’t ask a tool to check whether it has defaulted to the canonical frame, not usefully, because it has no way to know that it has. It doesn’t experience its own pattern matching as a choice. The check has to come from outside, from someone who has seen enough real organisations to know that the “your model is outdated” explanation, however often it’s correct, is not the only explanation, and that the fastest way to miss a deliberate structural decision is to assume you’re looking at an accidental one.
That’s the actual shape of AI-enabled work, at least in this instance. Not AI proposing and a person approving. AI proposing, quickly and well, and a person holding the proposal up against everything they know that the tool doesn’t, until what’s left has survived contact with a question the tool couldn’t have asked itself. The output that came out of those two hours wasn’t a compromise between what I would have written and what it produced. It was better than what the AI could do, and it was an order of magnitude faster than anything I could have done on my own.
The tool telling me to write the rest myself wasn’t a failure of the process. It was the process working. It had done what it could do well: propose fast, argue coherently, hand me five well-built wrong answers to think against. What it couldn’t do was know that all five shared the same blind spot, because the blind spot was its own.