Most organisations now have a line in their AI policy about human oversight. Someone reviews the output. Someone signs off before it goes further. On paper, that looks like governance.
In practice, it depends entirely on whether the person doing the reviewing still knows how to do the task themselves and knows what to look for in the details and around the edges.
It’s a pretty safe assumption right now that there are people who can do that, but what about in a year or two years?
Why this isn’t an immediate gap
You cannot spot a subtly wrong answer unless you know what a right one looks like, and you only know that from having produced it yourself, more than once, under real conditions. The first generation of reviewers is usually fine. They came up doing the work manually, so their judgement is intact even after the tool takes over the execution. The risk sits one generation down, with whomever they train, or fail to train.
If a new starter’s first exposure to a task is already AI assisted, they inherit the output without ever building the underlying capability the output is meant to replace. They can operate the tool. They cannot tell when the tool is wrong. Two or three years on, the organisation still has a named reviewer on every process, and still believes it is governed, while the actual skill the governance depends on has quietly gone away.
This is not a gap you notice quickly. It shows up as a slow drift in output quality that nobody can quite trace, because every individual step still has a name attached to it. It may also arrive as a critical failure when a review missing something they didn’t know they needed to look for.
The gap may be more urgent than you think
The drift is what happens when new starters are never given the chance to build the skill in the first place. There is a second version of the same failure, and it moves much faster.
The organisations most exposed here are not the ones with nobody left who can do the task manually. They are the ones with exactly one or two people who still can. What if those people are close to leaving? A retirement, a redundancy, a better offer somewhere else, and the capability does not decline gradually. It disappears all at once.
Both versions end in the same place. You are left with a reviewer with no basis to judge what the AI has produced. But the second one can happen in a matter of weeks rather than years, and it is easy to miss, because the organisation still looks fully staffed right up until the one person who actually understood the task in detail walks out the door. Unless the transfer of that knowledge to the next person has been made deliberate, before that key person leaves, the organisation finds out what it lost only when something goes wrong.
What this means for operating model design
Governance of AI is usually designed as a policy question. You document who approves, at what threshold, with what audit trail. Those are real questions, but they sit on top of a structural one that very often gets skipped. How is the organisation going to keep producing people who are qualified to approve.
That is only solved by deliberately keeping some pathway into the work that does not route through the tool, at least for the people who will need to judge it. It means rotation through manual versions of a process before automation, junior roles that still touch the raw task rather than only the AI’s output, and sitting next to the experienced reviewer as they complete a review and explain in detail what they are doing and why.
It also means treating the departure of anyone who still holds the manual version of a task as a named risk, not a routine leaver process. If there is one person left who could catch the AI being wrong, and no plan for who inherits that judgement before they go, the organisation is one resignation letter away from a governance step that, in a very real sense, stops existing.
None of that is about slowing AI adoption down. It is about recognising that the capability to supervise a system is a separate capability from the system itself, one that has to be built and maintained on purpose, because it will not survive by accident.