The Designing for Human Limits series
Why speed erodes human backup systems
AI is making work faster: Analysis, code, reports, decision documents – all can be drafted, formulated and polished now in minutes instead of hours or days. It feels effortlessly.
For managers, this looks attractive, as more output can be delivered with less manual input.
But what effect does this have on the expertise in the organization when effort turns into prompting and passive oversight?
Skills decay when work disappears
Surely, we will not replace humans any time soon – yet. Though, already the “Human in the loop” approach puts human into passive “consume” mode: Reviewing, checking, approving.
Yes, this preserves a form of control, but it does not retain the same capability within the organization.
Reviewing an output does not train the same judgment as creating it in the same way approving a recommendation does not build the same understanding as working through, and condensing all the available options does. Checking a deliverable does not replace designing it. How can the skills needed to oversee the AIs output be retained, let alone be developed without exercising the “creation muscle” in exactly these movements.
Automation removes learning loops
Automation removes friction, and this is useful. Many organizations have too much friction, reduction SHOULD be the goal here.
But friction is also where learning happens, for example:
- Designing a process exposes dependencies.
- Preparing a steering decision clarifies trade-offs.
- Defining a Key Result forces outcome thinking.
Just to name a view.
When these activities are delegated, the learning loop comes to a halt (or at least is externalized!). Read further in this deepdive article: AI Learning Loops: When Speed Kills Improvement
This creates a fragile system. It is highly accelerated when everything runs smoothly but stalls to a halt, or even worse, spirals into error territory, as soon as an exception occurs.
The organization can produce more, report more and process more. But when the system is wrong, fewer people can intervene meaningfully.
A useful image is an organization consisting only of middle managers.
Everyone coordinates and reviews.
But nobody observes, contributes and THINKS from the grassroots up.
AI operating models drift into exactly this pattern: humans are drastically reduced and only remain involved as monitors and approvers. Governance still exists. Meetings still happen. Progress is still reported. But the ability to challenge, redesign and manually intervene gets thinner.
What aviation gets right
Aviation has worked with high automation for decades.
Modern aircraft can automate large parts of navigation, flight management and landing. Yet pilots were not turned into passive observers.
The system was designed to preserve human capability to avoid exception cases turning into catastrophes. That is the relevant management lesson for AI-enabled transformation execution.
Four design principles
1. Keep critical decisions explicitly human-owned
In aviation, the pilot in command remains responsible, even when autopilot is engaged.
AI operating models need the same clarity.
Define:
- who owns the decision
- who can override the system
- which decisions must stay human-owned
- when escalation is required
- what evidence is needed before accepting an AI-supported recommendation
“Human in the loop” is often too vague – aim for “human in the lead” (i.e. clarify unambiguously accountabilities)..
2. Design roles around judgment and expertise
Pilots are not designed into the system as passive reviewers of autopilot output. Their role is to preserve operational judgment, which means: noticing when the aircraft is doing something unexpected, comparing instrument indications with the real flight situation, and take manual control when needed.
In the same way: Weak AI roles are built around checking outputs.
Strong AI roles are built around preserving business judgment.
That means people must still be able to:
- challenge assumptions
- detect context drift
- understand failure modes
- compare outputs with business reality
- decide when to escalate
- revert to manual execution when needed
Think: A product owner who only reviews AI-generated user stories may become faster at backlog handling, but weaker at product judgment.
3. Embed learning into the operating cadence
Airlines use operational data and reporting systems to constantly learn from operations to improve safety. This even is mandatory by FAA.
In the same way AI capability cannot live only in training sessions. Every AI-enabled workflow should constantly generate learning signals:
- which outputs failed
- where humans overrode the system
- where confidence was misplaced
- where handoffs broke
- where context was missing
- which capabilities need reinforcement
These signals belong into retrospectives, risk reviews and transformation governance.
All to answer a simple question: what did execution teach us about the capability of the system?
4. Delivery metrics only show half of the picture: Track capability signals instead
A flight can land safely and still contain warning signs: too fast, too high, unstable descent rate, late configuration, weak energy management or a missed opportunity to go around. If you only measure “arrived safely” or “arrived on time,” you miss important indicators for the continuous operation of the system you designed.
Yet, most transformation governance still focuses on delivery:
Timeline, budget, milestones, scope, output quality, risks, next steps.
But more importantly: “No need for a Retro if the output was delivered as expected”.
These remain relevant. But they are not enough when AI takes over large parts of execution.
Because a team can deliver faster while losing domain judgment. The same way a function can reduce effort while increasing dependency.
Add capability signals:
- quality of human overrides
- clarity of decision ownership
- recurring AI failure patterns
- escalation frequency
- manual fallback capability
- ability to explain AI-supported recommendations
- dependency on specific tools or individuals
- capability gaps observed during execution
Three questions before automating further
Before automating a workflow, ask:
1. What capability does this workflow currently train?
Because some manual work looks inefficient but builds understanding.
2. What happens when the system is wrong?
The critical test is detection of edge cases and how smooth the handover to human actors works.
3. Who owns the judgment?
Ownership means accountability. Without it, people will default to output review. That is too weak.
Final thought
AI can make organizations faster.
But that comes with strings attached.
If automation removes friction without preserving judgment, organizations become more efficient and more dependent at the same time.
The operating model needs to protect where humans must stay sharp: in roles, decision rights, governance, learning loops and leading indicators.
Authors Remarks
This is not an argument against AI-supported decision-making. In many contexts, well-designed human-AI systems improve speed, accuracy and outcomes.
The problem starts when organizations scale AI output faster than they redesign the operating model around it. Judgment, verification and accountability do not disappear. They accumulate somewhere in the system. If they are not explicitly designed for, decision quality becomes harder to protect.
