The Unlearning Company

The Designing for Human Limits series

Why Continuous Improvement Stops Improving

Organisations have spent decades learning how to learn.

They built feedback cultures, retrospectives, lessons-learned processes, continuous-improvement systems, employee surveys, coaching practices and performance reviews. The language differs by industry, but the underlying idea is remarkably stable: work produces experience, experience produces insight and insight changes future behaviour.

Something happens. Someone notices. Someone adapts.

This sequence is so familiar that it rarely receives much attention. Feedback is treated as an input, almost like data travelling through the organisation. If enough information flows and people feel safe enough to speak, improvement should follow.

But feedback does not improve anything by itself.

For feedback to matter, there must be a sender, a receiver and an actor. Someone observes a difference between what happened and what should have happened. Someone understands what that difference means. Then someone changes behaviour, judgment, a decision rule, a process or a system.

Until now, organisations could usually assume that these functions would remain connected through human work. The person doing the work experienced the result, the team discussed it, a manager intervened, a specialist noticed a pattern, etc. Even when the formal learning process was weak, some adaptation happened simply because people remained in contact with the consequences of their actions.

AI changes that assumption.

Increasingly, the visible behaviour inside an organisation is no longer produced entirely by a person: a recommendation is generated by a model, a decision document is drafted by an assistant, a customer interaction is handled by an agent – or a workflow changes course because a system classified a case, predicted an outcome or selected the next action.

When that behaviour turns out to be wrong, incomplete or inappropriate, the familiar language of feedback starts to become strangely imprecise. Who, exactly, should receive it?

While Learning Loops asked Why are organisations creating less learning?
This article asks: What happens when organisations no longer know where learning is supposed to occur?

Feedback culture had a hidden dependency

A feedback culture assumes that the receiver can change through the act of receiving feedback.

That sounds obvious, but it distinguishes feedback from ordinary information transfer. Feedback has direction. It points from an observed outcome towards a different future behaviour.

When a manager tells an employee that a decision was escalated too late, the purpose is not merely to document the delay. The employee is expected to recognise the pattern, interpret the consequences and respond differently the next time. The person who acted is also, at least in principle, the person who adapts.

The same logic exists at team level. A retrospective connects delivery experience to future ways of working. A post-incident review connects failure to revised judgment. Customer feedback connects an encounter to a change in service. Neither the meeting nor the report creates improvement. Improvement occurs when the receiver incorporates what happened into future action.

Human beings do this socially.

We interpret feedback in relation to intention, context, identity, norms and expectations. We can notice that an action technically succeeded while still violating what others needed from us. We can understand that a decision created costs outside the metric by which it was evaluated. We can change because someone explains an impact we did not previously see.

That process is imperfect. Feedback can be misunderstood, rejected, softened, politicised or weaponised. It can arrive too late or reach someone without the authority to act. Yet the mechanism is recognisable: experience changes the actor through meaning.

An AI system does not participate in that mechanism.

You can provide additional information to a model. You can correct an output, adjust a prompt, change a threshold, add a rule, modify the workflow, replace the model or retrain the system. These interventions may produce better future results. But the system has not reflected on its behaviour. It has not reconsidered what it did. It has not understood the effect it had on somebody else. It does not return to work with a changed interpretation of its responsibility.

The system has been modified and modification requires a modifier.

Correction can survive while learning disappears

From the outside, an automated system may look as if it is learning continuously.

Performance is monitored. Outputs are rated. Users accept or reject recommendations. Exceptions are captured. The model may even improve over time. The loop appears intact: signal, correction, better output.

At the level of the technology, this may be entirely accurate.

At the level of the organisation, something else can be happening.

Suppose an AI-supported customer-service process repeatedly produces answers that are factually correct but unhelpful in context. Customers complain. Employees intervene. The responses are corrected and the prompt is refined. Output quality improves.

Who learned?

Perhaps the operations team learned that the prompt was underspecified. Perhaps the model owner learned which examples to add. Perhaps frontline employees learned which outputs require intervention. Perhaps the vendor’s system incorporated a new pattern.

Or perhaps none of these actors developed a fuller understanding of why the original responses failed. The symptom was removed, the output improved and the workflow continued. The local correction succeeded without strengthening organisational judgment.

The central risk.

Automated systems can become better at producing accepted outputs while the organisation becomes worse at understanding the work those outputs represent.

Continuous improvement then continues in appearance. Error rates fall, cycle times improve, fewer cases are escalated, etc. At the same time, the people around the process may gradually lose contact with the reasons one answer works and another does not. They learn how to operate the system, how to classify its failures and how to trigger a correction. They do not necessarily learn the underlying practice.

Improvement becomes located in the artefact rather than in the organisation.

The system performs better. The company knows less.

When the actor and the learner separate

Human work traditionally tied performance and learning together, however imperfectly.

A person prepared an analysis and, through preparing it, learned which information mattered. A manager wrote a decision proposal and, through writing it, clarified the trade-offs. A team handled an unusual customer case and, through handling it, developed a more precise understanding of the customer’s situation.

The effort was not incidental to learning. Working through the problem produced the experience from which judgment grew.

Automation can separate these functions.

• The system generates the analysis. The person reviews it.
• The system drafts the proposal. The leader approves it.
• The system handles ordinary cases. The employee receives only the exceptions.
• The system recommends an action. The decision-maker accepts or overrides it.

The human remains formally involved, but involvement has changed. Production becomes supervision. Practice becomes review. Repeated exposure to the full task is replaced by selective exposure to outputs and anomalies.

This changes the path through which feedback travels.

If a person creates a weak recommendation and receives feedback, the creator can adapt. If a system creates the recommendation, the reviewer receives feedback about behaviour they did not produce. They may be accountable for the outcome while lacking direct contact with the reasoning that generated it.

The reviewer can correct the output. The process owner can change the workflow. The model team can adjust the system. The vendor can improve the product. Each actor holds a fragment of the loop.

No one necessarily holds the whole act of adaptation.

Organisations already struggle with fragmented ownership. AI adds another fragmentation: the entity producing the behaviour cannot own its consequences, while the people owning the consequences may not control how the behaviour was produced.

Feedback still arrives but its address has become unclear.

The receiver is replaced before anyone notices

Automation programmes usually describe the tasks that will change.

Drafting will become faster. Classification will become automated. Recommendations will become more consistent. Routine interactions will require less human effort. Review will replace production in selected parts of the workflow.

What rarely appears in the design is a map of the learning that currently happens through those tasks.

• Which judgment is formed by drafting?
• Which weak signals become visible during classification?
• Which assumptions are tested while comparing options?
• Which customer realities become apparent through repeated interaction?
• Which professional instincts are developed through ordinary cases rather than exceptional ones?

These questions are difficult because the learning is often informal. It does not sit in the process map. It is not recorded as an output. It occurs inside the work.

A task can therefore look like an attractive automation candidate precisely because the capability it develops remains invisible.

Once the task disappears, the organisation may retain the deliverable and lose the developmental mechanism behind it. The report still exists. The recommendation still arrives. The customer still receives an answer. Nothing appears to have been removed except effort.

Only later does the system discover that effort carried information.

This does not mean all manual work should be preserved in the name of learning. Organisations contain waste, repetition and friction that deserve to be removed. But automation decisions made solely through the lens of throughput cannot distinguish useless effort from capability-producing effort.

Because both look slow and both consume time. Yet, only one leaves the organisation more capable after the work is finished.

The last mile feedback problem

The problem becomes more visible when automated behaviour crosses organisational boundaries.

Consider customer feedback: a customer reacts to the organisation, not to its internal architecture. They do not care whether a disappointing interaction was created by an employee, an AI agent, a policy, a workflow or a model supplied by a vendor. For the customer, the company acted.

Inside the company, however, the signal may travel through several layers before reaching anyone able to change the underlying behaviour:

• The frontline employee sees the complaint but does not own the model.
• The process owner owns the workflow but not the training data.
• The model team controls technical configuration but lacks the customer context.
• The vendor can modify the product but does not own the organisation’s promise.
• The executive sponsor owns the outcome but may see only aggregated performance.

Each party can truthfully say that the relevant cause sits elsewhere.

Eventually, the complaint may be converted into a category, a score or a ticket. Once translated, it can be counted and routed. But part of its meaning may disappear in the process. A social signal becomes a technical defect. A disappointed customer becomes a failed response. A breach of expectation becomes a prompt issue.

The correction may be technically valid and organisationally incomplete.

This is how AI can replace the receiver of organisational feedback without any formal decision to do so. The feedback remains visible. Dashboards may show more of it than ever. Yet the relationship between experience, interpretation and authority has been split across a chain of actors who each receive only the part they can process.

The organisation becomes excellent at handling signals and weak at being changed by them.

Feedback without consequence

There is another reason this matters: feedback creates obligation.

Once a problem has been surfaced, somebody is expected to respond. The cost of feedback lies partly in what happens next: priorities may need to change, resources may need to move, a decision may have to be reversed or a process may have to slow down.

This is why organisations frequently celebrate voice more readily than adaptation. Asking for feedback is culturally attractive. Acting on it creates trade-offs.

Automation makes it easier to absorb feedback without disturbing the organisation. A model can be adjusted. An exception route can be added. A guardrail can be strengthened. A user interface can be changed. These interventions may be useful, but they can also contain the signal locally.

The underlying operating model remains untouched.

If employees repeatedly override an AI recommendation because it ignores an important contextual factor, the immediate answer may be to improve the recommendation. A deeper reading might reveal that the organisation never decided how that factor should influence the outcome. The system is being asked to resolve a trade-off leadership left ambiguous.

Correcting the model can then become another form of buffering.

The technical layer absorbs contradiction so that authority does not have to choose.

Over time, the system accumulates exceptions, rules, thresholds, confidence levels and escalation paths. Each modification makes sense. Together they can encode years of unmade organisational decisions.

At that point the AI system has not learned the organisation’s judgment – it has inherited its avoidance.

Who exactly is meant to improve?

The question sounds simple until it is applied to a real workflow.

When an AI-generated output causes a poor outcome, who should become better because of it?

• The individual user may need to improve at verification.
• The team may need to improve its review criteria.
• The process owner may need to redesign the workflow.
• The model owner may need to improve system performance.
• Leadership may need to clarify the trade-off the system is expected to apply.
• Procurement may need to reconsider the vendor relationship.
• Risk or compliance may need to change the boundaries of acceptable automation.

Several of these responses may be necessary at once. Yet unless the operating model assigns responsibility for adaptation, the most immediate correction usually wins. It is visible, actionable and narrow enough to complete.

Change the prompt. Add another check. Train users again. Escalate unusual cases. Document the limitation. The workflow resumes and the organisation records progress.

What remains unanswered is whether the same underlying failure will return in a different form. The output may improve while the decision logic remains incoherent. Users may become better reviewers while losing the ability to produce the work independently. The system may handle more cases while employees become less able to explain why its answers are acceptable.

A functioning feedback loop therefore needs more than an owner of the output. It needs an owner of adaptation.

Someone must be responsible for deciding where learning lands.

Designing an organisation that can still be changed

Future operating models will need to treat adaptation as explicitly as they treat automation.

That begins by separating three questions which are easily collapsed into one:

• What should the system do differently?
• What should people understand or do differently?
• What should the organisation decide differently?

A technical correction answers only the first.

Sometimes that is enough. A clear defect requires a clear repair. But when behaviour touches judgment, customer expectations, risk, fairness, professional standards or unresolved trade-offs, improving the system alone may leave the organisation unchanged.

The second design requirement is to preserve access to the reasoning behind important work.

People cannot learn from decisions they experience only as outputs. If human judgment will remain accountable for an AI-assisted decision, the role must provide enough exposure to assumptions, alternatives and failure modes for that judgment to develop. Review cannot be reduced to approval. An approval button creates a record of acceptance, not evidence of understanding.

The third requirement is to keep feedback close to authority.

Signals should not end in reports whose recipients lack the power to alter the system. Nor should model teams be expected to resolve organisational tensions through technical configuration. When AI behaviour repeatedly exposes a conflict between priorities, policies or stakeholder needs, the signal has reached leadership territory.

The fourth requirement is to examine successful automation with the same seriousness as failure.

A system can produce better results while weakening capability, narrowing understanding or increasing dependence. Delivery metrics will not reveal this. Leaders must ask whether the organisation could still explain, challenge and recover the work if the system became unavailable or began failing in unfamiliar ways.

Finally, organisations need deliberate moments at which correction becomes learning.

Not every model output deserves reflection. That would create an impossible burden. But recurring overrides, consequential failures, surprising successes and changes in decision patterns should trigger a review at the level where future behaviour can actually change.

The purpose of such a review is not to discuss whether the AI performed well but to determine who must now adapt.

The unlearning company

An unlearning company does not stop collecting feedback. It may collect more than ever.

Every interaction produces data. Every override becomes a signal. Every output can be rated. Every deviation can be classified. The organisation appears attentive, responsive and continuously optimised. Yet its capacity to be changed by experience declines.

Humans encounter less of the work through which judgment was formed. Feedback is routed towards systems whose behaviour can be modified but which cannot understand the social meaning of the correction. Responsibility for adaptation fragments across users, process owners, model teams, vendors and leaders. Local defects are repaired while the underlying decision logic remains untouched.

The company improves its outputs and loses its ability to explain how improvement happens.

That is unlearning.

It is not forgetting in the ordinary sense. The organisation may retain more information, more documentation and more recorded decisions than before. What disappears is the living connection between action, consequence and changed judgment.

Eventually, expertise becomes concentrated in systems nobody inside the organisation fully owns. Employees know when to approve, reject or escalate. Managers know which indicators to monitor. Governance knows where responsibility is formally assigned. But fewer people understand the work deeply enough to reconstruct it, challenge it or redesign it when the familiar failure modes no longer apply.

The organisation remains efficient as long as reality stays within the boundaries its systems recognise.

Outside those boundaries, it discovers what it failed to preserve.

Bottom line

Feedback culture was built for a world in which the actor receiving feedback could change through receiving it. AI breaks that link.

A machine-generated behaviour can be corrected, but correction does not automatically create organisational learning. Someone still has to interpret what happened, decide what it means and change the future system of action. When this ownership is missing, feedback travels through the organisation without finding a responsible receiver.

The danger is not that organisations will stop improving their AI systems. They probably will improve them, repeatedly and at speed. The danger is that they will mistake better system performance for stronger organisational adaptation.

Every time a human learning loop is replaced by an AI execution loop, leadership inherits a design decision: Where should the learning that used to happen through the work now take place?

If the answer remains implicit, it will usually be displaced. Into a model team without business authority. Into reviewers without sufficient practice. Into frontline employees handling exceptions. Into vendors outside the organisation. Or into no one at all.

Continuous improvement does not fail because the feedback disappears. It fails because the receiver does.

The future learning organisation will therefore be defined by more than its ability to gather signals or optimise systems. It will be defined by whether someone remains capable, authorised and obliged to be changed by what the organisation experiences.

Otherwise, the company may become faster, more consistent and more responsive with every iteration… and less able to learn.

And you might ask now: If output improves, KPIs improve, customers are happier and the business performs better… does the loss of learning actually matter?

The answer is yours, as is its consequences.