In my previous article I explored how AI is shifting the bottleneck in professional work from execution to judgement. Producing things has become fast and cheap, so deciding which of the things produced are actually good has become the real constraint. This article explores what to do about it.
When I use the word judgement here, I broadly mean the work of deciding whether an output is accurate, appropriate, and worth acting on. That includes approval decisions, quality checks, risk assessment, brand fit, client suitability, and strategic trade-offs. What these have in common is that they require a person to apply domain expertise to a specific output in a specific context, and to be accountable for the result. It's that category of work which AI cannot yet reliably do for itself.
The obvious initial response, when any capacity problem appears in an organisation, is to add more people. In many settings that does help: adding reviewers alongside clearer standards and escalation rules can improve throughput and consistency. But the volume of AI output is growing faster than most organisations can hire to keep up with it, and adding more people is at best a partial or temporary answer to a structural problem.
The organisations that handle this well won't hire their way through it alone. They'll design systems for how judgement gets applied and scaled. The end goal will be to build conditions in which AI does a large portion of the checking work, with humans focused on the decisions that genuinely require them. Most organisations have more ground to cover to get there than they realise.
A progression, not a switch
Scaling judgement isn't a binary choice between reviewing everything and reviewing nothing. It's a progression, and understanding each stage is what makes it possible to move through them deliberately.
The first stage is direct review, where a human checks every output before it goes anywhere. This is the right approach at first when you're new to a domain, when stakes are high, or when you don't yet have a reliable picture of how AI performs in your specific workflow. The goal at this stage is learning rather than efficiency, because what you observe here is what makes the next stages possible.
The second stage is sampling, where humans review a proportion of outputs rather than every one, using that sample to monitor quality, catch drift, and adjust when necessary. The logic of this stage transfers from mature quality systems in other industries, but the analogy isn’t perfect because AI outputs can fail in both systematic and sporadic ways simultaneously. A model can fail systematically if prompts, source data, or settings are consistently flawed, and it can also produce ninety-nine accurate responses followed by one highly plausible but completely wrong one, with no visible warning in the surrounding outputs. This makes sampling AI output a more demanding discipline than traditional quality control because the reviewer isn't just spotting an obvious defect. They're spotting a plausible falsehood, and that can require deep domain expertise rather than routine inspection. It's another reason why judgement doesn't scale like execution.
The third stage is designed oversight, where quality standards have been encoded into the workflow itself so that a significant portion of checking happens automatically before output reaches a human reviewer. A second AI pass can evaluate outputs against a defined rubric, flag factual inconsistencies against known sources, or surface potential issues before a human sees the result. More sophisticated implementations assign confidence scores to outputs. These act as priority signals, allowing reviewers to triage their attention and concentrate scrutiny where it's most likely to be needed, rather than treating every output the same. A simpler version of the same idea is a traffic-light system that routes outputs by risk level before anyone opens them.
These mechanisms need careful design because a priority signal is only as reliable as the criteria it measures against and a poorly calibrated system can create false security. They also require real preparation, including stable rubrics for what good output looks like, agreed definitions of quality, and usable source data to check against. For most organisations, that preparation can be a significant blocker in moving to this stage.
Designed oversight can only handle the portion of quality that can be defined clearly enough to check systematically: consistency with guidelines, factual accuracy, compliance requirements. By offloading those routine checks, it frees human attention for the decisions that genuinely require it, the context-dependent calls that depend on deep expertise and are unlikely to be fully encodable. The intention of this stage is to preserve human judgement for the hardest and most important calls.
Calibrating oversight to risk
The progression above isn't a single ladder that an entire organisation climbs together. Different tasks warrant different levels of oversight, and the right question isn't simply "how do we review less?" but "how much human supervision does this particular task actually require, and what are the consequences if the automation fails?"
A task that produces output used directly in a client-facing document carries different risk from one that feeds an internal first draft. A task where an error is immediately visible and easily corrected is different from one where an error might propagate silently through a process before anyone notices. Risk is the calibrating variable, and as execution scales, those calibration decisions need to be made explicitly and systematically.
It's also worth noting that this kind of top-down design only works if the organisation has visibility of where AI is actually being used. Many organisations are still grappling with employees using personal accounts or unsanctioned tools for tasks that carry real business risk, which means a risk framework can only be as good as the picture of execution it's built on. Implementing strong governance procedures can help on both counts, providing the visibility needed to calibrate risk and reducing exposure at the same time.
Organisations already make these risk-based judgements about human-generated work: some documents require a director's sign-off, others just need to be filed. What's new is that AI is increasing the volume of outputs across all risk levels simultaneously, which means those decisions need to be made deliberately rather than left to individual habit.
What this looks like in practice
The clearest public illustration of this progression has played out in software development. In February 2025, Andrej Karpathy, co-founder of OpenAI, coined the term "vibe coding" to describe a way of working with AI where you "fully give in to the vibes, embrace exponentials, and forget that the code even exists." Exactly one year later, he proposed a revised framing: "agentic engineering." His description: "you are not writing the code directly 99% of the time, you are orchestrating agents who do and acting as oversight." The goal, as he put it, is to "claim the leverage from the use of agents but without any compromise on the quality of the software."
That shift from "forget the code exists" to "act as oversight" reflects the progression above. It works particularly well in software because code can be checked against an objective standard by a machine: a compiler tells you whether it runs, a test suite tells you whether it behaves as specified. Software has unusually powerful tools for automated verification, which is why it's ahead of most professional domains in implementing this kind of oversight. The same progression is available elsewhere, but defining what good output looks like requires more work in domains where there's no objective test to fall back on.
Let’s consider marketing as an alternative example. AI makes it easy to produce content at scale with multiple versions of copy, creative variations, audience-specific adaptations, all generated in the time it previously took to produce a single asset. The efficiency gains are real, but they can be risky because studies have found that a significant proportion of consumers say AI-generated advertising makes them less likely to choose a brand. That finding comes with caveats because the effect varies by category, audience, and how visibly AI is involved. The judgement problem in marketing isn't only about whether any individual piece of AI-generated content is good enough, it's also about what happens when the volume of output starts to lower the average quality threshold of what gets published, potentially eroding brand distinctiveness, or reducing the care that audiences notice and respond to. A marketing team that scales its output without scaling the judgement applied to it is accumulating a risk that compounds quietly and is harder to catch than it would be in a domain with objective verification.
Financial analysis presents a different risk profile. The failure modes here are less about tone and more about substance: input data quality, formula logic, modelling assumptions, and whether the narrative interpretation reflects what the numbers actually say rather than what someone hoped they'd say. A finance team that produces more analysis than it can meaningfully check isn't more informed, it's more exposed.
The upstream condition
Both the progression and the risk calibration depend on one thing: being clear about what good output looks like in your domain. Much of what experienced professionals know about quality is tacit, internalised through practice and expressed through judgement calls rather than rulebooks. When standards are hard to articulate clearly enough to encode, that difficulty is worth examining. It can be a sign that the original intent wasn't precise enough, and sharpening it makes the encoding task more tractable. But some high-value judgement is genuinely hard to formalise even when intent is entirely clear, because it depends on context, timing, and the kind of pattern recognition that comes from years of experience in a specific field.
These articles are informed by my experience delivering AI capability programmes with Dixon AI, and by the Purpose, Execution, Judgement framework specifically (see link in the comments). The judgement design challenge is something I see across organisations and sectors, and while the pace and starting point vary, the direction tends to be the same: move from reviewing everything by default, to sampling by design, to building the systems that let human attention focus where it genuinely matters.