An AI agent can tolerate uncertainty when choosing another search query. The same uncertainty may be unacceptable when overwriting a trusted record or publishing a factual claim. A single autonomy setting cannot express that difference.
The design question is action-specific, but it has a hard boundary: can the system discover that its judgment is wrong in time to prevent unacceptable consequences? Without a meaningful way to challenge the judgment, accepting more uncertainty can become accepting undetectable failure.
Separate Uncertainty About Goals, Facts and Routes
“Uncertainty” can describe several different problems. A vague goal leaves the system unsure what success means. An incomplete evidence base leaves it unsure what is true. An open route leaves it unsure which step will be most useful. A creative task may simply allow multiple good results.
These situations need different responses. Clarify the goal. Gather evidence. Explore a permitted route. Generate alternatives. Giving an agent more freedom does not resolve all four.
A clear action contract can coexist with an uncertain investigation. The system may not know which source will settle a contradiction, but it can still know which repositories it may read, which records it must preserve and when the research budget ends.
The Hard Boundary: What Could Show That This Is Wrong?
A successful tool call establishes that an operation completed. It does not establish that the operation was justified. A deployed page can match the approved file exactly while publishing an unsupported claim. Execution verification and substantive falsification answer different questions.
Operationally, falsification means identifying evidence that would defeat a claim, interpretation or action rationale, then giving that evidence a route to rejection, correction or stopping. It is not a promise that every false claim will be caught. Passing the checks leaves residual uncertainty; it does not prove truth.
Before granting consequential autonomy, ask: what observation could make us reject this decision, how would we obtain it, and would it arrive before the damage becomes unacceptable? A test that reveals an error only after an irreversible commitment may be useful for learning but inadequate as an execution safeguard.
This is the boundary proposed here, not a universal scientific or regulatory rule. Where a meaningful falsification mechanism is unavailable, the alternative is a validated, bounded heuristic with an explicit failure-detection strategy. If neither exists, restrict the system to exploration or proposals; do not treat confidence or approval as evidence of correctness.
For preferences such as tone or visual style, specify an acceptance rubric and an accountable reviewer rather than pretending that taste has a factual falsifier. Factual premises and consequential effects still need their own challenge mechanisms. Falsification, Not Determinism develops that distinction.
The Difficult Cases Have an Incomplete Test Oracle
Recognizing when ordinary checks are inadequate is often harder than implementing the agent. A test oracle is a basis for judging correctness. The software-testing literature documents the difficulty of constructing one and approaches such as partial and metamorphic oracles. Open-ended interpretation makes this especially visible.
Look for a mismatch between what is easy to measure and what actually matters:
| Process | Available check | What can remain undetected? |
|---|---|---|
| Fix a software defect | Existing tests pass | An untested requirement or a weakened test |
| Extract a knowledge graph | Valid schema and supported edges | A consequential relationship that was never extracted |
| Research a contradiction | Citations resolve and match quotations | Missing counterevidence or an incorrect synthesis |
| Diagnose a service incident | A monitored metric recovers | A temporary improvement that hides the cause |
| Publish an article | Correct version is deployed | An unsupported central claim |
These are illustrative failure modes, not evidence that the processes cannot be automated. Each calls for a different challenge strategy. Test the behavior described in the defect report; sample source material for omitted relations; search specifically for counterevidence; monitor recovery beyond the first positive signal.
Three warning signs deserve particular attention: success is defined by a proxy, relevant failures are omissions, or feedback arrives only after the next commitment. The absence of a complaint or contradiction is especially weak when nobody has looked for one. Where even a qualified reviewer cannot identify an adequate challenge, “human in the loop” does not solve the epistemic gap.
The practical discovery exercise is to write down a plausible case that passes every current check and is still wrong in a consequential way. If the team can construct one, decide whether an added test, a dynamic heuristic or a narrower scope addresses it. If the team cannot assess such cases, record the evaluation gap rather than equating it with safety. Testing AI When Specifications Cannot Capture Reality explores this problem further.
When a Robust Dynamic Heuristic Is Necessary
A complete correctness test is not always available. An incident investigation, for example, may need to change its next diagnostic step as observations arrive. A useful heuristic can combine independent signals, track contradictions, detect unfamiliar conditions and escalate when the evidence no longer supports continued action.
“Dynamic” means that the response adapts to observations under governed rules. It does not mean that the agent can quietly rewrite its success criteria whenever a check fails.
A heuristic earns limited operational trust through evidence about its failure modes: representative and adversarial cases, known misses, sensitivity to changed conditions, monitoring latency and defined stopping behavior. Its observations should reach beyond the model’s own assessment. Another generated critique may help find problems, but agreement between model outputs is not independent confirmation.
NIST’s AI RMF calls for documenting evaluation limits and tracking risks where measurement is difficult. It does not certify a heuristic as sufficient. The owner must decide whether the demonstrated coverage, detection speed and residual risk justify this specific action. Until then, test in isolation or shadow operation without giving the heuristic authority over consequential changes.
There Is No Universal Confidence Number
A rule such as “execute above 90 percent confidence” is incomplete unless that score has a defined meaning and demonstrated calibration for the relevant task. A model’s fluent certainty is not enough to establish the probability of a harmful outcome.
NIST frames risk through likelihood and consequences and emphasizes that risk tolerance depends on the application and context. It does not provide a universal agent-confidence threshold. In practice, acceptable residual uncertainty must be chosen for the action and checked against representative cases.
For one action, the important question may be entity identity. For another, it may be whether the source actually supports a claim. For a publication, the approved content version and destination also matter. Combining these into one unexplained score can hide the condition that failed.
Turn the Boundary into Action Conditions
Start with authority: the action must fall inside an established scope. Evidence then needs to support the specific object, operation and intended result. Falsification identifies observations that would reject the decision; execution verification checks what actually happened. Both need detection timing appropriate to the consequences. Reversibility defines available recovery. Consequence limits establish how much damage or exposure is acceptable before intervention.
These conditions interact. An edit may be easy to revert locally yet expensive to undo after downstream systems consume it. A research call may be read-only while transmitting confidential material to an external service. “Read” and “write” are useful categories, but neither is a complete risk assessment.
Anthropic’s tool-design guidance recommends reducing ambiguity through clear tool contracts and defined inputs and outputs. Narrow operations make the conditions easier to enforce than a general administrative tool with broad credentials.
Work Through a Contradictory Claim
Imagine two sources giving different release dates for a software feature. The agent can search for a dated announcement and a versioned release note. It should preserve the conflicting statements rather than immediately choosing the more convenient one.
Finding another search term requires limited commitment. Adding a qualified interpretation to a draft requires stronger support. Replacing an accepted fact in a knowledge store requires an explicit acceptance rule and a record of what changed.
If the sources remain inconsistent, the useful result may be an unresolved finding with clear provenance. Completion can mean that the investigation reached its defined stopping condition. It need not mean that the contradiction disappeared.
Use Action Contracts Instead of Broad Permission
A publication action could require the following contract:
| Condition | Evidence required before execution |
|---|---|
| Scope | Named site, permitted path and authorized operator |
| Artifact | Identified content version and associated assets |
| Quality | Claim review plus build checks; neither substitutes for the other |
| Challenge | Counterevidence search and a route to reject unsupported claims |
| Approval | Acceptance tied to that artifact and destination |
| Recovery | Recorded prior release and a usable recovery procedure |
| Verification | Deployed version matches the accepted artifact |
This is a proposed engineering rubric. The exact controls depend on the site and its consequences. A personal draft repository and a public customer announcement should not inherit identical policies just because both contain Markdown.
A runtime can evaluate these conditions independently. When the artifact changes after approval, the approval may no longer cover it. When the destination changes, the earlier authorization may no longer apply. Explicit contracts make that distinction visible.
Human Approval Must Resolve a Real Decision
Microsoft’s experimental Magentic-UI includes configurable action approvals and mechanisms for human control. That is a useful implementation example, rather than proof that approval alone makes an action reliable.
A reviewer needs the proposed action, target, evidence, unresolved issues and expected consequences. “May I continue?” supplies little basis for judgment. A concrete preview can let a human resolve a preference or accept a particular consequence.
Approval should not quietly supply missing facts. A reviewer can authorize a known risk, but a system should still identify an unknown recipient or an ambiguous record before executing against it. Failed authorization, missing evidence and accepted uncertainty remain distinct states.
Test Escalation as Carefully as Execution
Evaluate cases in which the agent should act, investigate further, ask for clarification or stop. Include conflicting evidence, stale state, changed permissions, repeated requests and unavailable verification tools. A system that completes easy tasks but proceeds through uncertainty without limits is not ready for the same autonomy elsewhere.
Measure more than successful completions. Record inappropriate actions, unnecessary escalations, human correction effort and whether failures were detected before they spread. Thresholds should evolve from these observations and the owner’s tolerance for consequences.
The practical boundary is whether errors can become visible and actionable soon enough. Falsification and bounded heuristics make that boundary operational; neither eliminates uncertainty. The next article applies that boundary to a complete content task.