Before You Let an AI Act: What Must Be Visible

The short answer

Before you let an AI system act on your behalf, four things have to be visible: what it is trying to do, how it decides what matters, what actually shaped the result, and where its authority ends and yours resumes.

That is not a demand for full internal transparency. Nobody needs to inspect every weight, every intermediate step, every internal state. It is a narrower and more practical standard: enough visibility to govern the action you are about to delegate.

The reason to care is not tidiness. A system whose intent, criteria, and evidence stay inside the head of whoever built it can still work — right up until that person is unavailable, or the volume grows, or six months pass and nobody remembers why it does what it does. Systems like that do not fail loudly. They rely on memory, on heroics, on someone standing nearby to interpret them, and they collapse the moment scale or time is introduced. Legibility is not a cosmetic preference; it is an operational precondition for responsible delegation.

A workflow that works and cannot be governed

Consider an ordinary one. A company routes incoming vendor invoices through an AI step: it reads each invoice, matches it against the contract on file, approves what looks routine, and flags the rest for a human. It has run for months. The queue clears. Nobody is complaining. By every visible measure, it works.

Now ask four questions about it.

• What is it optimizing for? Nobody stated it. In practice it is optimizing for queue depth, because that is the number anyone looks at. It was never told that a wrongly approved invoice costs far more than a wrongly flagged one.
• What counts as routine? Nobody can say, including the team that runs it. "Routine" is whatever the system has been treating as routine, and that definition has drifted as the invoice mix changed.
• What did it check this against? Unrecorded. When a disputed approval surfaces, no one can reconstruct which contract version was in force, whether the amount was verified or inferred, or what changed between the run that flagged a vendor and the run that stopped flagging them.
• When is it supposed to stop and ask? There is no answer, so it never stops. A case it has no basis to judge looks identical, on the way out, to a case it judged confidently.

None of this shows up as a failure. It shows up as a system that is fine until the first consequential dispute, at which point the only person who can explain any given decision is the person who built it — and their explanation is a reconstruction, not a record. That is the actual risk of illegible delegation: not that the system is wrong, but that you cannot tell whether it is right, and you cannot find out afterward.

The four things that must be visible

These are not four documents to produce. They are four questions a system's own surfaces should be able to answer without a person translating.

1. Intent — what is this trying to do?

The goal it is pursuing, the scope it is working inside, the constraints that matter, the tradeoffs already accepted, and what would count as failure. Most operators hold this internally and never externalize it, because they have always been the review layer themselves. Delegation breaks that arrangement: a system cannot act on an aim that was never surfaced, and neither can the colleague who inherits the workflow next year.

What it looks like in practice: a stated purpose, a declared role and its boundaries, and a written definition of failure specific enough that a particular output could be judged to have failed it.

2. Judgment — how does it decide what matters?

What distinctions it is applying, what criteria govern selection and emphasis, what it treats as settled versus uncertain versus out of scope, and which cases it is not entitled to resolve on its own. This is the layer people resist most, because judgment feels personal and criteria feel reductive. But the constraint is unavoidable: judgment that cannot be articulated cannot be delegated. It can only be imitated, and imitation is invisible until it is wrong.

What it looks like in practice: criteria stated rather than implied, exceptions named, uncertainty marked as uncertainty, and explicit escalation conditions — the cases where the system must surface rather than decide.

3. Causality — what actually shaped this result?

Which inputs were used, which assumptions were load-bearing, which dependencies mattered, where inference is doing work that evidence is not, and what changed between one run and the next. This is the layer organizations skip most often, because skipping it costs nothing on the day you ship. It costs later: without it you cannot debug, audit, or improve, and the same mistake returns in six months wearing a new name.

What it looks like in practice: source anchors on claims, a visible line between evidence and inference, recorded dependencies, and change notes when the basis of a decision shifts.

4. Authority — where does the system stop and you resume?

What the system is doing right now — advising, analyzing, executing, translating — and what it will not do under any circumstance. Where authority sits, what you remain responsible for, and how much trust the output has earned. This is the layer that decides whether the human in the loop is exercising oversight or performing it.

What it looks like in practice: role stated plainly, authority boundaries that hold without someone watching the gate, explicit handoff points, and a maintained distinction between a system that supports your judgment and one that has quietly replaced it.

Enough to govern — not everything

The standard here is deliberately bounded, and the boundary is the useful part.

A legible system does not need to expose every internal detail. It needs to expose enough for a person to understand the nature of the task, inspect the distinctions that govern it, see what is load-bearing, recognize where the system is uncertain, and know where judgment must remain theirs. Below that line you are guessing. Above it you are reading telemetry, which is a different activity and mostly a way to feel informed.

This also rules out the usual substitutes. Fluency is not legibility — a system can sound confident without showing anything about what governs the confidence. Structure is not legibility, if the structure cannot be traced back to criteria or evidence. Polish is not legibility; it is frequently its replacement. And a successful response, a clean dashboard, or a complete event log is not legibility either, because none of them state what the system was aiming at or what it was entitled to decide.

The uncomfortable part is why this is rare. Legibility forces responsibility. If intent is visible, you can be wrong on record. If criteria are visible, your judgment can be examined. If causality is visible, the excuses evaporate. Ambiguity is comfortable, and most systems stay illegible because someone benefits from the ambiguity — usually without ever deciding to.

What this does not claim

• Not every AI system should be autonomous. Some work should not be delegated at all, and making it visible does not change that. Visibility is what makes delegation governable when delegation is appropriate — it is not an argument that it always is.
• This is not a demand to expose every technical internal. Model internals, intermediate states, and implementation detail are not the subject. The subject is the aim, the criteria, the load-bearing basis, and the boundary of authority.
• "Explainable" is not the same as governable. Explainability is a property of a model's output. Legibility is a property of the surfaces an operator actually acts on.
• A log, an audit trail, or a successful response is not sufficient evidence. Each records that something happened. None of them records what it was for.
• Nobody should have to serve as the permanent interpreter. If understanding a system's behavior requires the founder or the original engineer standing nearby, the system is not legible — it is dependent, and the dependency is a person.
• Visibility does not guarantee good outcomes. It makes decisions inspectable and failures locatable. That is a different property from being right, and it is the one you can build for.

Where this sits

This page is about the condition for delegating at all. The neighboring question — which decisions belong to the system and which must remain yours — draws the boundary; this one states what has to be visible for delegation inside that boundary to be governable rather than merely convenient. The two are usually needed together: a boundary nobody can see is not a boundary, and visibility with no boundary is just more information.

At larger scale, the same requirement is what lets an operator retain authority over a system you cannot fully comprehend in detail. The general form of the object is governable AI action under human authority — AI acting under your name with legibility, bounded delegation, reviewable memory, and inspectable action. For the larger structure this belongs to, see the Verse.