Decision Architecture for Operational Agents

How operational judgment is recovered, proven and given authority before agents act.

Our earlier essays make the case that agents need an architecture around them. The Loop Is Not the System shows what the popular agent diagram leaves out. How We Decide traces how judgment moved from managers into systems. Discovery and The Minimum Viable Pilot describe where to begin. Giving an agent access to operational data does not give it an understanding of how the operation decides. We take that argument as made.

This paper turns to the harder questions. How is judgment recovered from people who cannot fully describe it? How is the resulting logic proven before anyone relies on it? How is authority to act granted, limited and kept current? We answer each with the research behind it and work one case through from start to finish.

Our position is this. Agents can take a reliable part in consequential operational decisions only when the organization has made three things explicit and tested each against evidence: what its operation consists of, how it actually decides, and who is authorized to act. Together these form an operating architecture for decisions, which must be built before agents are deployed and maintained while they operate.

A decision with minutes to spare

An inbound flight will reach a connecting hub about 85 minutes late. Its aircraft is due out again 55 minutes after the original arrival time, with the same pilots, whose duty day is already long. The outbound flight carries passengers connecting to the evening’s last departures. The aircraft is due at the maintenance base tonight for a scheduled check, and its gate is booked for another arrival.

Crew control, maintenance control, passenger recovery and hub control each see part of this. The operations controller reconciles them while the options are still open, largely on judgment no one has written down. The scene is a conceptual illustration, not a deployment, and the kind of decision executives are now being asked to hand, in part, to AI agents. It raises three questions. What should the agent do? Who authorized it? And how will anyone know it is right?

Real disruptions show what an agent would inherit. After Winter Storm Elliott in December 2022, Southwest’s own review found that its aircraft and crew scheduling teams were pushed onto manual processes that could not keep up, and the airline committed to more objective escalation criteria [1]. The U.S. Department of Transportation recorded 16,900 canceled flights [2]. After the July 2024 global IT outage, Delta’s chief executive said a crew-tracking tool could not process the number of changes [3]. Neither event was an AI failure. Both describe the conditions an agent placed in that operation would inherit.

The decision as the unit of design

The unit of design is a recurring operational decision with measurable value at stake; novel, one-off decisions stay with people. At the hub, it is the recovery of a delayed rotation: a coordinated plan owned by the operations controller on duty, containing decisions owned by crew, maintenance, passenger and hub control, with a window that closes as each option stops being feasible.

The architecture is the set of governed artifacts agents consult while they work: a shared model of the operation, maps of the decision logic, a table of decision rights, oversight settings and a register of the value each decision should deliver. None of the parts is new. What agents add is action across systems, which calls for authority enforced where they act, logic applied consistently between agents, and a record of what each acted on and why. The sections that follow show how each artifact is built, tested and kept current.

Make the operation explicit

Return to the hub. The facts the controller needs sit in separate systems: flight operations, crew management, reservations, maintenance and gate planning. Each was built for its own function, and their definitions can differ. One system may measure delay from gate departure, another from takeoff. Some facts that matter are recorded nowhere, such as a captain’s report that the crew is tiring.

Research on airline disruption management describes the same pattern. A widely cited review found that, at the time, no general tool could handle aircraft, crew and passengers together in a single system, and that a gap separated the reality of operations control from the decision support available to it [4]. A later review notes that aircraft are typically recovered first, then crew, then passengers [5]. Each step works within choices already made by the one before it.

An agent placed in this environment inherits the fragmentation. It needs a shared representation of the operation, scoped to the decision: the entities involved, from flight legs and crew duty periods to passenger itineraries and gates; the states each can be in; the events that change them; and the constraints that bind them. Definitions are reconciled once, in the shared model, so every agent reads the same operation.

That representation is necessary, and it is not enough. It can tell an agent that the pilots are close to their duty limit and that the aircraft is due at the maintenance base. It cannot say which matters more tonight. It also brings conflicts into view. When two systems define the same thing differently, someone has to decide which definition governs. The model records that decision. It does not make it.

Recover how it actually decides

Disruption recovery has written procedures, and they handle hard constraints well. Crew legality and maintenance limits are never traded against cost. What they rarely capture is the rest: the trade-offs between propagated delay and passenger disruption, or between today’s recovery and tomorrow’s, and the unstated thresholds at which a controller swaps an aircraft instead of delaying a flight.

Asking experienced controllers to describe that reasoning does not recover it. Nisbett and Wilson’s review of the evidence concluded that people may have little or no direct introspective access to their own higher-order cognitive processes [6]. A controller can state the rule and still be unable to name the threshold that drives the call.

The reasoning is drawn out through real cases instead. The Critical Decision Method was developed for tasks with high time pressure and changing conditions; it reconstructs specific incidents and probes the cues and judgments that shaped them [7]. Contrastive questioning asks why two similar disruptions were handled differently, which brings unstated factors to the surface. The recovered logic is then modeled branch by branch.

Some branches should stay with people. Kahneman and Klein concluded that the quality of an intuitive judgment depends on how predictable the environment is and on the person’s opportunity to learn its regularities, and that subjective confidence is not a reliable guide [8]. Whether to sacrifice one flight to protect the wider network during a widespread disruption is such a judgment zone: rare, consequential and hard to learn from experience.

Other branches belong to people by regulation. Under U.S. rules, extending a pilot’s flight duty period for unforeseen operational circumstances requires the concurrence of the captain and the carrier, and an extension of more than 30 minutes may be taken only once before a required rest period [9][10]. The logic can show that an extension would make a plan feasible. It cannot take one.

This also settles how language-model agents are governed. The model reads what rules cannot, such as a captain’s report or a station’s message; it assembles options, explains trade-offs and drafts briefs. Formalized branches, such as legality checks, run as rules. Judgment-zone branches are prepared by the model as briefs for a person. Whatever the model proposes is checked against the governed rules and constraints before it can be acted on, because a fluent explanation is not evidence that a decision is valid. Test results then describe the logic that actually runs.

Prove the logic before trusting it

A logic map that convinces the people who built it has not yet been tested. The test is replay. The mapped logic is run against past disruptions held back from elicitation, and its choices are compared with what experienced controllers decided. It must meet a defined threshold before anything is built on it. Every disagreement is reviewed and classified as a gap in the model, an inconsistency between experts, or legitimate variation in judgment.

Replay is only as good as the history behind it. Records can be incomplete, past decisions inconsistent and conditions different from today’s. Replay is therefore the gate for building, not for live operation. Before a class goes live, its logic must also respect hard constraints and policy in every case tested, respond sensibly when information is missing, stale or conflicting, and hold up on novel and deliberately difficult cases. It then runs in shadow alongside the people who decide today, through a period that includes peak disruption, so its choices are compared prospectively before any are acted on.

Models built from experts’ own judgments have a long record. In Armstrong’s review, they forecast more accurately than unaided expert judgment in eight of eleven comparisons, less accurately in one, with two ties, and they can expose biases in how experts decide [11]. Camerer’s analysis indicated that they improve judgment slightly under almost any realistic task conditions [12]. This research concerns forecasting models, so we treat it as support for the principle, not as evidence about airline recovery.

Validation answers three separate questions. Fidelity asks whether the logic reproduces what experts did; replay tests it, and it is the gate for building. Consistency asks whether experts agree with each other on comparable cases. Where they differ, the difference is resolved against policy and stated priorities, not by majority. Such differences can be large and are usually invisible. At one insurer, executives expected underwriters’ premiums for the same case to differ by about 10 percent; the median difference was 55 percent [13]. This is an analogy; we found no comparable study of airline controllers.

Correctness asks whether the decision was good. In disruption recovery it is never fully observable, because the outcome of the option not taken is never seen. It is built up through outcome review, analysis of overrides and accounting for the value each decision delivers, with simulation of the alternatives filling part of the gap.

Decide who may act

In our approach, each decision class and exception branch carries one of five autonomy levels: the agent observes; prepares options; recommends, with a person actively accepting; decides under oversight; or decides on its own. Each also has escalation triggers and one named accountable role. This goes beyond decision support, where the system informs and a person always decides: here, authority beyond recommendation is earned one action at a time, on evidence. Exhibit 1 shows where these levels sit in the architecture as a whole.

Autonomy is granted on evidence. A class enters service only after a shadow period at the level below, and it rises above recommendation only if it has passed replay at its threshold and its actions are reversible in their consequences, not only in the system. A notice can be withdrawn from the system after passengers have already acted on it. The level is enforced in the systems agents act through and is lowered as readily as it is raised.

At the hub, agents for aircraft, crew, passengers and gates each recommend into a coordinated plan. That plan is a decision class of its own, owned by the operations controller. We set it to run at the level of the most restricted action it contains, so any plan that includes a cancellation is decided by a person. That is a design choice, not a property of every multi-agent system; other designs execute a plan’s permitted steps and hold only the restricted ones. In the case’s illustrative levels, opening the case runs under oversight; most recovery actions start as recommendations, and rebooking stays one because a passenger notice cannot be recalled; holding a departure for connections and extending a duty period stay with people, prepared by agents.

The plan earns its owner in the case’s central conflict. Passenger protections at the revised departure time would put the pilots over their duty limit, while the aircraft agent prefers a swap that breaks some connections. Because the agents share one model, the coordinating agent sees that the protections rest on an infeasible plan, holds them and puts the real choice to the controller. Multi-agent recovery with people in the loop has been studied for well over a decade [14]; the architecture adds explicit, enforced authority.

Time windows are part of each decision’s definition. Where a window is shorter than a realistic review, the action earns a higher level on evidence or stays with a person; it is never approved because nobody objected in time.

Regulatory authority does not move. Under U.S. rules for domestic operations, the certificate holder is responsible for operational control, and the captain and the aircraft dispatcher are jointly responsible for preflight planning, delay and dispatch release [15]. Regulators draw comparable distinctions: EASA’s June 2026 consultation draft on AI in aviation runs from assistance and human–AI teaming to advanced automation [16].

Keep it true as the operation changes

The operation the model describes will change. Schedules, fleets, labor agreements and regulations move, and experienced controllers retire with their judgment. Sculley and colleagues found that machine-learning systems commonly incur large ongoing maintenance costs, including from changes in the external world [17]. Decision logic is exposed to the same pressure.

The architecture is therefore maintained as an operating asset. Drift shows up in leading signs: rising escalations, clusters of overrides, new cases the model cannot classify, and falling agreement when replay is rerun on recent disruptions. Each override is reviewed, because it shows either that the model is out of date or that the person was wrong. Changes to the shared model and the logic pass through controlled change, with replay of the affected branches and approval by each artifact’s named owner. That control covers the governed artifacts; model versions, prompts and code need controls of their own. When the people who hold the logic move on, it is elicited again from their successors.

Human skill needs the same care. Bainbridge observed that skills deteriorate when they are not used, so an operator who has mostly monitored an automated process may no longer be an experienced one, and that the most successful automated systems may need the greatest investment in operator training [18]. Controllers who take over in the hardest disruptions need regular practice making those calls.

What leaders do first

Start with one recurring decision class that carries measurable value, such as recovery of a delayed rotation at one hub; our pieces on Discovery and The Minimum Viable Pilot describe how to choose it and run the first build. Four conditions need to be in place: experienced practitioners named and given time to take part; historical decision records to replay against; a risk owner with authority to approve autonomy levels; and a funded steward to keep the model current.

Measure a baseline from past disruptions before anything is activated: time to an accepted plan, propagated delay, misconnected passengers, legality exceedances, cost per disruption and consistency between controllers on comparable cases, normalized for severity and weather. Then stage the work: replay, shadow operation through a period that includes peak disruption, and autonomy raised one action at a time as evidence accumulates. We claim no results here; the hub is an illustration.

That sequence answers the three questions from the opening. What the agent should do comes from a shared model of the operation and logic proven against past decisions. Who authorized it is recorded in an autonomy level set on evidence and owned by a named role. How anyone will know it is right comes from replay, outcome review and a baseline measured before the agent acted, all kept current as the operation changes.

References