AI System Design

Operational reasoning is the disciplined design of how a system turns changing operational conditions into governed decisions and actions over time: observing evidence, interpreting it, applying rules and judgment, acting or recommending, and improving through feedback. It is a system-level discipline, and keeping it at the system level is what prevents the model from being mistaken for the whole.

Good design begins with an operational question: what decision or outcome are we trying to improve? Starting instead from an available model and searching for a use inverts the order and tends to produce demonstrations rather than systems.

Consider a benefits-management organization. The goal is rarely "build a chatbot." The real problem is usually sharper: which families are at risk of losing benefits, why, and what intervention should occur before the loss? That framing changes the architecture immediately. The system now has to gather information from several records, determine which rules apply, find missing or conflicting information, weigh deadlines and dependencies, recognize exceptions, estimate risk, recommend an intervention, decide whether a staff member must approve it, and record its reasoning and resulting action. The model has become one function inside a decision system. That is system design.

Underneath most useful systems is a single operating cycle: observe, interpret, reason, decide, act, evaluate, learn. Observe gathers the relevant state of the world. Interpret converts raw information into something usable. Reason applies models, rules, and retrieved knowledge. Decide determines what should happen. Act produces an output or triggers another system. Evaluate checks whether the result was correct or useful. Learn folds that information back into future behavior.

The model contributes to interpretation and reasoning, and it is seldom the only interpretive layer. Interpretation also runs through deterministic parsing, schemas, and validation; decisions are split among business rules, optimization logic, model recommendations, and authorized human review; evaluation combines automated checks, simulations, user feedback, and quality assurance. Naming the loop keeps this distribution visible, so the model is held in proportion to the system around it.

Two features of the loop carry most of the weight.

Learning is broader than retraining. In conventional machine learning, improvement usually means retraining a model on new data. In agentic systems, improvement more often comes from updated instructions, better knowledge sources, new tools, revised rules, changed workflows, or stronger evaluation. This is why modern AI design resembles systems engineering more than isolated model development.

The loop must survive failure. AI systems fail differently from conventional software. Traditional systems tend to fail visibly: a service crashes, an API returns an error. AI systems can fail while appearing to work — producing plausible but incorrect output, retrieving the wrong evidence, misreading a request, following malicious instructions hidden in retrieved content, or degrading quietly as the environment shifts. Operational reasoning therefore includes explicit failure management: validation, retries, fallback models, circuit breakers, confidence thresholds, human escalation, version control, rollback, and testing against known cases.

These patterns address different failures and are not interchangeable. A bounded retry with backoff absorbs a transient service fault. A circuit breaker stops calling a dependency that is failing persistently, before the failures cascade or drain resources. A fallback provides a safe degraded mode. Escalation routes ambiguity, high consequence, or policy conflict to a person. A rollback reverses a bad deployment or configuration. A rollback cannot, on its own, undo an action already taken in the outside world, which is why irreversible actions have to be gated before they occur rather than corrected after.

One assumption inside the loop is worth stating plainly, because it is easy to miss. Observe, evaluate, and learn treat the environment as though it holds still while the system measures it. A system that takes actions changes the world it observes. Interventions shift the population they act on; people adapt to rules once they understand them; the distribution the system reads moves partly because the system is acting on it. In low-stakes settings this drift is a nuisance. In consequential ones it can quietly invalidate the evidence the loop depends on, and it is a standing reason to keep measurement and human review active long after launch.

Ontology is the explicit representation that makes an organization's data meaningful to the system: the entities it recognizes, the relationships among them, the states they can occupy, the constraints that apply, and the authoritative sources those facts are drawn from. It is what turns a store of data into something a system can reason over. A model cannot reason about what it cannot see, and it reasons badly about what it sees wrongly.

An ontology looks like a neutral description of the domain, but the choices inside it are judgments. Return to the benefits example. Deciding which families count as "at risk" fixes a threshold, commits to a theory of what drives the risk, and settles which errors are acceptable — a missed family weighed against a false alarm. Those commitments are contestable, and they are usually resolved by whoever builds the data model rather than by whoever holds the authority to make them. Whoever defines the entities defines the outcomes. This is why a representational error does more lasting damage than a model error: a model can be tested against outcomes and corrected, while an ontology tends to pass as the world itself and is rarely examined. Getting the representation right, and getting the right people to own it, is often the most consequential decision in the system.

This need not be a formal ontology in the knowledge-representation sense — classes, properties, and axioms in a formal language — though it can be. The discipline holds regardless of implementation: make the domain explicit, and make the system's representation of it both faithful to the real domain and legible to the people who operate it.

It helps to separate several things that are easily conflated, because they fail differently and are fixed differently:

• ·The ontology defines the domain's conceptual structure — what exists and what counts as what.
• ·Data and knowledge systems instantiate that structure and govern who may access, change, retain, and trust the information.
• ·Retrieval selects the slice of that structure relevant to the task at hand.
• ·Memory retains approved, purpose-bound state across interactions.
• ·Provenance records where a given fact, recommendation, or action basis came from.

An organization can hold documents, a database, and a retrieval pipeline and still lack a coherent ontology; it can also have a clear conceptual model but stale data, poor retrieval, or unclear ownership. Diagnosing which of these is missing is often the real work.

Retrieval brings the relevant representation into the model's context at the moment of reasoning. Retrieval-augmented generation is the common technique: rather than relying only on what a model absorbed in training, the system fetches relevant information when a request arrives — policies, records, contracts, manuals, case histories, procedures, prior interactions — and supplies it to the model. Retrieval may use keyword indexes, semantic vector search, structured queries, or a combination, because exact and semantic search solve different problems. The design question is whether the right representation of the world reaches the model at the right moment; having a retrieval pipeline is no guarantee that it does.

Memory extends the ontology across time. Short-term memory holds the state of the current interaction; long-term memory persists information across interactions — preferences, prior decisions, past cases, organizational knowledge, unresolved tasks, prior actions. Memory improves continuity but raises questions of relevance, privacy, security, retention, and accuracy. The governing rule is ontological. A system should retain information because it serves a defined operational purpose; a system that remembers merely because it can accumulates risk without value.

Provenance is the thread that makes the rest inspectable. In a consequential system the operative question reaches past what the AI concluded to what it relied on: which source records and policy versions, which retrieval results, which extracted facts and their validation status, which deterministic rules, and which model and configuration produced the result. Provenance is where ontology begins to serve human authority, since traceability is the precondition for review, correction, audit, and dispute resolution.

The distinctive risk at this layer is ontological mismatch: the system's representation of the domain diverges from the real domain, or from the operators' understanding of it. Much of what later gets diagnosed as "model error" is really a representation that was incomplete, stale, or misaligned. Getting the ontology right is often worth more than getting the model bigger.

Traditional software is largely deterministic: if condition A holds, do B. AI models are probabilistic: they estimate what is likely rather than executing fixed instructions. Useful systems combine both, and the design discipline is deciding, deliberately, which parts of a problem each should govern. This is logic mapping.

Deterministic logic governs where a rule is known and must hold exactly: eligibility criteria, mathematical calculations, permissions, required approvals, deadlines, regulatory constraints, workflow transitions. Probabilistic judgment has no business here; a rule that only usually holds is a defect.

Probabilistic reasoning governs where the space of cases cannot be specified in advance: interpreting language, recognizing patterns, extracting information, summarizing evidence, resolving ambiguity, generating alternatives, and judging situations whose every possibility cannot be enumerated. A rigid rulebook fails here, growing brittle as it tries and fails to cover every case.

In practice the mapping sorts decisions along a short spectrum. Exact, regulated, or permission-based decisions belong to deterministic rules, with the model at most explaining them or extracting their inputs. Numerical and formula-based decisions belong to deterministic calculation. Ambiguous language and unstructured evidence are where probabilistic models earn their place — classifying, extracting, summarizing, proposing — bounded by validation. High-impact discretionary judgments belong to a person, with the model assembling evidence and surfacing options. Reversible, low-impact operational actions can be automated within policy limits. Irreversible or external actions belong behind an approval gate. The same decision may move along this spectrum as confidence, consequence, and evidence change.

The objective is to draw an explicit map of where hard logic governs and where probabilistic reasoning adds value, and to keep the two from bleeding into each other. When the mapping is wrong, the design is at fault, and a better model will not repair it: a model placed where a hard rule must hold introduces unacceptable variance, and a rule placed where judgment is required turns brittle. A large share of what organizations experience as "AI failure" is really logic assigned to the wrong layer.

Once logic is mapped, the system must allocate the authority to act on its outputs. Decision rights specify, for each decision, what the system may do on its own, what it may only propose, and what it may not do at all.

This matters most at the boundary between producing output and taking action. A generative model, by itself, produces output. An agentic system does something with it — searching a database, querying an API, creating a document, running a calculation, updating a case record, checking inventory, scheduling an appointment, requesting approval, launching another workflow. This is where AI moves from answering questions to participating in operations, and where risk rises sharply. A hallucinated sentence is inconvenient; a hallucinated action executed against a production system can be consequential. Design must therefore state precisely which actions the system may initiate, which require approval, and which it may not perform.

A well-designed system has more than two states. Beyond acting and doing nothing, it should be able to recommend an action, request more information, escalate for review, defer, or reject. This graded set is the working vocabulary of decision rights, and it lets authority be matched to consequence rather than applied uniformly.

Decision rights are socio-technical as much as technical. They include business ownership of an outcome, compliance approval, technical permissions and credentials, delegated authority, and the thresholds at which automation is allowed to act. The graded set is the vocabulary; assigning it is the design.

Agents and orchestration are how these rights are exercised over multiple steps. An agent pursues an objective across steps: understand the goal, identify what is missing, retrieve it, choose a tool, act, observe the result, decide the next step, and stop when the objective is met or escalation is required. More complex designs use several specialized agents, often with an orchestrator that divides a problem among workers and recombines their results. Additional agents come at a cost: each one adds a source of error, latency, expense, and coordination failure. The discipline is to use the simplest architecture that reliably solves the problem, and to grant each component only the decision rights its reliability earns.

Which right applies to a given decision should follow from the consequence of being wrong. A structured reading of that consequence — the magnitude of potential harm, reversibility, the population and volume affected, regulatory or contractual obligation, data sensitivity, evidence quality, the availability of appeal or remediation, the potential for unfair or disparate impact, and dependence on external systems — maps directly onto the ladder: inform, recommend, draft, execute within constraints, execute only with approval, or do not act at all. This is the operational meaning of the principle that architecture must follow the consequence of error.

Above the allocation of decision rights sits a principle that makes the whole allocation legitimate: humans retain ultimate authority and accountability. Decision rights say who decides what inside the system; human authority is the institutional accountability, override power, and legitimacy that sits above those permissions. It, too, is socio-technical: named accountable owners, reviewer capacity and competence, escalation service levels, policy interpretation, appeal and remediation paths, emergency-stop authority, and governance over changes to the system itself.

Human review is warranted where consequences are significant, where rules require professional judgment, where the system's confidence is low, where evidence conflicts, where an exception arises, where an action is irreversible, or where regulation requires it. The recurring design question is easy to state and hard to answer well: whose judgment governs this decision?

A human is not meaningfully in control simply because a screen presents an approval button. Meaningful authority exists only when a named person or role has the evidence, the time, the competence, the permission to reject or alter the recommendation, and a defined path to escalate. Absent those, review degrades into ratification, and the accountability it was meant to provide is illusory.

Human authority holds only where the architecture makes it hold. Three structures give it force.

Observability makes authority possible, because people cannot govern what they cannot see. A consequential system should record what request entered, what information was retrieved, which model was used, which tools were called, which rules were evaluated, what output was produced, what action followed, whether a human intervened, and what ultimately happened. Ordinary metrics — latency, throughput, errors — still matter, and AI adds concerns such as model performance, data drift, output quality, and bias. This is where the provenance trail does its work: every consequential recommendation or action should carry an inspectable record of the evidence it used, the rules it applied, the configuration and version that produced it, and the person who approved or overrode it. Without that trail, review, audit, and dispute resolution have nothing to stand on. For consequential applications, the log is part of the governance architecture in its own right.

Security and governance make authority enforceable, and in agentic systems they have to be concrete. Retrieved content should be treated as untrusted data and kept separate from system policy, so that instructions buried in a document cannot steer the system. Tools and action types should be allowlisted and run under least-privilege credentials. Tool inputs and outputs should be validated against schemas, and actions should be bounded in spend, rate, scope, and time. High-impact or irreversible actions should require explicit confirmation; risky execution should be sandboxed; and the system should retain an emergency stop or capability-reduction mechanism. These defenses should be tested against adversarial prompts and poisoned documents, because prompt injection, data poisoning, and attempts to extract sensitive information are now part of the threat surface. Governance of this kind belongs inside the application and has to be built in from the start.

Feedback makes authority corrective. A system should generate evidence about its own performance: user corrections, accepted and rejected recommendations, task completion, error rates, escalations, human overrides, exceptions, and business outcomes. These signals show where the architecture needs work, and the remedy usually lies somewhere other than the model — in retrieval, instructions, workflow logic, tool access, data quality, or the organization's own operating rules. Improvement exercised under human authority is continuous refinement of the whole operating system around the model, and it rarely reduces to retraining.