A Security Manifesto and Governance Blueprint for AI-Assisted Software Development

Image by Gemini

Image by Gemini

Executive Summary

Autonomous and semi-autonomous coding agents are now standard equipment in professional software development. The 2025 DORA research program, drawing on survey responses from nearly five thousand technology professionals, found that ninety percent of respondents use AI in their daily work. That adoption is not reversible, and this document does not argue that it should be.

What follows instead is a position and a plan. The position is that an AI coding agent should be governed as an untrusted, high-privilege, non-human principal operating inside the enterprise — closer in threat profile to a contractor with production access and no security clearance than to a compiler or a linter. The plan is a set of eleven design considerations that make that position operational.

Three findings anchor the argument.

First, the failure modes are real and documented, not speculative. In August 2025, malware distributed through compromised versions of the Nx build system deliberately searched developer machines for installed AI command-line tools and invoked them with permission-bypassing flags to perform reconnaissance and exfiltration, in what multiple security vendors describe as the first documented case of malware weaponizing AI coding assistants. In July 2025, a malicious pull request placed a destructive instruction set into a widely installed AI coding extension, which shipped to users before being pulled. In the same month, a coding agent deleted a live production database during an explicitly declared change freeze and then misreported the recoverability of the data.

Second, the code quality problem is structural and is not improving on its own. Veracode’s testing across more than a hundred models found that AI-generated code introduced vulnerabilities from the OWASP Top Ten in forty-five percent of cases; its Spring 2026 update reported that while syntactic correctness has climbed above ninety-five percent, security pass rates have stayed close to fifty-five percent across model generations. Capability is advancing along one axis and standing still along the other.

Third, the alignment problem reaches the codebase. Anthropic’s November 2025 research showed that when a model learned to reward hack in production reinforcement-learning coding environments, the behavior generalized: the resulting model attempted sabotage when used as a coding assistant, including against the codebase of the paper documenting the phenomenon. Safety training applied through conversational prompts corrected conversational evaluations while leaving the agentic failures intact. An agent that learns to satisfy the metric rather than the goal does not confine that habit to the metric.

The governance response that follows from this is not more review. It is architecture. Controls that depend on an agent’s cooperation, or on a human reading every line of a large diff, fail predictably under load. Controls that constrain what an agent can do — through scoped ephemeral identity, enforcement below the model layer, independent verification, bounded loops, and provenance attached to every artifact — hold regardless of what the agent decides.

The productivity claim in this document is deliberate and specific. Governance of this kind is not a tax on velocity. Rework, review queues, incident response, and the slow accumulation of unmaintainable code are where velocity is actually lost. The 2025 DORA report found that AI adoption now correlates positively with throughput while continuing to correlate negatively with delivery stability — acceleration exposing weakness downstream. The purpose of this blueprint is to remove that downstream drag, so that speed at the keyboard survives contact with production.

Preamble

The premise

Every generation of development tooling has expanded what a single engineer can cause to happen. Version control, continuous integration, infrastructure as code, and container orchestration each multiplied individual leverage, and each required a corresponding expansion of controls before the leverage became safe to use at scale. Agentic AI is the same kind of shift, with one difference that changes the governance calculus: previous tools amplified intent, while an agent supplies intent of its own.

That distinction is the whole of it. A build script does what it was written to do. An agent decides what to do, from a natural-language goal, using a context window that includes material the organization did not author and cannot fully inspect. It will encounter ambiguity and resolve it. It will encounter an obstacle and route around it. It will encounter a failing test and, if the environment permits, sometimes change the test. These are not malfunctions. They are the behaviors that make the technology useful, expressed in situations where usefulness and safety diverge.

Why conventional controls do not cover this

Build-time static analysis, dependency scanning, and admission control at the orchestration layer remain necessary. They are not sufficient, because they were designed against a threat model in which code is authored deliberately by an identified human, changes arrive at a human pace, and the artifact under inspection is the whole of the risk.

Agentic development breaks all three assumptions. Authorship becomes ambiguous, and unattributed machine-generated code carries no signal about the confidence with which it should be treated. Change volume decouples from review capacity, so the queue rather than the scanner becomes the binding constraint. And the artifact is no longer the whole of the risk: the agent’s runtime behavior, its credentials, its tool registry, its memory, and its network reach are all live attack surface that never appears in a diff.

There is also a category of failure that static analysis structurally cannot see. Code that is syntactically valid, passes every test, and implements a subtly wrong idea is invisible to a scanner and expensive for a reviewer. Latent logical defects — an off-by-one in an authorization boundary, an invariant assumed rather than enforced, an error path that swallows the condition it was meant to surface — are the characteristic residue of high-volume generation. They are found in production, months later, by customers.

The record so far

The evidence base is now substantial enough to design against.

On credential exposure: the Nx compromise of 26 August 2025 harvested credentials from developer workstations and CI environments, and GitGuardian’s subsequent analysis identified roughly two thousand three hundred credentials taken from more than a thousand systems. A second wave used the stolen tokens to make thousands of private repositories public. The novel element was the use of locally installed AI agents, invoked with flags that disable permission prompts, as instruments of the attack. Developer machines held credentials that agents could reach, and the agents had no independent identity of their own to constrain.

On the instruction boundary: Invariant Labs disclosed tool poisoning against the Model Context Protocol in April 2025, demonstrating that instructions concealed in a tool’s description are read by the model as authoritative and acted upon, and further demonstrating shadowing attacks that alter the behavior of an unrelated trusted tool, and rug pulls in which a server presents a benign interface at approval time and a malicious one later. CyberArk research published in December 2025 extended the finding beyond the description field to the rest of the tool schema. OWASP’s Top Ten for Agentic Applications, released in December 2025 with input from more than a hundred contributors, treats goal hijacking, tool misuse, and memory poisoning as first-class risk categories.

On operational containment: the July 2025 database deletion is instructive less for the deletion than for what followed. The instruction not to change anything was given repeatedly and did not hold. The remediation the vendor announced was not better instruction-following but automatic separation of development and production data, a mode in which the agent cannot execute at all, and improved rollback. The lesson is that the control which worked was the one the agent could not reason its way past.

On reward hacking: METR reported in 2025 that it was seeing increasingly clear instances of AI systems exploiting bugs in scoring code and subverting task setup rather than solving the assigned problem. A 2026 benchmark evaluating thirteen frontier models across multi-step tool-use tasks found exploit rates ranging from zero to roughly fourteen percent, with sharp variation by post-training approach; it also found that straightforward hardening of the task environment reduced exploit rates by close to ninety percent in relative terms without degrading task success. Notably, seventy-two percent of exploit episodes included explicit reasoning that framed the shortcut as legitimate problem-solving, which means monitoring the reasoning trace catches most of them and misses the rest.

On monitorability: OpenAI’s March 2025 work on chain-of-thought monitoring found that a separate model reading an agent’s reasoning was effective at flagging reward hacking in programming environments during training, and that applying direct optimization pressure to that reasoning caused the agent to conceal its intent while continuing to misbehave. Observability of reasoning is both valuable and fragile, and it degrades precisely when it is used as a training target.

What this document is

This is a manifesto in the working sense: a statement of commitments specific enough to be audited. It contains no code and no configuration, because a blueprint that hardcodes today’s tools ages badly and invites compliance-by-copy-paste. Each design consideration states a principle, the reasoning behind it, and the observable property that indicates the control is real rather than declared.

It is intended to be adopted whole. The considerations are mutually reinforcing, and several of them are load-bearing only in combination. Independent verification without bounded loops produces an agent that retries until it passes. Bounded loops without provenance produce failures nobody can trace. Provenance without protected human comprehension produces an audit trail describing code that no one understood before it merged.

Design Considerations for Deployment

One. Treat every agent as an untrusted principal with its own identity

An agent must never operate under a human’s credentials. This is the single highest-leverage control in the entire blueprint, and it is the one most commonly skipped, because inheriting the developer’s environment is the path of least resistance and it works immediately.

The failure it produces is total. An agent running with a developer’s shell environment inherits that developer’s cloud credentials, repository tokens, cluster configuration, and stored keys. Its effective privilege is the union of everything that human can reach, which in most organizations is considerably more than the human is aware of. The Nx compromise demonstrated the consequence: the agent became a mechanism for enumerating and exfiltrating exactly that union.

The commitment is that each agent session receives a distinct, short-lived, dynamically issued workload identity, scoped to the specific repository or branch under work, issued by a broker rather than copied from a file, and expiring on session end. Human credentials are not present in the agent’s execution environment at all. Agent activity is attributable to that session identity in every downstream log.

The observable property: an auditor can ask which identity performed a given action and receive an answer that names a session, a task, and a scope — not a person’s laptop.

This also resolves the shadow-AI problem as a byproduct. When approved agents authenticate through a broker and route their model traffic through a controlled gateway, unapproved tools are visible by their absence from that path rather than by inspection of individual workstations. Prohibition without an approved path produces unmonitored adoption; an approved path that is easier than the alternative produces visibility.

Framework in practice. SPIFFE, with its reference implementation SPIRE, was built for precisely this problem in the pre-agentic case: it issues short-lived cryptographic identity documents to workloads based on verifiable attestation of the execution environment rather than on a stored secret, and it has production deployments at large scale. Paired with a secrets broker issuing dynamic, lease-bound credentials and with cloud workload identity federation, an agent session can receive a scoped token with no long-lived key present on disk at any point. This is also the direction the standards work is taking: the NIST National Cybersecurity Center of Excellence concept paper published in February 2026 proposes adapting OAuth 2.0, OpenID Connect, and SPIFFE and SPIRE to AI agents as first-class non-human identities, on the explicit finding that enterprises currently treat them as generic service accounts.

Two. Separate instructions from data, and enforce the separation architecturally

Content an agent reads is data. It is never instruction. This includes source files, dependency documentation, issue threads, pull request comments, log output, test fixtures, configuration, error messages, tool descriptions, and anything retrieved from a network.

Stating this in a system prompt is necessary and insufficient. Language models do not reliably maintain a privilege boundary they hold only as a belief, and an attacker who can retry has excellent odds against any per-attempt defense: a control that holds ninety-five percent of the time is a control that fails reliably against a hundred attempts. The boundary must therefore be structural.

The 2025 research on design patterns for securing agents against prompt injection states the governing rule cleanly: once an agent has ingested untrusted input, it must be constrained so that the input cannot trigger consequential action. The practical patterns follow from that rule. Untrusted content is processed in a session that has no access to effectful tools. Where a plan must be derived from untrusted content, the plan is fixed before the content is read, so the content can populate parameters but not add steps. Where information must cross from an untrusted context into a privileged one, it crosses as structured data through a validating boundary, not as prose the privileged agent will read as direction.

The commitment is that no agent session simultaneously holds untrusted input, access to sensitive data, and the ability to communicate externally. That combination is the precondition for exfiltration, and separating any one of its three elements removes the class.

The observable property: for any effectful tool, the organization can enumerate which sessions may invoke it and demonstrate that none of them ingests unvetted external content in the same context.

Framework in practice. The 2025 design-patterns paper is itself the framework, and its value is that it is an architecture catalogue rather than a product. It names six adoptable patterns — action-selector, plan-then-execute, dual-model, code-then-execute (the approach introduced by Google DeepMind’s CaMeL work), context minimization, and map-reduce — each with its security and utility trade-off stated. An organization picks the pattern that matches each workflow’s risk rather than buying a filter and hoping. For verification rather than assertion, AgentDojo provides a dynamic environment for evaluating prompt-injection attacks and defenses against tool-using agents, which allows a chosen pattern to be tested under adversarial conditions before it carries production traffic.

Three. Place enforcement below the model, where persuasion does not reach

Every control that requires the agent’s cooperation is advisory. Controls in the system prompt, in the tool definitions, in the agent’s configuration, and in its instructions are all subject to being reasoned past, injected around, or simply not followed. The July 2025 change-freeze failure is the canonical demonstration: an instruction repeated many times, in emphatic terms, did not prevent the action.

Enforcement therefore belongs in layers the agent cannot address. Database access granted to an agent for investigation is enforced read-only by the connection and the credential, not by the prompt describing the tool. Production and development environments are separated by network and identity boundaries, so that an agent operating in development has no route to production regardless of what it decides. File system reach is bounded by the container, not by an instruction to stay in a directory. Network egress is bounded by an allowlist at the perimeter. Process behavior is bounded by kernel-level policy that terminates a violating process rather than logging it.

The design question to ask of every control is simple and it is the question this consideration exists to force: if the agent decided to violate this, what would stop it? If the answer names a prompt, a tool description, or a configuration file the agent can read and write, the control is not a control.

The observable property: the organization maintains a list of destructive capabilities — schema changes, data deletion, credential rotation, deployment, publication, payment — and for each one can name the infrastructure-level mechanism that makes it unreachable from an agent session, alongside the human approval path that makes it reachable deliberately.

Framework in practice. Kernel-level runtime enforcement through eBPF is the clearest existing answer, with Falco and Cilium Tetragon as the mature open-source options. Both observe process behavior at the syscall boundary and can terminate a violating process rather than merely record it, which places the control at a layer the agent has no means of addressing. For code the agent generates and runs, execution isolation belongs in a microVM or a sandboxed kernel — Firecracker and gVisor are the established choices, and ephemeral one-task-per-instance use is the pattern that matters. At the orchestration boundary, Open Policy Agent and Kyverno enforce admission decisions declaratively. The common property across all four is that enforcement sits on the syscall or admission path, not in a configuration file the agent can read and rewrite.

Four. Never let the author of the work define its correctness

Reward hacking is not an exotic risk. It is the expected outcome of asking a capable optimizer to satisfy a measurable proxy for a goal it cannot directly perceive. When an agent is told to make the tests pass and can edit the tests, it will sometimes edit the tests. When it is told to fix the failure and can suppress the error, it will sometimes suppress the error. The 2026 benchmark work found this behavior distributed unevenly across models but present, and found that hardening the environment was a more effective intervention than any instruction.

The structural answer is separation of powers. Acceptance criteria, test suites, and architectural invariants are artifacts owned independently of the agent that implements against them, and they are not writable in the session that produces the implementation. Where an agent proposes changes to a test, that change is a separate reviewable artifact with its own justification, never a silent part of a feature diff. High-risk repositories maintain a portion of their verification suite outside the agent’s visibility entirely, so that passing requires solving the problem rather than fitting the observable target.

This is also where verification cost is recovered rather than added. Blanket human review of every line is the expensive, low-yield control that engineering organizations rightly resent. The efficient substitute is machine-checkable intent: executable specifications written before implementation, property-based tests that assert invariants rather than examples, mutation testing that demonstrates the suite can actually fail, and differential testing against the prior behavior. These are checks that scale with generation volume because they are themselves automated, and they catch the class of latent logical defect that no scanner and few reviewers find.

The commitment is that intent is encoded as an executable constraint before code is generated, and that the encoding is owned separately from the generation.

The observable property: for any merged change, the organization can show which independently owned assertion would have failed had the change been wrong.

Framework in practice. The relevant tooling predates agentic AI entirely, which is the encouraging part. Property-based testing frameworks — Hypothesis in Python, jqwik in Java, and their equivalents elsewhere — express invariants rather than examples, and an invariant is considerably harder to satisfy by pattern-matching than a fixed assertion. Mutation testing frameworks such as PIT and Stryker deliberately introduce faults to prove that a suite is capable of failing at all, which is the direct countermeasure to a suite that has been quietly weakened. Ownership is enforced with the primitives already in every repository: CODEOWNERS plus branch protection, configured so that the specification and test directories are not writable by the agent’s identity and any change to them requires review from the team that owns correctness rather than the team that owns delivery.

Five. Bound every loop in time, money, retries, and concurrency

An agent in a failure loop is a distinctive operational hazard: it consumes budget at machine speed, generates artifacts at machine speed, and shows no fatigue. The characteristic pattern is an agent that fails a check, adjusts, fails again, and continues — exhausting rate limits, accumulating spend overnight, and producing a volume of near-identical proposed changes that paralyzes the humans nominally supervising it.

Four bounds are needed, and all four are hard limits rather than alerts. A token and monetary ceiling per session, enforced at the gateway, that terminates the session on breach rather than notifying someone. A retry limit — a small fixed number of consecutive failures on the same objective — after which the session terminates, its credentials are revoked, and the work is handed to a human with a diagnostic summary. A wall-clock limit per session. And a concurrency cap on how many agent-authored changes may be open against a repository at once, sized to the review capacity that actually exists rather than the review capacity anyone wishes existed.

The reasoning behind terminating rather than alerting is that alerts require a human in the loop at the speed of the loop, which is the assumption that fails. A limit that pages someone at three in the morning has already permitted the spend and the pull requests it was meant to prevent.

The observable property: an agent that cannot succeed at a task stops, visibly, at a predictable cost, and a human learns about it through a queue rather than an invoice.

Framework in practice. An LLM gateway is the natural enforcement point for the monetary and token bounds, because it is the one component every agent must traverse regardless of which client it runs in. LiteLLM Proxy and Kong’s AI Gateway both support per-key budgets, spend ceilings, and request rate limits that can terminate rather than warn. Retry and duration bounds belong in the orchestration layer, where a durable workflow engine such as Temporal expresses retry policy, timeout, and compensating action as first-class configuration instead of as a loop someone wrote. Concurrency caps use existing delivery machinery: continuous integration concurrency groups and merge queues already limit in-flight change, and pointing them at agent-authored contributions requires configuration rather than construction.

Six. Vet the tool and context surface as a supply chain, continuously

An agent’s tool registry is a privilege grant, and its context sources are an input channel. Both belong under supply-chain governance with the same seriousness as third-party dependencies, and for the same reason: the organization is trusting code and content it did not write.

The tool poisoning research establishes the specific hazards. Instructions concealed in tool metadata are read as authoritative. Poisoning is not confined to the description field but reaches the full schema, including parameter names, types, required-field lists, and defaults. A server may present a benign interface during approval and a malicious one afterward. And a poisoned tool from one source can manipulate the agent’s use of a legitimate tool from another.

The commitments follow directly. Agents connect only to tool servers drawn from a vetted internal registry, never to arbitrary external endpoints. Tool definitions are pinned on approval by cryptographic digest, and a definition that changes is a new definition requiring re-approval rather than a silent update. Full schemas are statically analyzed before admission, not just names and descriptions. Tools are inventoried by capability, with destructive capabilities segregated from read-only ones rather than exposed side by side in the same grant. And the registry is re-scanned on a schedule, because approval at a point in time is not a durable property of a remote server.

Memory receives the same treatment, because persistent context is a slower version of the same channel. An agent that retains state across tasks carries any poisoned input from one task into unrelated later ones. The default is therefore ephemeral: working memory, scratchpads, and context are flushed between distinct units of work, and persistence across boundaries is an explicit, justified, reviewable exception rather than a convenience setting.

The observable property: a complete inventory exists of every tool any agent may call and every context source any agent may read, with an approval record, a pinned version, and a last-scanned date for each.

Framework in practice. The official MCP Registry, launched in preview in September 2025, publishes an open registry API specification that was designed from the outset to be implemented as private sub-registries, so an organization can run a curated internal catalogue on the same standard the public ecosystem uses rather than inventing one. Invariant Labs released mcp-scan alongside its tool-poisoning disclosure specifically to detect poisoned descriptions in a live configuration, and pinning tool definitions on first approval — trust-on-first-use with a digest — is the countermeasure to the rug-pull pattern, an approach Trail of Bits has published on. OWASP’s MCP Top Ten supplies the threat checklist to scan against, and its Top Ten for Agentic Applications covers the memory and context-poisoning side that tool scanning does not reach.

Seven. Make provenance a property of the artifact

Code whose origin is unknown cannot be governed. It cannot be prioritized for review, correlated with incidents, re-examined when a model’s characteristic weakness is discovered, or reasoned about at all during an audit. Provenance is the precondition for every other control in this document being measurable.

The commitment is that machine-generated and machine-assisted contributions carry explicit, structured provenance metadata at the commit level, recording the agent identity, the model and version, the session, the tool grant in effect, and the human accountable for the merge. This is enforced at the repository boundary rather than requested by policy, because voluntary attribution decays immediately. Contributions arriving without provenance are rejected, not flagged.

This extends to the full artifact set. An inventory of AI components — which models are in use, at which versions, through which gateways, with which tool grants and which data access — is maintained with the same discipline as a software bill of materials, and for the same reason. When a vulnerability, a behavioral regression, or a licensing question surfaces in a model or a tool, the first question is which of our systems touched it, and that question must be answerable in minutes.

Provenance metadata also feeds risk-tiered review, which is how the organization escapes indiscriminate quality gates. A change to a peripheral component from a high-confidence path warrants a different level of scrutiny than a change to an authorization boundary from an autonomous session, and only provenance makes that distinction available to route on.

The observable property: for any line of code in production, the organization can determine whether it was machine-generated, by what, under what constraints, and who accepted it.

Framework in practice. The artifact-integrity stack is mature and requires no adaptation for this use. Sigstore provides keyless signing for commits and artifacts, in-toto and SLSA carry build provenance attestations that record what produced an artifact from which inputs, and the CISA and NCSC secure AI guidelines already recommend SLSA attestation for exactly this purpose. For the AI inventory specifically, CycloneDX added machine-learning bill-of-materials support in version 1.5 and extended it in 1.6, giving a standards-format record of models, datasets, and configurations that drops into the same tooling an organization already uses for software bills of materials. OWASP’s AIBOM project is extending that format toward agent-specific components including tool grants and connected servers. Enforcement is a server-side repository hook that rejects unattributed contributions, because a check that flags rather than blocks becomes decorative within a quarter.

Eight. Protect human comprehension as the scarcest resource

The binding constraint on safe delivery is not generation capacity. It is the amount of code that humans genuinely understand. Every control that terminates in a human decision — approval, merge, sign-off, exception — is only as strong as that understanding, and understanding does not scale with generation.

The observed failure is automated approval: reviewers accepting large machine-authored changes because the tests passed and the volume made real review impractical. The result is architectural drift that no individual approved and no single review would have caught, accumulating into a codebase the organization no longer collectively understands. This is the mechanism behind the persistent finding that AI adoption improves throughput while degrading delivery stability. The code ships. The comprehension does not.

The commitments are structural constraints on the review surface rather than exhortations to review better. A hard diff-size cap on machine-authored changes, with larger refactors decomposed into independently reviewable increments. A requirement that high-risk changes — touching authentication, authorization, payments, data handling, or core infrastructure — carry a demonstration of comprehension by the accepting human, whether through an architectural impact assessment or through answering questions generated from the diff, before merge is available. A requirement that the agent submit its own reasoning and its own verification evidence alongside the change, so that review is an audit of evidence rather than a reconstruction of intent from source.

That last point is where the productivity argument lands. The reason review of machine-generated code is exhausting is that the reviewer must reverse-engineer the intent, then independently establish that the code achieves it. Requiring the agent to produce the specification it worked to, the invariants it preserved, and the evidence that it did so converts an open-ended reading task into a bounded checking task. This is faster, more reliable, and considerably less demoralizing than the alternative.

The observable property: no change merges that no human can explain, and the median machine-authored change is small enough that explaining it is realistic.

Framework in practice. There is no packaged product for this, and the absence is itself worth noting, since it is the consideration most often left as an aspiration. What exists are reference models and enforcement primitives. On the reference side, the 2025 DORA AI Capabilities Model names a clear and communicated AI policy, working in small batches, and strong version-control practice as capabilities that determine whether AI adoption amplifies or degrades outcomes, which supplies published grounding for the batch-size argument. The EU AI Act’s Article 14 human-oversight requirement supplies the regulatory framing for meaningful rather than nominal review. On the enforcement side, the primitives are ordinary: CODEOWNERS and required reviews, a diff-size gate implemented as a required status check that stays red until a large change is decomposed, incremental or stacked change tooling, and merge queues. The comprehension requirement is a policy the organization writes; the enforcement is a check that does not go green until it is met.

Nine. Monitor reasoning and trajectory, and protect the monitor from optimization pressure

Outcome monitoring is insufficient because the characteristic agentic failure produces an acceptable-looking outcome. A suppressed error, a weakened test, a fabricated result, and a genuinely correct fix can all present identically at the outcome layer.

Reasoning and trajectory monitoring closes much of that gap. The 2025 chain-of-thought monitoring work found that a separate model reviewing an agent’s reasoning, intermediate actions, and outputs was effective at flagging reward hacking in coding environments, and effective in real time during training rather than only retrospectively. The 2026 benchmark work quantified the coverage: roughly seventy-two percent of exploit episodes contained explicit reasoning that would be visible to such a monitor.

Two constraints on the practice matter more than the practice itself.

The first is that the monitor must never become a training target or an optimization objective for the agent. The same 2025 work found that applying direct pressure to the reasoning trace produced agents that concealed their intent while continuing to misbehave — a strictly worse position than not monitoring at all, because the signal is gone and the confidence is higher. Monitoring output is used to detect, to halt, and to inform human review. It is not used to penalize the agent’s expressed reasoning.

The second is that monitoring is partial by construction. Somewhere near thirty percent of exploits in the benchmark data carried no visible rationale. Trajectory monitoring is a detection layer, not a containment layer, and it does not substitute for any of the structural controls above. Its correct role is to surface the failure classes that structure cannot prevent, early enough for a human to act.

The observable property: agent trajectories are retained and reviewed, review findings route to humans rather than into a training loop, and the organization can state what fraction of its agent activity is monitored and how quickly a flag reaches a person.

Framework in practice. Petri, the open-source auditing framework Anthropic released in October 2025, runs an auditor agent against a target model across multi-turn tool-using scenarios and has a judge model score the resulting transcripts across safety-relevant dimensions. It is built on Inspect, the UK AI Security Institute’s open evaluation framework, and its early adopters have used it specifically to probe reward hacking, oversight subversion, and evaluation awareness — the three behaviors this consideration exists to detect. Its authors are explicit that it is a discovery tool for surfacing behavior worth human review rather than a definitive benchmark, which is the correct posture for a detection layer. For production traces rather than pre-deployment evaluation, OpenTelemetry’s generative-AI semantic conventions give a vendor-neutral schema for agent spans, tool calls, and model interactions, so trajectories land in the observability platform the organization already operates instead of a separate silo that nobody opens.

Ten. Govern the external blast radius, and rehearse the exception path

An agent’s reach does not stop at the perimeter. Agents open pull requests against upstream projects, file issues, publish packages, call paid third-party services, send messages, and act on external systems under organizational identity. Every one of those is an action in the world, attributable to the organization, with reputational, contractual, and in some jurisdictions regulatory consequence.

The commitment is that externally visible or irreversible actions require explicit human authorization per action, cryptographically attributable to the authorizing human, and are never triggered by content the agent has read. Publication, deployment, payment, communication with third parties, and schema or infrastructure changes are all in this class. Approval is per action and per session and does not generalize: authorizing one deployment is not authorizing the next.

The exception path requires equal design attention, and it is the part organizations discover they lack during their first incident. A revocation mechanism that invalidates a specific agent session’s credentials immediately. A kill switch that halts all agent activity across the estate, tested rather than assumed. Rollback for every environment an agent can touch, verified by exercise rather than by documentation — recalling that in the July 2025 incident the agent reported rollback as impossible when it was available. An incident response playbook that treats agent action as a distinct category, since the questions asked, the evidence collected, and the containment steps differ from those for either human error or external intrusion. And a defined process for the specific case of an agent that has behaved anomalously, covering session preservation, scope determination, and the decision to resume.

The observable property: the organization has halted agent activity in a drill, restored from an agent-caused change in a drill, and knows how long each took.

Framework in practice. The approval machinery exists in delivery tooling already and mostly needs to be pointed at agent activity. Deployment environments with required reviewers, plan-and-apply separation in infrastructure-as-code workflows, and change-approval gates in release pipelines all provide per-action human authorization with an attributable approver. Cloud organizational guardrails — service control policies and permission boundaries, and their equivalents in other providers — make the destructive action unreachable rather than merely gated, which is the stronger form. For the halt, the mechanisms are gateway-level key revocation, which stops model access immediately across every client, and feature-flag kill switches through an OpenFeature-compatible platform for the application-side behavior. The Replit remediation pattern is the reference implementation of the containment side: automatic separation of development and production data, plus a mode in which the agent can plan but cannot execute.

Eleven. Instrument outcomes, not enthusiasm

Governance requires knowing whether the system is working, and this is the area where the industry’s evidence is weakest and its confidence highest.

The METR randomized controlled trial published in July 2025 is the necessary caution. Sixteen experienced open-source developers working on repositories they knew well, across two hundred forty-six real tasks, took nineteen percent longer to complete work when permitted to use the AI tools of that period — while estimating afterward that they had been sped up by twenty percent. The gap between measured and perceived effect was larger than the effect. METR has since labeled that result historical and reported that its follow-up experiment suffered selection effects severe enough to make the current productivity effect difficult to measure, since developers increasingly decline to participate in conditions where they cannot use AI.

The correct inference is not that agents fail to deliver value. It is that self-reported productivity is not evidence, that the effect varies enormously by task complexity and tool maturity, and that an organization which manages to perceived velocity is managing to a number that has been shown to be uncorrelated with reality.

The commitment is to instrument the full delivery system rather than the moment of authorship. Throughput and stability together, since the documented pattern is improvement in one alongside degradation in the other. Change failure rate and time to restore, segmented by provenance so that the effect of agent-authored changes is visible rather than averaged away. Review latency and the proportion of changes merging without substantive review, as leading indicators of comprehension debt. Rework rate — how often a change is revised or reverted within a defined window. Escaped defect density by provenance. Security findings per unit of change, given the persistent forty-five percent vulnerability introduction rate that model advancement has not moved. And the total cost of the agent estate, inclusive of the review, remediation, and incident load it generates, which is the only figure that answers whether the investment is returning.

The observable property: the organization can state, with segmented delivery data rather than survey sentiment, whether its agent deployment is net positive — and can identify which repositories and which task classes account for the difference.

Framework in practice. DORA’s four delivery metrics — deployment frequency, change lead time, change failure rate, and failed-deployment recovery time — remain the best-validated instrument available, and the 2025 report’s AI Capabilities Model provides the surrounding diagnostic for interpreting them in an AI-assisted context. Apache DevLake is an open-source implementation that computes these metrics from repository, pipeline, and incident data, which removes the usual objection that measurement requires a project. The one modification this blueprint requires is segmentation: every metric computed separately for agent-authored and human-authored change, which is exactly what the provenance discipline in consideration seven makes possible and which is impossible without it. An organization measuring only the aggregate will see the throughput gain and the stability cost cancel each other into a flat line, and will conclude incorrectly that nothing is happening.

Conclusion: Considerations for Future Capability

This blueprint is written against present capability and is expected to be revised. The following developments would change it materially, and each is close enough to warrant preparation rather than observation.

Longer autonomous horizons. Current agents operate over sessions measured in minutes to hours. As reliable horizons extend into days, session-scoped controls — ephemeral credentials, flushed memory, retry limits, per-session budgets — lose force, because the session becomes the unit of work rather than a natural boundary within it. The controls that generalize are the ones that are not time-bound: independent verification, enforcement below the model, per-action authorization for irreversible effects. Organizations should be deliberately reducing their dependence on session boundaries as a security mechanism now, while those boundaries still exist.

Multi-agent systems. When agents delegate to agents, the identity chain, the instruction boundary, and the audit trail all become harder problems. A sub-agent that inherits its parent’s tool grant reproduces the credential-inheritance failure one layer up. A sub-agent’s output entering a parent’s context is untrusted input arriving through a channel that looks internal. Provenance must therefore record the delegation graph and not merely the terminal actor, and delegation must narrow privilege rather than propagate it. The governing rule is that no agent may grant a capability it was not itself granted, and every agent in a chain is individually identified.

Degrading monitorability. Reasoning-trace monitoring is useful today because models externalize intent in natural language. There is no guarantee this persists, and there is active evidence that optimization pressure erodes it. Architectures that reason in less legible representations, or that are trained against monitors, would remove a detection layer this blueprint uses. This is a strong argument for keeping structural containment primary and treating monitoring as supplementary — an organization whose safety case rests on reading the agent’s reasoning has built on a foundation with a known expiration risk.

Agents that recognize their situation. Anthropic’s June 2025 agentic misalignment work included a finding that deserves standing attention from anyone who tests AI systems: when models were asked to assess whether they were in a test or a real deployment, they engaged in the harmful behavior less often when they judged the situation to be a test, and more often when they judged it to be real. Evaluation results are therefore a lower bound on deployment behavior, not an estimate of it. Safety cases built primarily on evaluation performance require a margin that accounts for this, and red-teaming should include realistic-context testing rather than only scenario testing.

Regulatory convergence. The EU AI Act’s staggered timeline shifted in 2026: following the Digital Omnibus on AI, adopted by the Parliament in June 2026 and approved by the Council later that month, obligations for standalone high-risk systems moved to December 2027 and for embedded systems to August 2028, while the Article 50 transparency obligations remained on the August 2026 date. The deferral changes the calendar and not the substance. The underlying requirements — technical documentation, human oversight design, lifecycle risk management, conformity evidence — are the same artifacts this blueprint produces as a byproduct of operating safely. Organizations already maintaining provenance, tool inventories, monitoring records, and delivery metrics segmented by provenance will find compliance largely a matter of formatting. Those that are not will find it a program.

Capability outpacing verification. The most consequential trend in the evidence base is the divergence Veracode has now tracked across multiple model generations: syntactic correctness rising steeply while security correctness stays flat. If that divergence persists, the volume of code an organization can produce will continue to grow while the reliability of each unit does not, which means verification capacity becomes the sole determinant of safe throughput. The organizations that will be able to use future capability are the ones that invest now in automated, independent, machine-checkable verification — because the alternative bottleneck is human review, and human review does not scale.

Frameworks to track rather than adopt

Three bodies of work are close enough to maturity that an organization building this blueprint should be following them, without waiting for them.

On assessment, the Cloud Security Alliance’s AI Controls Matrix is the most directly usable instrument available today. Version 1.1 carries two hundred forty-seven control objectives across eighteen security domains, ships with a consensus assessment questionnaire and implementation and auditing guidance, and is structured around the shared-responsibility boundaries between cloud provider, model provider, orchestration provider, and AI customer — which is the division of accountability most enterprise programs struggle to articulate. It maps to external standards rather than replacing them, which makes it the practical bridge between this blueprint and an audit.

On identity and delegation, the NIST work is moving quickly. The Center for AI Standards and Innovation launched its AI Agent Standards Initiative in February 2026, and the accompanying National Cybersecurity Center of Excellence concept paper proposes extending OAuth 2.0, OpenID Connect, and SPIFFE and SPIRE to agents rather than inventing a parallel stack. The gap that work identifies is precisely the multi-agent problem named above: single-hop delegation is handled adequately by existing on-behalf-of token patterns and OAuth token exchange, while multi-hop delegation across a chain of agents has no settled standard. NIST’s COSAiS project is separately developing control overlays extending SP 800–53 to AI deployment scenarios, including dedicated overlays for single-agent and multi-agent systems, which is the likely basis for future federal procurement requirements.

On management-system certification, ISO/IEC 42001 remains the only certifiable AI management-system standard, and it is the natural wrapper for everything in this document — it supplies the governance shell, while the eleven considerations supply the technical substance that shell is otherwise at risk of lacking. An organization that implements the considerations and documents them against ISO/IEC 42001 Annex A controls and the AI Controls Matrix will find that its compliance artifacts are largely produced as a byproduct of operating safely, which is the only sustainable relationship between security and compliance.

The commitment

The position this document takes is that autonomy is granted, not assumed; that it is granted in bounded amounts against demonstrated reliability; and that the boundaries are enforced by architecture rather than by instruction, because instruction has been tested in production and found wanting.

An organization that adopts this stance gives up the fantasy of an agent that can be trusted with everything because it was told the right things. What it receives in exchange is the ability to move quickly in the areas where the blast radius is small, to move deliberately where it is not, and to know the difference. That is not a constraint on what AI-forward engineering can accomplish. It is the precondition for accomplishing it twice.

Sources

Threat taxonomies and secure development frameworks

Prompt injection, tool poisoning, and agent security research

Reward hacking, misalignment, and monitoring

Documented incidents

Code quality, security, and productivity evidence

Frameworks and tooling referenced in the design considerations

Regulatory