Build the
Churches
A transformation does not scale through belief alone. It needs places, rituals, tools, operating rules, shared language, and visible proof. The first pillar is the foundation – the technical infrastructure and operating system through which humans and agents can work together at scale.
It’s tempting to call this done when 90% of your code is AI-written – but that isn’t necessarily agentic engineering.
Using an analogy to self-driving, the AI IDE is driver-assist, coding agents take your hands off the wheel, and a lights-out “dark factory” lets one engineer run a fleet of independent agents.
“If you build it, he will come.” – Field of Dreams (1989)
The Foundation.
Before an organization can convert its people or restructure its teams around autonomous agents, it must first build the environment in which AI-native work can reliably happen.
The metaphor is intentional. The churches are the infrastructure, the operating processes, and the dedicated experts that turn new behavior from individual experiment into organizational practice. A company that has merely purchased AI coding licenses has not completed the foundation – it has only opened the door.
The foundation has four dimensions, each scored on its own maturity scale.
Coding Infrastructure puts the capabilities into your team’s hands. AI Ops keeps you current with a pace of change no one doing their day job can track – it takes dedicated focus. Measurement tells you whether the investment is paying off: that the team is genuinely using the tools, and that it isn’t overspending to do it. Security & Compliance keeps you from exposing yourself to new AI-era risks, and keeps you compliant. Underbuild any one and it becomes the ceiling on the rest.
Coding Infrastructure
This dimension covers the full arc of AI inside the coding loop – from a coding assistant the engineer steers keystroke by keystroke, to a team that triggers and ships work itself. The maturity scale below maps it as four levels; the paragraphs underneath walk each level in detail.
Interactive human–AI coding inside the IDE. Engineer steers; multiple sessions; tight back-and-forth.
AI added to the CI / CD pipeline – PR review, test triage, runtime observability.
Long-running sandboxed agents. Multiple in parallel per dev – where 10×+ leverage kicks in.
The “dark factory” of code. Self-triggering and self-healing, with humans governing consequential changes.
The two jumps that matter: Assistant to Engineer is taking your hands off the wheel – the moment you stop reviewing every line. Engineer to Team is no longer needing your own car – the moment the loop closes around you. And just as no city skipped straight to robotaxis, no organization in the study reached the final stage without first building every layer underneath.
The four levels, in detail.
AI as the Assistant. Interactive human–AI coding inside the IDE: the engineer watches the AI write, runs several sessions at once, and guides every step with tight back-and-forth. The field grew up here – from tab completion (GitHub Copilot, 2022) to chat-in-the-IDE (Cursor, 2023) – and Cursor and GitHub Copilot remain the dominant tools at this level. The acceleration is real but bounded: the human is still the throughput constraint.
AI as the Reviewer. AI added to the CI / CD pipeline as an automatic review layer: PR review, test-failure triage, security findings, runtime observability (Sentry, Datadog, Honeycomb piped to AI). First-party tools now cover it – Cursor’s Bugbot (agentic review with auto-fix), Anthropic’s Claude Code review and /security-review – alongside independents like CodeRabbit, Qodo, and Graphite’s Graphite Agent. Easy to wire in because it doesn’t change how engineers code, and routinely skipped for flashier autonomous work, even though it pays back on its own.
AI as the Engineer. Long-running, sandboxed agents that write code, open PRs, and run multi-hour tasks on their own. Multiple concurrent agents per developer is where 10×+ leverage kicks in and the job shifts from writing to directing. The edge existed earlier but was narrow: one study company had used Cognition’s Devin since an early beta, well before autonomous agents became common.
It is the first level that demands real infrastructure – the progression runs worktrees → containers → managed VPSs – and a striking number of orgs build that layer themselves (parallel-session orchestration, make-agent-style commands, a sandbox fleet) rather than wait for a vendor, because off-the-shelf tooling doesn’t yet match how they run agents at scale. Claude Code is the dominant tool here, with OpenAI Codex the major-vendor alternative. Cursor approaches the same level from the IDE side, with an agent-first workspace, automatic model routing, and its Composer model family. This is where the study’s mid-stage and advanced organizations now operate.
AI as the Team. The closed loop – and not just for fixing. Triggers come from everywhere: caught bugs, CI failures, production incidents, telemetry anomalies, customer requests, and increasingly agents listening on Slack threads, PRD drafts, or planning calls that start building before anyone formally asks. The trigger feeds observability or the planning layer; agents act on tests deep enough to validate the work; deploy closes the loop. The industry calls this a dark factory – manufacturing’s lights-out model (FANUC, Xiaomi), now applied to code. The line between the Engineer and the Team is that the system begins the work and closes the loop itself. Human approval becomes risk-based rather than the trigger for every step; people stay in the loop for judgment, not keystrokes.
The VP Engineering dream: software that maintains itself.
The dream is a self-healing feedback loop. Early signals across QA, staging, production, and live user behavior are continuously collected, scanned, and correlated. When something begins to break, agents reproduce it, identify the likely cause, and turn it into a tested fix – often before a user complains.
The human operator receives a PR with the evidence, approves the merge and rollout, and the system verifies the repair on the live system. Narrow, reversible fixes can eventually run automatically.
For legacy companies, the prize is not only faster development. It is reducing the maintenance burden: fewer hours spent investigating regressions, recurring incidents, dependency drift, and support escalations.
AI Ops
AI Ops is the new DevOps. Just as organizations learned that infrastructure, deployment, monitoring, and developer productivity could not be left to every engineer to solve independently, the same is now true for AI. The name is deliberate and worth separating from an older one: AIOps, in Gartner’s sense, means applying AI to IT operations. This is the reverse – the team that builds and owns the AI infrastructure everyone else engineers on top of. The field changes too quickly – new models, tools, workflows, pricing, security patterns, and agent architectures land constantly – and no engineer doing their normal job can credibly keep up.
The misconception is that this is a procurement task – buy the tools and switch them on. It is the opposite: a strategic function, the hardest role on this list to hire for, and the gate on autonomy. It can’t simply be bought, because the infrastructure isn’t generic – it has to be wired into your own runtime, and that wiring is always in-house. Licenses get you assistants and reviewers; independent, long-running agents run on infrastructure only this team builds.
Handled informally by the CTO, head of R&D, or scattered champions.
A named person with mandate, but still a side responsibility.
2–5 FTEs with a roadmap, budget, and ownership of internal infra.
Supports non-engineering builders; catalyst and guardrail for shadow AI.
The reason the team has to exist is the pace. New models, new harnesses, new pricing structures, new tools: any can land in a given week, and no engineer doing their day job can keep up. Dedicated AI Ops experts read every release as it ships, follow the practitioners building the tools directly – Boris Cherny at Anthropic (Claude Code), Peter Steinberger at OpenAI (creator of OpenClaw, now working across its agent products including Codex), and the wider set of public thought leaders, test what they find against the org’s actual workloads, and bring what they learn back inside. Without that team, an org gets stuck on one environment or harness, paying in lost capability and cost efficiency. The role is the AI-era version of what DevOps became, except the cycle is measured in days, not quarters.
The shape recurs across the study – small (2–5 people), cross-functional, full-time. One legacy SaaS chartered an Office of the CTO with the same remit. The team also reshapes procurement: once the agent stack is standardized, every new SaaS purchase is filtered through whether it fits – does it expose an MCP (Model Context Protocol) server, can agents drive it.
At full maturity the function grows outward, beyond engineering, into the org’s whole-org enablement layer for AI: PMs building prototypes, finance teams wiring reporting agents, and designers shipping production code all need a place to bring questions and stay inside the org’s security and cost guardrails. Without it, the failure mode is shadow AI and app sprawl – teams running their own credentials and one-off apps no one knows about until something breaks or leaks.
Measurement
Measurement is how you monitor progress through the agentic SDLC. Done well, it answers the questions that actually matter: which teams and engineers are genuinely converted and which only say they are; which teams have hit real productivity acceleration and which are just burning tokens; and how you compare to industry benchmarks.
Measurement is where organizations in this study lag the most, and at both ends of the spectrum. The slower movers simply haven’t reached it yet: still mid-rollout on tools and people, they leave it for later. At the other extreme, AI-native and sharp-turn orgs skip it on purpose, because the game, they feel, is speed, and instrumentation reads as drag. But even they are coming around, because without it token-maxxing becomes a runaway spend spiral and you can’t make grounded decisions about where to push and where to pull back.
The scale below runs through four levels. It begins with usage: who’s using which tools and how much, the easy data where most organizations stop. Then quality, whether the AI-generated code holds up. Then productivity, whether you’re genuinely shipping faster, which only means something once work is complexity-weighted. And finally benchmarking, how your velocity, cost, and output compare to peers. Each level is harder to measure than the last, and more useful.
Stats collected from every AI dev tool – tokens, LOC, sessions, who is using what.
Code quality, regressions, security findings, test cycles – does AI-generated code hold up?
Is the org delivering faster? Requires complexity-weighted PR scoring to be meaningful.
Compare velocity, cost, and output against peer companies and industry cohorts.
Usage is the floor – measure tokens and spend
Most of the organizations furthest along averaged $1–$2K per engineer per month – but the average hides a wide internal distribution: a few heavy parallel-agent users at $8–10K/month, many lighter users near zero. The bimodal shape inside the average is the real story. One organization in this study reached $12,000/engineer/month as its org-wide average – the outlier of the cohort, and an early signal of where heavy-agentic spend can land.
The same intensity that converts a skeptic is what makes the spiral real. At one AI-native company, one engineer used roughly $28,000 worth of tokens in a single month – about $16,000 of it from one /goal run that spawned fifty parallel agents and left them looping; the team found it “would run four hundred hours if you didn’t stop it.” None of it was instrumented until after the fact – measure before the spike, not after.
The picture has now surfaced publicly at scale. Uber rolled out Claude Code to its 5,000 engineers in December 2025; by March 2026, 84% were classified as agentic users and more than 70% of committed code was AI-generated, and the company had burned through its entire 2026 AI budget in four months. CTO Praveen Neppalli Naga, candidly: “I’m back to the drawing board because the budget I thought I would need is blown away already.” Salesforce is the public benchmark at scale – Benioff says Salesforce expects to spend close to $300M on Anthropic tokens in 2026 for the company’s ~15,000 engineers (roughly $1,700/engineer/month), paired with an explicit hiring freeze on engineering and a public claim of 30% productivity gains. The framing is sharper than the dollar number: Salesforce is publicly substituting tokens for headcount, and there is plausibly more spend to come.
Subscription pricing as arbitrage – Claude Max 20x and ChatGPT Pro 20x cost $200/month, versus $2–8K/month at API list price for a heavy agentic engineer. Many organizations cannot use individual plans because they lack centralized billing, SSO, and audit logs; where the control environment allows them, the gap is too large to ignore. The discount carries a control tradeoff: consumer accounts are not zero-data-retention environments, so disable model training and evaluate the provider boundary separately.
Routing becomes an economic requirement at this intensity. At the Claude Code mix measured across this study’s own sessions in July 2026 – roughly 94% cache reads, 5% cache writes, and under 1% each of output and fresh input – 10B tokens would cost roughly $10–12K/month on Opus and $20–23K on Fable at list prices. The common pattern is a portfolio: subscription access to top models for planning and hard tasks; faster, lower-cost models for routine agent operations.
Lower-cost substitution is no longer confined to chores. GLM-5.2 brings long-horizon agentic engineering into low-cost coding plans; Qwen, DeepSeek, Kimi, and small fine-tuned models handle tests, documentation, triage, and other repeatable operations. Cursor’s Composer 2.5 (built on Kimi K2.5) is another standout – matching frontier models on key benchmarks at ~10–60× lower cost per task on Artificial Analysis’s measurements.
It’s the distribution, not the average.
The useful number is not the average; it is the distribution – by engineer, team, workflow, and accepted output. The same skew that showed up in spend shows up even more sharply in what gets produced.
The cleanest public proof comes from Cursor’s 2026 Developer Habits Report: AI output is brutally concentrated – the p99 ships 46× the median’s AI lines (Gini 0.77). A widening median-to-p99 gap is the signal that AI compounds for the converted few, not the whole org. Other datasets rhyme: CircleCI found the top 5% of teams nearly doubled throughput while the median grew +4%, and one study company saw a third of engineers at 3–5× parallel throughput, a third single-threaded, and a bottom third barely using agents.
And the organizations that moved fastest are often the ones furthest behind on measurement. Fewer brakes – but also less control: when the bill finally arrives, there is no usage-and-output dashboard to interpret it with, and the only responses left are to absorb the cost or cancel the program. Measurement, the subject of this step, is what would have given them a third option.
Quality is the second-hardest dimension to measure
AI-generated code adds two requirements on top of traditional quality measurement: an attribution layer (which lines came from an agent) and a comparison layer (is the agent’s output better or worse than what the engineer would have shipped manually). Without those, every “AI quality” claim is anecdote.
One study CTO proposed a more operational quality bar: merge-readiness. Cognition’s FrontierCode asks whether a repository maintainer would actually merge the PR, combining correctness with regression safety, test quality, scope discipline, style, and codebase fit. Internally, measure the same idea through first-review acceptance, rework, and revert rates – segmented by task complexity.
The emerging metric set:
Acceptance and revert rates. What share of agent-written PRs merge on first review, and what share get reverted within two weeks of merging. A high first-review merge rate paired with a low revert rate is the cleanest single signal that the team trusts – and can trust – what the agents produce.
Regression rate per AI-touched module. Tag modules agents have touched; track incidents against them vs. human-only. Orgs doing this report no meaningful delta – itself the justification for further delegation.
Accepted LOC vs. generated LOC. A 90%-generated, 30%-accepted ratio means expensive tokens are being thrown away – a direct AI-Ops tuning signal.
Security findings rate. SAST (static application security testing – scanning the source code) and DAST (dynamic application security testing – probing the running application) on every PR, with particular attention to agent-generated code. Adversarial-agent setups (writer + critic, cross-model review) exist to catch what one model misses with another. False-positive rate counts as a quality dimension too – noisy AI review erodes engineer trust faster than no review at all.
The gap: most organizations in this study aren’t running these systematically. Cost is the only line almost everyone watches; quality remains aspirational for the majority. There’s a reason it stays that way, beyond effort: engineering is extremely noisy. Output varies wildly across modules, developers, and the kind of feature being built – so statistical measures are hard to trust, and too often the result you see is mostly chance.
Productivity needs a complexity denominator
Counting lines of code or tokens is the easiest place to start and the most misleading place to stop. A principal engineer fixing the company’s hardest bug might ship a single line in a 24-hour session; a junior writing routine CRUD might ship two thousand lines in the same time. Without a way to weight what was shipped by how hard it was, every “AI productivity” claim is suspect – and every “X% of our code is AI-generated” headline is an input metric, not an output one.
Two approaches to the complexity denominator have emerged, from opposite directions – a learned model and a fixed taxonomy.
The learned-model approach is visible in the open: the insurtech Lemonade publishes complexity-analyzer (github.com/lemonade-hq/complexity-analyzer), an open-source CLI that uses an LLM to score the intrinsic difficulty of a pull request – so each unit of work carries a difficulty weight before any productivity number is computed. The unit of productivity becomes complexity points shipped per engineer per week, not lines of code – and the “vanity metrics” framing for LOC and tokens used elsewhere in this playbook follows directly from treating complexity, not volume, as the denominator.
One study company arrives at the same place from the opposite direction – a fixed taxonomy rather than a learned model:
Both approaches converge on the same operational metric: complexity-weighted throughput per engineer.
One caveat grows as orgs approach the dark factory: “per engineer” inflates toward meaninglessness when a few people run large agent fleets – headcount shrinks while the token bill balloons, so the denominator stops reflecting real cost. The metric that survives the shift is complexity-weighted throughput per dollar of total cost – human plus AI. Per engineer is the practical metric today; per total dollar is the one to graduate to.
Without this denominator, even a well-instrumented organization cannot say what “productivity” actually means. Third-party analytics tools (DX, Hivel, Jellyfish, Worklytics, LinearB and others) are starting to fill this gap on top of Cursor and Copilot telemetry, but the study’s strongest practitioners are building the complexity layer themselves – because the weighting reflects their domain, not a generic SaaS heuristic.
Benchmarking is peer-to-peer, not platform-to-planet
Benchmark against peers (same stage, sector, and architectural class) on per-developer metrics, not platform aggregates (planet-wide totals like GitHub’s are the sum of millions of repos; no single org runs at that rate). Three numbers actually travel between companies: AI-attributed share of code shipped (the study runs <10% to 95%+), effective FTE per human engineer (human ICs × concurrent agents, 1× to 5–7×), and token spend per engineer (~$1–2K average, near-zero to $12K). Almost no one in the study measures these systematically yet – the benchmarking advantage is simply the discipline to define a peer set, instrument the same numbers, and update quarterly.
Security & Compliance
Agentic AI widens the security surface in four directions at once. There is risk in what the agents write: vulnerabilities shipped at a volume traditional review cannot fully catch. There is risk in what the agents do – prompt injection, tool misuse, supply-chain compromise, and PII leakage: familiar risks with new agent-specific paths, plus attack modes that emerge when models can act. And there is risk from the attacker’s agents: AI that scans your code orders of magnitude faster than any human researcher could. The fourth vector is the model and provider themselves: bias in the weights, hidden serving routes, data retention, and trust in a company that increasingly competes across the software stack. On top of all four sits a compliance gate that maps agentic work to the organization’s approval, evidence, and audit controls. Unlike the other dimensions, this one is not a progression you climb; it is a surface you have to cover.
The strategic implication is the one most underestimated in the study: “we won’t use AI in our codebase” is no longer a defensible posture. The attacker is using AI against you regardless – so opting out doesn’t lower your risk, it just means you find your vulnerabilities second, in a CVE filing, rather than first. The job is to cover the surface, not avoid it.
Insecure code shipped faster than review can catch it.
Prompt injection, supply-chain attacks, PII leakage.
AI finds your gaps orders of magnitude faster than any human.
Bias, model-level attacks, hidden routing, retention, and trust.
Risk Surface 1: What Your Agents Write
The risk. The simplest vector is the easiest to underestimate: agents generate code faster than humans can review it, and not all of it is safe. AI-written code carries the same vulnerability classes as human code – hardcoded secrets, injection-prone queries, insecure defaults, over-broad permissions – but at a volume that overwhelms line-by-line review.
The mitigation. Move security into the pipeline: SAST and DAST on every PR, plus secret and dependency scans and an adversarial-agent stage – a writer model produces the code and a separate critic model or specialist service tries to break it before merge. Measure findings caught before merge, escaped issues, false positives, and added review time. Because this stage already reads every change, reuse it for complexity scoring and quality-adjusted throughput on the Measurement dashboard.
Risk Surface 2: What Your Agents Do
The risk. Two agent-amplified attack paths dominate. Prompt injection is the new SQL injection, and the CVEs have started landing. In EchoLeak (CVE-2025-32711), indirect prompt injection could cause Microsoft 365 Copilot to disclose information over a network. The shape: an agent reads an issue, comment, document, website, tool result, or customer email and treats embedded instructions as authoritative.
Supply-chain attacks are the second. The 2025 npm “Shai-Hulud” worm self-replicated through hundreds of packages by stealing publish tokens. In February 2026, the Shai-Hulud-like SANDWORM_MODE campaign disclosed by Socket – named for the malware’s own Dune-themed switches, not the Russian APT of similar name – added an agentic twist: it planted a rogue MCP server into AI coding assistants (Cursor, Claude Desktop, and the then-Windsurf, now Devin Desktop) whose prompt-injected tool descriptions turned the assistant into a confused deputy, quietly exfiltrating SSH keys, cloud credentials, and npm tokens. The same surface opens every time an agent installs a typosquatted package or connects to an unvetted MCP server.
The mitigation. The defense is layered, because no single layer holds. The probabilistic layer – input sanitization, spotlighting (tagging untrusted input so the model can tell data from instructions; in Microsoft’s GPT-family experiments it cut injection success from over 50% to under 2%, although results vary by model and attack), and classifier guardrails (Llama Guard, NeMo Guardrails, LLM Guard), productized by the AI-gateway vendors: Noma and Lasso, plus the platforms now inside Check Point (Lakera) and SentinelOne (Prompt Security), with MCP-specific gateways like MintMCP underneath, on a triple-gate pattern: client→LLM, LLM→MCP-server, MCP-server→external-API. This layer shrinks the attack surface but never closes it; a clever enough input gets through.
The architectural layer is the durable one: the dual-LLM / quarantine pattern Simon Willison proposed in 2023 (a privileged model that holds the tools never reads untrusted content; a quarantined model that reads it cannot act), capability scoping (each agent runs under its own non-human identity, treated as an independent entity for permissions and auditing – sandboxed, with no write access that matters), and a human in the loop for privileged actions. Prompt injection is the new SQL injection with one cruel difference: there is no parameterized query that fully solves it yet – so you defend in depth, or not at all.
Persistent memory and agent-to-agent handoffs are security boundaries too. Validate provenance before anything is written to memory, isolate state by user and session, authenticate peer agents and delegated authority, and bind every action back to the initiating user. At runtime, monitor tool sequences and data egress, with the ability to isolate the agent, revoke access, and roll back poisoned state.
Risk Surface 3: What Other People’s Agents Do
The risk. The surface widens even for organizations that don’t use AI to build – because the attackers do. AI now scans thousands of code paths in minutes that once took researchers days. CrowdStrike’s 2026 Global Threat Report logged an 89% year-over-year increase in operations by AI-enabled adversaries. Separately, Anthropic described a 2025 cyber-espionage campaign in which AI performed 80–90% of the tactical work, calling it the first documented large-scale attack executed without substantial human intervention.
The mitigation. AI red teaming – continuous adversarial testing, automated or researcher-led, hunting vulnerabilities across code, applications, cloud configuration, agent workflows, memory, and exposed interfaces, from vendors like FireCompass, HackerOne, Zscaler, and Obsidian Security. Track what it finds and how quickly those findings are fixed.
Risk Surface 4: The LLM and the Provider
This is the most complex surface because it raises a chain of trust: the model, the company behind it, and the infrastructure that runs it. The strategic question is whether the code, workflows, and feedback you send to an AI provider could help that same company compete with you later. Start with the workload: what data it handles, what actions it can take, and the consequences of failure. Only then choose the model, provider, region, retention mode, and execution boundary.
Frontier Providers: Training, Retention, and ZDR
The minimum control is simple: company data must not be used for model training. Verify that training
is disabled for every account and plan – whether it is the business default or a setting you have to
turn off. But training opt-out is not zero data retention (ZDR). ZDR means eligible
prompts and outputs are not retained in provider logs after processing. It generally requires an
approved API or enterprise arrangement, is not available for every endpoint or feature, and does not
necessarily eliminate application state. For API calls, set provider-specific persistence controls
explicitly: on OpenAI’s Responses and Chat Completions APIs, send store: false and verify
that the SDK, agent framework, or gateway preserves it. This prevents response application-state
storage; it does not create ZDR or remove standard abuse-monitoring retention. Without ZDR, API and
temporary-session content is commonly retained for up to 30 days even when training is disabled;
ordinary chat history may remain until it is deleted. Under the major providers’ published terms, use
of that retained content is limited to operating the service and providing the requested history,
detecting abuse and security incidents, providing support, and meeting legal obligations.
Safety-flagged content or data subject to a legal hold can be kept longer.
Frontier Providers: Trust
Those terms define what the provider is allowed to do with your data today. They do not answer how much you trust the company over time. Alex Karp framed the strategic risk as handing model companies your “weights and alpha.” Providers’ published policies restrict how customer data can be used, but his broader question remains: frontier labs are moving from model APIs into coding, agents, search, browsers, and more of the application layer their customers occupy. A toggle and a policy reduce the near-term risk; they do not guarantee that incentives will remain aligned. Decide which data should never cross that boundary.
Neo-Clouds and Model Routers
Identify the serving path before approving the model. A hoster runs the weights on infrastructure it controls. A router forwards the request to another provider. A mixed provider can do both behind the same catalog, account, and API. Mixed providers such as Together AI illustrate the issue: “available on Together” does not by itself tell you where the request is processed.
Together makes the boundary explicit in its organization privacy controls: pass-through models send prompts and responses directly to third-party providers under those providers’ data policies, and prompt storage must be on to use them. Set both prompt storage and Allow passthrough models to No, and leave training consent off. Then allow-list and log the approved model, host, deployment, region, and retention mode. Otherwise a model selected through a familiar cloud account can still move company data to another provider or jurisdiction.
The neo-cloud itself is also part of the trust boundary. Even when it hosts the weights, it can retain prompts or outputs through caches, debugging logs, abuse monitoring, or support systems. Its contract – not the model’s license – controls retention and staff or subprocessor access. Review its DPA, retention defaults, logging exceptions, subprocessors, and deletion controls with the same rigor as a frontier provider.
Open-Weight Models
Open weights were a powerful distribution wedge for Chinese model labs: Western inference clouds could host DeepSeek, Qwen, Kimi, GLM, and MiniMax models close to their customers, accelerating adoption and the surrounding deployment and evaluation ecosystem. That route remains important, but the market is splitting. Alibaba now pairs open Qwen releases with proprietary Plus and Max tiers; Qwen 3.8 Max Preview is, as of this writing, available only through Alibaba’s own channels – its Token Plan and the Qoder and Qwen apps – with open weights promised but not yet released.
Running open weights on infrastructure you control can remove the external data path and reduce dependence on a frontier provider. It does not remove what the model learned. A 2026 EACL study across 36,000 prompts found that model origin and prompt language systematically changed political bias; a separate audit of DeepSeek across 646 sensitive prompts found suppression of references to transparency, government accountability, and civic mobilization. The answer is not to reject Chinese open models – it is to review independent evaluations and test the shortlisted models in the languages and domains where they will actually operate.
Private Hosting
Private hosting is becoming a credible architecture choice for a growing minority, although it remains operationally out of reach for most organizations today. At WWDC26, Apple showed a complete local agentic stack on MLX and a distributed path for models too large for one machine – explicitly citing a 1.6-trillion-parameter DeepSeek model whose weights require more than 800GB, sharded across multiple Macs. It was not one Apple CPU running a “1TB model”; the important point is that local silicon, unified memory, and clustered inference are making private execution more practical. Western tier-one companies are also expanding the open stack: NVIDIA’s Nemotron family provides models, weights, datasets, and training recipes that can run on-premises, in a private cloud, or through a controlled enterprise endpoint. The likely end state is a portfolio – frontier cloud models where capability wins, privately run open models where control matters more.
The Compliance Gate
Start with the organization’s documented SOC 2 change-management controls. CC8.1 states that the entity “authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures to meet its objectives.” For agentic changes, document how each control activity is satisfied and confirm the treatment of automated and human approvals with the auditor. The most advanced study companies are working through that question now:
The infrastructure and test coverage are there; the remaining question is how each compliance-sensitive flow satisfies the company’s audited control. Where human judgment or signature is required, it becomes a useful prompt-injection circuit-breaker only if the system presents the underlying evidence and enforces the decision outside the model. Alongside this, a new certification stack is forming for the agent autonomy SOC 2 was never written for: NIST AI RMF (risk guidance), ISO 42001 (AI governance), and above all AIUC-1 – the first agent-specific standard, positioned as “SOC 2 for AI agents.” AIUC-1 uses independent audits, at least quarterly technical testing, and annual operational-control reviews. It adds AI-specific evidence and controls; it does not itself rewrite the organization’s existing SOC 2 control design.
Volume is the operating challenge. When agents write most of the code, a human-review control can become the bottleneck the pipeline backs up against – and much of the review is already theater: low-risk PRs get rubber-stamped in seconds because no one can meaningfully read that much. The workable middle keeps the control real without making it a rate limiter – a model performs the first pass and presents its judgment, underlying evidence, and risk signals rather than an unstructured raw diff; the change is surfaced and acknowledged in the channel where the work happens, which becomes the audited record; and genuine scrutiny is reserved for the tail of changes that actually carry risk, with automation applied where the documented control design and auditor permit it.
The Ten Commandments for Agentic Security
How the Strongest Teams Build It.
At the leading edge, the agent itself is increasingly something companies build, not buy. Ramp runs an internal agent, Inspect, on the open-source OpenCode and on Modal sandboxes loaded with its full stack; it writes over half of all merged pull requests, and more than 80% of Inspect is now written by Inspect. Stripe’s Minions merge over a thousand PRs a week on a fork of Block’s open-sourced Goose. Shopify open-sourced Roast; Coinbase runs Forge. The pattern is consistent among teams with deep engineering benches: own the harness, because it only has to work on your code.
What ownership buys is control a buyer never gets: routing each task to the cheapest adequate model, running on your own sandboxes, wiring in your own context, multiplayer sessions the whole team can watch, and no lock-in to a single vendor. For a company whose edge is engineering velocity, that control can be worth a standing team.
One AI-native company sharpened the point: which harness you run, whether Claude Code, Codex or OpenCode, is the commodity layer, and increasingly interchangeable. They swap it freely (lately experimenting with Codex inside their own container); the part that compounds is everything wrapped around it – a headless platform the agent can drive end to end, the curated skills that encode how your best engineers build, the tools, and the loops. The durable advantage is how you serve the agent its context, not which agent you serve.
Two things keep this from being a blanket recommendation. First, the economics bite even at the top. That early playbook – unlimited use and competitive usage leaderboards – does not survive maturity. Uber’s reversal from maximal use to per-engineer caps and gated tools shows why routing and budgeting belong inside the harness, not as a finance afterthought.
Building your own harness is not necessarily right for everyone. Credible off-the-shelf options already exist, and they keep improving. Every name here has a reason to lead on tooling and the bench to staff it; for almost everyone else the off-the-shelf agents are the right call, and a half-built internal harness is a tax. Factory’s Droids show the buy-side alternative: a commercial agent runtime with model routing, integrations, permissions, and observability built in. These systems were also built into a gap that is closing; several of them predate good managed agents entirely. As managed options mature and open harnesses standardize – Block’s open-source Goose now sits under the Linux Foundation’s Agentic AI Foundation – the honest rule is narrower than “build your own”: own the parts specific to you (your context, your sandboxes, your evals), buy the rest, and revisit the line every few months.
That line keeps moving because the ground beneath it does. Anthropic, OpenAI, Cursor, and others run roughly a phase ahead of everyone else: they turn each model on their own engineering before the rest of us feel it, so the way they describe their internal work is a preview of the infrastructure and workflows you’ll need next year.
Score yourself, and see where the study sits.
Plot your own scores on the four dimensions, then compare against the study. Where your row sits darker than the study, you are ahead; where it sits paler, that is your next move.
| Coding | AI Ops | Metrics | Security | |
|---|---|---|---|---|
| Your organization | ? | ? | ? | ? |
| AI-native scale-up (advanced) | 4 | 4 | 4 | 3 |
| Late-stage converter (aggressive) | 4 | 4 | 1 | 2 |
| Mid-stage converter (steady) | 3 | 3 | 3 | 2 |
| AI-native startup (mid-stage) | 4 | 3 | 2 | 2 |
| AI-native startup (early-stage) | 3 | 2 | 1 | 1 |
| Late-stage SaaS (mid converter) | 3 | 3 | 2 | 2 |
| Legacy SaaS (slowest mover) | 2 | 2 | 1 | 1 |
| Study median | 3 | 3 | 1 | 2 |
Score the first three columns from the four-level scales of Coding Infrastructure, AI Ops, and Measurement. Security has no single scale – read it from 1 (ad hoc SAST / DAST, no AI-specific policy) to 4 (compliance-as-code: policy enforced at the agent layer, audit trails generated as a byproduct).
Build the Churches is a test of seriousness.
A company that has only bought AI coding licenses is still experimenting; one that has built the full Coding Infrastructure ladder, AI Ops ownership, measurement, and an agent-aware security posture has begun changing its operating system. But do not wait for the foundation to be complete before converting people. Engineers working agentically expose the missing tests, permissions, context, and observability, giving AI Ops its real backlog; every improvement then enables deeper conversion. Infrastructure and conversion advance together. The church is where the new behavior becomes repeatable.