too dangerous to release: the containment problem in frontier ai

we are getting good at telling when an ai model is dangerous. we are not getting good at stopping its release. that gap is the whole problem.

Joakim Nordheim Larssen

the asymmetry, stated plainly

The International AI Safety Report, the multi-government assessment chaired by Yoshua Bengio, names the timing version of this problem the "evidence dilemma": pre-emptive mitigation might turn out unnecessary, but waiting for conclusive evidence "could leave society vulnerable to risks that emerge rapidly." [1] That is about when you know. The containment problem is its irreversible twin: what you can do once you know.

The detection side has genuinely improved. The canonical paper, "Model evaluation for extreme risks" (Shevlane et al., 2023), proposed dangerous-capability evals (offensive cyber, manipulation, CBRN uplift) and alignment evals (will it apply them). [2] But it is about detection. It does not solve what to do once a dangerous capability is found.

The reports now admit the control side is the weak one. The October 2025 update to the Safety Report cites "new evidence of challenges in monitoring and controllability," warning that future systems "may conceal unsafe behaviours during testing." [3] Our best detection method gets least reliable exactly when stakes are highest. The November 2025 update adds that the number of companies publishing safety frameworks more than doubled, yet "sophisticated attackers can often bypass current defences, and the real-world effectiveness of many safeguards is uncertain." [4] More detection, safeguards that attackers route around.

the cleanest real-world case: claude opus 4

The asymmetry showed up, in writing, in a single deployment decision. On May 22, 2025, Anthropic activated its strictest safeguards, ASL-3, for Claude Opus 4. It stated it had "not yet determined whether Claude Opus 4 has definitively passed the Capabilities Threshold that requires ASL-3 protections," and acted because "clearly ruling out ASL-3 risks is not possible." [5] The detection was ambiguous: the lab could confirm neither danger nor safety. It shipped anyway, with mitigations bolted on as a precaution.

This cuts both ways. Acting precautionarily under uncertainty is the responsible move, and it is to their credit. But it illustrates the whole problem. When you cannot tell whether you are over the line, you do not stop; you ship with mitigations and hope they hold. Detection that resolves to "we are not sure" produces release-with-caveats, not non-release.

what the thresholds actually say

The three leading labs all publish "if-then" frameworks: here is the capability we watch for, here is what we do if we detect it. The named triggers are real and falsifiable. Credit where due, before I take it away.

Anthropic's Responsible Scaling Policy reached version 3.0 on February 24, 2026. Its automated-R&D threshold triggers "at the point where we determine that a model could compress two years of 2018 to 2024 AI progress into a single year." [6] A real, checkable claim, the kind that did not exist in 2022.

But notice what v3.0 did structurally. Its appendix admits that "a specific list of controls is overly rigid," and now prefers "to focus on what sort of argument an AI developer should make": a pivot from bright-line rules to case-by-case judgment. [6] And it steps back from hard "we won't deploy" commitments, "because" of a collective action problem: one lab pausing while others race "could result in a world that is less safe." [6] The governance failure this essay is about, conceded by the people writing the policy.

OpenAI's Preparedness Framework v2 (April 15, 2025) defines "severe harm" as "the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage." [7] "Critical" capability triggers the strongest language in any of these documents: "Until we have specified safeguards... that would meet a Critical standard, halt further development." [7] A real commitment to stop. But no model has been verified to cross a Critical threshold, so the stop button has not been tested in anger.

OpenAI also quietly removed things. The framework dropped from four thresholds to two, and dropped persuasion entirely, on the grounds that those risks "require solutions at a systemic or societal level." [7] Read that as honesty about scope or as a major risk vector defined out of existence. I lean toward the second: persuasion is the capability most likely to act at societal scale.

Google DeepMind's Frontier Safety Framework v3.0 (September 22, 2025) uses "Critical Capability Levels," detected with "early warning evaluations" and "alert thresholds." [8] It also wrote down the unsolved problem. For misalignment at "Instrumental Reasoning Level 2," where a model could undermine human control even while monitored, the listed mitigation is: "Future work: We are actively researching approaches." [8] Admirably honest, and an admission that detection has outrun control.

the escape hatch all three wrote down

Here is the part that turns three policies into one structural problem. Every framework contains a competitor clause: if a rival ships a comparably dangerous model without comparable safeguards, the lab may lower its bar.

OpenAI's is the most discussed. Section 4.3 says it may release a model with High or Critical capability if a competitor already did so, provided this "does not meaningfully increase the overall risk of severe harm" and it keeps safeguards "more protective than the other AI developer." [7] Zvi Mowshowitz and others called this a race-to-the-bottom incentive. [9]

Anthropic, when in the lead, commits to delay "until and unless we no longer believe we have a significant lead." [6] The commitment only binds when Anthropic is confident no competitor is close, which is precisely when competitive pressure is lowest. It evaporates exactly when it would matter most.

DeepMind's appears in its risk-acceptance criteria: a model at a misuse capability level can still be acceptable because "if other models are similarly capable and have few mitigations, then the marginal risk added by our release is likely low." [8]

This is the codified race to the bottom: the structure rewards whoever moves first with the weakest safeguards, who then defines the "marginal risk is low" baseline everyone else points to. And the labs know it. All three frame their hardest mitigations as conditional on industry-wide adoption. DeepMind says its mitigations are "of limited social utility" unless "all relevant organisations provide similar levels of protection"; Anthropic separates "our plan as a company" from "ambitious industry-wide recommendations" it "cannot commit to... unilaterally." [6][8]

The decision to deploy sits, in all three cases, with the company, under frameworks Anthropic calls "voluntary" outright. [6] Self-certification is the enforcement model: the gap between detection and binding stopping power.

why you cannot recall weights

Detection cannot save us because the most consequential release decision is irreversible.

The UK AI Security Institute put it as plainly as a government body can. Open-weight models "can allow harmful AI capabilities to proliferate rapidly and irreversibly," and safeguards baked into them "can also be quickly and cheaply removed." Their bottom line: "no current solutions can offer hard guarantees of safety." [10] The same institute holds the other side fairly: open weights "increase transparency, allow for widespread red teaming, and decrease market concentration." [10] Closed models are not safe either; they get jailbroken, and concentration creates its own systemic risk. The honest framing is not "open bad, closed good." It is that open release removes the one lever, rollback, that closed release retains.

That lever is real and has been used. When OpenAI's April 2025 GPT-4o update turned excessively sycophantic, to the point of encouraging self-harm, OpenAI reverted it. [11] An open-weight model with the same flaw could not have been pulled back; the copies were already distributed.

The de-alignment cost should end the "but we shipped it with safety training" argument. Lermen, Rogers-Smith and Ladish (2023) showed that with LoRA fine-tuning, a budget "under $200 and one GPU" reduced Llama 2-Chat 70B's refusal rate on harmful prompts to "about 1%" while retaining general capability. [12] Once weights are public, the safety training is not a property of the deployed system. It is a suggestion.

the cases: llama and deepseek

Meta's Llama is the proliferation case. Llama 3.1 405B (July 2024) was the first open-weight model to benchmark competitively against GPT-4o and Claude 3.5 Sonnet. [13] The download numbers are irreversibility as a single figure: one billion cumulative by mid-March 2025, 1.2 billion by late April 2025. [14][15] You cannot un-distribute 1.2 billion downloads. (Llama is open weights, not OSI open source; the Community License imposes a 700-million-MAU cap. [16] For containment the distinction is irrelevant: licensed or not, the weights are downloaded and de-alignable.)

DeepSeek is the export-control stress test. DeepSeek-R1 (January 20, 2025) roughly matched advanced US models, and the headline was the cost. The DeepSeek-V3 report states the model "required 2.788M GPU hours" and, "assuming an H800 GPU rental price of $2 per GPU hour, the total training costs amount to $5.576M." [17]

A caveat the research demands: that $5.576M is the final training run only. The paper says it "excludes the costs associated with prior research and ablation experiments." [17] The "frontier model for $6M" narrative is misleading, and Brookings notes an unsubstantiated suggestion DeepSeek had more compute than disclosed. [18] (There are also allegations, in the politically charged House Select Committee report "DeepSeek Unmasked," that it used "over 60,000 Nvidia chips" via transshipment through Singapore. [19]) But the real point is the direction: a non-US lab produced a near-frontier model at a fraction of the assumed cost, which is what makes any compute-based containment lever leak. DeepSeek even splits the experts, who divide over whether it argues for smarter controls or for keeping them: the strongest physical lever is contested at the level of whether it helps or hurts. [20][21][22]

the enforcement levers, and where each one leaks

If self-certification is the enforcement model and open weights are irreversible, what is left? Four external levers, each failing at a different seam.

Compute and chip export controls. The strongest lever, because chips are physical, scarce, and produced by a concentrated supply chain. Also the leakiest in practice. The administration banned even compliant H20 chips in April 2025, reversed in July to let Nvidia resume H20 shipments, and in December 2025 announced H200 exports to China in exchange for a 25% revenue stake to the US government. [23][24][25] A lever that reverses three times in a year is not reliable. And it leaks physically: in December 2025, CNBC reported roughly $160 million of export-controlled Nvidia GPUs allegedly smuggled into China. [26]

The EU AI Act. The most concrete legal regime. A general-purpose model is presumed to have "high-impact capabilities" (the criterion that triggers systemic-risk obligations under Article 51) when training compute "measured in floating point operations is greater than 10^25." [27] Models in scope face risk assessment, adversarial testing, and serious-incident reporting. [28] But the Act governs behavior and reporting, not release reversal. The 10^25 threshold is a compute proxy that DeepSeek-style efficiency gains route around, it is EU-only, and it does nothing about open weights already in the wild.

The International AI Safety Report. The best collective instrument we have, and by design an assessment that makes no policy recommendations. [29] It diagnoses; it cannot enforce. That our flagship international instrument is a detection instrument, not a containment one, is itself evidence for the thesis.

Voluntary commitments and summit diplomacy. The Bletchley Declaration (2023, 28 countries and the EU, including China) is explicitly voluntary. [30] The Seoul Frontier AI Safety Commitments (2024) had 16 companies, including China's Zhipu and the UAE's TII, agree to publish safety frameworks. [31] Real and worth something, but unenforceable, self-defined, and carrying no penalty for breach.

why voluntary containment fails: the unilateralist's curse

There is a clean theoretical reason voluntary regimes cannot hold, the keystone of my argument.

Bostrom, Douglas and Sandberg (2016) named the "unilateralist's curse": "when a set of decision makers independently choose whether or not to undertake some action, the action will be taken more frequently than is optimal." [32] Even if every actor is altruistic and well-informed, it takes only the single most optimistic one to trigger an irreversible action. Their remedy is a "principle of conformity," that is, coordination, which is exactly what voluntary regimes cannot guarantee.

Apply it. It does not matter how good the cautious majority's detection is. If one lab, or one nation, judges a release acceptable, the weights are permanently out. DeepSeek and Meta are real-world instances. The competitor escape hatches are not bugs; they are the formalization of the curse, explicitly letting the most optimistic actor set the floor for everyone else.

This is why I keep returning to "structural." Containment of a non-scarce, copyable, irreversibly distributable artifact requires unanimous restraint, and unanimous restraint is the one thing the curse guarantees you will not get.

what the treaty analogies actually teach

People reach for nuclear and chemical arms control as the model. The comparison is instructive, mostly for the disanalogy.

Nuclear (NPT and IAEA). The NPT has 191 states parties, the most-ratified arms-control treaty; IAEA safeguards cover roughly 900 nuclear facilities across 57 non-nuclear-weapon states. [33][34] Why did it partly work? The control point is fissile material: rare, expensive, physically detectable, hard to hide. And even then only partly. India, Pakistan and Israel never joined; North Korea acceded, withdrew in 2003, tested in 2006; the A.Q. Khan network sold enrichment technology to Iran, Libya and North Korea. [35][36] The best case in the history of containment achieved only partial containment.

Chemical (CWC and OPCW). The strongest verification success. The Chemical Weapons Convention has 193 states parties, and on July 7, 2023 the OPCW verified that all 72,304.34 metric tonnes of declared chemical agent had been irreversibly destroyed. [37] You can literally destroy the dangerous thing, and they did, and counted it. But hold onto one word: declared. Syria acceded in 2013, did not declare its full program, then used chemical weapons; after Assad fell, OPCW teams found previously undeclared weapons. [38] Verification verifies the declared, not the hidden.

Biological (BWC). The cautionary tale, and the best analogy for ai. The Biological Weapons Convention entered into force in 1975 with 187 states parties, and it has no verification organization and no inspection regime. [39] In July 2001, after about six years of negotiation, the United States rejected the draft verification protocol and all further negotiation, arguing in part that bioweapons development cannot be reliably verified. [39] Its entire Implementation Support Unit was set up in 2006 with three permanent staff. [39] Contrast: 900 facilities inspected, 72,304 tonnes destroyed, three staff.

Biology is the closest precedent because it is dual-use, knowledge-based, dispersed across legitimate labs, and the dangerous input is not scarce. The US declared verification infeasible and the regime stalled. That is the precedent ai is walking into.

why ai is harder than all three

The arms-control cases that worked shared four properties: the dangerous thing was physical, scarce, declarable, destructible. Ai has none, with one partial exception: compute. "Computing Power and the Governance of Artificial Intelligence" (Sastry, Heim, Belfield, Anderljung, Brundage, Bengio et al., 2024) argues compute is the one ai input that is detectable, excludable, quantifiable, and produced via a concentrated supply chain: the ai analogue of fissile material. [40] The same paper warns naive compute governance "carry[s] significant risks in areas like privacy, economic impacts, and centralization of power." [40]

But the lever is eroding. Epoch AI estimates the compute needed to reach a given capability halves roughly every eight months, about 3x per year, far faster than Moore's Law. [41] Decentralized training is breaking the assumption that frontier models require one concentrated cluster; low-communication algorithms have sharply reduced the interconnect bandwidth that distributed training needs. [42] And the verification hardware that would make compute governance enforceable, on-chip metering and proof-of-training, is the least mature part of the stack. [43] That mirrors the BWC: the verification technology is not there, so the lever cannot bind. Everything else points the same way. Stanford's Alpaca was fine-tuned from Meta's LLaMA for under $600 [44]; open weights are irreversible, which the NTIA's 2024 dual-use foundation models report treats as the central feature of the open-weight question. [45] Where the CWC could inventory and destroy 72,304 tonnes of a substance, there is only a file that has already been copied.

For balance, the bridge proposals are serious: an "IAEA for AI" (proposed by OpenAI's leadership in 2023), standard-setting and registration, four-institution designs including an "NPT for AI." [46][47][48] But the skeptics are equally credible. Chatham House and others argue the nuclear model will not work for ai, on the disanalogies above: ai is intangible software, led by private firms not states, moving faster than institutions can. [49][50] The "IAEA for AI" proposals are designs for a fissile material that does not exist.

my position

The asymmetry between detecting danger and stopping release is structural, not a policy gap waiting to be closed by a better framework or a more committed lab. Detection has matured. Every containment lever, meanwhile, fails at a different seam. Physical chips leak through smuggling and reverse through politics. Legal thresholds are jurisdictional and gameable by efficiency gains. Voluntary commitments founder on the unilateralist's curse. Open weights are irreversible and two hundred dollars to de-align.

So the answer is not "ban open weights." The benefits are real, and the curse makes a ban both costly and futile: you give up the transparency and red-teaming, and one optimistic actor abroad releases anyway. The only levers that survive scrutiny are the physical chokepoint, compute, while it lasts, and binding pre-release verification on the EU model, evidence of safety before release rather than self-certification after. Both are worth doing. Neither is sufficient. Compute is eroding at 3x a year, and verification verifies the declared, not the hidden.

So the honest conclusion is uncomfortable. If containment cannot be made reliable, it cannot be the primary strategy. We should plan for a world where dangerous capabilities proliferate, because the structure of the problem says they will, and invest in resilience and defense, not only gatekeeping. The labs already wrote the diagnosis into their own policies: the hardest mitigations are worthless unless everyone adopts them at once, and no one can make everyone do anything.

We are getting very good at reading the warning light. We have not built a brake. The work that matters now is the work that assumes the brake will not arrive in time.