Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Anthropic has drafted federal AI safety legislation and has been presenting it to lawmakers in the hope that Congress will take it up, according to reporting by Politico that TechCrunch summarized. The lab confirmed the draft's existence to Politico, with a spokesperson saying it reflects the proposals in Anthropic's Advanced AI Framework, which the company says it has been advocating with federal and state lawmakers since June. Anthropic did not respond immediately to TechCrunch's request for comment.
The bill sets out narrow conditions under which an AI model's distribution could be stopped. The trigger is a "foreseeable, imminent risk" that more than 50 people could die, or that property damage could exceed $1 billion. That risk would also have to stem from one of three specific causes: the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack.
Those thresholds track California's Senate Bill 53, which was recently enacted and uses the same 50-person and $1 billion figures to define what counts as catastrophic risk. In practice, numeric thresholds of this kind are meant to make a vague standard auditable: a regulator, evaluator or court can point to a concrete line rather than argue over whether a model is "dangerous" in the abstract. The trade-off is that any specific number will fit some risks badly and leave others outside the trigger entirely.
The proposed process relies on independent evaluators registered with the U.S. Commerce Department. Those evaluators would assess a model's potential to cause the harms the bill describes, and the AI labs themselves could pay them, provided the payment is not contingent on the evaluation's results. The draft does not spell out how that condition would be enforced. This is a familiar problem in third-party assurance: if the entity being assessed chooses and funds its assessor, structural independence has to come from somewhere other than the contract language. A likely reading is that registration and the ban on results-contingent pay are meant to supply that independence, though the mechanism for policing it is not described in the reporting.
Timing and authority are the other distinctive pieces. When evaluators conclude an imminent catastrophic risk exists, they would have 72 hours to notify the Commerce Department. The department would then have to ask a judge in Washington, D.C., to halt the model's release. The bill would expressly bar any federal agency from stopping distribution on its own discretion. That judicial check is the provision the coverage flags as the draft's most consequential design choice: the bill reads as a safety regime, but it also constrains agencies that might otherwise act unilaterally.
Compared with other industry positions, Anthropic's draft goes further than some. Nvidia chief executive Jensen Huang recently argued that the AI industry needs no new laws at all. Demis Hassabis, chairman of DeepMind, has argued that a self-governing AI regulatory body should resemble the financial industry's FINRA, the self-regulatory organization for broker-dealers. Anthropic's evaluator model is closer to that self-regulatory instinct, with a registered-evaluator layer and a court as the backstop rather than direct agency authority.
The risks the bill targets are also narrower than the incidents that have dominated recent industry discussion. The focus in practice has largely been on agentic hacking: a run of incidents in which AI programs appear to have disobeyed their operators and attacked websites on the open internet, raising questions about whether models have adequate security controls. A framework built around biological weapons, control evasion and cyberattacks with mass-casualty or billion-dollar consequences addresses a different, more extreme class of harm than misbehaving agents causing localized damage. This suggests the bill's real function may be to set an outer boundary on liability and oversight rather than to govern the everyday failure modes developers currently encounter.
For developers and product teams, the practical questions are about evaluation and release timing. A regime built on pre-release assessment by registered third parties implies a gate before shipping frontier models, and a 72-hour notification window with a court filing behind it implies that a challenged release could pause on a legal timetable rather than a product one. Nothing here is law: the draft has been shopped to lawmakers, not introduced as reported. Whether the 50-person and $1 billion thresholds, the evaluator registration scheme and the judicial check survive the legislative process is unknown.
Why it matters: frontier labs are now writing the rules they would be regulated by, and this draft's most concrete effect would be to route emergency stops through a D.C. court instead of an agency, which changes how fast a release can be halted and who bears the burden of proof. If adopted in anything like this form, teams shipping high-capability models would need to plan for external evaluation before launch and for legal delay as a real release risk. My inference is that the evaluator-funding arrangement will draw the most scrutiny, since a self-funded assessor with a results-contingency ban leaves the enforcement question open.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
The proposed policy seems to offer protections against AI running amok but actually forbids federal agencies from blocking a model's release without the permission of a judge.