AI #186: The World Takes Notice

In the wake of Jacob Coxon's resignation and the resulting preference cascade, events escalated quickly. The mainstream media picked up the story. Anthropic CEO Dario Amodei said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.

The public took notice, raising the salience that AI might kill everyone and roughly doubling people's estimates of how likely that is to happen, from a mean of about 15% to 30%. Many politicians called for regulations, guardrails and emergency hearings in Congress. Avoiding political polarization was described as the most important thing.

David Sacks, Mark Zuckerberg and Jensen Huang convinced Donald Trump to fully conflate existential risk with opposition to data centers, and Trump went Full Hoax on AI existential risk. There is also a systematic effort to launch hit jobs against those who warn that AI might cause everyone to die, with the first concrete target being METR and extreme fire focused on Effective Altruism.

Separately, Anthropic reported attempts to use Claude for malicious purposes, including fraudulent distillation attacks at scale attempted by all the major Chinese AI labs, several of which also silently passed massive amounts of user data to Anthropic. Anthropic reports doing a remarkably good job containing such threats, though it is unknown whether all perpetrators have been caught. A new model solved a Millennium Prize, and a capabilities review for Astra was published. OpenAI took 25% of its engineers, pointed Astra at its own systems and kept them there until they ran out of security issues. Senator Chris Van Hollen sent OpenAI six pages of questions about Astra, concerned about the lack of monitorability and evidence of alignment.