The Preference Cascade Is Only Getting Started

Zvi Mowshowitz wrote that a preference cascade about existential risk from AI is in progress and that continuing it is necessary. He cited new AI Impacts survey results indicating the average AI researcher estimates about an 18% chance AI will cause human extinction or similarly permanent and severe disempowerment, with Asian researchers reporting higher risks than Western researchers. He also referenced Politico polling showing nearly two-thirds of Americans say there is at least a moderate risk AI will destroy humanity, with a mean chance around 30%-33% and a median on the order of 10% and only a small partisan split.

Mowshowitz noted a Yale School of Management flash poll of executives in which 93% disagreed with President Trump's characterization of AI catastrophic dangers as a hoax. He quoted Elon Musk supporting some form of AI regulation and a standalone regulatory agency. He cited Matthew Yglesias arguing that safety-focused policy and technical work are not doomsaying, and outlining an order of operations: light-touch transparency and evaluation rules, tough export controls, moderate-touch rules, negotiation with China, then an ambitious end-stage framework. Mowshowitz wrote he worries whether there is that kind of time.

Mowshowitz pointed to op-eds by Daniel Kokotajlo in The Free Press, Steven Adler in The New York Times, Will Knight in Wired, Stephen Witt in The New York Times, and Hayden Field in The Verge. He referenced a Jacob Coxon AMA, including Coxon saying the core disagreement with Anthropic leadership was about the inevitability of a race, that he was surprised people were ready to engage with existential risk, and that scaling laws could still plateau. Mowshowitz also cited Bilal Chughtai's statement after resigning from Google DeepMind, in which Chughtai said he believes AI has the potential to kill us all and that alignment is difficult and unsolved.

Mowshowitz wrote that the current cascade is insufficient and that underlying problems must be solved. He said overcoming political opposition will be necessary, naming Nvidia and a16z, which he said for now has the rhetorical allegiance of the President and is doing things like planting METR hit pieces in the New York Post. He announced a quickly thrown together conference, AGI.WTF, at Lighthaven September 22-23.