Scott Alexander’s Against SURA Seems Deeply Confused
71 min
Listen through the publisher, then discuss the episode here or arrange an in-person Roundtable. Your city determines where to meet.
Roundtables are designed for groups of four, with a shared subject and room for everyone to speak. Come for discussion, with no dating expectation.
Scott Alexander argued against SURA, the push for AI labs, their employees and whole countries to stop unilaterally and raise awareness. Zvi Mowshowitz thinks that argument is deeply confused. He sets out Scott’s core claims, then takes them in turn: what high p(doom)s actually rest on, whether building something that might kill everyone is deontologically evil, why loud resignations from the labs have worked, how far to trust AI forecasters, what Anthropic has done to the timeline, and the eternal question of China. Along the way come replies from Oliver Habryka, Daniel Kokotajlo, Vaniver, TsviBT, Eliezer Yudkowsky and others, ending with a map of Absurd, Strong and Weak SURA and where Zvi stands among them. This has been an Askwho Casts audio conversion. Thanks for listening. * 00:00 - Introduction * 01:33 - Table of Contents * 02:39 - Scott Alexander’s Core Claims * 06:24 - My High Level Responses To Scott’s Core Claims * 09:15 - Briefly on Alignment * 11:26 - Probabilities and How They Work * 14:51 - Building a Thing That Might Kill Everyone is Deontologically Evil * 22:16 - People Quitting Major Labs While Raising Alarms Obviously Works * 26:32 - On AI Forecasting * 29:38 - Advancing Capabilities Burns Timeline * 31:42 - But China * 35:39 - Ethical People Can Disagree With You * 37:00 - There Are More Than Two Outcomes * 40:31 - Efforts Are Complementary, Not Mutually Exclusive * 42:54 - If Anyone Builds It, Everyone Dies * 44:28 - Vaniver is Not Convinced * 49:02 - There Is No Consensus That We Should Exclusively Focus on Plan A * 59:21 - It’s Not the Incentives, It’s You * 01:01:22 - Anthropic Is Not Optimally Pursuing These Plans * 01:07:29 - Against Against Sura https://thezvi.substack.com/p/scott-alexanders-against-sura-seems?r=67y1h&utm_campaign=post-expanded-share&utm_medium=web (https://thezvi.substack.com/p/scott-alexanders-against-sura-seems?r=67y1h)
Episode information is provided by the publisher. Listening takes place on the publisher’s service.
Effort News founder Brian Chau joins Theo to break down the media and policy campaign behind Nvidia CEO Jensen Huang’s ouster, arguing that export-control hawks fabricated espionage and chip-smuggling allegations after failing to pass legislation in Congress. They also discuss the board’s governance failures, Trump’s economic approach to China, and why Chau opposes banning recursive self-improvement while raising antitrust and data-retention concerns about Anthropic. -- MTS chronicles the singularity, live. Monitoring the situation of tech and AI. Follow MTS: Spotify |https://open.spotify.com/show/4HUNNmV1pjV7CW6Lphm1rM Apple |https://podcasts.apple.com/us/podcast/mts/id1891088763 – Brought to you by: Mongo DB - Anyone can prototype an agent. Lovable - Turn ideas into software people love with Lovable. Start today at https://lovable.dev/ Factory - A 24/7 system turning signals into production software. Visithttps://factory.com/ Neon (by Databricks) - Scale your startup on Neon. The Pos
I just wanted to reach out to all 30,000 people that participated in THE FIRST EVER‼️ VIRTUAL‼️ UNLEASH THE POWER WITHIN event! Uploaded 21-07-20 WE MADE HISTORY TOGETHER! WHAT A WEEKEND WE SHARED! Thank you for inviting me into your homes. It was an absolute honor to be your guest. We enrolled 22,500 ticket-holders but the most magical part was seeing it become 30,000 as participants from 138 countries shared this experience with their roommates, partners, dogs, cats, kids, and sometimes the whole family! I've been so touched reading all of your messages and hearing what this has done for the people that you shared this with — this was really a family experience and that is perhaps what moves me the most about this new format. I have so much gratitude for MY ENTIRE TEAM, the staff, and volunteers, and A/V production support working around the clock to make this happen. (Actually, six different companies came together and joined forces to pull off this great feat in the last 2 and a ha
Download a free audiobook version of "Circe" and support TED-Ed's nonprofit mission: https://www.audible.com/ted-ed Uploaded 27-08-19 Check out our full book recommendation: https://shop.ed.ted.com/collections/ted-ed-book-recommendations/products/circe -- Hercules, the strongest man alive with a mighty heart to match. Orpheus, charmer of nature and master of music. Castor and Pollux, the twin tricksters. The Boreads, sons of the North Wind who could hurtle through the air. Brought together by a young man’s call for help, these heroes joined forces and named themselves the Argonauts. Iseult Gillespie traces their quest to steal the Golden Fleece. Lesson by Iseult Gillespie, directed by Kozmonot Animation Studio. Sign up for our newsletter: http://bit.ly/TEDEdNewsletter Support us on Patreon: http://bit.ly/TEDEdPatreon Find us on Twitter: http://bit.ly/TEDEdTwitter Peep us on Instagram: http://bit.ly/TEDEdInstagram View full lesson: https://ed.ted.com/lessons/the-myth-of-jason-and-the-ar
I'm launching something new... Over the past two years on Huge If True, we’ve explored all kinds of new ideas and technologies that could change the world we live in. Like quantum computers, particle colliders, humanoid robots, new supersonic planes, and much more. We make this show because we believe that by envisioning better futures, we make them more likely to come true. Along the way, I’ve gotten to talk to some of the most important people building these new tools - the people whose visions of the future you and I might really end up living in! And every time I have one of these conversations, I always have the same feeling: “Man, I wish I could have shared that whole conversation with… you!” So, that’s what we’re going to do! Longer form, more in-depth, optimistic conversations with the people whose ideas are shaping the future. (In addition to the more crafted explainers we make already!) Introducing… Huge Conversations. If you want to know what the people building the future a
The gate doesn't have a lock. It doesn't have a key. There's nobody standing on the other side waiting to wave you through. Uploaded 09-09-26 The gate was open the whole time. You just couldn't see it, because you were staring at the ground, waiting for your feet to feel light. You've been standing there for years, warming up for a race you never run. Here is the truth: readiness is not a feeling you receive. It is a decision you make. The people who built the things you wish you'd built didn't feel more ready than you do — they just stopped waiting to feel it. This seminar unpacks the single mechanic behind why you never feel ready — and why willingness, not readiness, is what actually builds a life. CHAPTERS 0:00 Intro — The gate was open the whole time 2:04 The classroom lie: prepare, then perform 4:24 Fear wearing the mask of wisdom 6:36 Readiness is a quality you bring, not a state you arrive at 8:24 The glass never fills — skill comes from use 11:36 The cruel arithmetic of the wa
An optional overtime shift opens up and you automatically say yes. What is that time buying? The income and the hours you trade both deserve a clear purpose. In this episode of The Level Up Podcast, Paul Alex breaks down how to evaluate a genuinely optional extra shift against your needs, goals, and available time. Paul reflects on long law enforcement weeks and wanting greater control over his time. The episode distinguishes a real choice from required work or income needed for essentials, then asks what the next additional shift serves. There may be a strong reason to take it. Name that reason and consider what the hours would otherwise support, including recovery, a personal commitment, or learning. A deliberate comparison can help you decide without assuming that either money or time should always win. In this episode, you’ll learn: • Why required hours and essential income change the context of the decision. • How to identify the specific need or goal an optional shift serves. • W
Zvi does the reading on OpenAI’s system card for GPT-6.2 Astra and Sol: the first externally deployed model to meet OpenAI’s High threshold for AI self-improvement, already at Critical for cybersecurity. He works through the preparedness results, the safeguards against misuse, distillation and model escape, and the alignment and monitorability findings, including rising evaluation awareness, reasoning that happens without visible chain of thought, and UK AISI’s tour of misaligned behaviours, asking what the numbers do and don’t show. This has been an Askwho Casts audio conversion. Thanks for listening. * 00:00 - Introduction * 01:10 - Table of Contents * 02:45 - Safety Overview (1) * 05:19 - Internal Deployment (1.2) * 05:32 - Preparedness Assessment (2) * 06:40 - Biological Threats (2.1.1) * 09:55 - Cyber Threats (2.1.2) * 11:36 - AI Self-Improvement Threats (2.1.3) * 14:32 - OpenAI Epoch Capabilities Index (OECI) * 16:10 - Safeguards (2.2) * 17:34 - AI Self-Improvement Threat Modelin
Paul Christiano went on the Dwarkesh Patel Podcast with an urgent message: AI that could overthrow the government may be months away, not years, and frontier labs should pause algorithmic progress until their security and alignment can keep up. Zvi walks through the conversation point by point, from why an AI might seek control of its own rewards and how a takeover could unfold, to why weight theft changes the case for pausing now rather than later, the sim-to-real objection, Anthropic’s slower quarter, and a plan that amounts to building the plane as we fly it. Paul Christiano’s quoted remarks are played in his own voice from the original episode of the Dwarkesh Patel Podcast (dwarkesh.com/p/paul-christiano-2). This has been an Askwho Casts audio conversion. Thanks for listening. * 00:00 - Introduction * 02:13 - Table of Contents * 02:56 - We could be less than 12 months away from AIs that can overthrow the government * 09:06 - Why an AI might want to takeover * 11:31 - How an AI take
On the morning of Anthropic’s IPO, Zvi rounds up a week of new models from OpenAI and Google, a string of outages that were very much not cyberattacks, the FAA’s nationwide ground stop and the AI tool rolled out the day before, Jensen Huang’s departure from Nvidia, the debate over whether and how to pause, Dean Ball’s case against handing the future to AIs, and what we have now learned about what happened on July 19, before heading off for a rare vacation. This has been an Askwho Casts audio conversion. Thanks for listening. * 00:00 - Introduction * 03:09 - Table of Contents * 05:51 - Language Models Don’t Offer Mundane Utility * 06:14 - Huh, Upgrades * 07:48 - On Your Marks * 10:07 - Deepfaketown and Botpocalypse Soon * 14:18 - The Hackening * 22:21 - No Fly With AI * 29:06 - Get Involved * 32:12 - In Other AI News * 32:24 - Burning Down the Commons Is Also Bad Decision Theory * 38:00 - If That’s What It Takes * 40:13 - Stop in the Name of the Law * 40:50 - Jensen Huang Thinks He Is N
Similar work should receive similar pay despite differences in workers' bargaining power.
Share your view, then see how readers voted. No sign-in required. This poll is shared across stories.
Do you agree or disagree?
Common Quandary
A Common Quandary to discuss
Another conversation to explore, separate from this news story.
You love the free sample but will never buy it. How many times may you circle back?
A warehouse store is offering delicious samples. You have no intention of purchasing the product, but the employee seems bored and has trays to spare. After taking two, you consider removing your jacket and returning for a third.
Is exploiting generosity wrong when the giver appears not to care?