Cassie: A tiny group of people is trying to render us obsolete, but we can stop them. I'm Cassie Pritchard, a labor organizer.

Garrison: And I'm Garrison Lovely, a journalist and the author of Obsolete, and this is Organize Against the Machine. All right,

Cassie: So, Garrison, why did we decide to start this podcast?

Garrison: I was your reply guy on Twitter. I saw that you had takes that were interesting and unconventional for a leftist. And after a few attempts, I got your attention, slid into your DMs, The conceit was like, I wrote this book. It's called Obsolete. Cassie's copy is coming in the mail soon. Writing a book in twenty twenty six. It's kind of a strange move. Writing a book about AI in 2026, even riskier. The topic moves so fast, people are not really reading books as much we kind of need to get a lot of attention on this issue to build the movement that we think is necessary And it just seemed like a good way to take the ideas from the book and translate them into real-world action. And Cassie In addition to having really sharp takes, I was excited to work with you because you're an organizer. You have a lot of experience on the other piece of the puzzle. I've spent the last eight years doing journalism and other jobs and thinking about like the problem, how to articulate it and get people to understand what's going on, but much less on how do you get them to actually take actions in the world.

Cassie: That felt to me what was really compelling about the idea of us having a podcast. Other than just like once we got talking, we clearly had a lot of rapport and a lot of similar ideas about this issue and a lot of similar frustrations. But that there is this particular gap, right? That on the one hand, there are the people with the technical familiarity who are really worried about all these different kinds of AI risk. It's not all just quote unquote X risk, the idea that the machines will actually cause human extinction or take over the world or something like that. Very sci-fi, very Terminator. But there's a lot of different kinds of risk that we face from this technology. Risks to our democracy, risks to the integrity of the education system, risks to the concentration of corporate power and the disempowering of labor. And there's a group of people who really understand the technical side of this, the dynamics of this very insular and kind of weird community that are at the heart of developing AI. And then there's people with political and organizing experience. And like the two do not generally meet. Like this seems to be the big problem is the people who know how to do politics, the people on the left who are interested in mass mobilizations to advance the cause of working people and protect people from harm from trillion dollar tech oligopolies, are remarkably uninterested in these particular risks, and the people who are interested are I don't want to be overly harsh, but have almost no theory of organizing or politics, and almost kind of a disdain for it as a rule. A real like visceral distaste for getting down in the muck and doing the work of persuasion and coalition building. So it feels like this is a really necessary kind of unity between these two halves that neither one works to address the issue on its own, and we need to in fact start creating some kind of communication between folks who you would expect would be interested in organizing on these issues and the folks who have the detailed technical knowledge and historical background on what's going on and why it is genuinely alarm.

Garrison: Yeah. I wrote this Jacobin cover story about AI existential risk and the three-sided debate around that question, which we'll get into later in the show. The conclusion of that article was something like the people who focus on AI's immediate harms and the people who focus on its longer term existential risks are kind of f on the same side, where they both want to regulate these companies and have them prioritize things beyond just profit and growth. But as I was writing the book, You know, it has elements of that thesis, but it became clear to me that like these two groups actually desperately needed one another. And by these two groups I mean like the AI safety doomers, which is a pejorative that they often don't identify with, and then the kind of critics of the technology and people more broadly on the left who get organizing and dispositionally are are primed to approach this issue in the way that we think is necessary, but only if you take the issue seriously. It's kind of this great tragedy that people who have, I think, the best understanding of the technology and where it's going and the best track record of predicting it are dispositionally so anti movement building and politics and democratic processes to some extent. And then the people who build movements and think about power every single day, every single second, basically, have, for a bunch of reasons, sort of refuse to look at the technology as a serious potential threat, not just like it doesn't work sometimes and that causes problems. But like, no, what if it works really well at disempowering laborers, at concentrating power in governments, at surveilling people, at killing people autonomously, at hacking, at like all of these things that we're starting to see glimpses of? I think the trend lines have been pretty clear, which is that the technology has been improving at a large range of activities. And the people who say it will stop have been disproven by reality. over and over again. And I think they're just like these totally different worlds of beliefs where you can see people saying like AI is a bubble, it's going to pop any minute now, and the tech technology hasn't gotten any better or it's hit a wall. At the same time as it's like coming up with a bunch of new math breakthroughs that would potentially some of them be worthy of the highest awards in mathematics and autonomously hacking into multiple other companies. and escaping the control of these companies that are supposed to be the experts in controlling and aligning these systems.

Cassie: Although of course a lot of people still on the left are skeptical that those models did escape control, that they did autonomously engage in hacking, there's still a real reluctance to buy into the idea that the models can even be this capable, which goes to the heart of like why we have this difficulty, right? There's this strong incentive or disincentive to like take the technology seriously because you don't wanna look foolish, right? You don't want to buy into the marketing. But Even if it were intentional, even if OpenAI had like prompted their models to hack and commit felonies on purpose, that would also be bad and call for a regulatory oversight and legal framework. It's definitely quite frustrating. But I think we should tell the story of what is happening. And I think to start that story, we have to start with who these people are. What is their mission? What are they trying to accomplish? And How did this rush to AGI come about in the first place to get to this point where a trillion dollars of private investment is being poured into this just in the United States alone?

Garrison: The story of this industry is a story of true believers. The people at the vanguard of the AI companies are people who believed in artificial general intelligence, which they think of as like a mind or a machine that can outclass humans at basically all cognitive tasks. They believed in that idea long before it was taken seriously by other people. And it's still not taken seriously by many people, but it's kind of hard to dispute the fact that the people who believed in it were the best at building increasingly capable and autonomous AI systems. I don't like the term AGI. I think that we should understand these machines as machines that make labor itself. And in the book, I call that the obsoleting machine. The no context required version is universal labor replacing machines. And when you think of it that way, it's like, we need to race China. To build universal labor replacing machines, like starts to sound a bit sillier. People will say we need to raise China to build AGI and not really entertain the implications of that for workers, for democracy, for power concentration, for wealth inequality. But like calling it that, OpenAI defines AGI as a highly autonomous system that outperforms humans at most economically valuable work. That's like written into their charter. And if you look at the statements of these CEOs, they'll often talk about it as like an economically relevant activity or like labor replacing machine, Dario Amade, the CEO of Anthropic. He talks about how half of all white collar entry level workers could be put out of the job by AI in three to five years or something. And a lot of people are like, well, he's just saying that because he's talking his book, he's trying to hype up his technology. It does get a bunch of attention on how capable and autonomous these systems are. Which is helpful for investment in some sense, but it also creates really strong pressures to regulate and potentially stop the industry from doing this because putting lots of people out of work is not a popular thing to do. And OpenAI has these economists that they've hired. And there's been reports that those economists have been told to like downplay how much labor automation is actually happening. I think like as shit's getting more real, these companies are realizing, like, this is actually. bad for us that people think that they're going to be put out of work and or killed by the technology that we're building. And there was reporting recently that Dario Amade was getting pushback from investors or prospective investors in Anthropic ahead of their IPO. And the investors are like, hey, stop scaring people. This is bad for business to be talking about extinction. And like one of the biggest, most popular takes on the left about AI existential risk is that it's this like marketing scheme to talk about this. And I just think the evidence for that is just not very good because again, it's like not good for your business to be convincing policymakers that if you continue on your present course, everybody might die and also before that be put out of work.

Cassie: Right. It does feel like there's a real culture that was built that's very inward focused where these people were talking to each other amongst themselves for a very long time. They developed a certain kind of discourse, certain kind of norms, ideas, jargon, shibboleths that were all built around this less wrong, rationalist, effective altruist. It's a very niche Community that, on some level, as I understand it, they pride themselves on having niche heterodox concerns. They don't care about the things other people care about, right? They're different. They're more perceptive, they're more intelligent. And this is a core thesis of how they understand themselves and how they understand their community. And when you build a small, inwardly focused community that defines itself in negation to the majority of everyone else and like what normal people believe and what normal culture is and what normal political incentives are. You like prove your identity in that culture by constantly articulating these things that would be alarming or nonsensical or otherwise off-putting to the mainstream. It can get very insular and very weird, and then now they're coming into contact with everyone else again. And they're remembering, right, yeah, actually These norms and philosophies that we developed are actually quite upsetting to other people. And I think we're starting to see that political backlash, and we're starting to see that also with investors who are maybe outside of the valley. I do think this gets underrated as an explanation for why they talk so much about existential risk, why they talk so much about recursive super intelligence where the machines Build the next even smarter machines on and on in an infinite loop of constant improvement that ultimately leads to some runaway process that disempowers all of humanity. Even in their like utopian good scenarios, that seems to be something that they believe will happen. Ideology and community norms really get underrated in the sort of theory of mind that people have who are skeptical of AI, of what drives these people and how they operate. And I think part of understanding how this is a community of true believers, like you're talking about, is to realize like how far back this goes. They were saying these things decades before there was any money in it. Decades before there was any plausible technology. Like the Transformer architecture was decades away, and people were going to conferences with a bunch of other niche weirdos talking about this potential scenario that they foresaw. And very strangely, their reaction to identifying this potential scenario of recursively self improving machines that disempower or destroy humanity was to go out and figure out how to build them.

Garrison: That's a weird approach to this.

Cassie: It's like hard to even analogize. Like, I've tried a few different analogies. Like, it's as if early scientists in like the seventeen seventies correctly figured out that climate change would happen if you burned a lot of fossil fuel and they went out and started a bunch of coal mining immediately. Like we have to be the ones to steer fossil fuel in the right direction for the future because otherwise it'll wipe out humanity. We better get a head start. Like that's a weird approach. Or given how much of the existential risk fear or the fear of robots otherwise becoming, you know, misaligned reminds me of kind of a classic American paranoia of slave revolt, right? You know, it's like if a bunch of Portuguese guys in like 1460 were like, you know what would be really bad is if we lived in a society where there were a ton of people who we enslaved and then they got really upset about being slaves and they got out of our control and killed us. You know what we should do? We should go find a bunch of people to enslave and start new countries where we're ten percent of the population and the slaves are ninety percent of the population, so that we can make sure that it doesn't happen. Like it's just again, it's like difficult to draw a clean analogy because it's so fundamentally insane and unlike what anyone has ever done, I think, on some level, that

Garrison: Nuclear weapons is kind of the analogy here, right?

Cassie: Yeah, nuclear weapons is the closest.

Garrison: And the climate one, they would say, like, well, like climate change is just overwhelmingly bad. And so harnessing it doesn't make sense. Whereas, like, if you build the superintelligence and give it the right goals, it could create a utopia on Earth for everybody. That's the kind of dual use nature of the technology that makes the analogies not quite work. Even though I think it is worth calling out that like when I learned about the concept of AGI and existential risk, I was like, we should just not build that then, right? And I think most people have that reaction. But then this small sliver of people, they learn about it and they go, like, I should be the one to build it. And that means like there's this intense selection effect. These people were so self-assured in their own righteousness or intelligence or both that they could handle this gargantuan responsibility. And now these people, because they've been very successful at building these incredibly valuable AI systems. They're seated with the heads of the G7 countries at the same table. And they are the ones dictating how this technology plays out along with the Trump administration. That's horrifying. These people who are like by selection chosen to build this technology that they recognize to be existentially dangerous without the consent of basically anybody who would be affected by it, which is everybody. And they are the ones. Because they did this, who get to decide how it plays out. It would be way better if we had five random people in charge of all these companies or something. Like we did sortition, because they wouldn't be nearly as power seeking and megalomaniacal and obsessed with like the upside or trusting themselves. And it's like a crazy situation. And then it's even crazier that the left wing broadly doesn't see it this way because they're like, well, it doesn't actually do anything. So it's not a big deal that. It's only being controlled by this handful of people. Or just sleepwalking into this horrible dystopia or annihilation, if you take seriously these arguments.

Cassie: You have a lot more expertise in this than I do of the particular and peculiar history of this community and these firms and these technologies and the sort of leading luminaries of AI. So how long ago did people first come up with this scenario of artificial general intelligence, of recursive self-improvement? When did they identify this future case that they became convinced? was an almost inevitable reality that they would have to be personally responsible for adverting. Like when did that happen and who emerged as kind of the leading players?

Garrison: Samuel Butler is like the first guy to kind of put down this idea of losing control of the planet to machines back in the 1860s. And his take was like all humans should get together and fight against this. And then in Dune by Frank Herbert, you have the Butlerian jihad, where they like fight AI and the machines and make sure that there's no machines that think like people. Doom

Cassie: And that was written in the sixties, right?

Garrison: is in the 60s, yeah. And then you have people like Alan Turing. And a contemporary of his, IJ Good and John von Neumann, talking about the concept of human level AI and recursive self-improvement, although they didn't use those terms. And IJ Good is one of the first to really crisply talk about this, where he's like posit that an ultra intelligent machine exists and it can make new machines better than humans can do everything better. This is like the last invention man need ever make, but we should be very careful about like what goals we give the machine. And that's in like the 60s. This is an idea that people come to or rediscover continuously. And then in the nineties, there's this guy, Eliezer Gidkowski, who, among other people, really focused on artificial intelligence as this incredibly important technology. And he initially wanted to build it and thought it would like save the world and end war and poverty and do all these amazing things. And then he started an organization to advocate for this and got Shane Legg, who's one of the co-founders of Deep Mind, on to the idea of artificial general intelligence. And then DeepMind in 2010 is the first company started to build AGI that is still with us. There were other companies further back, but DeepMind is like the earliest one that's still around. And that was Demis Asabis, Mustafa Soliman, and Shane Legg. And Demis seems like his motivation is like solving science. The mission statement of Deep Mind was. Step one, solve intelligence.

Cassie: and then step two, use it to solve everything else.

Garrison: And so it's this like ultimate technical thinker's fantasy where you just come up with something that's like so smart that any problem you throw at it, it will just come up with the best possible solution. It's very appealing, right? Like you don't have to solve Israel-Palestine yourself. You don't have to figure out climate change. You can just like come up with a machine, ask the machine. And the thinking was also like, we have to do this first because we'll do it safely. We're gonna make sure nobody else is building it. We'll spend our time figuring out how to get the machines to do what we want, and there won't be any race. And then, of course, that inspires OpenAI to exist. And one of the big motivations there was Elon Musk and Larry Page, the Google co-founder, they had this big falling out because Larry Page famously is fine with human extinction, reportedly, you know, but there's like many different stories independently. Talking about this in the Jacobin article I started with, Google co-founder Larry Page thinks superintelligent AI is, quote, just the next step in evolution. In fact, Paige, who's worth about 120 billion, has reportedly argued that efforts to prevent AI-driven extinction and protect human consciousness are speciesis and sentimental nonsense. That's one of the richest people in the world. It's pretty concerning. And Elon like freaking out. He tries to buy DeepMind, doesn't have enough money, not yet a trillionaire. DeepMind goes to Google. And then Sam Altman smells an opportunity, pitches Elon on starting like the nonprofit open source counterweight to Google, because otherwise Demis could be the AGI dictator. There's emails where they talk about it this way. And so they clearly saw this technology as incredibly powerful, to the point of concentrating all the power to control the world in one person, which has never happened. We've never even been that close to it. And so they start open AI then Some people at OpenAI have the same kind of, we don't trust Sam Altman. We should try and get him pushed out. Doesn't work. They go out to start anthropic. Caveat that anthropic formerly contests this narrative, but there's reporting in the New York Times that Dario Amade and others went to the board at OpenAI, tried to push Sam out, and failed and then started Anthropic. And this has happened with a number of other companies and the entire industry. It's just this fractal pattern. of distrust of the people currently building it and the belief that I will do it better. And so you go off and then it creates this multi headed hydra of companies that are all racing to build the first human level AI which will become superhuman. And then question mark, question mark, question mark, profit, extinction, disempowerment, who knows?

Cassie: It's honestly so ironic because again, even though there's this extreme disconnect between political organized left and AI safety people, rationalist effective altruist folks that work on AI, the cultural experience of the AI safety people is so much like the experience of leftists. We're in an institution together, we're in some kind of party formation or whatever, and the guy in charge has the philosophy all wrong, and we need to schism and we need to break off and form our own sect, and then our own sect is gonna break off from that. And it's just like believing that in like some sort of teleological destiny and we're going to achieve socialism and we have to be prepared and be the exact right people for it. And so when there's this internal disagreement we have to schism and split off into a little new microsect that's even purer and even cleaner and like has all the right ideas and none of the wrong ones. It's just like the left at its sort of nadir in the West of subcultural and countercultural tendencies instead of being a mass politics. And luckily I think Today on the left, we're moving away from this and sort of rediscovering mass politics and democracy and persuasion and what it means to really build a fighting coalition that can win. But it is funny to see this mirror image because I think these groups think of themselves as completely different and unrelatable to the other, right? And are mutually disdainful. It seems very clear that the AI safety kind of crowd. is extremely disdainful of the left as like insufficiently rational and sentimental and emotional and whatever. And obviously the left is very disdainful of these people as masturbatory and grifters who are just trying to pump up this technology. But they have a certain mirror image of each other in some ways, which on the one hand is unfortunate because the mirrored features that are similar drive you towards being ineffective. But that maybe gives me a little bit of hope that there might actually be more of a substrate for cooperation ultimately than people actually realize. Partially because there is real ideological motivation here. There's a kind of vulgar materialism that suggests or pretends that the very rich and the very powerful are people who operate without beliefs. They are sort of paperclip machines, to use the AI analogy.

Garrison: Just more profit, more ROI.

Cassie: They're just like mindlessly converting everything they can into more profit. That they have a single goal that they just absolutely maximize and optimize for and there's nothing more to them than that. And they will like consume anything in their path. And there's obviously some truth to this, but we've also seen how these people are ideological. I mean, I don't know how you can look at Elon Musk, for example, and the way his brain has been completely fried by his own platform and not think that these people are vulnerable to ideological Commitments, right? Clearly they are. But also, I think it's so interesting and so underreported, and people on the left are so unfamiliar with how deep this goes back. I mean, the ideas themselves going back literally hundreds of years, but even the players in the game today, Yadkowski talking about this in the 90s, like firms were coming into existence quite a long time ago, and then DeepMind gets founded in the 2010s. And it doesn't appear at that time to be something that's gonna be imminently lucrative. And yet these people are extremely convinced. And Sam Altman might be someone who's just genuinely a pure grifter who just wants to get rich. I could actually believe that.

Garrison: I mean, like, he doesn't have equity in open AI. And he does have equity in other companies that have relationships with open AI. So he's gotten quite a bit richer from Open AI's success. But it's still the case that like he could be worth fifty billion, sixty billion, hundred billion dollars, and he's worth like single-digits billions, which is obviously an insane amount of money. But somebody told me who worked at OpenAI that Sam chose to prioritize power over money. And the evidence was like, if you don't have equity, there are certain decisions that you can vote on as a board member that other board members who do have equity have to recuse themselves from, which is like a pretty straightforward I'd rather have one more vote on these decisions at the expense of tens of billions of dollars potentially. And then there's also the narrative kind of thing of like, well, I'm just doing this because I I love it, I think is what he told Congress. But Yadkowski, I got to interview him for the Jacobin piece. Back in 2023. And he told me that people, at least the ones who started the industry, really were not doing it for money. They were doing it to be in the room where it happens. And by that he means AGI will be created. It'll be created by a company. And the people at that company are going to potentially decide the fate of not just the world, but like the light cone, the observable, affectable universe that humans and the things that we create. can change. And that's like a pretty crazy belief, but it follows from really taking seriously this idea of creating a superintelligence that can improve itself and that can be scaled up far beyond human population levels. And I have this chapter called Why Build the Doom Machine, which gets into Yadkowski and this question of like why are they doing it if it's so dangerous? The whole thesis is like Yadkowski's fingerprints are all over the industry and the beliefs motivating the people who created it. And it's like, the room where it happens? Well, here's Yadkowski saying in twenty ten, I think that the future of humanity boils down to more or less nine people and a brain in a box in a basement. I think that saving human species eventually comes down to metaphorically speaking, nine people and a brain in a box in a basement. And so if you actually think that, you will do anything to be in the room where it happens. And that's hard to model.

Cassie: There's a certain kind of person who would do anything. They want to be one of nine people deciding the fate of humanity or the fate of the galaxy or whatever they've convinced themselves of, as absurd as it may or may not be. I think conversely there are a lot of people who would be entirely uninterested in that. I think most people actually would probably be pretty uninterested in I don't want that responsibility, right?

Garrison: They like shit their pants. They'd be like, I don't want That's terrifying. Yeah, just start sweating bullets right away.

Cassie: And then there's like a whole other group of people, and this is the group I would count myself among, who if there really were a decision or a set of decisions at that level of stakes, I wouldn't want to be one of nine people making that decision. I would want to set up some kind of meaningful democratic structure in which everyone was empowered to have some agency in how this decision was made, because that feels like the right thing to do. Those are my values that People deserve to have agency in their lives and democracy, including economic democracy, which includes therefore the development and deployment of new technologies, an essential good and something that allows people to flourish and feel that they have some measure of control over their lives. So I think it is worth teasing apart that this is like a particular kind of person with a particular kind of personality that looks at this problem and says, Strap those

Garrison: Icarus as like a role model. It's like, we just needed better wings, obviously. It's a technical problem.

Cassie: Yeah, he made his wings out of wax. That was stupid. I would I would build much better wings. He didn't build any kind of thermoregulatory mechanism. Exactly. They approach it like engineers, right? Like, you should just like build better wings, as opposed to understanding the basic premise of hubris as a fatal flaw. And I think the reason that that's worth teasing out is to impress upon people that yes, these people are serious. They really do mean it. They really do believe it. And the thing that they desire ultimately is control. They desire power on a world historic scale. Some people say, well, they're not gonna attain that level of power because the technology fundamentally isn't capable enough and it never will be. To which I would say they've done a remarkably good job of accumulating power so far.

Garrison: They were not sitting at the G seven table a year ago.

Cassie: Right, exactly. This very small niche community with very strange heterodox concerns that most people find alienating or fantastical or infantile have put themselves in a position to control trillion dollar corporations and be sitting down with world leaders talking about a regulatory framework in this country or even internationally about the technology that they're building. So they actually seem rather good at accumulating control. In addition to Having an insatiable appetite for it. That alone feels to me like there has to be some democratic counterforce because these people have demonstrated an appetite and a competence for accumulating power. You sort of envision a tripartite debate right now around AI. That there's warriors, there's boosters, and there's critics, and there's something missing. from all of these responses to this vast accumulation of power that's happening very rapidly. And I think we should get into defining what are these three parts of the debate, what characterizes each of them, and what's missing.

Garrison: So the warriors are people who are like worried about AI. If it gets smarter than people, we could lose control of it. We could lose control to it and potentially go extinct or could concentrate enormous amounts of power in a small number of hands. And they take the technology very, very seriously. And they'll often say, Well, I'm not worried so much about today's AI. And they might even be pro today's AI on net, think it's like does more good than harm. But Future AI, often artificial general intelligence or superintelligence being the thing they're worried about, that could just become more powerful than all the people put together. And then you've got the boosters who want to speed everything up. They're worried about overregulating the technology. They think it's obviously going to be great to build as capable and autonomous of AI as possible. You'll cure all these diseases and you'll get rid of poverty and like Anyone standing in in the way of the technology is doing a bad thing. And also it's futile because it's inevitable that this technology will be built anyway.

Cassie: And so both of those groups, broadly is it fair to say, they're within the AI world to some extent. Like that's where the core of the Warrior group and the booster group, they're either in the industry or they're in these foundations and groups that are associated with it, or they're at least in the cultural milieu of tech and San Francisco and rationalism and effective altruism, right? Like those two sides belong to this niche world.

Garrison: Yes, caveat like, you know, I think the boosters a lot of them are concentrated in the San Francisco Bay Area. I think the boosters don't have a ton in common with the EAs or the rationalists. They'll be aware of the same kind of memes and their Twitter timelines, like there might be a lot of overlap between them. But I think in a weird way, the Warriors are way more optimistic about the potential capabilities of the technology. You'll see boosters tweet things like AGI will revolutionize apps, like iPhone apps. And it's like, I mean, yeah, probably would, but that seems like not the place to focus. Like the Warriors are the ones who are talking about Dyson spheres that are like built to harvest all the energy produced by the sun, not like a hundred years from now or a thousand years from now, but like maybe next decade or something. They really, really believe that AI could become incredibly capable on relatively short timelines. And I think if you really believe that the technology could become vastly superhuman across the board, it's kind of hard not to think it's a risk of some sort. I think a lot of this just comes down to like, do you believe in the capabilities getting to some level in the future or not? The last group is the critics. The distinguishing thing here is just being quite skeptical of AI's capabilities today and its capabilities in the future. And This is a group that wants to focus on the harms the technology is having now, tends to focus on harms downstream of the technology failing in some way, like algorithmic bias or machine vision that can't recognize black people as much as white people. And so people taking tests getting like monitored by their universities don't get recognized as not cheating or something. And like that's bad. We should stop that. This set is more concentrated in academia, less so in in the Bay Area. And then you have people like Ed Zitron, who's like a prominent commentator, and he's like, Really big on AI as a bubble. The critics just basically deny that the technology is that capable at all and definitely deny that it will like become superhuman or generally intelligent in the future. These camps are often at odds with each other and they'll kind of line up in these interesting ways where the critics and the safety people, the warriors, will sometimes agree on like some broad regulation or something. But then the critics and the boosters might agree on open weight models where anyone can download and modify. The AI models, they might agree on that because the boosters like it because it's a software kind of tech culture thing. And the critics, some of them work with open weight software and it like distributes the power to more people. And then the warriors are kind of like, I don't know, open weight models, they're fine right now, but if they get really good at helping people hack or make bioweapons and you release the model, you can't take it back and the safety guard rails won't work because you can just remove them if the weights are public. And so it's Complex and you'll have people like Nancy Pelosi and Mark Andreessen on the same side of a policy issue against Elon Musk and the SEIU and the National Organization for Women. And that's confusing. That's because there's this three-sided debate, and most debates don't have three sides, or at least the most prominent ones that animate political discourse.

Cassie: It feels to me like a sign that the debate is malleable and immature to some extent. When you're getting a debate that cuts in weird ways across the normal coalitions and the normal worldviews that otherwise bind people's values and beliefs together into kind of coherent bundles, that to me suggests that people have not really figured out what they actually believe about this issue. Whatever this issue is that's cutting in these strange ways. Once people have stronger feelings about it, usually the battle lines become a little more clear, I think. Even if it is kind of a supermajority coalition that cuts in some ways across political party identification, for example, you still don't get this like rotation from issue to issue, sub-issue to sub-issue. In a world where there's a mature political discourse around AI and the risks that it presents, I would expect there to be a much more stable community on each side that would self sort on each new issue as it came up because they belong to some kind of broader coalition. You know what I mean?

Garrison: Camps do line up with political persuasions. The warriors are kind of aligned with the effective altruists, rationalists, AI safety. That's not a political persuasion, but it's like a distinct set of subcultures that have a lot of things in common. And then the boosters are like libertarian tech founder types. They're fairly right leaning and like very pro free markets. And then the critics line up with the left broadly and like the social justice left in a lot of cases. Most people just don't have any position. They're not. familiar with these camps, I think in practice, they just take the long-term risks seriously, the short-term risks seriously. The short-term risks are what people focus on the most in at least these studies that they've done. A lot has changed in the last few weeks, even. So who knows what it would be now. And there's a study where they ask people, are you more worried about these immediate harms from AI or like these existential risks, long-term risks from it? And they come in more worried about the former, the immediate harms. And then if you show them a bunch of headlines about existential risk or like Nobel laureates saying this is an extinction risk. They become more concerned about the existential risk, but they don't become less concerned about the immediate harms. And so it's like not so clear that you actually have to pick between trying to solve the algorithmic bias question and also the like, can we keep control of super intelligent machines question? Obviously there's finite political capital in some sense or like finite like attention, but I think this is often presented as this very zero sum thing. The Jacobin article, one of the big upshots was like We don't have to pick between these two things and we should see ourselves broadly being in the bucket of like pro humanity and organizing to make sure this technology benefits humanity, not just the capitalists who are currently financing it and controlling it.

Cassie: Yeah, that belief in trade offs is very strong and it's very interesting to me. I've definitely encountered this quite a bit, where I went on Twitter and said something to the effect of what's the harm in l letting the people who are really concerned about existential risk express that politically? If you are skeptical of that, but otherwise opposed to AI and the sort of people behind it and big tech generally, why wouldn't you Militate around this extremely emotionally gripping, alarming thing that does appear to be motivating a lot of people to feel extremely negatively towards this technology and the people building it and the companies that are currently experiencing massive valuations because of it. The response I got was that people were quite convinced that somehow if we not even took existential risk seriously ourselves, but even just kind of like flattered the people who think it's a problem by giving it any kind of legitimacy as a political interest, that we would lose somehow, that it would blunt our ability to actually oppose AI. There was a real clear sense that this is a zero-sum interaction. Either you deny that the technology has any capabilities or will develop capabilities in the future, or you give in. To the technology, the firms, and the boosters and the warriors, who I think, if you're on the critic's side, are often just kind of like melded together into a single entity. The Warriors and the Boosters are both just people who believe the technology can do stuff, that it works at all ever. And then there's everyone else. That really is understood as the main fault line, which is interesting to me, but I think we're gonna need more of that kind of empirical evidence that shows that in fact you can be worried about the sci-fi sounding, futuristic, very alarming tail risks, and you can still oppose the technology or want to regulate the industry for very prosaic, immediate, provable reasons like misinformation, algorithmic bias, AI psychosis, people harming themselves or others as a consequence of delusions that have been reinforced by AI, what it's doing to the education system. These are all immediate near term things that we can take seriously. And they're not lessened in any way by entertaining what might be longer or medium term risks.

Garrison: One of the things I heard when I interviewed some people in the critic camp was this AI existential risk is an attempt to take all the air out of the room. If you tell a policymaker, hey, this technology is going to get so good that it could literally kill everybody, you, your family, everybody you've ever known and loved, and all humans forever, then you have people being like, this technology is also biased against minorities. If they take both of those things seriously, one is like kind of the biggest deal ever, even if the other problem is also worthy of working on it and it's important to solve it. I get this. It's kind of a scary thing to internalize existential risk as a concept. And it can lead to fanaticism and can lead to prioritizing that issue over all others. One of the arguments for working on X risk or taking it seriously is the like asymmetry of being wrong, where if you overinvest, work really hard on addressing X risk, and then it turns out for whatever reason, the capabilities just never get there. Or somehow the AIs just do exactly what we want, and that doesn't create all kinds of other problems, which pretty unlikely. The tech just actually hits a wall, doesn't get any better. We can't figure it out for decades. And it's like, wow, that was kind of silly, I guess, to invest in that. I don't want to downplay. You could potentially have invested a lot of money and political capital and effort in preventing something that ends up being not such a problem or not a problem at all. But the other side is like, say you don't invest in it, and say the technology doesn't hit a wall, it continues to improve. And It's actually really hard to get it to do what we want, but we can't tell. And the companies just keep making more and more capable and autonomous AI systems. They automate more and more labor. They automate their own research and development. And then they produce like these systems that take on a larger and larger share of total responsibility and labor. And then at some point down the line, they just are doing most of the stuff. And it turns out like they actually weren't really working in our interests and just like we're pretending to, and suddenly, like, you know, they do the worker uprising, but they like just have more labor power than we do, and we're just not in control of the planet anymore and potentially all dead. And that asymmetry of outcomes, you just don't want to be wrong in that direction. This idea eats everything, right? I

Cassie: I feel like this is where there's another weird kind of mirror image between the AI safety crowd and the left. Because the people building this will talk about their you know, their P doom, their probability of doom, which I mean there's a whole separate tangent about casting your beliefs in like the language of mathematics as a way of establishing like a higher degree of rigor than actually exists. But people building this technology or or rationalist adjacent community who w wanna talk about the risk of this technology, try to put a number on the probability that the technology will result in human extinction or some other equally terrible scenario, human subjugation or whatever. I just feel like there's a weird lack of humility in whatever number you put on it, even if you're putting some amount of uncertainty on your prediction. If you say, you know, my P doom is like five percent. So therefore I think we should build the technology, blah, blah, blah. Or there's people who put it much higher and still think it's worth pursuing the technology.

Garrison: Which is insane by the way. If

Cassie: It's it's completely insane.

Garrison: it's like point one percent you should not build it, obviously.

Cassie: But then there is a different kind of lack of humility on the left, where people essentially are saying, My P doom is zero, and I'm perfectly certain in it. By all means, keep flipping the coin because I know it's gonna come up heads every single time. I think one of those is more legible to us as a kind of epistemological arrogance, which are the people who are like, there's a 10% chance it's gonna kill everybody, but I can build it. It'll be fine. That's very easy to read as hubris. But I still think there is a kind of hubris or at least epistemic. arrogance in feeling very confident that you know the probability of these major tail risks is zero. It doesn't actually feel that different to me when I look at it. I feel like there has to be a lot of uncertainty for everyone involved. From the critics side, you hear people characterize the technology in all these kind of pejorative ways. It's just a stochastic parrot, it's just fancy autocomplete, it's just a chatbot, it's just this, it's just that. We can't generate anything novel, can only just regurgitate what's in its training data, and it'll always hallucinate and confabulate. Whatever. But we don't even have a reason to believe that necessarily this particular architecture will be the way AIs are built 10 years from now. It might not be LLMs and the transformer architecture. It could be something completely different. This technology is itself recently invented. There's no reason to believe there won't be further inventions that could fundamentally change the capabilities, even if the current norm for building AI models does hit a wall, it seems entirely plausible that we might then switch onto a different technological track that doesn't share some of these problems and can advance further in new and other and unanticipated ways. So In light of having a little bit of humility about not knowing where this is gonna go ultimately, it seems incredibly prudent to me that we would want some kind of democratic regulatory infrastructure. This is the other thing. With the idea of it taking all the air out of the room, X risk, I'm a big believer that winning is the best strategy, you know? Like if you if you win the debate on X risk and successfully pass legislation that's capable of regulating the AI companies and the technology they produce, it seems to me that it would actually be a lot easier to then produce further regulations that are targeted at much smaller issues like algorithmic bias, right? You've already done the big thing. You've already established whatever legislative regime or regulatory oversight bodies or agencies that you would need to control this technology, that's going to make it easier to control the technology in whatever ways are important to you, as opposed to having nothing. It's the one shot theory. You get one shot, so you gotta make sure it's the right thing. I don't really believe in that. I th I I think actually that any successful hit you score weakens your opposition and strengthens your own position, right? So I can't really see any strong arguments against having some kind of political coalition that's interested in regulating AI.

Garrison: I just wanna second basically everything you just said there, where it's like we don't know so many things about how this plays out. We don't know if LLMs will be enough to get to AGI or superintelligence. We don't know if it's possible to build those things ever. Just understand that they're gonna keep trying. If they hit a wall, all that money and all that talent will be redeployed to trying other strategies. And the notion that you could rule out all of those strategies working without knowing what they might be is like incredibly arrogant. And so the humble position is the one of like, yeah, I don't know. It seems like a lot of people who have a lot of credible expertise on this topic, they disagree, but like actually 70% of them think that existential risk is above 0.5% of the leading AI researchers surveyed in the biggest survey of them in 2023. I would guess it's higher now. Like Joshua Bengio, the most cited scientist in the world, wrote out all these arguments for why rogue AIs might arise. It's like very lucid, and you could try to argue against it, but I think it's actually pretty hard. And I think that it's not even necessary because the policy recommendation that I think we should organize around is stop the race to replace us, ban further research towards the universal labor replacing machines. And if you care about that topic for existential risk reasons. Great. This is the best strategy for fighting X-risk. But if you also are worried about AI using too much energy and water or AI causing all kinds of harms in education or discrimination, whatever, we should want to stop further progress towards these machines that are very dangerous if they were built and the mere pursuit of them is democratically illegitimate. And so I hope that that demand will organize people. But the label and the kind of idea that I think is the one that I hope we can organize around is called AI reform. This is like me looking at these different camps and being like, well, I do take the AI safety position pretty seriously, but the community, the classic AI safety of like, we just have to solve this technical problem of figuring out how to align the machines with our intent. intend the right things and then it'll be fine. I'm caricaturing it a little bit, but I think that is actually how a lot of people were thinking about it. That doesn't seem it. And then the critics tend to not buy the technology's capabilities or that they'll ever get there. What is something that takes the best of these various camps and leaves out some of the baggage or the failure modes? And the thesis for AI reform is something like AI is neither impossible nor inevitable, but it's potential. And that potential is being squandered. Because deep learning, the technique that's underlying all these different AI models, deep learning is amazing, but it's being pointed at the wrong things by the wrong people for the wrong reasons. We should build a movement, stop the race to replace us, and democratically point AI towards what we actually want from it. And like I don't know what that is, right? I have some ideas. Like we want cures for diseases and we want mitigations to climate change and we want things that will actually make us more democratic and wiser, not just help us cheat on our tests. We don't know what the end state should look like. AI reform is a process. It's a way of approaching the problem that's rooted in democracy and letting people collectively decide how to govern this technology and how to structure our institutions so that we can get the best out of the technology and deliver results that we would all be happy with if we got to think about it first.

Cassie: I completely agree. And I think one point I want to pull out of that is this question of what do we wish all this investment was pointed at? What do we wish this technology was pointed at? When you're building the labor replacing machine, there's kind of a natural incentive to build it to replace the labor that's already happening. And that's I think mostly what people don't want. What people want from revolutionary new technologies is for it to do things human beings can't do. Like human beings can't analyze trillions of different potential candidate molecules for curing diseases. We actually do need machines that are capable of parsing enormous data sets and modeling a bunch of different possibilities in an efficient way to do that for us. And by doing something that we're incapable of doing manually, potentially we get these great dividends that could actually benefit a lot of people. I think we want technology to do the things that are hard for human beings. We don't want technology to do the things that we currently do and derive some sense of purpose or political and economic agency or enjoyment from, but those are precisely the near-term things that are going to be profitable to deploy a labor replacing machine to do. It's going to be more profitable to deploy it to cheat on your test, replace the local graphic or print artist who used to make Posters and flyers and advertisements for people. There are people who would like to replace human art making more generally, or of course replace you at your job. I mean, that's the most straightforwardly profitable proposition of all. I think this speaks to therefore like a fundamental inability of this technology to be deployed to the ends that people actually desire as long as it's in the hands of unregulated private capital. It will always go towards the thing that is exactly what people don't want. out of potentially transformative, radically powerful new technologies. They want it to be directed at doing the things we can't, or if it's going to replace some of our labor, at least the labor we really can't stand, the labor we really hate, you know, do the dishes for me, iron my clothes for me, but don't replace my job or teach my kids or replace my conversation with my doctor with some AI agent pretending to be a healthcare worker and diagnose me. Like those things make us uncomfortable. And the only solution, therefore, is some kind of formal process of economic democracy. And that means we have to use the state and we have to use politics. We have to build coalitions. Like there's just no substitute. You can't get to using AI, using deep learning, using machine learning for the things people might actually want out of that technology. Just through like negative consumer sentiment about AI and AI companies. There's no amount of like calling it slop that's gonna direct all of that investment towards curing diseases instead of making slop. You know what I mean?

Garrison: That's such a good point because so much of the energy is towards stigmatizing the use of the technology. And like, you know, I hate slot. I hate seeing AI writing. I don't like the vast majority of the images that I see. There's occasionally like videos or whatever that are funny and you know it's fake and it's like Trump with like his hair gigantic. You know, it's just like something absurd, fine. But so much of it is not desirable, stigmatizing that without like going upstream and looking at how this technology is being developed and understanding that you can't just address it at the consumer level or the platform level. You have to go upstream and look at what are they making, how are they making it. There's this belief in some places that you could just use regulation, unions or or something to protect jobs from being automated by AI and robotics. That'll work in some cases, but if they actually build the universal labor replacing machines and then eventually robotics that can compete with humans after that. It's not going to matter if your job is protected by statute or by a collective bargaining agreement. The economic activity is going to flow to wherever those machines are allowed, and countries will be falling over themselves to allow the machines because it will generate so much wealth for those places. We don't have any choice but to get organized and go upstream. And again, you don't have to believe in all of the sci-fi stuff to want to change what's happening now and redirect the industry away from replacing us. Because even if they don't succeed, they're doing a lot of stuff that we do not like that poses a lot of risks and is creating a lot of harms right now for people.

Cassie: So I think it's also worth talking about what brought us each to this place where we're interested in this concept of AI reform. How did you ultimately move to this position?

Garrison: I first read about AGI and existential risk back in like twenty fifteen, twenty sixteen. I encountered it through this blog called Wait But Why, which wrote this like long series of posts about superintelligence and extinction and presented it as like it's either going to be annihilation or utopia. Like you either lose control of the superintelligence and it kills everybody because it just has some desires that are just like outside of what we wanted or what we intended, or It solves all of our problems forever and cures all diseases and solves aging and poverty and all of these things. And then I read Nick Bostrom's superintelligence, which that blog post series was like based on. And I took a class in college called Minds and Machines, which is exploring artificial intelligence and philosophy of mind and consciousness. My thinking was something like, Yeah, this AI thing is pretty interesting. If AGI is possible, that's like big if true. I understood why it was so important if this thing could exist, but I just didn't really think of it as a thing to work on or spend that much time on because it just seems so far away and speculative. And I did encounter people who are working on the issue and I was kind of like, Yeah, but there's so much suffering right now. There's so many like real problems that we can try and solve right now. You're just doing like math problems and you're thinking about what Would the robots do? You know, how would they get out of the box? I was just skeptical, I think, of the motivations of people working on this. I got into left politics after Trump was elected the first time. I had been a kind of radical liberal or something who believed in like prison reform bordering on abolition, you know, legalizing all drugs and liberalizing immigration policy. But capitalism, I was like, well, you know, it's better than the other systems. I was working at McKinsey out of college and I was working for immigrations and customs enforcement. through McKinsey when Trump became president the first time. And I was tasked with building the hiring models to comply with this executive order directing ICE to hire 10,000 new deportation officers. And I was horrified at his election and and and what this meant would have been tripling the workforce of people who were already the lowest on the competence and the virtue level of federal law enforcement agents were already doing all kinds of horrible things. And just seeing McKinsey go along with it, even as we raised a lot of objections internally. It was in fact helping ICE hire these people and optimize their deportation and removal process at the expense of people they had detained. That was just incredibly radicalizing, just to see this institution, this paragon of American capitalism, not only fail to resist this really unjust situation, but like aiding and abetting it and then applying the kind of McKinsey mindset of optimizing to this process that was affecting real people. I kind of like had both of these things in my head and I didn't see this. I thought the AI thing was a big deal and I thought that we should organize society very differently, not around the profit motive. And then I got into journalism by writing about my time at McKinsey, first anonymously for current affairs. And then I became a source for some articles about ICE and about Rikers Island, this other project I worked on. And then I wrote more about it and I kept going with journalism, but like was just doing things I thought were interesting and important, like writing about prison reform, psychedelics, and solitary confinement and pescatarianism and progress studies and like all these kind of random things. And then Chat GPT comes out. I had avoided writing about AI because I was like, well, other people have got that covered. They've studied linear algebra, they know how the technology works. And then the quality of journalism covering AI wasn't so great once like a lot of people started writing about it. And it wasn't amazing before that either. And so I was less worried about like coming in and Leroy Jenkinsing some bad article about this important topic. It was more like, I think I could bring up the average. And I realized that there was just like really wide open lane of thinking about the political economy. Of artificial intelligence and thinking about it from a materialist perspective. These companies are becoming incredibly valuable and they're doing it off of the prospect of automating basically all labor. People previously were writing about AI as like either a technology or just like a normal business or something. And looking at this intersection of like political economy, AI, and risk was really fruitful. Kind of the intersection of left-wing analysis with taking the technology seriously was kind of like. landing on an undiscovered continent and you could just look around and like point at things and be that's a new species. There's so much low-hanging fruit, basically. And my life's work now is to try and like bring this perspective to more people. And I think we desperately need people to be organizing around this issue and stopping the race to replace us, which is, you know, what led us to to this podcast.

Cassie: Yeah, it's it it's interesting. You were aware of this much earlier than me and in more the context of the safeties folks and I'm not saying exactly that you came out of the warriors camp where I I

Garrison: I think I was like I most identified with that camp, but then I like realized all these problems with the kind of classic way of thinking about it. It's like, okay, you align the AI to like the ruling party of whichever government's in charge of the country that builds it. Like that doesn't seem great. What about everybody else who will be put out of work and not have access to the machine? Like all these questions that a lot of people just don't think that much about, even though they're thinking about this technology like day in and day.

Cassie: Day out. Or I mean, not even aligning it necessarily even to a government at all, but aligning it to people like Peter Thiel and Elon Musk and Sam Altman also feels pretty uncomfortable. It's interesting because you came out of sort of more of that background. I definitely was firmly, I would say, in the critic camp for I mean, until pretty recently. Part of that is just I've been a socialist and an identifying as part of that community and like being in DSA and whatever. On and off since like 2016. Like the time you were first hearing about AGI and the idea that this could be a big problem. So when this sort of emergent left consensus started to develop, I was like, yeah, okay, sure. I'm used to tech grifterism. I've seen how crypto went. I've seen how NFTs went, prediction markets. Become sort of inured to all of these grandiose claims by big tech that it slotted in naturally to a lot of pre-existing archetypes, mental models that I had about how that world works and my relationship to it as somebody who's interested in socialism or economic democracy. I didn't have any particular road to Damascus moment where it was like, shit, this is actually a big deal. All of a sudden I got it. It was really bit by bit. One of the first things is my mom's an English professor and she would consistently describe to me what AI was doing to her experience of the classroom and just like the active teaching itself. And it became clear to me that even well before we're reaching anything like AGI, two, three, four years ago, it was already becoming incredibly disruptive. She's like, why am I grading something that a machine wrote? Like this is pointless. Also just it like introduced a huge amount of intolerable amount of dishonesty into the relationship between teacher and student and It seemed like it was fundamentally making the job impossible. Not that education hasn't had plenty of dysfunction always, but there's a difference between something being dysfunctional and in need of reform or improvement, and it becoming just a complete farce. Where there just like straight up is not contact between the person trying to teach and the person ostensibly trying to learn. You're just talking to a robot. There's not a human on the other side of that relationship anymore. And The more I thought about it, the more alarming that seemed to me. At the same time, I was seeing the image and video models get a lot better very quickly. I remember when that first like Will Smith eating spaghetti video came out and

Garrison: I was like, well, like two or three years ago, like

Cassie: Yeah. There were these people talking about it as if it was this dramatic, incredible achievement and it's like it's horrible. Aesthetically and like kind of morally repulsive to look at.

Garrison: But it's also fascinating, right? Like that that video is just there's something about it.

Cassie: There is something about it. I mean But contrasted with the hype. Okay, yeah, there is this dishonest group of people who really want to over-emphasize what this technology is capable of, to serve their own valuation-chasing incentives, and I can look at it with my own two eyes and see that it kind of sucks. And then within like a year, that changed. And it kept changing. And it kept changing faster and faster and faster. And started getting more coherent. And we went from This is like actively kind of a horror show to well, it looks pretty realistic, but you can still always tell people have seven fingers. And then that stopped being a thing. And then it was like, well, but it can't really be coherent beyond about three or four seconds of video. And then that stopped being a thing. And it just seemed like every argument that dismissed this technology based on its capabilities was falling apart in front of my eyes very quickly. And it didn't seem tenable to actually keep holding the same beliefs about where this was ultimately capable of going. And then about a year, year and a half ago, maybe, I actually tried using Frontier models for myself. And I was like, fuck. Like that that I think was kind of the nail in the coffin. And there is a way in which the stigma against using them that exists on the left, and I'm not saying anybody has some moral or political imperative that you must use them in order to understand them and fight them. But it is just s simply true that because so few people on the left use the frontier models There really is a poor perception of what they're capable of and also people are not tracking the dramatic increase in capabilities that's happening very quickly. Like just in the amount of time I've spent vibe coding with Claude, and I like I can't code or program at all myself, and I have no interest in learning, so I'm like, whatever, Claude can do it for me. I don't wanna relinquish things I actually care about, but like I don't wanna be a coder. I think I first started vibe coding with it like at the beginning of this year. I was interested in building an organizing app based on my experiences in the Starbucks workers union, and I felt like there was this missing technology that we needed to like put organizers and rank and file people in the same bucket together, and this doesn't exist. And I've been complaining about this for like years, and I was like, somebody should build this. So I started doing it with Claude, and it took like a week and a half to stand up a prototype that actually worked. And I was just like, this actually has capabilities. This is labor I could not do, and I can't afford to hire someone to do for me because I'm some unemployed former barista who's like interested in making an app. But it turned out Claude actually could build this prototype. And in fact, in the time since I first started tinkering with it, it's gotten massively more competent at doing that. And that's like one kind of labor, and I think most people still believe there needs to be a human in the loop, like overseeing it and managing it and making sure what it produces is good. It became very clear to me that I needed to have a much higher degree of uncertainty and humility about like, where is this going? What could it possibly become? At the same time as I was also, again, already realizing that it was breaking down something as fundamental as the educational relationship. Human beings have been Teaching each other how to do things. We've been passing along skills and knowledge and information from older generations to younger generations to prepare them to go out into the world and be successful and self-sufficient literally forever. And it seems like in just a few short years, AI had destabilized this very, very fundamental, universal way that humans relate to and care for each other. And so when you put those two things together. This is actually quite horrifying already, what it's doing to us and human connection. And it's getting much more capable right in front of my eyes very rapidly. I realized I had to start taking the risks more seriously, and that started mostly thinking about economic risks, labor deplacement risks, but also the concentration of power risk, what this could mean for democracy. If information production and distribution and control was concentrated in a very small number of people, and even if they couldn't replace all the jobs, the extent to which they could replace human beings in delivering violence through drones, through automated weapon systems, Those are very concerning for democracy. I've been pretty reluctant to meaningfully contemplate X risk, and maybe that's partially just residual lefty, like everyone thinks it's stupid and I don't want to be the person going out on a limb and saying something that everybody's gonna call me a moron about. Like,

Garrison: It's cringe.

Cassie: It is, it's cringe. You're like a credulous dumbass. If you say, like, I'm a little worried that they could invent the extinction machine. But honestly, the last few months it's like getting harder and harder to not take that seriously. Like I'm still mostly concerned about what perfectly aligned or reasonably well aligned AI does in the hands of the kinds of people who are building it and owning it. That really worries me. But I'm start I'm starting to get a little more worried about misalignment, I'm not gonna lie. I mean, we're seeing these hacks that seem to be autonomous machines escaping control and committing felonies, and that seems that seems like worth worrying about. We definitely started from very different places, but both came to realize the inadequacies of some of these camps that exist in the AI discourse and the need for something that was fundamentally different, that was able to piece together multiple different perspectives and put them in a coherent value set that was dedicated to the idea that people should have some say in this.

Garrison: I've seen a lot of people in the last few weeks with these autonomous AI hackers that had gone rogue and were creating secret message boards within OpenAI's infrastructure and self-radicalizing and helping each other hack their way out of open AI until it culminated in them attacking another company with no humans involved. I've seen a lot of people being like, you know, I was actually wrong. I think that this AI risk thing is real. I think like if you're one of those people who's been skeptical in the past and publicly dismissive of of this. That's fine. I think it's important to update on new information and for people who do that to be praised for it because that's just like how persuasion happens. And I think it's reasonable to be like, yeah, there wasn't enough evidence before that this type of thing could happen. Maybe there's some theoretical possibility that AIs could become misaligned, capable, and autonomous enough to do harm in the real world, but we just haven't actually seen that evidence yet. And I'll like reserve my judgment until we do. That's a totally reasonable position. And then, like, that's actually happening. And some people are like, I don't believe any of this. They wanted this to happen, or they're not telling us something, or something like that. It's like, look, in the fullness of time, there will be government investigations, there will be third party investigations. I'm willing to stake my credibility as a journalist who covers this topic on the fact that it will be largely as it was described and probably will look worse in some ways. That, like, no, these AIs actually did go rogue and they actually did. break out of the company and hack into multiple other companies to get an answer key. Which is like a weird thing that's like hard to relate to as a human. But they're not like humans, but they're also not like normal machines either. But I digress just to say like, you know, if you are wavering, if you're open to these ideas, we're trying to create a space where it's like not cringe to be like, yeah, no, I'm actually worried about losing control of AI, or I'm worried about AI replacing all white collar labor and then blue collar labor after that, once the robots get good enough. And I think we should organize and stop it. That's great. That's what we're trying to do here. And I don't think that means you're giving Sam Altman a win. In fact, I think it's the thing he really doesn't want to happen the most. So if you really want to stick it to him, then get organized.

Cassie: Yeah. And I mean honestly, even if it is cringe, okay, some people are gonna cringe at it. And it's like I'd rather be a little bit cringe than be blindsided by the autonomous drone kill swarm. You know what mean? Like I'd rather be a little cringe now and risk being a little over eager about certain risks than find out too late that and real drones have replaced human military and policing capacities to a point that enables genuine autocratic crackdown, with no one in the loop to like mutiny or say no. That's something I'm genuinely quite worried about.

Garrison: And it doesn't require believing anything that's like beyond an extrapolation of the t technology as it currently exists. Yeah.

Cassie: Exactly. I don't think you have to use words like intelligence even if you don't want to, or reasoning. I mean intelligence is like a hard enough problem to describe in human beings. It's like a pretty fuzzy concept that is like famously hard to define and famously hard to research and understand as is consciousness. So there's a lot of leeway to have skepticism and different beliefs about this and still recognize that we should take action to prevent real harms that can happen to real people. And I think it's perfectly natural to have started from a place of skepticism again because the tech industry is this kind of boy who cried wolf problem. Every single technology is revolutionary. That's table stakes. The business to business software as a service platform is revolutionary. It's gonna transform everything, right? The latest widget or gadget is always revolutionary. So dismissing that out of hand with AI is totally normal. I think that's like a very natural human response. They always oversell everything. The problem is this time, even if they are still overselling it in some ways, it could still be incredibly harmful exactly as it is right now, let alone if it gets even a little bit better.

Garrison: It could be that it is overhyped, especially in the near term, and still incredibly transformative in many, many terrible ways. And I think in the long term, the full scope of possibilities is on the table. Some people are like, I think AGI won't happen for like 10 or 20 years. That's still crazy. The three-body problem is about these aliens are coming in like hundreds of years. And they're like, we got to organize the entire planet to be ready for this eventuality. And instead, we have this thing where it's like, Kind of equivalent, but in much shorter timelines. But because there's uncertainty about it or something, we'll just do approximately nothing compared to the actual stakes. Right. Like if you think there's a 1% chance of extinction at the hands of AI, very standard economic models where you only account for humans in America who are currently alive, it's still like spend multiple percentage points of GDP every year for 20 years to bring down the risk a little bit. If you take it even a little bit seriously, it justifies a World War II level of mobilization. And we're spending a billion-ish dollars a year on safety for these systems and like many, many hundreds of billions each year on making them more capable. Very few people are organizing to stop the technology's advancement. They might be organizing against local instantiations like data centers, which that slows the industry down a little bit. But you could block all the data centers in the US and it would still not stop the development of these machines, assuming that they're possible to be developed, which again, like I think you cannot write it off as a possibility. The only way to really stop it is to get organized and recognize that we're all in this together. We don't want to be replaced. We can defeat the industry, but we actually have to get our shit together.

Cassie: And I think education is again kind of a perfect example of how all the existing institutions appear to be failing us. Everyone directly involved, students and teachers alike seem to recognize that this is a problem. Even though students are using AI to cheat constantly and epidemically, when a commencement speaker talks about AI, they like b boo pretty intensely. Like how

Garrison: Time magazine selected its person of the year for twenty twenty five, and it would this time it was the architects of artificial intelligence. Interesting.

Cassie: They seem quite aware that this is not serving their values or their interests, even at the same time as they feel compelled to use it because of competitive incentives or whatever. There's not meaningful action being taken by the people leading universities, and in fact many of them are actively capitulating. They're trying to figure out how to implement more AI because they want to feel like they're at the cutting edge, they're at the bleeding edge. How do we make our university incorporation of AI feel sexy and exciting? It's a clear example to me of where a lack of democracy is allowing these harms to continue. The people directly affected are aware that they're not being served by this technology as it's presently conceived of and deployed. But because they don't have a direct voice in the way these institutions are organized, nothing is being done to prevent. The harms that they're experiencing, and in fact it's being accelerated by the people in charge. This is, I think, broadly emblematic of where society is right now. We need to develop a better institutional response at institutions that already exist. But I think we also need to start thinking about new institutions. I think back to the American history of mass membership organizations or the women's Christian Temperance Union, you know what I mean? Or even like The American Legion ending up writing the GI Bill or the NAACP or the Southern Christian Leadership Conference, which is like not exactly a mass membership organization, but like organized a lot of institutions that already had membership in the form of churches, brought those institutions together and their membership around a concrete political cause with like discrete objectives that could be achieved and in fact was achieved through intensive organizing. We don't have a lot of those kind of institutions anymore. Those kinds of groups have largely given way to lobbying entities and legal advocacy and think tanks and policy shops and the NGO model is very different.

Garrison: It's come professionalized.

Cassie: It's become very professionalized. And we're seeing how those institutions are proving really vulnerable to wanting to integrate AI. And I think we might have to look towards reviving a kind of mass politics and broad-based membership style institutions to be able to sufficiently combat this.

Garrison: This technology is advancing at a moment where things couldn't be worse almost. Our institutions are already failing under the strain of the authoritarian desires of the Trump administration, social media, the flow of capital wherever it can be put most profitably at the expense of the people who their jobs are being deindustrialized. At the same time, it's creating this incredible generational opportunity to Have like a villain in this industry and what they're trying to do that creates this incredible collective action problem, but also the perfect conditions for basically everybody recognizing that, I don't want my job to be replaced. I don't want the business I started to be completely eaten by philanthropic making some app that makes it obsolete. I don't want my kids to stop learning at school. I don't want to Die. And then the people in charge are just so unsympathetic and so explicit about their desires to build this thing that they recognize as being an existential threat to the further existence of humanity. They're like asking us to come together and get organized and create the laws that are needed to stop them from building this technology and then to create the institutions, the industrial policy, the regulations. culture, the norms around building the types of AI that we actually want and social infrastructure for the type of world that we actually want to live in. There's this like trench run from like Star Wars, where if you do it just right, you can not only stop our obsolescence, but also create a realignment of society around helping everybody, like protecting the interests of not the elites, but like the masses. And creating institutions that are fit for purpose in light of what's technically possible now and what might be possible in the near future. It's so, so desperately overdue.

Cassie: It is, it is. There wouldn't be any downside. There'd be a lot of advantage to rejecting the philosopher king model of AI safety and AI luminaries and building a democratic alternative. To the extent we're successful in building that democratic alternative, like you're saying, we can orient that power in a lot of different ways. We can orient that towards thinking more explicitly and intentionally about how we manage our economy in general for common good. Direct investment and productivity where it is going to do the most good for the most number of people. So I just really don't see a downside. I really don't see a downside to saying, you know, we've had enough of people anointing themselves the great leaders that are going to decide our fates for us. In fact, we should decide our own fates for ourselves.

Garrison: Yeah. No, you said it perfectly. Bringing it back to the AI reform idea, the important thing about it is that it's a democratic process. It's an appeal to giving people the right to decide decisions that affect them. We're gonna try and figure that out together, the two of us, but then also with our audience and in coalition with people who are thinking about how to improve the world in light of this technology. AI reform right now is this idea and what it becomes is really up to all of us who want to get involved. AI reform doesn't pretend to have all of the answers, but it's focused on this question, which is in light of what AI can do now, what it might soon be capable of, how do we democratically build a future worth inhabiting? And, you know, that's just not something that I and or Cassie or anybody can just decide unilaterally. It's just a thing that we have to apply this like framework and and this model to the world. And the first step of that is like, Letting people know about it and kind of it's this invitation to to figure it out together.

Cassie: Absolutely.

Garrison: This has been the very first episode of Organize Against the Machine.

Cassie: let's get organized.

Garrison: If you enjoyed the show, the best thing you could do is share it with a friend and rate and review it wherever you found it. Our theme music is Problem Child by Ian McFarland.

Cassie: Thanks for joining us.