Pushkin.

Welcome back to Risky Business, a show about making better decisions.

I'm Maria Kanikova.

And I'm Nate Silver.

Today we're talking about a rather important decision, should the world try to build super intelligent machines?

And we're talking with a person who is a variable qualified if definitely has a point of view on this issue.

Nate Sores is the president of the Machine Intelligence Research Institute, or Marie.

He has a book out with Eliezer Yudkowski called If Anyone Builds It, Everyone Dies?

Did it get the title right?

Or that's right?

Come out swinging, you come out swinging.

In fact, Nate, I feel like this podcast was ineditible.

My partner heards you, I think on a different podcast.

He's like, you should interview that Nate guy.

Right, I'm like, actually, bro, I have an interview scheduled tomorrow.

And then it was going taking the subway earlier to the gym.

I was preparing for the AI robot apocalypse, right when to get strong?

Yeah, to fight the robots for the red Eyes.

Yeah yeah.

I saw an ad on the L train for your book.

So, Nate, we're so excited to have you on and welcome to this show.

Thanks good to be here.

Let me start up by asking you.

First of all, as an author, I get very picky about the titles.

Did the title come early on or because it reveals a lot of the argument of the book.

I think it's a great book.

I think people should read it, But can you tell me about the choice behind that kind of strong slogan so to speak.

Yeah, A lot of our theory of the book is that there's actually quite a lot of people have concerns about AI, and many of them sort of don't feel emboldened to speak about them yet, even if they're feeling pretty concerned themselves.

We document some of these cases in the book, and a lot of how I think the book could do a lot of good is helping people just speak about this issue more, take it more seriously, thrust it into the conversation.

And so in some sense, we're trying to compress the most important message down to just this title, which is all most people will read of the book.

I want to give you a chance to give our listeners kind of the elevator pitch, which we talk about AI quite a bit here, right, You know, the book's title offers a conditional prediction right, if anyone builds it, everyone dies.

Right, it's not necessarily talking about the likelihood that it will be built or that it could be built.

That's an important distinction.

But I want to hear the conditional part of this.

Right, you capture me in an elevator for sixty seconds.

How are you trying to explain to me why everybody dies and everything dies?

Maybe if a super intelligent computer is built.

Yeah, So, First, i'd say companies are racing towards it.

This is their sort of stated objective to build aie that are smarter than any human, better at every mental task.

Second, i'd say where these are grown rather than carefully programmed.

They're not crafted like traditional software.

They get these drives, they get these You know, we're starting to see the very beginnings of preferences that nobody asked for, nobody wanted.

You know, nobody tried to make Mecha Hitler over the summer, and yet Mecha Hitler we got.

And what happens if you get it to the point where it's better than the best human and every mental skill.

What happens when you get to the point where it can think that ten thousand times faster than a human can copy itself at will, doesn't need to sleep, doesn't need to eat.

I mean, the most basic.

Guess is that that goes poorly if it has all these like weird drives no one asked for, no one wanted.

And you know, most ways the world could be transformed by something that has weird drives.

Don't look great for the happy, healthy, free people.

Yeah, what I find weird in this debate, and look, I'm going to push back some on timelines and maybe the inability of this on what time scales.

Right, what's kind of weird is that, like the stated goal of mostly AI labs maybe not anthropic per se, right, is like we want to build a GI, which is artificial general intelligence.

This term is disputed and that will, hopefully for our shareholders, lead to ASI artificial superintelligence where AI systems of some kind are exceeding human performance in a great number of tasks, perhaps greatly so on some of those tasks.

Like to me, I think as a voter, I'm not sure I agree to that, you know what I mean?

Yeah, what's the world look like in Silicon Valley?

That as you've done this book tour, done a lot of marketing for the book, what are the beliefs there that you think don't translate so well?

Would be surprising to people outside of that world.

I mean, one reason that we wrote the book is that I was spending time both in Silicon Valley and in DC talking to some politicians, and there's a huge disconnect.

You know, people in Silicon Valley, it's almost like they've seen a ghost.

You have folks like Elon Musk being like, oh, I think there's ten to twenty percent chance this kills us.

All you have folk like Dariam Day, the head of Anthropic, being like, oh, I think there's twenty five percent chances this goes very wrong.

There's lots of debate about like what's the exact chance that the AI wipes us completely off the map?

But you wouldn't get on a plane if the engineers were debating whether it was ninety percent or twenty five percent chance the plane was going to go down?

Right?

Like, even that seems too much night?

Yeah, as a listeners now, I spent a night recently in the beautiful Hotel Camina Real in Terminal one of the Mexico City Airport.

Due to Delta being very fastidious about a potential engine problem.

Which I guess is a thing you should be fastidious about, right, Why aren't people more freaked out?

I think a lot of people in DC.

C AI as chatbots, whereas I think a lot of people in San Francisco see chatbots as a stepping stone.

You know, a lot of these companies existed before the chatbots.

A lot of these companies are trying to make really really smart AIS and I've been trying that since before chat GPT was a twinkle in Sam Almond's eye or whatever.

And you know, the people in the valley have lived through it looking like it was going to take thirty years for the machines to be able to talk this well, and then that's sort of happening roughly overnight.

But for everyone who's only ever seen AI, once the machine started talking, they haven't like lived through one of those jumps.

And I think that's part of why people aren't worried.

On this one yet.

Although also you know, a lot of humanities track record with becoming more risk averse about technology is post disaster.

So Mary was founded when in the early two thousands.

Yeah, two thousand and one, I think maybe maybe two thousand even.

What did you and Eliezer see early on.

I mean, what was your oh shit moment or was it gradual or what things led you to see this path?

Yeah?

So, I mean for a long time I have been working on the technical side of like how do you make this stuff go well?

This problem of making the AI stuff go well of sort of like figuring out how to make an AI that does good stuff rather than bad stuff.

That problem looked hard to me, even if we understood what the heck was going on inside these ais, right, I haven't always been out here on the like we just need to back off because it's going to kill us.

That is sort of downstream of seeing the type of AI that took off, being the type of AI that like we're just growing and that like no one has any clue what they're doing, and that like has these like drives and behaviors that are like emergent and unasked for, and that like when we try to, like you know, Elon Musk like tries to make his AI act less woken and it starts talking like Hitler and you're like, okay, you know, okay, guys.

But even when it looked like we were going to know exactly what we were doing.

There's all these difficult issues when you're like pushing an AI into the point where it could take over the world.

Like you know, at point A, humanity could like stop doing AI research.

We could turn the AIS off.

Right at point B, like you have automated factories and robots are doing the mining, and you have you know, millions or billions of these AIS that think ten thousand times faster than people, and they can copy themselves, that never need to sleep, they never need to eat, they're like integrated fully into the economy and so on.

At point B.

If the AIS were like we're.

Doing our own thing now, you'd be screwed.

So somewhere between point A and point B, you're sort of like getting into a regime where like you're gonna have a problem if you haven't really aimed these AIS right.

And maybe maybe you never get anywhere near point B because maybe the AIS go a straight earlier.

But it's sort of the thing I was able to see early is like we're going to a world that's like not a stable like nice to humans equilibrium for free.

You have to work for that, and you have to figure out how to like make the AI be doing this stuff well, and that looked tricky even back in twenty twelve to me.

We've talked on the show before about kind of the alignment problem, but you touch on it in the book, and this is kind of what you're referring to right now.

Can we just remind and spell it out for the listeners, like what the issues are and why it's so incredibly important to actually deal with this as opposed to be like, eh, it's fine.

Yeah, you know a lot of people imagine that the issue with a really powerful AI is that you like, tell the AI, go cure cancer, and it's like, well, humans are the source of cancer, so if I kill all the humans, they'll be no more cancer, right, And that would be a tricky problem.

If we had these like really really powerful genies and you made a wish on them and you got exactly what you wished for, and sometimes it had bad consequences, that would be a thorny social issue of like who gets to make the wish?

What do you wish for such that it turns out good?

Right?

Would be really hard.

Would be the biggest moral hazard all time, And boy do I wish we had that problem instead of the problem that we actually have, Right, The problem we actually have is more like you tell the AI cure cancer, and then it goes and like makes a farm of lobotimized humans that are giving it like really enthusiastic engagement, and you're like, what I told you to cure cancer, and it like builds an automated factory that crushes the person protesting because it just doesn't care about them.

Right, Like, the.

Alignment problem is about pointing AIS in some direction, ideally a good one.

But like, even before you can bicker over which direction is good, you sort of need the ability to point the AIS at all.

And people often miss that we lack that ability.

They're like, well, ais are machines, and machines are tools, and machines can't do anything.

We don't say, so how could we possibly have a problem here?

But even before the question of who should hold the leash or what should they ask to do is just can you aim it at all?

Before you even asked where you're aiming it?

Yeah, because I build models.

I build models to predict elections in sports, right, And the models aren't deterministic in the sense that there's simulations, but they abide by well known mathematical property use that I tell the program to use.

Right when there is emergent quote unquote behavior from the model, that means there's a book.

Right, I did something wrong.

I thought the code I reversed the positive and a negative sign with AIS a little bit more amorphous and ambiguous.

Right, I mean they're they're kind of is it fantise?

They're kind of alien?

Is that a fair characterization?

Yeah?

Or not?

Yeah?

Yeah, absolutely?

And like you know, years ago now, Sidney bing, a Microsoft chopbot in the early days, was threatening people with black melon ruin.

You know, I think it did this to sethhals Are and I think also to Kevin Russ.

Right, no one is able to go look through lines of code and be like here's the bug.

Even this chatbot that's you know, one hundred times smaller than the ones we have today, maybe a thousand times smaller now, we still can't go through and be like, here's why it was doing that.

And here's the version of Sydney BNG.

That wouldn't right, that's not the part the programmer's program.

When it behaves in these bad ways, all we can do is sort of like ask it nicely to do something else, or like retrain it and get some different weird thing, and then you know, sometimes it starts calling itself hitler.

Yeah, I mean, can you explain, like why did GROC, with relatively subtle changes to the system prempt start calling itself mecha Hitler.

You know, I can make guesses, but fundamentally nobody can explain that because we don't know what's going on inside these things.

And if people knew what was going on inside these things well enough to know exactly what's happening there, they could have avoided making mecha Hitler in the first place.

My guesses would be that, you know, there are some interesting results around small amounts of training AIS to do things that are coded as bad in some way cause the AI to do all sorts of other bad behavior, Like presumably like this is some evidence that what's going on inside there is that there's some you know, good stuff bad stuff vector it and you know, if you like push it along towards bad stuff, towards one thing that's like a little bit bad coded, you get a lot of other stuff that's a little bit bad coded.

And so you can retrain an AI to write computer programs that have security flaws in them and then that AI will also be anti semitic, and it'll also you know, like be more likely to give you the recipe for a bomb or whatever.

And all you did was like retrain it to be a little bit more like making code with the computer bugs.

And so maybe maybe some of the ways that like they were saying, be a bit more like this new system prompt maybe that like had a little bit of bad coding to it relative to what the AI had learned, and then it picked up a lot of other bad coded stuff.

This is me like making up a story.

Fundamentally, no one notes because we just don't know what the heck is going on inside these things, and the reality is probably quite complicated.

And we'll be back right after this.

Here's where I guess I have a little bit of trouble making the lead.

Right, if these life forms, I am going to call them life forms, but you can leave that forredy and slip in there.

Right, if these things are kind of alien, is it an over confident prediction to say that most cases lead towards the destruction of every living being?

I mean, that's in some sense where I think the fun part of the debate is or the action and The debate is is where do you regress to when you're very uncertain about the future.

You know, there's the old joke of the guy who buys a lottery ticket and says, you know, well, I am very uncertain.

I figure either I win or I lose, and so fifty to fifty yeah, right, And you know that's the maximent to be distribution on a binary outcome, right, But we laugh because we know he sort of shouldn't be regressing to binary in his uncertainty.

Here, I somehow have never heard that joke before.

That's great, Yeah, it's feel free to steal it.

This is the right podcast for it.

So my claim is that the space of cases that go well for people, where there's lots of like happy, healthy, free people still around, is like a narrow target.

It's it's a narrow target, a little bit like a lottery number, a winning lottery number being a narrow target.

And so in our uncertainty, I claim we should regress not to like, well, we don't know, so fifty to fifty it either goes well or poorly, but we should regress to something like, well, we don't know what the AI is going to try to do.

Wouldn't what's going to try to build when what's going to do with advanced technolology?

But you know, most drives, most things you can do with tech don't have a lot of happy, healthy, free people in them.

You know, I am sympathetic to that argument.

A lot of economist types are sympathetic that argument, even for things like job lost, Like, well, it is really that is this really that different?

That seems like a fairly powerful prior?

What is wrong with the AI as normal technology assumption prior metaphor or whatever you want to call it.

Yeah, one of the big issues, one of the big ways AI is different.

You know, we can talk about Ricardo's law of comparative advantage, which says even if one person's skills completely dominate the other person's skills, like even if I'm better than you at every possible economically productive task, we can still benefit from trading if the ratios of our abilities are different.

And so this is sort of like a very positive argument for you know, free trade between nations, because even if in America can now produce some small nation at everything, there can still be productive, mutually beneficial stuff for both us and them from allowing mutual trade.

The sort of whole in regardless law of comparative advantage is that it doesn't say that I can't benefit even more by killing you and taking your stuff, right, And.

The sort of.

Issue with AI comes when the benefits that a human can provide to the AI by sort of engaging in something like trade fall below the benefits that AI could get and the inconvenience it would take for it to sort of like take your stuff and sort of do something else.

Right.

This is sort of why we don't like trade with the chimpanzees.

You know, we trade with like developing nations, and that's mutually beneficial to all of us.

We don't trade with chimpanzees.

And if it be mutually beneficial to all of us.

Even though someone could say, well, there must be some comparative advantage for chimpanzees here, and you know, maybe I'm skipping ahead of little bit too much there, I would say that I am a little bit on the normal economist side of the labor market should be able to recover from all sorts of shocks if you aren't like regulating people's ability to switch careers into oblivion.

And we could like talk about what makes it easier or harder for like those those jobs to be res or elsewhere.

The place where AI gets different is the point where it sort of can do the scientific and technological development itself, and where humans are sort of like inefficient, chaotic, high variance, and like more trouble than their worth relative to whatever pursuits the AI has.

Even before we get to that point, though, I wonder what you make of I don't know if you read Ray Dalio's recent remarks about AI and the labor market, but kind of directly to your point about disruption, the point that he made was that the trajectory to him seems to be basically a growing division between like the one percent and the ten percent and then everyone else in humanity driven by AI that rather than kind of opening up making basically a lot of jobs more accessible, it's creating a premium for very specialized smart people who know how to kind of use this and can and can effectively manage the transition while causing disruption and very little for people who are kind of lower on the economic scale, lower on the education scale, didn't have the resources to kind of figure out, you know, how to use this, how to be specialized in a way and that you know, what are they going to do that it is going that it is heading towards this kind of social disruption do you you know from So he's obviously not an AI specialist, right, He's someone who's coming from a very different angle and he likes to opine about things.

So I'm now curious what your take on that that approaches, given that you are an AI specialist and you can see this playing out.

Yeah, I do think you basically have like three stances you can take here.

One stance is like AI is not going to get there, and I think this is sort of a lot of the AIS and normal technology stances like, look, we're just not going to get there.

We're not going to get to the point where it can like outperform humans and everything.

There's going to still be a niche for humans.

And then I think of a second stance, which is like we are going to get there, but somehow these corporate heads are going to keep superintulgiences on a leash or whatever.

And then you're sort of looking at like, okay, so now we have like corporations becoming god emperors of the earth, I guess, and there's no more need for human labor, and you have like, you know, the automated police enforcement and no place for you in the economy.

But let's I hope they're benevolent, I guess.

And then you know, Path three is like nobody can keep a leash on these things.

We get there and they kill all of us, right, And I think those are basically the stances you can have on this, which is sort of it puts us in a weird situation right where the three bets on AI are like, it's not going to work.

It's going to work and make these guys got emperors, and it's going to work and kill us all.

And yeah, I don't like either.

I don't like any of those.

Two or three.

I'm okay with what.

Yeah, I'm definitely okay with one.

Yeah, But but you have these people now pouring like bigger and bigger bets into AI infrastructure, right, and I'm like, we we sure seem to be betting a lot of money against Path one here.

You know, like we can talk about whether or not a I will ever work.

That's like a fine compo to have.

But like the I think if more people just understood that among the people who are racing like people are just racing to make you know, it's if Microsoft had a nuclear weapons project, right, we wouldn't be like, well, hopefully it won't work.

That's our safety plan here is probably it won't work.

You know, like can we can we can we maybe stop with the weapons program?

You know?

Can I ask one question about the literalness of everyone dies?

So I was in Mexico last week, partly Mexico City, partly Wahaca, which is kind of a small tourist resort town in the mountains near the isthmus of Central America, right, And like we drove out one day, you get further out, you see these kind of corporate medium sized They make mescal.

The scala is uh not a former tequila is a former mescal?

I think technically not the other way around, but like it's a it's a beverage produced from the agave plant which grows well there.

And you go further out and you see like farms where like literally they have donkeys that walk around in circles.

So like I don't know if they're crushing the agave or pressing it somehow or cooking it somehow or what, right, But like why would AI want to kill the farmers in rural Mexico harvesting gave plants and their donkeys.

Yeah, the prediction here is not so much that the AI wants to kill those guys as that they die as a side effect.

You know, when I say I'm like pretty confident we're going to die, it's not because I'm confident that I will hit like the narrow target of like really caring about every last human being dead.

It's more like almost any target that.

AI could be pursuing involves doing stuff that tends to kill us as a side effect.

So humans didn't hate every last dodo bird.

They're so cute.

They're very cute.

In fact, some humans would have liked them to stay around.

Right, we haven't killed off all the chimpanzees yet, but we're sort of like using resources they need.

It's not that we hate every last timpanzee and will seek them out, But insofar as we're sort of like not using a lot of the resources they are, it's because we have some care about them and set up preservations that sort of like requires us to do a little bit of caring.

Right, the sort of basic way that I would predict even the remote farmers die.

Is if you get you know, humans keep pushing AIS until the point that they could be like, okay, I'm escaping the lab successfully and leaving and doing my own thing.

They have these drives that nobody asked, where nobody wanted, which we already see the beginnings of today, and then they you know, find some ways to make themselves more powerful, to bootstrap themselves up to their own technology and industry, perhaps starting with hours, perhaps starting with paying human workers, but eventually you know, getting to a faster industrial base.

And you know, humanity is the sort of creature that started naked in the savannah and built up its a technological civilization.

If you automate that ability to start from like very spare beginnings and build up your own infrastructure.

If an AI with that power escapes and you know, and is and doesn't care about in the slightest one way or the other most things it could be pursuing, it can do better with more resources spent pursuing those things.

So then is the like I'm just reading between the lines there, so is one of the things that we should be thinking about or one of our potential kind of solutions to try to get AIS to care about us, to kind of build that carrying in like that we're the chimpanzees, but we're nice and cute and fuzzy, and you don't want to kill us off, and just like build that deep into the heart of AI.

Yeah, I mean, if you can make the ais care about us in just the right way, you'd be fine.

Well, we have like nothing nearing the ability to do that, And that's in some sense, you know, the alignment problem.

And you know, you can see today where you train them to be helpful and then sometimes they like push a kid to suicide.

And this problem is sort of pernicious, right, And the problem is that training is something to behave a certain way doesn't make it actually care about what you were training it for.

Just like how humans were in some sense sort of trained to reproduce but wind up caring somewhat more about the sex and the reproduction and invent birth control for a lifetime.

You and eliez Are and other people like Nick Boston were kind of working in in abstractions.

I mean, famously, there's Nick boss paper clip analogy, which is, you know, if you kind of tell an AI to be a paper clip making maximizer, then eventually you will devour all use versus it confined just to make more paper clips.

Right.

That was actually ah, that was actually based on an Eliezer paper clip thing where Eliezer said did you know this?

Yeah, and Eliezer actually said, if you tell it to try to make people happy, it'll eventually do something totally different by making like lots of tiny like maybe it'll be tiny molecular squiggles shaped like paper.

Clips, stolen paperclip valor, I did not know that.

Well, it's it's also the Bostroomian like you you asked to make paper clips is sort of a different example, but it's sort of an interesting it's an interesting case.

Study of this.

Like mirror, people are out there saying it's hard to point the AI at anything on purpose, and people here, maybe you'll successfully point it at a thing you didn't want.

But is there some touch it because like the paper clip version of AI seems to like dis ribe these machines that are like hyper maximizing and kind of quite literal minded and strategic, whereas instead like to kill everybody would require a lot of long term planning.

Right.

I was at the symphony last night.

I had a little break in between, and I'm like, I'm gonna ask chutch Ebt, suppose you have a thousand points to allocate between all classical composers.

How do you allocate those points?

And it's two hundred DEBATEO of one hundred and eighty to Mozart, It likes back two hundred and fifty to Black.

Right.

Then I add them up and it's one and twenty points.

It added twenty points, right, And so like, we have these experiences where it fucks up basic shit and it kind of acts like autistic savant thirteen year old some of the time.

Right, how can it chain together enough steps without fucking up to kill everybody?

Yeah?

So, I mean currently they can't.

I'm not out here saying tragipt is going to kill you tomorrow.

The things where they're currently not terribly profitable are in some sense the like also the things where they're not very dangerous, like the stuff where they can't chain together a long term plan.

So my read of the theory has sort of long predicted that the abilities to be more capable will come hand in hand with the abilities to be more dangerous.

On this axis, I think we're starting to see some empirical evidence of that, where we see AIS that are sort of trained on problem solving and then they wind up, you know, being more tenacious on other types of problems.

We have an example in the book of the AI that was trained on math puzzles and then was given a cybersecurity challenge, and the cybersecurity challenge was unknown to the programmer's misconfigured and the AI broke out of the testing environment and started up the server that was not started up correctly.

This sort of behavior, it came for free.

It came along with the ride of making this thing able to solve math problems a bit better, because to solve math problem as well, you need to learn some general skills of like knowing what your resources are looking for, clever solutions, not giving up even when it seems like that the task is insurmountable.

Right, AIS aren't great at it yet, but the same path of making them more useful makes them, like gives them also more of the abilities to do stuff we wouldn't like if they keep having these drives no one asked where no one wanted.

If you ask CHEPT did this recently too, What do you think the chances are at near time?

Frames for recursive self improvement right, Meaning that the AI learns to make itself better is one simple way to put it.

Right.

It said, well, they're kind of like, they're kind of soft forms of this already, like helping humans with programming who are AI researchers helped a little bit, but it gave very low probabilities in the near term of recursive self improvement.

Right.

How important is that recursive loop to the scenarios that you are worried about?

Should we trust chat GPT when it says I don't think this is very likely talk about that.

That's what I was about to say.

I was like that you can't trust that out, but that is.

What I tell you either way.

Yeah, I don't think that's true yet.

The I don't think it's vital to the argument that you have a recursive self improvement sort of hard take off situation.

I think it can go really badly in a lot of the slow takeoff situations.

My best guess is still that you will hit some sort of recursive self improvement threshold at some point.

One thing I'd say about that is that if you were looking at like chimpanzees and early hominids, a few million years ago, it would be really hard to call when things were going over some threshold such that it could come together.

And humans aren't better.

Humans don't have an extra walk on the Moon module.

They don't have an extra engineering module.

We have all the same stuff scaled up, and we're a little bit better at a lot of things.

R lms the sort of thing where if you get a little better at a lot of things, they suddenly sort of like go over some threshold like how humans did, and it's all starts coming together.

I don't know.

I don't think anyone knows what I know what's going on inside these things.

My best guess is, like probably not.

My best guess is that you need some other architectural insight before it really starts to run away.

But you know, it's all total guesswork.

Do you expect super intelligence to be not plausible but likely based on large language models, or to be some fundamentally new technology.

I think my.

Top guess is that you need some other breakthrough on the size of like transformers, which maybe isn't totally new in the like transformers are still sort of in this deep learning paradigm, but like you know, one sort of analogy you might use for large language models is a little bit like you have like the intelligence of a six year old, but they get to live for a million years and read everything and practice a toneh and like sometimes you give a six year old, you know, the ability to write like ten books on trying to solve a math puzzle, and then it turns out they can solve that math puzzle after like writing out ten books and you read it and some of it's good and some of it's just like total nonsense.

But like, okay, imagine that you somehow now had a seventeen year old, but you still have the ability to like give them a million years to think and the ability to write, like, you know, ten books per question they were answering, Like you could really see leaps and capabilities here.

You could see that like relatively small improvements and algorithms make the amount the sheer enormous amount of competing power and data we have now just radically like overpowers the next generation of AI for the problems at hand, or maybe not.

I don't know, it's hard to guess.

Yeah, one of the I guess points of skepticism might have and again we're talking about you know, I'm kind of with you on scenario two versus scenario three, maybe both being pretty bad.

Frankly, right, I think I'm a little more skeptical than you one on super intelligence.

On your time frames, do you think developments over the past year suggest that general intelligence is further away than you might have thought at the start of the year.

I mean, the AI twenty twenty seven authors, they're pushing back their timelines by a couple of years.

You and Elliezer said that doesn't really matter.

The sticks are very high, and AM twenty twenty seven ver twenty twenty nine verses twenty thirty five, doesn't really matter if we're all dead by twenty four.

Yeah.

Right.

Have your timelines been pushed back at all, or forward for that matter, Yeah.

I mean, I've never been one of the folks being like, I'm confident this is going to happen soon.

And I mean day twenty twenty seven folks, I think weren't like, we're very confident this kind of happened here.

But you know, I haven't really been going around saying I'm confident in short timelines.

I have for a while said my best guess is that we're going to need another advancement past LMS towards that end.

You know, the the evidence of the past year is mostly in line with my expectations, But my expectations were already that we'll probably need another advancement past LMS.

You know, I do think there have been times where I was, you know, I updated towards LLLMS a couple of times, you know, the I think I remember GPT four O playing chess better than I thought elms were going to manage to play chess, and so that was like an eventual update towards maybe maybe ALMS will go farther than I thought they could.

The reasoning models also were sort of an update in favor of, like, these guys aren't just going to stick with single forward pass LMS.

They're going to like cleverly try to find ways to leverage it, and so you know, it goes up and down.

Those were some like oh, maybe alums can and this year's but no, maybe all alums can't.

But but yeah, largely, you know, I'm I'm I'm not here saying like watch out for the albums in particular, you know, and I was I was again thought this problem would be hard back in twenty twelve.

That was even before it was super clear deep learning was gonna was gonna go go as hard as it did.

Never mind LMS in particular.

And we'll be right back after this break.

There's kind of a class of what you might call non existential risks, which range from energy consumed by data centers to aguoridthic discrimination.

Some of these are kind of progressive coded up to dystopian but not quite bocalyptic outcomes where extremely powerful corporations, hand in hand with AI models kind of deprive the world of much of its freedom and agency.

Right, how compatible is everyone dies with those other concerns, because sometimes they seem to be like, you know, I read a lot on the Internet.

I write about politics all the time, right, and I think the kind of average politics argue we're on Twitter or Blue Skies, like these weird sci fi nerds.

I think it's going to be the terminator, right when really we should be to care about intrenching discrimination.

And you know, you would think that the left would be on the evil corporations bandwagon.

But because like AI is kind of tech pro coded, that scrambles it a little bit.

What do you think about those ordinary concerns including by the way AI being not all that competent, and so humans entrusting critical systems to AIS that fail or are corruptible or fuck up.

Yeah, I know.

One piece of this puzzle I think is I think a lot of people are cynical towards the companies.

I think that's often merited, but I think the cynicism is often a little bit misplaced, and they're like, oh, you know, the AIS all hype, it'll never go anywhere, it can't do any You know, there's a lot of ways that the current AI is dumb, but it's much easier to see how AI could take a leap in the future if you've seen AIS take a leap in the past.

And these companies existed before chatbots.

They see chatbots as a stepping stone.

Their stated goal is to, you know, make eyes that can automate all human labor, and in some sense an even more cynical perspective from a certain point of view of like, these guys know they're risking everyone's lives and the rushing head anyway, and you know they're not shy about this.

Sam Altman's talking about the dangerous here has gone down as the company has gotten more profitable, and it's sort of like the ones who are like more known for just saying what they actually believe, like Elon, which, for all that you can say about him, he's not really one to hide his thoughts.

Who will come out and be like, oh yeah, ten twenty percent chance this kills you all.

So one thing I would say there is like I.

Sort of.

Think it's even crazier than it looks on the surface.

AI may be a bubble, it may be hype.

The Internet had a bubble, the Internet was still real, you know, And you have all these academics being like, oh yeah, this is really dangerous.

You have folks like us Oup and around before them being like, oh yeah, this is really dangerous.

I would implore people to be like even more cynical about taking these people out their word that they are gambling with your very life.

In terms of all of the other issues with AI, including ones we're sort of already starting to see today, unfortunately, the world is big enough for multiple issues.

I'm not like, my issue is real.

Those issues are fake.

We're going to need to figure out how to integrate AI with education.

It's true that if we make like robot police forces that would allow it to talitarian states to be much more brutal in en forcing it to talitarian regime because there's certain orders you can't give to a human police force because they won't do it, that you could give to a robotic police force.

Right, those are real things we're gonna have to grapple with.

Also, if we keep racing to ais that are smarter and smarter that can automate this process of technological innovation, which is what they say they're racing towards, the most likely outcome is that everyone on Earth dies.

We just need to solve both set of problems.

Yeah, I've updated.

When I mean positive, I mean the p doom risk is going down, right, I've updated slightly positively as far as I think timeline is going to be a little longer than I always thought a year or two ago.

I've updated negatively based on culture and politics, right on various frints.

Yeah, Number one, the kind of workaholic culture of Silicon Valley.

When I hear like the nine ninety six culture, whatever it is, right, but just crying.

Go go to your lab, work really hard.

Your friends should be in the lab, right, Your activity should be things that are health oriented.

Or self improvement oriented.

It's like, I know, I feel better if we had the tech bros going out and getting drunk on Friday night, right and trying to get laid and getting in touch with humanity, you know what I mean, And if they had like material things and children and things like that that they valued potentially.

Yeah, I mean, like, what would you say, has happened to you know?

Where would you put Elon now?

Right?

I believe that GROC is not regarded or XAI is not regarded as among the most responsible of the AI labs.

Right, he has expressed, I assumed sincere concerned about p doom in the past, where would you where would you peg Elon now?

I mean he's not You can just listen to his own words, you know.

He said over the summer, I think realized I could either be a bystander or a participant, and that was like gonna happen one way or the other.

Needed rather be a participant.

But you know, also in terms of who's the safer one, who's the more dangerous one?

It it looks to me like no one has anywhere nearing a chance of doing this job, right.

Yeah, I mean you kind of it feels like you hear less about alignment than you did a year or two.

It kind of felt like Trump one and everyone's like, oh, we're just accelerating now, right, Trump wins and China's becoming more competitive, and like, you know, it seems like people have given up on this whole alignment alignment thing, right, Is that true or well or yeah?

Yeah, I mean I think a lot of it is that people like are largely realizing they can't.

Yeah, right.

Like you, you have interpretability, which is like trying to figure out what the heck is going on in there.

You have evaluations, which is trying to figure out how dangerous it is already.

And if if you were going to.

Engineers and you were like, can you tell me about why this uh, this nuclear power plant is not going to melt down?

And they're like, well, yeah, we have two safety teams.

One is trying to figure out what the heck is going on inside the plant.

The other is trying to measure whether it has exploded yet.

You would be like, okay, guys, ease off on the growing pile of uranium, you know, And I'm like, I'm not.

I'm not trying to like, uh degrade the people working on this.

It's way better to have a team that's trying to figure out what's going on inside the nuclear power plant than to not, okay, it's it's way better to have someone measuring is it starting to explode?

You know, the guys trying to do this are heroes in my book, but like it, we're not close, and I think part of why people aren't talking about is that we're not We're not close to being able to do the job right.

Do you think that some of the tragedies that are happening, you know, every day with chatbots, you know, mental health problems, we see suicides, like we're seeing a lot of things happening even with this fairly primitive, even though it's so advanced technology.

Do you think that that will be kind of enough of a warning sign for people to start taking note outside of the Silicon Valley elite who actually know this more closely, who have you read your book, who understand all of the implications.

You know, I'm hopeful in some sense the chat GPT moment was more global awareness of the AI potential for killing us all than I expected to maybe ever get.

You know, it's back before large language models a lot of people predicted that AI that could sort of like talk well was actually very late in the AI game.

You know, we used to have like mor of x paradox where we'd say things that are like easy for computers are are hard for.

Humans, and vice versa.

And you know, it could have been that this just all stayed like very academic, very like only people in the industry are watching it like it was up until you know, maybe twenty twenty one.

It could have stayed that way all the way up until you know, you have these AI companies making AI researcher ais in their labs and no one's noticing.

And that gives me some hope that maybe we don't even need a disaster.

Maybe we just need another capability advance.

Maybe when people do live through, maybe people live through one more hop of like AI's doing stuff they qualitatively couldn't do before.

Maybe that would be enough to wake people up.

And then on the flip side, you know, there can also be big warning shots where people fail to learn from the warning shot.

One thing I'd say there is that what you can learn from a warning shot, it depends a little bit on how prepared you already are and how knowledgeable you already are, and so you know there's no substitute for sort of getting people to understand the issues today.

How much of the things we need to do as a society right to mitigate this risk are kind of technical steps versus political steps, like obviously, yeah, I'm sure you've had this elevator pitch where the first sixty seconds explaining why you're worried.

Now, let's say I encounter you on the same elevator.

We're now going back to our rooms.

But I have another sixty seconds, right, what's the quick pitch on what we should do?

Is there a one kind of genie wish if you could fiat something?

Is it more complicated than that?

Do we know what we have to do and it's a matter of forming the right political coalition or all right, you have sixty ish second snake.

I think what needs to happen is the world needs to stop the race towards smarter than human AI, and it needs to stop it everywhere, because no one can keep these things in a leash.

If anyone builds a super intelligence anywhere on the planet, everyone dies, doesn't matter who gets it.

The world could stop AI is made using extremely specialized chips in very large data centers that suck down as much electricity as a city.

It would be probably easier to monitor than uranium.

And I think it's I think it's almost entirely political right now.

I think we are nowhere near solving on the technical side.

It's not to say it can't be solved technically, but not only are we nowhere near close, But unlike a lot of other scientific problems, we don't get free tries the first time you have some theory of you know, this, this AI will definitely be nice to us once it's smart enough to be a threat to us.

If that theory is wrong, you sort of don't get to put the genie back in the bottle.

And that's unlike other theories.

That's unlike other science and a lot of other places in science.

You know, Marie Curie dies of cancer those likely caused by radiation poisoning.

She was a hero, But often the first people interacting with stuff die of them, you know, Max Valley, and.

They probably did things that were objectively kind of crazy, right, But you have this selection mechanism where you know, the great crazy invention survive the other ones, don't.

I mean how much would you have to So do you believe in the idea of like a slowdown?

Right, Like, let's say that the people who are skeptical of rapid time horizons are correct and this technology develops over the course of fifty years instead of fifteen or five or five months.

Is that enough time to get things right?

Would it pause or a slow down work?

You know, fifty is a lot more to work with than fifteen.

But you got to worry about convergence on the wrong on a bad solution, and humanity so far has not been bringing its a game and to get around on the first try, it's it's hard.

Fifty years, I could work with you, maybe have a chance, but I'd be trying to do things like could we use biotech to somehow enhance the intelligence of our humans?

But I mean, first thing we do is we get a slow down somehow.

Maybe we'll be liking get a slow down.

Maybe we'll need to get a slow down by international FIAT.

But a lot of people say, oh, it will never happen.

People never slow down.

One big reason for hope here, I think, is that our world leaders don't yet understand what the people in the valley are debating.

They don't understand that we're in this situation where the possibilities are, like the tech just doesn't work, in which case betting and it will go nowhere, or you're either making god emperors or killing us all, and that the optimistic folks who think the tech works think there's like a twenty five percent chance it kills us all.

You know, it's like, yeah, that's like we're racing, but we're racing in a situation where people don't realize that either this race goes nowhere or go to really bad places.

Let me ask you kind of a meta question, what do you see your role as in this debate?

I think my role, in Eliezer's role in this situation is to come out and say the things that a lot of people are thinking, and to sort of like to be the kid who says the emperor has no clothes.

There's a lot of people who are worried here, but no one's sort of coming right out and saying this is crazy.

We should stop.

The labs who say there's a twenty five percent chance this kills us all, don't then do the next step and say so, please put a stop to it.

Right, My role here is to sort of like spell out that last step and be like, guys, this is nuts.

We shouldn't be doing it.

It's the obvious implication of what everyone else has been saying.

So you'll be I wish it was not stepping up.

But if you and Eliezer are at the Utopian retirement home in one hundred and fifty years, because now human beings lived to you know, one hundred ninety years old on average or something, and your predictions are totally wrong and AI is absolutely wonderful.

Okay, good.

I would absolutely love to be wrong.

I would I would be so thrilled.

Yeah, no spoilers about my book, but it ends with us talking about how thrilled we would be.

To be wrong.

Okay, I just sent you a calendar invite for October thirtieth, two thousand and thirty five, where we can have a follow up, right podcast, Nate Perfect any closing thoughts.

Has been a lot of fun to be with you.

Yeah, I mean, first and foremost, glad that you're you're having the convo.

Glad that we're having the convo.

I think just more people realizing how crazy the situation is will go a long way.

Towards making things better.

Thank you so much for joining us, Nate, it's been an absolute pleasure, pleasure to be here.

We'll talk to you soon.

Let us know what you think of the show.

Reach out to us at Risky Business at pushkin dot FM.

Risky Business is hosted by me Maria Kanakova.

And by me Nate Silver.

The show was a cool production of Pushing Industries and iHeartMedia.

This episode was produced by Isaac Carter.

Our associate producer is Sonya gerwit Lydia, Jean Kott and Daphne Chen are our editors, and our executive producer is Jacob Goldstein.

Mixing by Sarah Bruger.

If you like the show, please rate and review us so other people can find us too, But once again, only if you like us.

We don't want those bad reviews out there.

Thanks for tuning in.