AI researcher Nate Soares describes how he talked to people in the industry about the threat from superintelligence for a decade. He took it to politicians. They were like: "That's crazy. We shouldn't allow that to happen."
"And I've been talking to people in the AI business about these issues for over a decade. And they often don't want to hear it. They have all these objections like, oh, well, won't the AI automatically turn out nice for this reason? Won't it always have to listen to what we say for that reason, blah, blah, blah? And I would have these like big long arguments. And then when I first started going to talk to politicians, I would sort of lay out the basic issues of like these guys are trying to make machines that are radically smarter than any human. We're growing these things. We don't understand what's going on inside them. We have no ability to sort of like make them actually care about what we want them to care about. It's possible in principle for these machines to get to the point where they can go toe to toe with humanity as a whole, not individual humans, but like outstrip humanity, the ability to make their own infrastructure, make their own technology, make their own civilization. This is a crazy thing to be rushing into. And I went into these conversations prepared for a really long back and forth. And a lot of these politicians were like, oh, that's crazy. We shouldn't allow that to happen. And I was like, yeah. And also what happened to the three hours of back and forth and all of these, you know, like, what about this? What about that? And it sort of turns out that people have a harder time understanding something when they are being paid a ton of money to not believe it. You know, and so it turns out that actually a lot of people really can't just understand the issue of like, hey, maybe if you're racing to create a radically smarter set of machines that nobody understands, that just might go wrong. Turns out that's kind of easy."
💬 Discussion
AI researcher Nate Soares describes how he talked to people in the industry about the threat from superintelligence for a decade. He took it to politicians. They were like: "That's crazy. We shouldn't allow that to happen."
