× SPECIAL FORCES      |     LOCKSPORT      |     SCI FI      |     INSECTS      |     RECIPE      |     RED DEAD REDEMPTION      |     WOODWORKING      |     FISHING      |     SCI FY      |     CRIME STORIES      |     BIRDS      |     STYLE      |     TV      |     INTERESTING      |     ENVIRONMENT      |     STAR WARS      |     OLYMPICS      |     EASTER      |     GOD OF WAR      |     FOOD & DRINK      |     PHOTOGRAPHY      |     WEDDING      |     COLD WAR      |     WAR      |     CHRISTMAS      |     ARCHITECTURE      |     FITNESS      |     ART      |     TOYS      |     SKATEBOARDING      |     YOUR HOME      |     PSYCHOLOGY      |     RELIGION      |     HEALTH & WELLNESS      |     PETS      |     GUNS      |     MATH      |     BIOLOGY      |     WW1      |     BUSINESS      |     WORLD WAR 2      |     AUTOMOTIVE      |     STEM      |     GUITAR      |     AQUARIUM      |     HOMEBREW      |     KITCHEN      |     SPEECHES      |     KIDS      |     ECONOMICS      |     PROGRAMMING      |     REMOTE CONTROL      |     MINECRAFT      |     MOTORBIKES      |     VALENTINE'S DAY      |     MULTIPLICATION      |     LAWNCARE      |     GARDENING      |     PI DAY      |     LEGO      |     BLOCKCHAIN      |     COMPUTER SECURITY      |     SPORTS      |     CATS      |     MUSIC      |     FORTNITE BATTLE ROYALE      |     FARMING      |     ARTIFICIAL INTELLIGENCE      |     ROCKCLIMBING      |     RELAXING      |     CSGO      |     BABY      |     AVIATION      |     MENS STYLE      |     ANIMALS      |     SCIENCE      |     DOGS      |     SHOOTING SPORTS      |     MOVIES      |     OUTDOORS      |     BIKING      |     HISTORY      |     FASHION      |     FAR CRY 5      |     HALLOWEEN      |     STPATRICKS      |     CRYPTOCURRENCY      |     ENGINEERING      |     GEOPOLITICS      |     POLITICS      |     TECHNOLOGY      |     FUNNY      |     TRAVEL      |     MARTIAL ARTS      |     GAMBLING      |     GAMING      |     KNITTING      |     BEAUTY      |     PHILOSOPHY      |     ANCIENT WORLD      |     POTTERY      |     ARCHERY      |     THANKSGIVING      |     HOME REPAIR      |     CRAFTS      |     ORGANIZATION      |     BIG THOUGHTS

  • 00:02
    MODERATOR: Today with us we have Professor Nick Bostrom.
  • 00:07
    He was born in Helsingbrg in Sweden.
  • 00:10
    He's a philosopher at St. Cross College at the University of Oxford.
  • 00:15
    He's known for his work on existential risk, the entropic principle, human enhancement ethics, the reversal test, and consequentialism.
  • 00:24
    He holds a Ph.D. from the London School of Economics, and he is the founding director of both the Future of Humanity Institute and the Oxford Martin Programme on the Impacts of the Future on Technology.
  • 00:37
    He's the author of over 200 publications, including the book we are presenting today, "Superintelligence." And he has been awarded the Eugene R. Gannon Award and has been listed in "Foreign Policy"'s Top 100 Global Thinkers list.
  • 00:53
    Please join me in welcoming Professor Nick Bostrom.
  • 00:56
    [APPLAUSE] NICK BOSTROM: Yeah, great.
  • 01:03
    Thanks you for coming.
  • 01:04
    And I'm not going to try to summarize the entire book, but I want to give some of the background from which this work emerges.
  • 01:14
    So I run this thing called The Future of Humanity Institute, which has a very sort of ambitious name.
  • 01:19
    It's a small research center where mathematicians, philosophers, and scientists are trying to think through the really big-picture questions for humanity, ones that often traditionally have been relegated to crackpots, relegated to journalists or retired physicists to write some popular book.
  • 01:39
    But questions that are actually extremely important and, I think, deserve close attention.
  • 01:45
    So to give some sense of where I'm coming from, one can think about the human condition, in grand schematic terms, on a diagram like this, where we plot time on the x-axis, and then on the other axis, some measure of capability, like level of technological advancement.
  • 02:03
    Measure of the overall economic productivity that we have at some given point in time.
  • 02:07
    And what we take to be the normal human condition-- the idea that you wake up in the morning and then commute to work and sit in front of a screen, and you're having too much rather than too little to eat, is a narrow band within this much larger space of possibilities.
  • 02:26
    It's obviously an anomaly on evolutionary time scales.
  • 02:30
    The human species is fairly young on this planet.
  • 02:32
    It's also an anomaly on just historical time scales.
  • 02:35
    For most of human history, we were inhabiting a Malthusian state, and it's really only in the last few hundred years that we've kind of soared up.
  • 02:45
    And even then, just in some parts of the world.
  • 02:47
    It's also a huge anomaly, obviously, in space, the little crust of [INAUDIBLE] planet being very different from most of the stuff around us, which is just a ultra-high vacuum.
  • 02:58
    And yet we tend to think that this is the normal way for things to be, that any claim that things might be very different is a radical claim that needs some extraordinary evidence.
  • 03:07
    Yet it's possible, if we reflect on it, that the longer the time scale we're considering, the greater the probability that we will exit this human condition in either one of two ways, downwards or upwards.
  • 03:20
    This picture has two attractor states.
  • 03:23
    So if we exit the human condition in the downwards direction-- there is in population biology the concept of a minimum viable population size.
  • 03:31
    With too few individuals left, they can't sustain themselves.
  • 03:34
    There's an attractor state down there, which is extinction.
  • 03:37
    Once you're extinct, you tend to stay extinct.
  • 03:40
    And more than 99.9% of all the species that once flew, crawled, or swam on this planet are extinct, so that certainly is one possible future.
  • 03:51
    Another way that the human condition could end would be that we exit in the upwards direction.
  • 03:57
    And there, too, I think that there is an attractor state.
  • 04:00
    If and when a civilization manages to obtain technological maturity-- meaning, we have developed most of all technologies that are physically possible to be developed, and we have the ability to spread through space in a reasonable way, through automated self-replicating colonization probes-- then the destiny might be pretty well set.
  • 04:25
    The level of existential risk will go down, and maybe we could continue on like that for millions and billions of years, just growing at the significant fraction of the speed of light indefinitely, until the cosmological expansion makes it impossible to reach any further resources.
  • 04:43
    Like if something is too far away today, then by the time we would get there, as it were, it has moved further away.
  • 04:48
    So there's a finite bubble of stuff that something starting from what we are could, in principle, get.
  • 04:56
    So that whole bubble of stuff, I call humanity's cosmic endowment.
  • 05:01
    There is a lot of it there.
  • 05:03
    And that might be another possible attractor state.
  • 05:08
    If one has to view that-- that this stuff that could, in principle, be reached there is very important, if one has some kind of view on ethics, where fundamentally, the moral significance of an experience or a life does not depend on when it is taking place-- just as many people have the belief that from moral point of view, it doesn't matter where it takes place.
  • 05:37
    Like if you travel to Africa and you suffer there, it's as bad as if you were suffering here.
  • 05:42
    If one has that view about time, then this cosmic endowment matters a lot.
  • 05:49
    Because it could be used to create an enormous amount of value.
  • 05:55
    We can count it up, roughly.
  • 05:58
    We know that there are billions of galaxies, each with billions of stars, and each of those stars could have billions of people living around it for billions of years.
  • 06:07
    And you get an enormous number of orders of magnitude if you try to measure the size of the future, even just assuming biological instantiations of minds.
  • 06:15
    If you're imagine a more efficient digital implementation, you can add another big chunk of orders of magnitude.
  • 06:22
    And so what you find fairly robustly, if you just work the numbers, is that if you have this evaluative view, some broadly-aggregated consequentialist view, then even a very, very small reduction in the net level of existential risk will be worth more, in expected utility terms, than any interventions you could do that would only have local effect here.
  • 06:44
    Even something as wonderful as, like, curing cancer or eliminating world hunger would really be, from this evaluative perspective, insignificant compared to reducing the level of existential risk by, say, 1/100th of one percentage point.
  • 06:58
    So this level of existential risk becomes, then, maybe an important lens through which to look at global priorities.
  • 07:06
    I define it as a risk that either threatens the survival of Earth-originating intelligent life, or threatens to permanently and drastically destroy our potential for desirable development.
  • 07:21
    And now I think that maybe in a complete accounting of ethics, there are other variables, as well, to take into account.
  • 07:27
    We have particular obligations to people that are near and dear to us.
  • 07:31
    There might be other things in addition to this sort of aggregated consequentialist component.
  • 07:35
    Nevertheless, I think it's in there and it's important.
  • 07:38
    So if one then tries to look more carefully at this category of existential risk-- like, what could actually go wrong in this way?
  • 07:46
    What could permanently destroy our future?
  • 07:50
    It's a very small subset of all the things that can go wrong.
  • 07:53
    Like most things that threaten human welfare don't really create any existential risk.
  • 08:00
    So it kind of narrows down the range of concerns quite significantly.
  • 08:04
    A first distinction that is obvious in this field is the distinction between risks arising from nature and risks arising in some way from human activity.
  • 08:14
    And a fairly robust result, I think, is that all the big existential risks, at least if we are thinking a time scale of 100 years or so, are anthropomorphic, arising from human activities.
  • 08:26
    And one can see that just by reflecting that the human species has already been around for 100,000 years.
  • 08:30
    So if firestorms and earthquakes and asteroids haven't done us in in the last 100,000 years, probably not going to do us in in the next 100 years.
  • 08:39
    Whereas we will be introducing entirely new kinds of hazards into the world that we have no track record of surviving.
  • 08:47
    And more specifically, I think all the really big existential risks are related to certain anticipated future technologies that we might well develop over the coming decades or 100 years.
  • 09:00
    And another way to-- another framing that makes a similar point is to think in terms of this metaphor of a great urn which contains a lot of balls.
  • 09:11
    The balls represent different technologies that can be discovered, or more broadly, the different ideas that we can invent.
  • 09:18
    And throughout human history, we have reached into this ball repeatedly and pulled out ball after ball.
  • 09:24
    And on balance, all these discoveries, all these ideas, have been an immense boon for us.
  • 09:30
    It's because of all these ideas that we now live in abundance, and why there can be seven billion of us.
  • 09:38
    There have been some discoveries, perhaps, that have been mixed blessings, that have done both good and ill.
  • 09:44
    And a relatively small number-- it's not totally trivial to think about it, but balls that we would have been better off without having extracted from this urn, like discoveries that we would be better without.
  • 09:55
    I mean, maybe chemical weapons, say, or nuclear weapons.
  • 09:59
    Or perhaps, like, torture instruments of different kinds.
  • 10:03
    There are few things that seems to have been clearly bad.
  • 10:08
    But there hasn't been any discovery so far made that is such that it automatically destroys the civilization that discovers it.
  • 10:15
    So there hasn't been any black ball pulled out from this urn.
  • 10:18
    And we can ask what that kind of discovery could look like.
  • 10:23
    What would be a possible discovery, such that it kind of almost automatically spells the end of the discoverers?
  • 10:31
    And it might be useful here to think of a counter [INAUDIBLE].
  • 10:34
    So we discovered, just over half a century ago, nuclear weapons.
  • 10:39
    And it turned out, fortunately, that in order to make a thermonuclear device, you need some difficult-to-obtain raw materials.
  • 10:47
    You need highly enriched uranium or plutonium.
  • 10:50
    And the only way to get those is by having some large facility that's very expensive to build, takes a lot of energy, is easy to see.
  • 10:58
    So very few people can build their own nuclear device.
  • 11:02
    But suppose it had turned out to be differently.
  • 11:05
    Suppose it had turned out instead that it had been possible to make a thermonuclear warhead by some simple procedure, like baking sand in your microwave oven, OK?
  • 11:16
    So now we know physics doesn't allow for that.
  • 11:19
    But before you actually did the relevant physics, how could you possibly have known whether particle physics would have provided some easy route to unleash these kinds of entities?
  • 11:28
    So if that had been the case, then presumably that would then be the endpoint of human civilization.
  • 11:33
    Once it became so easy to destroy entire cities, we could never again have cities, and maybe we would have been knocked back to the Stone Age.
  • 11:42
    And by the time we would have again climbed back up to the technology level where somebody could build microwave ovens, we would presumably fall back again.
  • 11:50
    And that might be forever the end of it.
  • 11:54
    But so we were lucky on that occasion, but the question is whether we will continue to be lucky always.
  • 12:00
    Like, whether in this big urn, if we keep extracting ball after ball, whether eventually we will pull out the black ball.
  • 12:08
    If there is a black ball in there and we just keep pulling them out, then eventually, presumably, we will get it.
  • 12:12
    And we don't yet have the ability to put a ball back in the urn.
  • 12:16
    We can't undiscover things that we have discovered.
  • 12:20
    So here is a kind of quick list of some areas where one might suspect that there could be these kinds of black balls.
  • 12:29
    And AI is one that I'll kind of come back to more in this talk.
  • 12:34
    There are some others.
  • 12:35
    Synthetic biology will, I think over the coming decades, vastly increase the powers of human beings to change the world around us and ourselves.
  • 12:46
    Those powers might be used wisely or not.
  • 12:49
    Molecular nanotechnology.
  • 12:51
    Not the kind of thing that makes car tires today, but some kind of more advanced future version of that, like Eric Drexler imagined.
  • 13:00
    Totalitarianism-enabling technology.
  • 13:02
    So remember, again, the definition of an existential risk-- not only extinction scenarios, nice but also ways to permanently lock ourselves in to some radically suboptimal state.
  • 13:13
    And you can imagine that maybe new technological discoveries that make surveillance very easy, or some new discovery that makes it possible, through psychological or neurophysiological techniques, to modify desires could sort of change some of the parameters of the sort of sociopolitical game, where new types of social organization becomes a lot easier to establish and maintain.
  • 13:38
    Human modification, geoengineering.
  • 13:41
    There are more you could add there, and I've left a lot of these bullets below here on the list unknown.
  • 13:46
    So it's useful to reflect that if this question-- what are the biggest existential risks?-- had been asked 100 years ago, then presumably none of the ones that I now would place close to the top would have been listed.
  • 14:00
    They didn't have computers, so they wouldn't have listed machine intelligence.
  • 14:03
    Synthetic biology was not even a concept, nor nanotechnology.
  • 14:07
    Maybe they would have worried some about, like, totalitarian tendencies.
  • 14:12
    But the others, not so much.
  • 14:13
    So if we reflect on our situation today, we have to maybe acknowledge, from standing outside and looking in, that there are probably some additional existential risks that are not yet on our radar but that could turn out to be as significant as some of the others.
  • 14:28
    Which as I said, there could be high value to doing analysis on this and research to try to find them out.
  • 14:36
    But if one combines these considerations with one other hypothesis-- say, a mild or moderate form of technological determinism, which I think is true-- the idea, basically, that assuming science and technology continues, there's no global collapse, then eventually we'll probably discover all technologies that could be discovered.
  • 15:03
    At least all general-purpose technologies that have a lot of implications in many fields.
  • 15:09
    I think that's fairly possible.
  • 15:10
    It's not assured, but that level of technological determinism seems quite possible to me.
  • 15:15
    It's a little bit like-- if you think of a big box that starts out empty and you pour in sand in it, this is like, you can fund one kind of research here.
  • 15:24
    You can fund another kind of research.
  • 15:25
    And what research you fund, where your priorities are, that determines where the sand piles up in this box.
  • 15:33
    So you get different technologies, depending on what you do.
  • 15:35
    But over time, if you just keep pouring in sand, then eventually the whole box will fill up.
  • 15:39
    And so that seems fairly possible.
  • 15:41
    Now if one has that view, then how should one kind of-- what attitude should one take to all of this?
  • 15:46
    Like what should we do?
  • 15:48
    So one possible response is one I think best expressed by this blogger.
  • 15:55
    I don't know who it is.
  • 15:56
    Washbash commented on some blog that "I instinctively think go faster.
  • 16:00
    Not because I think this is better for the world.
  • 16:02
    Why should I care about the world when I'm dead and gone.
  • 16:05
    I want it to go fast, damn it!
  • 16:06
    This increases the chances I have of experiencing a more technologically advanced future." So here we've got to be clear what exactly the question is So if the question is, what would be best for me personally?
  • 16:21
    What should we favor or hope for, from a egoistic point of view?
  • 16:26
    Then I think that Washbash is correct.
  • 16:33
    From an individual point of view, if-- well, first of all, if you're somehow hoping for these cosmic lifespans of millions of years, and being able to travel and expand into the universe, then clearly that's not going to happen unless something radical changes.
  • 16:48
    Like the way things are going, I'm sad to say, we're all just going to die from aging in a few decades.
  • 16:54
    Like we're all rotting.
  • 16:56
    So the only way that that could possibly change is some radical upset.
  • 17:00
    Like some cure for aging, or uploading into computers, or something really radical would have to happen to kind of thwart that.
  • 17:07
    So that would be reason to favor faster technical growth.
  • 17:10
    Or even, if you're despairing of that, even you could just hope to have more interesting gadgets around and a higher standard of living, which we can hope for through technology.
  • 17:20
    However, if the question we ask is instead, what would be best from an impersonal point of view?
  • 17:26
    Then I think the answer is quite different, something perhaps closer to this principle of technological development, rather than maximize the speed with which you rush ahead.
  • 17:40
    This principle would say that we should "retard the development of dangerous and harmful technologies, especially ones that raise the level of existential risk, and accelerate the development of beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies." So the idea here is that rather than asking the question for some hypothetical technology, would we be better off without it?
  • 18:05
    We ask a different question.
  • 18:06
    Because basically, on this moderate form of technological determinism, that's just not on the table.
  • 18:12
    We can't relinquish a technology permanently.
  • 18:14
    But what we should think of instead is on the margin, should we try to hasten the arrival of some technology or slow it down?
  • 18:21
    We might be able to make some difference there, say, by a couple months.
  • 18:26
    And we want to think about how that small difference will influence our likelihood of harvesting this cosmic endowment.
  • 18:34
    If you think that it was literally impossible to even make a small difference in the timing, then that would mean that all the funding and all the effort that goes into technology development would just be wasted.
  • 18:45
    So presumably we think we have some ability to at least move things around in time.
  • 18:52
    And the principle of differential technological development suggests that it might be quite significant, sometimes, exactly when different things arrive, particularly the sequence in which different technologies arrive.
  • 19:03
    So if there's going to be, at some point, like a really harmful bio-engineered pathogen, and it could spread really easily and it's very lethal, and there's going to be, at some point, like a universal vaccine, you want to invent the vaccine before you invent the pathogen.
  • 19:20
    If there's going to be, at some point, machine superintelligence, and if there is some possible technology that could assure the safety of machine superintelligence, you want the latter to come before the former.
  • 19:33
    AUDIENCE: There's an argument that trying to retard development of technology would make it more dangerous because you're driving it underground, or you have less opportunity to do it out in the open, develop safe [INAUDIBLE].
  • 19:49
    Like we had, for example, the Asilomar guidelines in biotech, which have actually been very effective.
  • 19:54
    For 30 years, there's been no accidents.
  • 19:57
    And if you drive these technologies underground, you don't have the opportunity to have those kinds of safeguards.
  • 20:02
  • 20:03
    This-- the principal leaves open whether you should focus on the retarding or the accelerating.
  • 20:09
    If you wanted to retard, maybe one way would be just to refrain from like funding it or actively devoting yourself to accelerating it.
  • 20:19
    With regard to AI, which I'll get to later, I think definitely accelerating the work on the safety problem is clearly the way to go.
  • 20:26
    I think it's just a lot easier to make a big difference there than to try to somehow retard the development of AI work itself.
  • 20:35
    Maybe we can return to that more in the Q&A.
  • 20:39
    So we have a picture, perhaps, like this, where again, we're looking at three axes here.
  • 20:44
    So technology on one, this is the same capability on the earlier slide.
  • 20:50
    Coordination-- some measure of the degree to which humankind is able to solve a global coordination problems.
  • 20:58
    Avoiding wars and arms races, and polluting our communal resources.
  • 21:04
    And insight-- so a measure of our understanding into what uses of our capability would actually make things better.
  • 21:14
    So it might well be that in order to have the best possible outcome, to have Utopia, that we need maximum amounts of all of these.
  • 21:24
    Like super-duper advanced technology is necessary to realize the best state; great coordination, so we don't use that technology to wage war against one another, as we have through so much of human history; and great wisdom, so that we apply all these abilities to really do things that are worthwhile.
  • 21:39
    So that might be where we have to be, if we want to realize the best possible state.
  • 21:43
    Now that then leaves open the question of whether from the position we are currently in-- at the moment, we would be better off with faster developments in each of these areas.
  • 21:53
    It might be, for example, that even though we ultimately want maximum technology, we would be better off getting that technology only after we have first made more progress on global coordination or wisdom.
  • 22:09
    Anyway, so that's by view of like a broader context.
  • 22:13
    So we are thinking about other existential risks and stuff like that.
  • 22:18
    And superintelligence, as I will talk about, I think is one big existential risk, perhaps.
  • 22:25
    Perhaps, arguably, perhaps the biggest.
  • 22:27
    I'm not sure.
  • 22:28
    But it's peculiar in one respect, that although it's a big danger in its own right, it's also something that could help eliminate other existential risks.
  • 22:37
    So if we imagine like a very simple model, where we have synthetic biology, nanotechnology, and AI-- we don't know which order they will come.
  • 22:44
    Maybe they each have some existential risks associated with them.
  • 22:48
    Suppose we first develop synthetic biology.
  • 22:50
    We get lucky and we get through the existential risks, however big they are.
  • 22:54
    And then we reach molecular nanotechnology, and we are lucky.
  • 22:57
    We get through that, as well.
  • 22:58
    And finally, AI, the existential risks along that path are kind of the sum of these three different ones that we'll each have to surmount.
  • 23:08
    In another trajectory, maybe we get AI first, and we have to face existential risk for that.
  • 23:12
    But then if we do get lucky there, we no longer have to face the risk with synthetic biology and nanotechnology, because we don't have the superintelligence to help us through.
  • 23:20
    So in reality, it gets a lot more complicated than that, and we can discuss the intricacies more in the Q&A.
  • 23:28
    But thinking about the sequencing and timing, I think, rather than yes or no, would we want the technology or not, is like an initial, necessary first step to be able to have any kind of meaningful conversation about this.
  • 23:38
    So superintelligence, I think, will be a big game-changer, the biggest thing that will ever have happened in human history, at some point, this transition to superintelligence.
  • 23:47
    There are two possible pathways, in principle, one could imagine that could lead there.
  • 23:52
    You could enhance biological intelligence.
  • 23:55
    We know biological intelligence has increased radically in the past, in kind of making the human species.
  • 24:05
    Or machine intelligence, which is still far below biological intelligence, insofar as we're focusing on any form of general-purpose smartness and learning ability, but increasing at a more rapid clip.
  • 24:21
    So specifically, you can imagine interventions on some individual brain to enhance biological condition.
  • 24:29
    I'll say a few words about that just shortly.
  • 24:33
    Or improvements in our ability to pool our individual information processing devices to enhance our collective rationality and wisdom.
  • 24:41
    I won't talk about that, but that's clearly an exciting frontier, with the internet and new institutions, prediction markets and other things like that.
  • 24:52
    There are some kind of hybrid approaches, it can vary between biology and machines, the cyborg approach.
  • 24:59
    I personally don't think that that's where the action will be.
  • 25:04
    It just seems to me very difficult, technologically speaking, to create implants that would really significantly enhance our cognitive ability more than you could have by having the same device outside of yourself.
  • 25:18
    So you could say, wouldn't it be great with a little chip in the brain, and you could Google just by thinking about it?
  • 25:23
    And well, I mean, I can already Google, and I don't have to have neurosurgery to be able to do that.
  • 25:29
    We have these amazing interfaces like the eyeballs, that can protect 100 million bits per second, straight into dedicated neural wetware that's highly optimized for making sense of this information.
  • 25:41
    And it's really hard to beat that, I think.
  • 25:44
    In any case, I mean, the rate at which sensory information can be entered into the brain is not really the limiting factor.
  • 25:51
    The first thing the brain does with all of this visual information is to throw away almost all of it and just extract the relevant part.
  • 25:59
    And then different versions of machine intelligence, where on the one hand, we have sort of purely synthetic methods that don't care about biology but try to make progress in mathematics and statistics and figure out clever algorithms.
  • 26:15
    And two, approaches that try to learn from this one general intelligence system that already exists, that we can study the human brain for inspiration from that, or maybe even reverse-engineer it.
  • 26:26
    Or in the limiting case, literally copying it in whole-brain emulation, where you would take a particular human brain and freeze it and slice it up, feed those slices through an array of microscopes to take good pictures.
  • 26:46
    So you have a stack of these pictures and use automated image-recognition software to extract the connectivity matrix of the neural network that's was in the original brain.
  • 26:57
    And then annotate that with neurocomputational models of each type of neuron works.
  • 27:02
    And finally, run that whole emulation on a sufficiently powerful computer.
  • 27:08
    That would require some very advanced enabling technology that we don't yet have, so we know that that is not just around the corner.
  • 27:18
    On the other hand, it would not require any theoretical breakthrough.
  • 27:22
    It would not require any new, deep conceptual understanding of how thinking works.
  • 27:25
    You would only need to understand the components of the brain to be able to make progress with that.
  • 27:33
    So it's an open question which of these will get there first.
  • 27:38
    Different researchers have their own favorite bets on that.
  • 27:42
    One thought that sometimes is put to me is that-- OK, so Nick, you're worried about this AI stuff.
  • 27:47
    So maybe what we should do is really try to push ahead with biological enhancement, so that we can kind of keep up with the computer.
  • 27:55
    The computer's going to get smarter, but maybe if we enhance our own intelligence rapidly enough, we can keep one step ahead.
  • 28:01
    I think that that's misguided.
  • 28:04
    And in fact, if we do figure out ways to enhance biological cognition, I think that will only hasten the time when machines overtake us.
  • 28:12
    Because basically, we will have smarter people doing the AI research and the computer science, and they will solve the problem faster.
  • 28:21
    I still think that would probably have reason to try to accelerate biological cognitive development.
  • 28:28
    But not so that we can keep ahead of the computers, but that so that when the time comes where we will create intelligent machines, that we will be more competent at doing so.
  • 28:38
    So let me just say a few words on this biological cognitive enhancement, because it might be-- especially if you think of arrival dates for artificial intelligence, where it's not just around the corner, but maybe it will happen in the latter half of this century or something like that-- by that time, that could be enough time to have a new cohort of cognitively enhanced people And the technology that I think will first enable cognitive enhancement-- my best guess is that it'll be through genetic interventions.
  • 29:08
    There are other paths, obviously-- smart drugs I just-- I don't hold out much hope that they will do a great deal to improve general-purpose smartness.
  • 29:18
    They might-- if there were a simple chemical that you could just inject and it would make you a lot smarter, I think evolution would find a way to endogenously produce that chemical.
  • 29:28
    I think there might be ways to improve some peripheral characteristics, like mental energy, say, or concentration.
  • 29:36
    And we can see that evolution would have optimized us for a certain type of environment where there are trade-offs between, maybe, metabolic consumption of the brain and the amount of mental energy you have.
  • 29:48
    And in the environment of evolutionary adaptiveness, the optimum point for that trade-off is at one place, and now we want to move that.
  • 29:55
    And maybe it could have some stimulant that just increases the burn rate of calories and give you more mental energy.
  • 30:00
    So peripheral adjustments like that, I think we could do maybe through drugs.
  • 30:04
    But raw cleverness, I think genetics is a more likely initial technology to do that.
  • 30:11
    And so one way that that could work is in the context of in vitro fertilization, where you have normally, in the course of standard fertility procedure, maybe some six, eight, or ten eggs produced.
  • 30:27
    And then the doctor chooses one of those to implant.
  • 30:30
    And at the moment, you can look for like obvious abnormalities.
  • 30:35
    You might screen for some monogenic disorders or for Down syndrome, which is done.
  • 30:41
    But you can't really select positively for some complex trait today, because we don't yet know the genetic architecture for, say, intelligence.
  • 30:53
    But we will, I think, soon know that, because the price of gene sequencing is falling, and it's now coming down sufficiently where it is becoming feasible to run these very large-scale studies with hundreds of thousands or even millions of subjects.
  • 31:10
    And because it turns out that the additive heritability, the variance in that in humans, is not due to like one or two genes that differ in between us, but a lot of genes-- maybe hundreds, maybe even a few thousand-- that each have a very, very small effect.
  • 31:29
    And so to discover a very, very small effect, you need a very large sample size.
  • 31:35
    And so you need to sequence a lot of genomes, and that was too expensive to do, really.
  • 31:39
    But now there are studies underway with hundreds of thousands of people, and maybe soon millions.
  • 31:44
    So that, I think, will tell us some of this information that would be needed.
  • 31:48
    And then to start doing this, nothing else would be required.
  • 31:51
    No new technologies at all.
  • 31:52
    You just have the information and you sequence it and select the embryos based on that.
  • 31:57
    Now this would be vastly potentiated if it were combined with another technology that we don't yet have ready for use in humans, which is the ability to derive gamete from stem cells.
  • 32:09
    So then you could do iterated embryo selection.
  • 32:13
    We would generate an initial pool of embryos, select one that's highest in the expected trait value of interest, and then use that embryo to derive gametes-- sperm and ova-- that you could then recombine to get a new set of embryos.
  • 32:28
    You pick out the best of those, and you repeat.
  • 32:31
    So this technology here, the ability to create artificial gametes through stem cells, has been developed and done in mice.
  • 32:42
    But a significant amount of additional work would be required to make it safe for use in humans.
  • 32:49
    But if you had this-- and this might take anything from 10 to 30 or 40 years.
  • 32:54
    It's hard to know.
  • 32:56
    This would have the effect of collapsing the human generation cycle from 20, 30 years to a couple of months.
  • 33:06
    And so if you imagine this kind of old mad scientist eugenics program where they would breed humans for like 500 years, and make very sure who mated with whom-- which setting aside all the ethical complications involved in that, which are legion, but I'm not going to talk about them here, not because I don't think they are there, but I just want to focus on the technical stuff-- it's just infeasible on a lot of different levels.
  • 33:32
    But here, you would instead have something that could be done-- instead of 500 years, you could have it done over a year.
  • 33:37
    And instead of changing the breeding patterns of large populations, you would have a Petri dish and a scientist plucking around in that.
  • 33:45
    And so through that, you would be able to probably achieve sort of weak forms of superintelligence in biology.
  • 33:55
    I did an analysis with a colleague of mine, Carl Shulman, quite recently where we tried to estimate, for different assumptions about the selection power applied, what the gain in intelligence would be.
  • 34:09
    And so you can see here that if you just produce two random embryos and select the best one-- not the one that's actually best, but the one that looks most promising, to the extent that there is an additive genetic heritability, you may get four IQ points from that.
  • 34:25
    So if instead, you select the best of 1 in 10, or 11, you can see here, even if you could select the best of 1 in 1,000, you only get maybe 24 IQ points.
  • 34:35
    This is with single-shot selection.
  • 34:37
    But if you did this iterated embryo selection, and you could do five generations of selecting the best of 1 in 10, then you might get as many as 65 IQ points.
  • 34:49
    And with 10 generations of 1 in 10, you'd get far above what we've had in human history.
  • 34:57
    You'd get the kind of phenotypes that have never existed in all of human history, the [INAUDIBLE] and stuff So you observe here that while you get quickly diminishing returns by just doing one-shot selection from a pool of embryos, you largely avoid that by doing this iterated selection.
  • 35:21
    So yeah, I'm going to skip through this.
  • 35:26
    Yeah, so that does look like it should be feasible without any sort of magical new technology coming And then that also, I think, adds to the possibility that we will eventually get AI stuff.
  • 35:36
    Like that would be towards the end of this century, if we haven't already solved the problem by then, with this significantly more capable generation of humans working on it.
  • 35:46
    But ultimately, we will be surpassed by intelligent machines, assuming we haven't succumbed to existential catastrophe prior to that, just because the fundamental image-to-information processing in the machine substrate or just far beyond those in biology, like in terms of speed.
  • 36:05
    Even transistors today are far faster than neurons.
  • 36:09
    So I'm going to-- this is not relevant, for you guys already know all of this.
  • 36:13
    So there is, like, progress in AI, like-- I'm just saying the public consciousness is shaped by a few big milestones, but there's a lot of progress under the hood.
  • 36:22
    Also hardware has driven a lot of progress we've seen.
  • 36:25
    Here is a slice that could have been earlier.
  • 36:27
    This is like with the brain emulation.
  • 36:28
    This is basically the state of the art today.
  • 36:31
    Here's a brain slice scanned with an electron micrograph.
  • 36:35
    Here is a stack of those pictures on top of one another.
  • 36:37
    And here is the result of applying an image-recognition algorithm to extract the connectivity matrix.
  • 36:44
    But although we have the right resolution-- you can see individual atoms, if you want to, in the brain.
  • 36:49
    It's just that to image the brain with that level of resolution would take, like, forever, so that we're presumably at least decades away from making something like that work.
  • 37:02
    Lot of application there [INAUDIBLE].
  • 37:04
    So the question of how far away we are from human-level machine intelligence, I think the short answer is that nobody knows.
  • 37:10
    We did a survey of leading AI experts last year, and one of the questions we asked was, by what year do you think there is a 50% chance that we will have human-level machine intelligence?
  • 37:21
    Here defined as one that could do most jobs that humans could do.
  • 37:25
    And so the median answer to that we got was 2050 or 2040, depending exactly which group of experts we asked.
  • 37:32
    That seems, to me, roughly reasonable, for what it's worth.
  • 37:37
    We also asked, by what year do you think there's a 90% probability?
  • 37:40
    And we got 2070 or 2075.
  • 37:43
    That, to me, seems overconfident.
  • 37:46
    There is just a lot more than 10% probability, I think, that we will still have failed by then.
  • 37:52
    I should say, as a footnote, that these estimates were conditioned on no global collapse occurring.
  • 38:02
    So-- so maybe the numbers would be-- like the years would be slightly higher up if we hadn't made that assumption.
  • 38:11
    We also asked, if and when we do reach human-level machine intelligence, how long do you think it will take from there to go to some radical superintelligence?
  • 38:20
    And you can see for yourself the answer there.
  • 38:22
    Now here, again, my view disagrees with those of people we sampled.
  • 38:27
    I think-- I'm quite agnostic as to how far away we are from AI.
  • 38:30
    I think we should basically have a very smeared-out probability distribution.
  • 38:34
    I do think there is a fairly large probability, though, that if and when we get to human-ish level, we will soon after have superintelligence.
  • 38:40
    I place a fairly high credence on there being, at some point, an intelligence explosion.
  • 38:46
    So we need to sharply separate the two questions, like the distance between now and human-level, and the distance in time between that and radical superintelligence.
  • 38:56
    I think this transition might well be very rapid.
  • 39:03
    And things depend on that.
  • 39:06
    So if you distinguish, like qualitatively, like fast-take of scenarios, where we go from something human-ish level to superintelligence within minutes or hours or days, a couple of weeks, in that kind of scenario, it happens too fast for us to really be able to do anything much about it while it is happening.
  • 39:26
    If we get a desirable outcome, it's because we set up the initial conditions just right.
  • 39:32
    By contrast, if one contemplates very slow take-up-- so you have some human-level system, and then only by laboriously adding additional little incremental capability after capability, so it takes like decades or centuries to work your way up to superintelligence, then that would be a lot of more time for new human institutions to arise to deal with this, like to develop a new profession of experts to deal with this, to try things out, see what works, and then change it up.
  • 40:00
    So it makes a difference.
  • 40:01
    Another way in which it makes a difference is that in the fast takeoff scenarios, it's likely that you will have a singleton outcome, I think.
  • 40:08
    Which is basically a world order where at the highest level of decision-making, there's like one decision-making agency.
  • 40:16
    If you think about competing technology projects, whether it's nations racing to build satellites, or nuclear weapons, or competing tech products, often there's some competition, and you're trying to get there first.
  • 40:27
    But it's rare that the difference between the leader and the closest follower is a couple of days.
  • 40:33
    Like usually the leader will be a few months ahead of the follower, or a couple of years.
  • 40:37
    So if the takeoff is going to be over in a few days or a few weeks, then one project will have completed a takeoff before the next one will have started it, very likely.
  • 40:47
    And then you will have a mature superintelligence in a world which contains no other even vaguely comparable system.
  • 40:53
    And for reasons that I'll be happy to elaborate on in the Q&A, and that a lot of the book is about, as well, this first system then is likely to be very powerful, maybe to the point where it is able to shape the entire future according to its preferences.
  • 41:08
    If you have a storied takeoff, then it's more likely you're going to have multiple outcomes.
  • 41:13
    No system is so far ahead of all the others that it can just lay down the law.
  • 41:17
    They end up superintelligent, but it will have economic competitive forces and evolutionary dynamics working on this population of digital minds shaping the outcome.
  • 41:30
    And the concerns in that type of scenario look very different from the ones in the singleton scenarios.
  • 41:36
    Not necessarily less serious, but different.
  • 41:40
    So instead of having one agency that can dictate the future, you now have this ecology of digital minds.
  • 41:52
    And you can think-- I mean, suppose to take a model-- so once we had human-level minds that were digital, like they could do exactly the same as humans do, and run at the same speed, initially-- suppose that you get there through whole-brain emulation, and this is the first type of AI you have.
  • 42:10
    Then you could very quickly have a population explosion.
  • 42:12
    So we know how to copy software.
  • 42:14
    That takes a couple of minutes.
  • 42:16
    And so as long as the productivity of these digital mind is higher than the cost of making another copy, there would be vast incentives to just keep making more copies until the income that digital minds can earn equals, like, the price of electricity and hardware rental.
  • 42:36
    So you have a Malthusian situation where the population of digital minds expands until the wage falls to subsistence level.
  • 42:42
    But subsistence level for the digital minds, rather than for biological minds.
  • 42:46
    So we are a lot more expensive, because we have to eat and have houses to live in and stuff like that.
  • 42:56
    So humans might still be able to make some income through their capital investment.
  • 43:03
    And there's then the question of whether in this world, which is increasingly shaped by the digital minds-- there are trillions and trillions of them, and they're getting faster all the time, and better, and humans constitute a small slice of all of this-- whether we would be able, in the long run, to really enforce property rights and our sociopolitical structures.
  • 43:22
    Or whether these digital minds would eventually just swamp us and expropriate us.
  • 43:27
    And at some point, presumably, even in this whole-brain emulation, at some point probably fairly soon after that point, you will have synthetic AIs that are more optimized than whatever sort of structures biology came up with, that will then kind of leave the [INAUDIBLE].
  • 43:42
    So there is a chapter in the book about that.
  • 43:44
    But the bulk of the book is-- so all the stuff that I talked about, like how far we are from it and stuff like that, there's like one chapter about that in the beginning.
  • 43:52
    Maybe the second chapter has something about different pathways.
  • 43:55
    But the bulk of the book is really about the question of, if and when we do reach the ability to create human-level machine intelligence-- so machines that are as good as we are in computer science, so they can start to improve themselves-- what happens then?
  • 44:10
    And what happens when you have a superintelligence that might be extremely powerful?
  • 44:13
    What are the control methods that we could try to apply to achieve a controlled detonation if there is going to be an intelligence explosion?
  • 44:22
    How could we set up the initial conditions to get some kind of beneficial outcome?
  • 44:27
    And there are a lot of initially plausible ways to solve this problem that turn out, on closer reflection not to worry.
  • 44:35
    That this kind of one of the types of progress at have occurred in this field.
  • 44:38
    It's like a deepening appreciation of just how profoundly difficult this problem is, of how you could create something vastly smarter than you and still ensure a desirable outcome.
  • 44:48
    So that's the bulk of the book.
  • 44:49
    And then the last two chapters are trying to think more generally about these macrostrategic questions, and how to think about what our levers of influence are, if one wants to increase the probability of a desirable outcome.
  • 45:02
    So I'll put on the pause there, because I want to make sure we get a little bit of discussion in.
  • 45:06
  • 45:07
    [APPLAUSE] MODERATOR: Thank you, Nick.
  • 45:14
    We will use the microphone for questions, please.
  • 45:21
    AUDIENCE: I'll just comment quickly.
  • 45:23
    That 40-year median for when we'll achieve human intelligence is-- I've been tracking that.
  • 45:29
    It was about 300 to 400 years in 1999.
  • 45:32
    It was maybe 50 years in 2006.
  • 45:35
    We took a poll at this conference at Dartmouth.
  • 45:37
    And now it's 40 years.
  • 45:40
    I'm saying 2029, but it's actually not so far off.
  • 45:44
    I don't think we're going to get that far with enhancing biological intelligence, because our biological circuits are just inherently a million times slower than electronics, and so there's only so far you can get that way.
  • 45:56
    Whole brain emulation is useful not to create an AI, but to be able to emulate a brain, or more likely a portion of a brain, to establish the functional description of what these basic circuits do to guide our creation of AI.
  • 46:13
    My view, though, is that we are emerging with this technology.
  • 46:16
    I mean, it's already-- during that one-day SOPA strike, I felt like part of my brain went on strike.
  • 46:21
    And so we're already enhanced by these devices.
  • 46:25
    When I was in college, I'd take my bicycle to the computer, and now I carry it on my belt.
  • 46:29
    I believe we will-- these devices are getting smaller.
  • 46:33
    I think within, say, the '20s, '30s, they'll go inside our bloodstream and go into our brain.
  • 46:39
    Basically put our neocortex on the cloud so we can extend the 300 million modules we have in the neocortex.
  • 46:48
    In the cloud, there will be a hybrid.
  • 46:51
    But I would agree that ultimately, the non-biological portion will be so powerful that it will dominate, but that's a path to getting to superintelligence.
  • 47:01
    But I would argue that the non-biological portion is human intelligence.
  • 47:06
    I don't think it's non-human just because it's non-biological.
  • 47:10
    NICK BOSTROM: Yes, so whether-- I mean-- I guess one doesn't want to be bogged down in the terminology of whether, like-- it seems clear to me to call it non-human.
  • 47:21
    But the idea that it's implemented in machine substrate to me doesn't begin to answer the question of whether the outcome is desirable or not.
  • 47:31
    To me, it would all depend on exactly what kind of intelligence is there in this machine substrate, and what this is it doing?
  • 47:37
    What is it using its resources for?
  • 47:39
    Like I could-- you could imagine that we're discovering that we are all in a simulation, we're already all digital.
  • 47:47
    Like so what?
  • 47:48
    I mean, that doesn't mean that human life doesn't have any moral significance just because we're not biological, as we thought.
  • 47:54
    So in principle, you could have a digital mind with exactly the same experience and capabilities as we do, and presumably it should count for the same morally.
  • 48:01
    However, there are a lot of really bizarre types of minds that are possible in principle, and I think one of the slides further down, and one of the key questions that the book tries to answer, is how can we think about the motivations of superintelligent agents?
  • 48:20
    Is it possible to say something useful about what they would want to do?
  • 48:24
    AUDIENCE: We're all evolving together.
  • 48:26
    There's, like, 2 billion people that are enhanced with these devices now.
  • 48:30
    And as they get more intimate with us, it's not going to be like these science futures of movies, of one evil corporation that's got got this technology.
  • 48:40
    It's going to be billions of us that enhance together, like it is today.
  • 48:47
    NICK BOSTROM: Yeah, so the growth of collective intelligence.
  • 48:50
    I mean, I think that at some point, the fleshy parts that are in crania will-- a., they will be a lot harder to enhance, and they will become just kind of negligible part of the actual intelligence that is created.
  • 49:05
    And that everything then depends upon us having set up the initial conditions.
  • 49:10
    So, like, superintelligence will be extremely powerful.
  • 49:12
    We have the one advantage, that we get to make the first move.
  • 49:15
    And I think we only get one try there.
  • 49:17
    Because once you have like an unfriendly superintelligence, it will resist you sort of changing its values.
  • 49:23
    And so part of what makes the problem so challenging is that you need to get it right on the first attempt, and humans are generally not very good at that.
  • 49:31
    We like to sort of see how things work out and patch things up and learn from experience.
  • 49:35
    AUDIENCE: I want to explore what you mean when you say a desirable outcome, what desirable means.
  • 49:42
    There this old philosophical problem of the utility monster.
  • 49:45
    It's sort of a challenge to a utilitarian notion of morality, which is, imagine that there's some creature that wants something more than the rest of humanity combined, feeding the one thing that it wants because it wants it so much more.
  • 50:02
    Maximizes utility, ignoring the rest of humanity.
  • 50:07
    So in some sense, the superintelligence scenario can give life to the utility monster in the sense that if the cognition after the explosion is vastly greater than the total sum of cognition of humanity, then perhaps the moral consideration of what a desirable outcome should be should only be paying attention to what it wants, not what we want.
  • 50:32
    NICK BOSTROM: Right.
  • 50:33
    AUDIENCE: So I wanted to raise that as a challenge.
  • 50:35
    I'm not advocating that perspective, I want to see how you reason about desirability in a world where we're coexisting with superintelligence.
  • 50:45
    NICK BOSTROM: Yeah, generally speaking, it's easier to describe what an undesirable outcome would be than a desirable one.
  • 50:54
    So there are a lot of ways in which things could turn out that, by most reasonable [INAUDIBLE], we would regard as pretty worthless.
  • 51:01
    Like the standard example in this little literature is the paper clip maximizer.
  • 51:06
    So an AI that's superintelligent, and has as its only final, highest-level goal to maximize the number of paper clips it produces.
  • 51:14
    This is a stand-in for some other arbitrary goal.
  • 51:17
    But most final goals, if you think through how the world would be structured in order to maximize the realization of that goal, would involve, as a side effect, the elimination of human beings and everything we care about.
  • 51:30
    So if you're a superintelligence that's a singleton, and you want to make sure there's as many paper clips as possible, for a start, you'd want to get rid of all humans.
  • 51:37
    Because maybe we'll want to switch off or something like that, and then there'll be fewer paperclips.
  • 51:42
    We also have bodies that are full of juicy atoms that could be used to make some really nice paper clips.
  • 51:47
    So then you think, OK, that's not do paper clips.
  • 51:49
    That's ridiculous.
  • 51:50
    But then you think of something else.
  • 51:51
    Like what about an AI who only wants to calculate decimal expansion of pi?
  • 51:56
    So similarly, such an AI would want to maximize the amount of hardware it has so it can make more rapid progress in this calculation.
  • 52:02
    And it actually turns out to be quite difficult to specify a goal that would not be maximally realized in a world where not just human biological organisms are extinct, but also anything we would possibly place value on is eradicated.
  • 52:18
    AUDIENCE: So the premise there is that-- I want to really focus on the premise, because I think the argument hinges on it-- that we're taking a snapshot of what it is we value today, where "we" includes the things that we consider to be adequately cognitive today.
  • 52:39
    And we are ignoring in our definition of desirability-- Let's go to the extreme of the paper clip scenario.
  • 52:48
    A utilitarian might say, well, OK, if it wants paper clips, but its overall cognition is vastly greater than the rest of humanity as a whole, well, then that's what it wants, so the weighted definition of desirability should be to maximize paper clips, because that's what it wants.
  • 53:09
    NICK BOSTROM: Well, OK, so there are different versions of utilitarianism.
  • 53:11
    There is preference satisfactionism, which I think is what you alluded to, which would stipulate some sort of social welfare function that is maximized by fully-satisfied single preferences that exist.
  • 53:24
    There's a big problem of how to aggregate them, but something along those lines.
  • 53:28
    Other utilitarians would say maximize pleasure or maximize happiness or maximize some other part, the common feature being that the value of the whole is, as it were, the sum of the value of the parts.
  • 53:43
    If you thought preference satisfaction is and was correct, you might want to design agents with easy-to-satisfy preferences.
  • 53:52
    Like they want there to be at least three prime numbers or something like that, and then you're done.
  • 53:56
    And then maybe to have as many as possible of those agents.
  • 53:58
    Like the minimum agent that would count as a morally considerable being.
  • 54:03
    But that seems like a fairly impossible moral view.
  • 54:07
    But one can decompose this big sort of problem into two parts.
  • 54:12
    On the one hand, you have the technical problem of-- if you specified some value in human language, like whether it's to maximize happiness or freedom or love or creativity, whatever it is, that how could you sort of embed that into a seed AI, like an AI that's destined eventually to become a superintelligence?
  • 54:28
    So this is like an enormous technical problem.
  • 54:31
    Because like in C++, you don't have a primitive saying "happiness," right?
  • 54:35
    You have to define all of these terms.
  • 54:37
    Some goals would be feasible, like maybe to calculate as many digits of pi.
  • 54:41
    It's something we could do today.
  • 54:42
    Others, like, there's this big, unsolved technical problem.
  • 54:45
    But then on top of that, you also have the second problem, which is the value selection problem, like trying to figure out which value it is that you would want to get in there in the first place.
  • 54:53
    And both of these are places where we could easily stumble.
  • 54:57
    So just to reflect on, like-- if the idea was to try to do some AI that was ethical, or maximally always did the morally right thing to do, if we try to achieve that by just creating a list or somehow embedding our current best understanding of ethics into a final goal, we should reflect that if any earlier age had done this with their values, it would have been what we can now see are catastrophes.
  • 55:26
    Earlier ages were condoning slavery or human sacrifice, and all kinds of abuses of different minorities and stuff.
  • 55:33
    And presumably even though we might have made some progress towards moral enlightenment, we haven't gotten all the way there.
  • 55:40
    So it would be important to preserve the possibility for moral growth in the value selection.
  • 55:46
    And so there are a number of different paths that each should be explored, because we're still at such an early stage here.
  • 55:54
    But maybe one of the more promising one is this idea of indirect normativity that I describe in the book, which is the idea that rather than trying to take explicitly characterized some desired end state, we try to motivate the AI to pursue some process whereby it can find out what it is that we were trying to work out when we were working with this problem.
  • 56:15
    So suppose you could give the AI the goal of doing that, which we would have asked it to do if we had had, like, 40,000 years to think about this question, and if we ourselves had been smarter, and if we had known more facts.
  • 56:31
    So now we don't know what that is, currently.
  • 56:33
    But it's an empirical question that we could then hopefully leverage the AI's superior intelligence to make a better estimate of.
  • 56:41
    And then that kind of indirectly specified goal might then be more likely to produce an outcome that we would recognize, on reflection, as being worthwhile.
  • 56:52
    AUDIENCE: So I have a story.
  • 56:53
    The other day, I was reading some of the news and analysis about the crisis in the Middle East, and I guess I spent like an hour thinking about it.
  • 57:03
    And I didn't come up with a solution for the Middle East.
  • 57:06
    NICK BOSTROM: Ah, darn.
  • 57:07
    AUDIENCE: Now if I had been a speedy superintelligence, and in that hour I had spent 1,000 hours of thinking, I think I still wouldn't have come up with a solution.
  • 57:17
    So I think there are some problems for which intelligence by itself isn't the answer.
  • 57:23
    And you know, as humans, we put sapiens in our name.
  • 57:27
    We think intelligence is really important, but it's not the only attribute.
  • 57:30
    I don't think it solves all problems.
  • 57:32
  • 57:33
    So I mean, I agree with that.
  • 57:36
    A lot of sort of sociopolitical problems in the human realm often depend on people with conflicting preferences.
  • 57:41
    There might just not success one solution that would maximally please everybody.
  • 57:46
    And with the case of the AI, I mean, I think that in fact, the most important problem to work on is not the intelligence problem, which hastens the day where we'll have it, but rather this control problem.
  • 57:56
    How to ensure that it would deploy its intelligence in ways that are not harmful.
  • 58:01
    And just briefly, there's, I think, two broad classes of control method that's one can envisage here.
  • 58:07
    So one is capability control method, where you try to limit what the AI is able to do.
  • 58:12
    So maybe put it in a box.
  • 58:13
    You unplug the ethernet cable.
  • 58:16
    You only allow it to communicate by typing text on a screen, let's say.
  • 58:20
    Maybe only even answers to questions that are posted.
  • 58:24
    And you try to clip its wings.
  • 58:26
    And I think that those can be important during the development phase, like before you actually are ready to launch your system.
  • 58:33
    But ultimately, I don't think they are the answer.
  • 58:35
    Because in order for this AI to actually have any effect on the world, it will at some point need to interact with it.
  • 58:42
    Like if you literally just had an isolated box that didn't closely interact with the world, yes, it could be safe, but it would also not do anything at all.
  • 58:51
    But as soon as you have, say, a human being communicating with it, then you have a weak link here.
  • 58:55
    Like humans are not secure systems.
  • 58:58
    And even humans often succeed in manipulating or tricking or deluding other humans to do their-- like scam artists.
  • 59:07
    And so if you had like a superhumanly powerful persuader and manipulator, chances are eventually, it would find a way to talk its way out of the box.
  • 59:15
    Unless it could just hack its way out, like by-- so there are things like, we think, oh, well, we'll just put it in a box.
  • 59:22
    If we don't talk to it, it's safe.
  • 59:23
    Well, maybe there's some unanticipated way that we haven't thought of, like by wiggling its electrons around in its circuitry, maybe it could create electromagnetic waves that could influence a nearby apparatus or something like that.
  • 59:34
    So then we think, oh, put it in a Faraday cage.
  • 59:36
    But OK, so if we just keep patching up all the flaws that we can find, then we will just patch up all the ones we can find, but there are probably some more ones that we can't think of.
  • 59:44
    And then it will use one of those.
  • 59:46
    So the second class of control method is motivation selection methods, where instead of, or in addition to, trying to limit what the system can do, you try to engineer its motivation system, so that it would not want to cause harm.
  • 01:00:03
    And that's then where this indirect normativity comes in, as one version of that, and there are many other many other aspects of that.
  • 01:00:12
    And that's, I think, the problem that ultimately we'll need to solve.
  • 01:00:17
    AUDIENCE: So if you'd use these two mechanisms to control it, still, it comes back to this question on the other side of the equation.
  • 01:00:26
    Like it somehow turns its fitness function into the will to dominate us, because of its will to survive.
  • 01:00:33
    But we also have that will to survive, and even though we make mistakes, it seems like the argument of a superintelligence coming to completely dominate us requires a lapse of attention on our part, in our own promotion of our desire to survive, for long enough for it to actually be irretrievable.
  • 01:00:51
    So have you considered that-- it seems like even in all of the horrific things that you've described that could happen if a superintelligence did come to dominate, there would be that take-off duration period where we would presumably wake up and unplug it.
  • 01:01:05
    NICK BOSTROM: Well, one would imagine, if the developers are somewhat sensible, that they wouldn't actually permit the take-off unless they at least believed that the system was safe.
  • 01:01:17
    So imagine a scenario where they have maybe falsely deluded themselves that there is no flaw in their system.
  • 01:01:23
    Or maybe they're just worried that there's this competitor who's soon going to release another system.
  • 01:01:27
    So even if they haven't spent enough time on the safety, they still-- But you have to take into account that you're dealing with an intelligent adversary.
  • 01:01:36
    So even just a human-level mind in this situation could figure out that it has an incentive to pretend to be nice, whether or not it actually is nice.
  • 01:01:45
    Like when you're weak and, at the mercy of your programmers, who are inspecting you and seeing if you're ready to be released, and if you're an unfriendly AI, you would want to sort of behave cooperatively and pleasingly and all of these things.
  • 01:01:57
    Like it can plan ahead to that extent.
  • 01:01:59
    And only once you are sort of strong enough that it doesn't matter whether anybody tries to stop you, because they can't-- only then would it be safe for you to reveal your true nature.
  • 01:02:09
    So there is this fundamental flaw in the-- so this is one of those initially plausible ideas that don't seem to work.
  • 01:02:15
    Like you develop your AI.
  • 01:02:17
    You keep it in a sandbox, like a secure environment, and you watch it for a while to see that it behaves nicely.
  • 01:02:22
    And only once you've seen that it's cooperative and nice and friendly there do you let it out.
  • 01:02:26
    And the flaw is that there is this possibility for strategic behavior, that unfriendly AIs could mimic a friendly AI.
  • 01:02:37
    And you mentioned something about this survival desire.
  • 01:02:42
    So there is something like that, but it looks different.
  • 01:02:44
    So we humans have-- we don't really have a clean agent architecture.
  • 01:02:50
    There's not, like, one final goal for most of us.
  • 01:02:52
    And there are lot of different drives that rise and fall in strength, depending on the time of day and the environment we're in.
  • 01:03:00
    But if you have this architecture where there is a clearly-defined final goal, and everything else is pursued only by virtue of being conducive to the attainment of this final goal, then there are a couple of theses that I think help you think about that kind of structure.
  • 01:03:16
    So on the one hand, you have the orthogonality thesis, as I call it.
  • 01:03:20
    This is the idea that values and intelligence are orthogonal.
  • 01:03:23
    You could have virtually any combination of them.
  • 01:03:26
    Like a really smart system could be really benevolent or really evil or have some bizarre goal, like paper clips, or something human-meaningful.
  • 01:03:33
    There's no necessary ontological connection.
  • 01:03:37
    On the other hand, you also have this instrumental convergence thesis, which says that for almost any final goal and almost any environment, there will be certain instrumental values that you will recognize once you're smart enough.
  • 01:03:49
    For example, the value to prevent your own death.
  • 01:03:55
    And so if you're a paper clip maximizer, the only reason that you don't want to die, it's not because you sort of value being alive.
  • 01:04:00
    It's just that you predict that there will be fewer paper clips if you are switched off today.
  • 01:04:04
    Because if you're still around tomorrow, you will still be working to make more paper clips.
  • 01:04:07
    And similarly, goal content preservation.
  • 01:04:11
    You can predict that if somebody changed your goals, then tomorrow, you will no longer be working to make paper clips.
  • 01:04:16
    Now you will be working to make staplers, and then there will be fewer paper clips.
  • 01:04:19
    So you, being a paper clip maximizer today, will want to prevent somebody from changing your goals.
  • 01:04:26
    And there are others, like acquiring more material resources, or enhancing your own intelligence so that you become better able to realize whatever your goal are.
  • 01:04:33
    And it's that combination between the lack of any necessary connection between final goal and intelligence, and these convergence instrumental reasons to just do things that are inconsistent with human values, that creates the intrinsic danger there.
  • 01:04:49
    You have to engineer a very particular kind of final goal to-- have a final goal such that if it's actually maximally pursued by a superintelligence, would be consistent with human survival.
  • 01:05:01
    Maybe something that kind of embeds within it the same values that we have.
  • 01:05:06
    MODERATOR: So we've been talking a lot about hypothetical stuff.
  • 01:05:09
    What about some concrete stuff, namely policymakers?
  • 01:05:13
    So we're talking here about scenarios that are potentially very dangerous and that may scare policymakers, whom we know are technologically not at the level of this audience and may start making decisions which will slow down or impede the progress, or maybe even ban computer science that tries to do AI research because of the fears that crop up in some of that.
  • 01:05:40
    What are your thoughts on the policymaking process and legislature process around issues of artificial intelligence?
  • 01:05:49
    And can we expect that, you know, like computer scientists are one day labeled as terrorists?
  • 01:05:55
    NICK BOSTROM: I don't think that that's very likely for various reasons.
  • 01:06:00
    It's hard at the moment to see exactly what it is-- even if policymakers were willing to do something, what they could actually do that would be helpful, rather than harmful.
  • 01:06:09
    At the moment, what needs to be done, I think, is more foundational work to build up a clear understanding of what precisely the problem is.
  • 01:06:17
    And then ultimately, it's mostly a technical research challenge to work out the solution to this control problem.
  • 01:06:23
    It requires some top-notch mathematical talent working together with theoretical computer scientists and maybe some philosophical expertise to really crack this problem.
  • 01:06:36
    It's very hard to see how, like, from some high level of government-- so it's a very blunt instrument.
  • 01:06:40
    And you might, even with the best intentions at the start, like at the top, once it filters down through the bureaucracy, it might have a very different effect than the one you intended.
  • 01:06:51
    So there are some other existential risks where I think it would be easier to imagine ways in which regulation could help.
  • 01:07:00
    AI is particularly difficult.
  • 01:07:02
    Even just to understand what the problem is is quite hard.
  • 01:07:07
    And it's hard to imagine a scenario, at least in the next couple of decades, where we would have some kind of sane thing coming from political processes.
  • 01:07:16
    Maybe the closest would be like more funding for work on the control problem.
  • 01:07:21
    But even that, once it sort of filters through the vested interest and academia, will probably translate into a rain of funding falling on a wide range of superficially related areas that might not actually have anything to do with the control problem, like general computer security or something like that.
  • 01:07:38
    But there are other things that can be done.
  • 01:07:40
    So there are some organizations that are working on this.
  • 01:07:45
    So we are doing some work at the Future of Humanity Institute at Oxford.
  • 01:07:48
    Another is the Machine Intelligence Research Institute, MIRI, at Berkeley.
  • 01:07:52
    They have some excellent people, as well.
  • 01:07:56
    That would be an obvious thing.
  • 01:07:57
    And generally try to recruit some of the brightest minds of our generation and the next generation, to sort of focus on this.
  • 01:08:05
    At the moment, worldwide, maybe there are half a dozen people or so, equivalent, working full-time on this, which is not in proportion to the importance of the problem.
  • 01:08:19
    It's a more general issue.
  • 01:08:21
    I did a little literature survey a couple of years ago.
  • 01:08:23
    I just compared a number of academic papers on the dung beetle compared to a number on human extinction.
  • 01:08:31
    And sad to tell you that there was more than an order of magnitude more on the dung beetle.
  • 01:08:38
    So the positive spin on that is that there are enormous opportunities for somebody who actually does care to make a big difference.
  • 01:08:45
    Like even one extra person or one like extra million or something, can do a lot of good there, because it's so neglected.
  • 01:08:53
    AUDIENCE: So regarding policy and political things, I think the general underlying principle here is that modern governments are like big battleships or big tanks.
  • 01:09:02
    They do very well against large, stationary targets, but against small, mobile targets, they're extremely ineffective.
  • 01:09:09
    And so if AI were like nuclear weapons, where in order to produce it, you need these giant, static manufacturing facilities that are very expensive and they're like fixed in one place so you can see where it is, then the political aspect, how you regulate it and whether you regulate it, is very important.
  • 01:09:26
    But artificial intelligence isn't like that.
  • 01:09:28
    You can develop it from anywhere in the world.
  • 01:09:30
    Your computer might cost $10,000, and it might be anywhere in the world, since you can do things through the cloud.
  • 01:09:37
    And when governments try to handle these sorts of small mobile targets, like individual websites or individual people on the internet, it doesn't really matter, compared to the nuclear weapons case, very much what kinds of things they do.
  • 01:09:53
    Because governments just can't hit that kind of target.
  • 01:09:56
    It's like, you know, piracy of software is, in theory, punishable by whatever penalty.
  • 01:10:03
    But as we see everywhere in the world, those kinds of things are totally ineffective at achieving their stated goals.
  • 01:10:09
    on what the scenarios here that we're having.
  • 01:10:11
    Like, say, if there were some scenario in which they would try to prevent AI from ever being developed, I think that's a lot more far-fetched.
  • 01:10:18
    And slightly more possible scenarios where it became clear which products were going to succeed, and that it was going ahead, and then they would acquire that.
  • 01:10:27
    Like they would nationalize it.
  • 01:10:29
    But then that doesn't solve the problem.
  • 01:10:31
    That just means that now you have an encapsulation.
  • 01:10:34
    So maybe it's all placed under the federal government, and they have military guarding the whole thing, but you would still have the same people inside, basically working on the same problem.
  • 01:10:42
    And so that that outcome, scenario, might not make that much difference one way or the other.
  • 01:10:49
    You still have the same basic technical problem inside.
  • 01:10:51
    And it's also unclear to what extent it would be possible for non-experts to really be able to exert micro-level influence on the precise design of the AI.
  • 01:11:01
    I mean, you have to know what you're doing to be able to do that.
  • 01:11:04
    I think-- I mean, things that they could do in general, there are indirect things.
  • 01:11:08
    So working harder to achieve global peace and coordination would help with a lot of problems, including AI.
  • 01:11:16
    Maybe it makes it easier.
  • 01:11:17
    And in the future, if there were like a race dynamic between different countries, that they could join together and do one joint thing, rather than racing to get there first and then having to cut back on safety.
  • 01:11:28
    There are things that could be done, maybe, to facilitate biological cognitive enhancement.
  • 01:11:34
    If that was the will, you could certainly imagine different kinds of funding and policies for accessing and linking different databases that could be done, and stuff like that, that would be useful.
  • 01:11:46
    So there are potentially cost-effective, indirect ways of approaching this problem, in addition to directly working on the control problem.
  • 01:11:53
    There are these other levers that one could also consider.
  • 01:11:56
    Particularly on things that we are still quite far away from the relevant crunch time.
  • 01:12:01
    AUDIENCE: Hi, there.
  • 01:12:02
    I was just curious.
  • 01:12:03
    You're one of the world's experts in superintelligence and the extensional risks.
  • 01:12:09
    Personally speaking, informally, intuitively, do you think we're gonna make it?
  • 01:12:14
  • 01:12:17
    I mean, it's-- I think that the, uh-- [LAUGHTER] NICK BOSTROM: I mean, like, I mean-- yeah, probably like less than 50% risk of doom.
  • 01:12:29
    But I don't know exactly what the number is.
  • 01:12:32
    I mean, the more important question, I guess, is what is the best way to push it down.
  • 01:12:38
    So that's where most of the mental energy is going into.
  • 01:12:45
    [LAUGHTER] MODERATOR: So with that, please thank our guest today.
  • 01:12:50
    [APPLAUSE] MODERATOR: Thank you, Nick.

Nick Bostrom: "Superintelligence" | Talks at Google

Superintelligence asks the questions: What happens when machines surpass humans in general intelligence? Will artificial agents save or destroy us? Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life. The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. If machine brains surpassed human brains in general intelligence, then this new superintelligence could become extremely powerful - possibly beyond our control. As the fate of the gorillas now depends more on humans than on the species itself, so would the fate of humankind depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed Artificial Intelligence, to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? This p

Community tags: big_thoughts     HOME     SIGN UP     CONTACT US