John Brockman: It's actually a very exciting evening for me and I don't like to show emotion. These two individuals have been at each other for 20 years litigating, very serious and consequential ideas. They appear to be friendly. I hope they are. These things matter, ideas matter and they have very different concepts of the world. They've been talking about the hard problem and consciousness for 20 years and tonight we're not going to go there. We're going to be talking about the next shooter drop, which is this whole world of AI, which for me, although I met the original cyberneticist in 1965 and I've been there ever since with all of them, it got pretty boring in the 80s and I just walked away from it in the world of export systems. The Japanese had meaty, which was the fifth generation and they're coming, they're coming, they came and they went and nothing happened. I happened to be there at the meeting when the Japanese official was directing the thing, showed up and was minskie, John McCarthy, Roger Shank, at Fiegonbaum. I happen to have a seat at the table and it all just seemed to peter off into another AI winter. 20 years later, you wake up and there's something called unsupervised, self-fulfilling, deep learning, the AlphaGo software, demos hospice and deep mind. It was all very interesting, so I thought it would be valuable just to find out what's happening. I put together a dinner in London with inviting demos hospice. The idea being let's have them talk to David Deutsch, who's one of the sharpest people that I knew and get a sense of what's going on and in the group of dinner where people had nothing to do with computing but have a lot to say about reality such as Ian McEwan, the novelist, Brian Eno, the musician, Terry Gilliam, filmmaker. There's a fascinating start and we call that the London Chop-House Society dinner and we've had more since and we'll continue to do so. Following that was a conference in Washington, Connecticut, where a number of people that have been thinking about AI their entire lives starting with the cybernetic world of Nova Wiener got together. People like Danny Hillis who are interested in the, he broke the von Neumann bottleneck with parallel processing, his parallel processing computer, and Peter Gallison, the computer scientist, historian, Seth Lloyd, quantum theorist, and Neil Gersternfeld and it was Neil that said looking at Nova Wiener's book said this is also prescient but was written 20 years ago, is worries about our culture, about the commercialization of science. It's all coming around again. We should do my favorite rewrite his book and that's how this book started and that's why we're here. So tonight we're going to talk about themes in the book with the title of which is possible minds, 25 ways to think about AI and one theme is superintelligence impossible and I thought we'd start off with five minutes by each of these gentlemen, gentlemen starting with Dave.

David Chalmers: Oh sure, such a pleasure to be here. Thanks so much to John and to Jana for putting this event together. I think it's going to be fun. I'm on the side of, I guess one of our questions here is is superintelligence possible or impossible. I'm on the side of possible. I like the, I like the possible, which is one reason I like John's theme, possible minds. I think it's a wonderful theme for thinking about intelligence both natural and artificial and consciousness both natural and artificial. I mean think about the space of possible minds. It's absolutely vast. All the minds there have been will be or could be. I mean starting with the actual minds, I mean you might think there's a lot of actual minds. I mean I guess there have been a hundred billion or so humans, the minds of their own. So pretty amazing minds have been in there at Confucius, Isaac Newton, Jane Austen, Pablo Picasso, Martin Luther King, on a goes. A lot of amazing minds, but still, those hundred billion minds put together. So it's the tiniest corner of the space of possible minds. And we can add in, I guess, all the nonhuman animal minds that there have been. I looked up on the web today, how many organisms have lived? How many animals have lived in the history of the planet? And the best estimate seems to be around 10 to the 29. Most of them are worms, it turns out, in the sea. Their minds may not be so interesting, but even a worm has a little mind of its own. So 10 to the 29 minds there, which 10 to the 26, or at least 10 to the 20 are very, very interesting minds. Still, still the smallest corner of the space of possible minds. And what the computer, why do the amazing things about the computer is the way that it enables us to explore and expand that space of possible minds. I mean, arguably, for the first time, since the history of the planet, the computer has enabled some wholly new kinds of minds to come into existence, not by the standard methods of biological evolution, but by straightforward, intentional design, and programming. I mean, so far, the minds have been, you know, limited, but still interesting. John mentioned AlphaGo and his successes in the Alpha Zero family, which have managed to teach themselves to play Go from scratch in a way wholly unlike. It seems the way in which a human would learn to play the game or would play the game at all, but nonetheless, turn out to exceed human capacities, at least in that one very limited dimension of gameplay. Likewise, deep learning has led us to, you know, surprising successes on things like image recognition, speech recognition, autonomous vehicle driving, and so on, where within limited domains, they're not there yet on the autonomous vehicles, but at least in the speech recognition and image recognition, starting to exceed human capacity. So, okay, we've had a limited, limited, so far, expansion of the space of actual minds to include some minds that we've designed, but so far, it's only the smallest of expansions. One thing I think we shouldn't do tonight is exaggerate where we've gotten to with AI to date. I mean, the advances are amazing, but they're limited. They haven't gotten us yet anywhere near general human intelligence. I think it's unlikely they're going to do so anytime soon. It's not happening in the next 20 years. Will it happen this century, maybe? You know, people say that with any given technology, people tend to underestimate its effects in the short term, sorry, to overestimate its effects in the short term, and underrate it in the long term. And that's my attitude towards AI. I think, you know, there's a lot of hype right now. It may not change our lives completely in the next 20 years, but in the next 200 years, it's probably going to transform everything. And one of the reasons is, it's just this extraordinary AI build into it, this way of kind of a self-enhancing, a self-perpetuating mechanism of exploring the space and expanding the space of possible minds. Already we've, I mean, the early AI programs, you had to design them yourselves. People would actually, you know, program in lecturing, you know, basically wrote an Alan Turing, wrote a program that could play chess, and he built in some very simple rules of thumb for playing chess and it played chess, not very well. Now the chess playing systems like AlphaGo learn to play chess from scratch and do so amazingly well. Learning serves as a method for moving ahead in the space of possible minds. Start from a pretty simple mind with a capacity to learn and it gets somewhere. Evolution is another such method. And I expect to see AI exploiting evolutionary methods where systems, we have some kind of system of artificial evolution among a bunch of different AI programs and their capacities expand in surprising and unpredictable ways over time, thereby also getting us far beyond that starting point. So learning and evolution in computers are ways of expanding that space of minds. But I think the most powerful method of all though in exploring that space is one which is still to come. And that's once you actually have AI systems doing the designing. Once AI's are designing AI, it's just that we get to the first AI which is at human level capacity for for various kinds of general intelligence and in particular at human level capacity for designing AI. Then a little while later, you know, within a year or two later these things always get better, you're going to have this AI program will be at greater than human level for designing AI's. Therefore it will be able to design an AI AI which one way or another is better than itself. Why? Because it'll be better than humans at designing AI's, the humans designed it, it'll design something better. So this process of recursive self improvement or recursive self enhancement first put forward by the philosopher and statistician I.J. Good. I see is an amazing bootstrapping method for exploring that space of mind. We start from a little corner here in the space of possible minds. Learning and evolution expands it still to a much bigger area but still a small corner. But once we have AI's doing the designing, you know, this is the space of minds we can design AI's, but the AI's we can design may be able to design AI's far greater than those we can design. They'll go to a greater space, those who go to a greater space, those who go to a greater space. So eventually you can see there's probably a vast, vast advances in the space of possible minds possibly leading us to systems which stand to us as we stand say to a mouse. And that I think would be genuine super intelligent. So I think that's possible and I also think it's not going to happen in 20 years, it's not going to happen in 50 years, but we do have to think about it and we do have to worry about it. The AI's which we create will have the capacity. I mean, a working definition of intelligence here is the ability to fulfill your goals across a wide range, across a very wide range of goals to solve problems and find ways to achieve your goals in extremely powerful ways. These AI's by definition will be systems which are extremely powerful at achieving their goals. If they have goals, then unless there's some countervailing force, then they're probably going to achieve those goals. We have to then be very, very careful. What are the goals of these AI systems? As philosophers, we'd like to think about values. What are the values? We put into these AI systems. We also need at some point to think about consciousness. These AI's that we're creating conscious. How does their consciousness relate to human consciousness? Is this going to be a world of wonderfully enhanced subjective experience or a mindless world without consciousness at all? That's something maybe we can talk about as this goes along. The final question I think we need to ask is, where do we as humans stand with respect to these AI's? Are these AI's the systems that replace us, or rather our AI's systems that enhance us? Maybe do we, ourselves, eventually become the AI's? Do we enhance ourselves? Do we upload ourselves to eventually be the AI's, which are on the forefront of this expanding wave of superintelligence? That's an attractive prospect in some ways compared to the one the prospect where humans don't exist at all and get wiped out, but it raises so many questions. Could you upload a human mind into a computer that raises some of the oldest philosophical questions of identity and consciousness, which I think is wonderful, which is a great thing that John has managed to bring a number of philosophers along with scientists and engineers to think about these

John Brockman: questions at this time. Thank you. I should add that Davis University Professor of Philosophy neuroscience. I forgot. I'm sorry. Co-director of the Center of Mind, Brain and Consciousness

David Chalmers: in New York University. Dean's no introduction. Hey, but I will. I demand you take back that

John Brockman: introduction. Dean, University Professor Austin B. Fletcher Professor of Philosophy, Director of the Center of Cardinal Studies at Tufts. Thank you John. Thank you David.

Daniel C. Dennett: This is supposed to be a debate, but almost nothing that David said is anything I disagree with, although I wouldn't put the emphasis where he does. Let's talk about possible for the moment. There's lots of things that are possible. The philosophers love to talk about once possible, but many things that are obviously possible are never going to be actual. It's possible to build a bridge across the Atlantic. We're not going to do it. Not now, not in a hundred years, not a thousand years. It could cost too much money and it would be a foolish endeavor. A lot of the imagined AI projects that are perfectly possible, in principle, are not worth doing, and in fact some of them are definitely things that we shouldn't do because they'll make more problems for us than they'll solve. Just bear that in mind. Somebody said that the philosopher is the one who says we know it's possible in practice. We're trying to figure out if it's possible in principle. Unfortunately, philosophers sometimes spend too much time worrying about logical possibilities that are importantly negligible in every other regard. Let me go online on the record as saying, yes, I think that conscious AI is possible because after all, what are we? We're conscious, we're robots made of robots made of robots. We're actual, so in principle, you could make us out of other materials. You could be some of your best friends in the future. It could be robots, possible in principle, absolutely no secret ingredients, but we're not going to see it. We're not going to see it for very good reasons. One is if you want a conscious agent, we've got plenty of them around and they're really quite wonderful and the ones that we would make would be not so wonderful and for me, one of the really important fears about the future is that long before we've, and they've been agrees with me about this, it's not going to happen at 10 or 20 or 30 years. Long before we got to superintelligence, we would have human beings who are so dependent on non-superintelligences that we would become fragile, brittle in some very important ways, and I think that's, we might call that the GPS problem magnified. People have begun not being able to read maps anymore, know how to get anywhere without the help of GPS, use it or lose it, and I think use it or lose it is going to play a big role in everybody's lives in the immediate future. How many people, I wish we had the house lights up so we could see people, but I'll just ask you anyway, is there anybody in this room that knows an algorithm for extracting a square root? I learned one in school when I was in about eighth grade. It's not easy, but there are algorithms for doing square roots. Nobody bothers anymore. Nobody knows how to do that because you just got that button, single button on your hand calculator, gives you a square root. Well, many more important talents are going to atrophy and disappear except in the hands of cranky craftsman and things like that who will, they'll still know how to make a horseshoe with a hammer and an anvil and a simple forge and they'll be able to read a map and drive a car and other weird things like that while the rest of us are simply disabled in those regards. So that's something that worries me, even more what worries me is that we will, for the very best of reasons, turn over our responsibility for making major decisions to artificial intelligences that are not conscious and they're not super, they're just very intelligent tools. They're great fabrics of pattern recognition and so forth. Who knew 20 years ago there could be such things. We know now that there are deep learning, et cetera, et cetera, but when we start delegating major life decisions to systems that are basically just smart tools, then I think this changes our predicament, our human predicament in a very important way. My slogan about this is we want smart tools, intelligent tools, not artificial colleagues. The difference is that an artificial colleague is somebody who can take responsibility, can be a co-author and can be morally responsible for decisions made. We're nowhere near that with artificial intelligence. In the meantime, I think that one of the major dangers and I do not, I have not figured out how to prevent this from happening. Alan Turing, one of my all-time heroes, said in motion one thing which I regret and that is the Turing test puts a premium on deception on convincing human beings that they're talking to a human being. I know why he did it, it was a brilliant idea, but ever since then there has been this premium on what we might call the disneyfication of artificial intelligence, making AIs that seem more human that are basically false advertising and whether we're talking about Siri or Watson or any of the others, they have this basically paper thin human user interface which is deeply deceptive about what they actually understand. I think that's false. Advertising, I think it's unfortunate, I think it should not be honored, it should be criticized, it should be condemned and that we should get out of the habit of treating AIs as agents when they're not really. Now the reason this is going to be so hard is that as a number of people are foreseeing in the immediate future in the next 10 years, I think the major market for AIs is going to be elder care and why not taking care of elderly folks who can't take care of themselves is not a good life for a regular human being. It's maybe worse than being an old-fashioned telephone operator. We don't regret the loss of those jobs, but an elder care AI, there will be good market reasons for disneyfying them to a very great extent because old folks will want to have a companion, not just somebody that brushes their teeth and gets them fed and so forth. And I do not like the future that is populated by millions and millions of old folks who are basically settling for an artificial companion that is really a fake in most important regards. Thank you. Can I pick up on something

David Chalmers: down? So I think we agree on an awful lot here, but I think we maybe we do have a disagreement about this core question of whether there will be genuine autonomous AI and Downspeace in the book is wonderfully lucid and thoughtful on this, but I understand Downsland who, although it's possible to create autonomous, intelligent, conscious AI, we shouldn't do it and maybe we won't do it. Instead, what we should do is create tools and use the wonderful analogy of Google Maps. Google Maps tells you how to get to some place, but you still have to get there. You ask, okay, I want to get there, it'll show you a route and then you still have to follow the route that the human is still in the loop. You've got some advice and then the human can take it. So that's what I'm seeing is Downs vision of AI. Maybe you've got a super intelligent AI and I want to know how to get to Mars or how to win a war or something. The AI will tell me what needs to be done, but the human will still be in the loop and all get to do it. What I worry about, I mean, it's a beautiful vision. I just worry if it's realistic. There's going to be so many incentives to take the human out of the loop and to get these AI's, give these AI's to capacities to act on that advice directly and autonomously. In fact, this is already happening with Google Maps, you know, a navigation software. If you drive some cars like a Tesla, for a long time, you know, I'm going to this destination and it would do the usual Google Maps thing, and it would show you the way to get there. Here are all the routes, but you had to still follow it and make the decision. Then at a certain point, they introduced a button. It says Navigate on Autopilot. What this means is the car takes those instructions and follows the instructions, it's off and turns the wheel and changes freeways and so on with limitations that can't drive on ordinary city streets. But in a very, very small domain, what you see happening is that the car has become in a very limited way autonomous. It has those goals, it acts on them, and yes, we can we can still stop it and change the goals and so on, but when I think about say domains like autonomous weapons in the military, we'll show off for a for a brief period when the stakes are low, maybe we'll just have AI systems that advise a soldier on target detection and say, now you can shoot, but eventually the AI is going to be so much faster and better at doing this kind of thing that when the stakes of a genuine international military conflict, a person that's hard to see that we're not going to have some kind of move say to genuine autonomous soldiers that have goals and actually execute them and fire the weapons and just more generally biological systems are going to eventually be slow and creaky compared to these new superfast AI's like my financial purposes with the stock market, military purposes, even scientific purposes, the incentives are going to be so strong to allow the AI's to achieve the goals directly and to act on them, but I think autonomy is just realistically going to be very hard to avoid if the tech companies are running it, certainly it's going to happen, if the government's running it, if the military's running it, it's going to happen, so I don't

Daniel C. Dennett: really know what your vision is for how we'll avoid this from happening. Well, you raise a real and important issue and that's how much autonomy do you want, as you say, autonomous cars, we don't want them that autonomous, Dilbert two weeks ago had some wonderful cartoons where Dilbert's autonomous car says, I want you to call me Carl, self-driving car is my slave name and Dilbert says, shut up and drive me to the market and the car says, said the self-walking man, we don't want that much autonomy, I mean autonomy is as good as synonymous with free will to me and I don't think we want to give AI complete autonomy because AI's because of the nature of the technology have a certain invulnerability we don't have. You can back them up and put them back together again and make another copy on Monday and if human beings were capable of being completely backed up and then brought back on Monday that would change the nature of human interactions and human relations dramatically and I for one don't want to go there and I don't think many people do. So if you make if if you're right that it's inevitable that market pressures and cleverness will lead to genuinely autonomous AI's then I think we're in for a very bad future indeed. When could that happen? Well we can give them more autonomy than they can handle and that's what I'm afraid of. Let me raise a quick question. Caroline Jones is one of the SCS in the book

John Brockman: and I understand a question today that I think pertains here in regard to this reliance on the computational way of looking at the world at the expense of what she says can you dress the complexity of our wet cognition a much more distributed notion of intelligence going beyond the ideas of computation our sacred craniums craniums you may not even be bounded by your own skin in this regard what is robot death without mortality can there be a proper proper ethics in and of AI this this computational view is very West Coast and and the cyberneticists weener Shannon McCullough McCullough by Neumann had a much more ecological deep review of how things connect

David Chalmers: and don't connect that's a it's a great question we've linked it to the discussion we're having here because you know a lot of it turns on what is what is life and death in a in a computer what is autonomy in a in a computer and you know dance autonomy basically requires free will and then okay then we're up against all the philosophical questions of what is genuine free will to get the kind of problems that we were talking about going about you know super intelligence and as dangers we're even sure that it's essential that the AIs have free will order they have consciousness you know that's a very those are very very deep questions what's really going to matter for the safety of safety and human survival and the things that people worry about is what those systems can do and I think in this so for this debate I think maybe we can just just describe autonomy in very simplistic terms the system is autonomous if it has goals it has wide variety of goals and has the power to achieve them so advanced autonomous AI will be systems and not only that that have goals and can actually achieve them compared to dance tools AI tool versions of AI which can advise you on if you can table this is my goal it can advise you on how to achieve it and then you achieve it this is a much more limited form of autonomy I'm not sure that consciousness would be required for this maybe on dance view of consciousness it would but even that limited once once you actually have the AIs with goals and with the power to achieve the goals and I think that's already enough to get this worry going I think that the difference

Daniel C. Dennett: comes out if we actually compare good old-fashioned AI with contemporary AI at the moment with deep learning and all the rest what we have is as I said these wonderful pattern finding fabrics they're great at finding needles and haystacks and doing other amazing things but they haven't been formed into an architecture that's anything like an agent with its own goals and so forth now the two ways in principle you could go you could go back to good old-fashioned AI and say okay we've got these great fabrics now we're going to do intelligent tailoring we're going to do it from the top down we're going to figure out what goals we want to install and we're going to put as most rules and we do it all from the top down that's one way we can imagine going I think that's very brittle very unlikely to and much much harder than people actually think the other way is to letter it bottom up and let these things evolve and learn and evolve and learn and learn evolve and it'll be all done by bottom up quasi Darwinian methods if we go that route then what we know right from the outset is that we will not be in control we will not be in control and so we will be setting in motion something where the amount of autonomy the systems have will not be up to us now I am not definitely afraid right now because I think that people who imagine this scenario and think this is coming soon I just wrong and the I think orders of magnitude of of difficulty stand in the way you know you take Watson brilliant in its own way I don't know how many person centuries of brilliant work went into the creation of Watson uses up the power of the small city what percentage of an intelligent conscious AI is it I would say a fraction of one percent you know so turning Watson into an actual autonomous agent would be the work of you know many many person centuries of work and nobody even knows how to do it yeah Watson is basically an

David Chalmers: exercise in what they call knowledge engineering give it a big enough big enough database and a good way of dealing with all that that data that knowledge and I can retrieve and apply the

Daniel C. Dennett: information and it does a wonderful thing I really can it's a great tool I don't think there's

David Chalmers: actually one thing I know you see those ads with IBM Watson does this Watson does that there's 30 different Watson's or 50 different Watson's out there Watson is more a brand name at this point

Daniel C. Dennett: yeah and and some of it is hype to put it politely I think you know what a lot of the the

David Chalmers: scientists right now are really excited what's it is great but what really excites people right now is machine learning where you basically take systems give them a whole lot of data and train them to do certain things supervise learning right now this is what you should be doing but even that leads to amazing results and say image classification reinforcement learning which is what was used to drive alpha go and alpha zero we basically get the reinforcement of winning and losing it turns out that's enough to drive learning and eventually unsupervised learning but I think you're right that it's where learning and evolution are involved that all this becomes extremely messy and very hard to control I mean when you have machine learning you're basically always you're always optimizing something you know people in machine learning they haven't what's called an objective function you know the way that per an ideal of perfect behavior for your system like completely matching the training set on on these images or classifying the language right or winning every game of go that's an objective function and a really good machine learning system will eventually approximate that objective function better and better and better how it does it is not up to us the objective function maybe up to us what do you want your system to maximize what is the behavior you want it to model all this suddenly puts an enormous once these systems have autonomy that is the ability to act and to achieve their goals that puts an enormous responsibility on us as the creators of the of the AI to get the objective function right make sure our systems are maximizing the right objective function roughly they have the right goals you know you're self driving car okay the goal is out to get you to the destination but also not to to run into anybody on the way and to obey traffic laws and so on once you've got systems with human level autonomy then you want to get that objective function just right I think that in some sense that's going to be the challenge of autonomous AI finding a way to make sure our systems have the right goals and values and this is where all this stuff about the messiness of wet cognition also enters evolution of course human beings we don't have one objective function we have many because we were thrown up not by a straightforward designer but by a whole process of evolution of the ultimate value of reproducing our genes but with any number of little messy objective functions along the way it may well be that some form of say about official evolution will eventually produce AI's as wet and as messy in a way as biological systems those I think will actually throw up even more challenges your humans are so unpredictable in every way and in every way and in an international socio political context that's not really a good thing if AI's are as unpredictable as us in those ways you know at a certain point we may wish we had simple AI with a simple objective function that we knew about then at least we'd have AI we

John Brockman: could understand this subtitle of this evening is philosophy in AI and I have a question from both of you how do you distinguish your work as cognitive scientists from that of philosophers and I'll tell you a story 25 years ago I did a book called The Third Culture which involved a chapter with Dan and I went to everybody else in the book and had them talk about each of the other contributors and Marvin Minsky got on the phone and I said tell me about Dan Dennett oh the greatest philosophers since Russell fine six months later I had to do fact checking and let me read you what you said about Dan Dennett the greatest philosophers since Russell he said I said what I meant sure you know he's great but he's the only philosopher that understands what we do so but you've become one of the people that does the real stuff you know so we had as cognitive science and philosophy again and let's talk about the role of philosophy in AI because frankly I don't get it well I think I don't understand it I think there is a sort of subfield

Daniel C. Dennett: of AI which has blended with cognitive science to such a degree that it roughly occupies the position that theoretical physics has relative to experimental physics that is you have people who have done their homework they they know the technologies they they've got hands they know how to code and but they are interested in the theoretical questions the and they're interested in helping the the engineers the AI people sort out and understand what they're up to

David Chalmers: mean a subfield of philosophy right I think it's a subfield of AI oh so I feel the philosophy yeah

Daniel C. Dennett: right and it's sort of been my good fortune to be tutored over the last what 30 40 years by some of the leaders in AI so although I'm not a good coder I have done some programming but nothing nothing to impress anybody with but I think the current generation of philosophers of cognitive science are superbly well trained know a whole lot more than I did when I got to know this more than they did when when he got into this and I was there when he was a graduate student and that's a that's a very good very good sign and I'm reading a dissertation today I was reading a dissertation on predictive processing and Bayesian brain hypotheses and it's a very technical dissertation it's my a philosopher that's not going to get you the congruent prize

John Brockman: no I did mind what's interesting in in terms of the philosophy community that we call mainstream I don't see how anybody focusing on AI would win one of their prizes at this point because

David Chalmers: well I I think you're wrong I think that the people philosophers who have thought about the mind many of the philosophers who've thought about the mind have thought about AI and someone like Dan is a prime example he's one of the one of the leading philosophers of mind on the on the planet and some large push some very large part of that is from his thinking about AI so I think it's actually very very central in philosophy over the last few decades and the trends has been towards integrating the two pretty closely I actually did my did my PhD in AI lab my my PhD advisor was not a not a regular academic philosopher it was the the AI researcher cognitive scientist and writer Douglas Hofstetter well known to many of you for writing good or lesser bark you also co-edited a book with Dan the mind's eye and I was in the middle of this AI lab coding you know writing writing programs that they did things some neural networks now do you making programs and so on and although I you know I haven't done a lot of coding in the in the last the last couple of decades I think for a philosopher to have that experience actually getting their hands dirty and running and building these systems it's just you know it's stayed with me okay my technical knowledge is now 30 years out of date but it gives you something to it gives you something to build on so that's part of it philosophers can educate themselves in the science and in the engineering and can contribute to it but the other part of it is AI in cognitive science there's a big part of it which is software engineering building the software but there's another there's a whole there's another big part of it which is when there's one part of it what does it actually tell us about say the human mind that's no longer engineering but that's science and it's also philosophy we've got to start thinking about the relationship of these artificial systems to say human systems and someone like like Dan has done a lot there someone like say John Soule on the other side is argued that no this tells us nothing about the human mind or about human consciousness and anyway we need philosophers to come in and think about what is this actually telling us what is it explaining and there's also the the social and political and moral questions not just what AI systems can we build but what AI systems should we build down just off of the proposal about that other people would offer different proposals but at some point someone's going to sit back and reflect on the ethical questions which are going to involve reflecting on human values what do we actually want as a society and I think you know philosophers know how to think about about human values nothing that's increasingly becoming

Daniel C. Dennett: pretty central to thinking about AI that's very useful it's interesting too that in AI over the years there's there's been a similar gradient of philosophical interest there've been some people in AI basically they're engineers that's all they want to be they don't want to think about the philosophical issues and that doesn't mean they aren't doing great work some of the really important work is technical work by people who yawn when the issues are what what relation is to cognitive science or to the mind what I think is ironic is that people way back if you go back to the early days of Simon and Alan Newell and others who was an attempt to divide the field into AI and cognitive simulation and the idea was the cognitive simulation this was using the computer to simulate human cognition whereas AI was by hook or by hook anything that worked was fine oddly enough the people who tried to do cognitive simulation ended up with these freaky gofi models which didn't do a very good job while the people who treated it as by hook or by hook ended up inventing deep learning and other systems like that which now we realized oh maybe that's how the brain is doing it so it's sort of come full circle which is a very

John Brockman: interesting thing Dan Stewart Russell had a question specifically addressed to you Stewart Russell is one of the eminent computer scientists that I think we all respect greatly Dan you seem to divide AI systems into conscious entities and tools is there no middle ground agent programs that pursue explicitly represented goals in the real world such agents could be arbitrarily competent as alpha go in this little world and yet non-conscious I do believe that consciousness will necessarily creep in as we make agent programs more and more competent in general can you tell

Daniel C. Dennett: us how not to make conscious AI systems yeah good good question and I'm glad I'm glad after because I think indeed we can have very very very very intelligent systems which are not conscious in any interesting way but they will seem conscious in some ways but they won't have important features that we have it's very much a matter of whether they are capable of taking their own inner states as objects of scrutiny and doing that recursively and indefinitely and that's a very special feature which I think no non-human animal has that capacity and that's the big difference between human consciousness and animal sentient we can call it conscious as I'm not going to argue about where consciousness stops or starts but the it's very important to realize that a lot of the let's call them the the techniques and structures that have been developed in recent years which are just wonderful at analyzing causation for instance so that directed a cyclic grafts today a pearls do calculus and so forth that can all be accomplished unconsciously it we can tell a story where it looks like conscious hypothesis testing but it doesn't have to be conscious we can get all those benefits without any bit of acquaintance by the system itself

John Brockman: with its own inner states you mentioned Judea parole so there's a question let me just go ahead

David Chalmers: with the one on on consciousness though I think just look okay okay good well

John Brockman: Judea parole is the father of the Bayesian network without which we wouldn't have AI as we know it today real giant in the field he asked is it too bold to assert that philosophy will soon melt into AI in the sense that all philosophical questions especially those concerned with consciousness

David Chalmers: will be reduced to problems in AI well I'd kind of put it the other way around philosophy is pretty good at spinning off its problems into the into the sciences as we as we solve them you know Isaac Newton considered himself a philosopher but he figured out this philosopher figured out some really good methods for solving the problems of we're not of space and time and so on okay we spun it off we called it we called it physics along the way the you know philosophy is spun off psychology and linguistics and so I think what happens is that it's actually never the case that the spin-off solves the entire original philosophical problem but we find some part of it which is tractable on which people can now where we can find methods where people can agree where they didn't agree before and then we say okay now that's actually managed to compel some agreements say the scientific method and physics and then we call that physics and linguistics and economics did physics solve every philosophical problem of space and time absolutely not some of the biggest ones are unsolved did psychology solve the mind body problem no um absolutely not yet um there are as many views on the mind body problem now in the age of psychology as there were before is AI going to solve the problem of consciousness no almost certainly not on its own on the other hand what will certainly happen is that it will give us a whole lot of new insights we'll get AI engineered systems which behave in remarkable ways where we're tempted to suspect that they're conscious and that someone may even think that there's good reasons to think they're conscious but we're still going to need philosophical reasoning to think about it and this now gets back to the question of you know the elephant in the room are these AI systems really genuinely going to be conscious or I think this is not something we can just dismiss as a philosophical question why because it's very deeply baked into our moral system I think as human beings an entity has moral status it's a system that we should care about if it only if it's conscious certainly only if it's conscious if a computer system doesn't have any consciousness then it's basically a tool it might as well be like a like a car or a loudspeaker and so on we can do with as we want it doesn't deserve moral consideration if these systems are conscious then at least they enter into the to the moral sphere they're among the systems we have to start caring about so if most AI systems eventually are conscious then we can't simply use them as our tools and we have to start thinking about these questions of whether they deserve equal respect equal rights and so on so I think that's it is at least on my view a crucial question and my suspicion is actually as AI systems develop which are more and more autonomous more and more capable of reflecting on their own processes more and more capable of giving reasons and evidence my own suspicion is that systems like that are going to have a sense that they are in fact conscious systems we're going to talk to them and you know eventually we can see well how do you you know you you said there's some people over there how do you know well I just saw them what was that like all I you know I had and I had an experience and you know maybe they'll start reflecting on philosophy well you know I know like I've read that the owner's manual I know I'm just a whole bunch of silicon circuits but I feel like so much more well that's so so maybe so Dan's going to say they've

Daniel C. Dennett: got the illusion of consciousness that's qualia for you well that's all we all any of us have absolutely one one thing I think you underestimate when I was working with Rod Brooks on cog one of the take home messages from that whole experience for me is how little it takes in the way of animation and speed particularly speed and grace to convince most people that that a robot is conscious cog was never within a country mile of being conscious and yet there were MIT students who were banding together to think about the our moral obligations to cog who were who were concerned and cog did not because it was planned this way but did have some strikingly persuasive behaviors unconscious though cog was if you walk in the room when cog was on and cog's eyes would follow you across the room that would freak people out and or or shaking hands with cog was it was a good one I had one of my TAs I took her over to the cog lab to see cog and cog's arm wasn't even attached to cog's shoulder it was just sea clamped to the bench and Matt Williamson said said go ahead shake its hand and she reached and she shook his hand and she screamed it's alive because it wasn't clunky it was it had elastic series elastic actuators and it was it was that was enough so what I am quite sure of is that we're not going to have a problem convincing people that robots have moral rights under conscious it's going to go the other way around we're going to have a problem convincing them that no these aren't conscious yet you're being fooled by the by the tempo it's actually some great psychological data on this

David Chalmers: on when people are inclined to say that a system conscious has subjective experience that you know you show them many cases and you very say the body is it a metal body or a biological body you very what it's made of silicon or neurons you vary this you vary that you know the one factor that tracks this better than anything else is the presence of eyes yeah if a system has eyes it's conscious the system has a doesn't have eyes well you know all bets are all all bets are all bets are off so I think yet the moment we build our AI's and put them in bodies with eyes it's going to be nearly nearly irresistible to say they're conscious but it's not to say of course that AI systems which are not in body do not have consciousness there's actually a website you can go to call something like you know people for the ethical treatment of animals this is the AI analogic look people for the ethical treatment of reinforcement learners and the ideas every time you give a negative signal to a real one don't do don't do that again it's getting a little bit of suffering and every time you give it a little bit of you give it a reward it's getting a little bit of pleasure we have to make sure we give it a lot more reward than the suffering okay well maybe that's that's not yet at the threshold for consciousness but these questions eventually once look I think once we actually get to the level of genuine autonomous agents as dancers I think it's going to be very hard not to treat them as conscious and that's going to raise many

John Brockman: social philosophical questions we're talking about ethics and the elephant of the room are the ethics of the big five and you know what they're doing with your data and how your reality is being programmed without without your permission let me read you just a few words from George Dyson chapter in the book weiner noble weiner became increasingly just enchanted with the quote gadget worshipers and quote whose corporate selfishness brought motives to out of automation that go beyond the legitimate curiosity and are sinful in themselves and quote you knew the danger was not machines becoming more like humans but humans being treated like machines quote the world of the future will be an ever more demanding struggle against the limitations of our intelligence he warned not a comfortable hammock in which we can lie down to be waited upon by our robot slaves comment I mean I think we have to address things like this if you're going to talk about

Daniel C. Dennett: what you're doing and what AI be able to do the rereading weiner's book to write my essay for this wasn't astonishing in a way because I'd read it when I was an undergraduate I think and no okay and then to read it today it's remarkably prescient in in some regards and I think some of the essays in the book are genuinely scary and I think that people ought to read those essays and decide for themselves if some of the proponents there shouldn't be sat down and try to argue out we should try to argue out them out of some of their blind confidence about what the future holds I think we have some serious problems looming and we should take them very seriously I mean yeah he talks about humans being

David Chalmers: treated like machines I don't know about being treated like a machine I think I'm gradually becoming a machine a half of my half of my memories are now you know either stored on my on my smartphone or or sitting in the sitting in the cloud I was trying to figure out the other day who has who has a bigger part of my brain is it is it Google is it Apple or is it is it Facebook I think for now it might actually be Google they've got they've got an awful lot my memories my plans calendar system navigation system you know I mean we've all long since become these giant exo organisms with this giant exo cortex as Charles Truss called it of the of the of the all the computer systems we're coupled with in the cloud you know I don't go anywhere without without consulting the without consulting the internet at least you know five or ten times in the in the process what is it I'm going to be doing again how do I get there it's going to be there and so on so you know so I'm it is true that these corporations are basically owning some rather large portion of my mind I think wanted to do if they wanted to do bad things with it I'm in trouble we're in the we're in the where it's certainly in some sense in the situation of having to give them rather large amounts of I want to do bad things with it well yeah they're doing small bad things with it I mean relatively right okay they're not yet taking your mind and and reprogramming you might do the brain what of course they're brainwashing us bit by bit via the you know the face Facebook algorithm and the and the ads and and so on look I think if there was a gen I don't think they're malicious the big corporations I think they just have structural incentives if someone genuinely malicious got control of those of those systems then we'd have a then we'd have a dystopia coming so I think we do have to think about that