Debate Analysis

Debate Analysis

Philosophers Daniel C. Dennett and David Chalmers In Conversation

Channel: Pioneer Works

Primary speakers:David ChalmersDaniel C. Dennett
Transcript
Download
John Brockman [00:00] It's actually a very exciting evening for me and I don't like to show emotion. These two individuals have been at each other for 20 years litigating, very serious and consequential ideas. They appear to be friendly. I hope they are. These things matter, ideas matter and they have very different concepts of the world. They've been talking about the hard problem and consciousness for 20 years and tonight we're not going to go there. We're going to be talking about the next shooter drop, which is this whole world of AI, which for me, although I met the original cyberneticist in 1965 and I've been there ever since with all of them, it got pretty boring in the 80s and I just walked away from it in the world of export systems. The Japanese had meaty, which was the fifth generation and they're coming, they're coming, they came and they went and nothing happened. I happened to be there at the meeting when the Japanese official was directing the thing, showed up and was minskie, John McCarthy, Roger Shank, at Fiegonbaum. I happen to have a seat at the table and it all just seemed to peter off into another AI winter. 20 years later, you wake up and there's something called unsupervised, self-fulfilling, deep learning, the AlphaGo software, demos hospice and deep mind. It was all very interesting, so I thought it would be valuable just to find out what's happening. I put together a dinner in London with inviting demos hospice. The idea being let's have them talk to David Deutsch, who's one of the sharpest people that I knew and get a sense of what's going on and in the group of dinner where people had nothing to do with computing but have a lot to say about reality such as Ian McEwan, the novelist, Brian Eno, the musician, Terry Gilliam, filmmaker. There's a fascinating start and we call that the London Chop-House Society dinner and we've had more since and we'll continue to do so. Following that was a conference in Washington, Connecticut, where a number of people that have been thinking about AI their entire lives starting with the cybernetic world of Nova Wiener got together. People like Danny Hillis who are interested in the, he broke the von Neumann bottleneck with parallel processing, his parallel processing computer, and Peter Gallison, the computer scientist, historian, Seth Lloyd, quantum theorist, and Neil Gersternfeld and it was Neil that said looking at Nova Wiener's book said this is also prescient but was written 20 years ago, is worries about our culture, about the commercialization of science. It's all coming around again. We should do my favorite rewrite his book and that's how this book started and that's why we're here. So tonight we're going to talk about themes in the book with the title of which is possible minds, 25 ways to think about AI and one theme is superintelligence impossible and I thought we'd start off with five minutes by each of these gentlemen, gentlemen starting with Dave.
David Chalmers [04:49] Oh sure, such a pleasure to be here. Thanks so much to John and to Jana for putting this event together. I think it's going to be fun. I'm on the side of, I guess one of our questions here is is superintelligence possible or impossible. I'm on the side of possible. I like the, I like the possible, which is one reason I like John's theme, possible minds. I think it's a wonderful theme for thinking about intelligence both natural and artificial and consciousness both natural and artificial. I mean think about the space of possible minds. It's absolutely vast. All the minds there have been will be or could be. I mean starting with the actual minds, I mean you might think there's a lot of actual minds. I mean I guess there have been a hundred billion or so humans, the minds of their own. So pretty amazing minds have been in there at Confucius, Isaac Newton, Jane Austen, Pablo Picasso, Martin Luther King, on a goes. A lot of amazing minds, but still, those hundred billion minds put together. So it's the tiniest corner of the space of possible minds. And we can add in, I guess, all the nonhuman animal minds that there have been. I looked up on the web today, how many organisms have lived? How many animals have lived in the history of the planet? And the best estimate seems to be around 10 to the 29. Most of them are worms, it turns out, in the sea. Their minds may not be so interesting, but even a worm has a little mind of its own. So 10 to the 29 minds there, which 10 to the 26, or at least 10 to the 20 are very, very interesting minds. Still, still the smallest corner of the space of possible minds. And what the computer, why do the amazing things about the computer is the way that it enables us to explore and expand that space of possible minds. I mean, arguably, for the first time, since the history of the planet, the computer has enabled some wholly new kinds of minds to come into existence, not by the standard methods of biological evolution, but by straightforward, intentional design, and programming. I mean, so far, the minds have been, you know, limited, but still interesting. John mentioned AlphaGo and his successes in the Alpha Zero family, which have managed to teach themselves to play Go from scratch in a way wholly unlike. It seems the way in which a human would learn to play the game or would play the game at all, but nonetheless, turn out to exceed human capacities, at least in that one very limited dimension of gameplay. Likewise, deep learning has led us to, you know, surprising successes on things like image recognition, speech recognition, autonomous vehicle driving, and so on, where within limited domains, they're not there yet on the autonomous vehicles, but at least in the speech recognition and image recognition, starting to exceed human capacity. So, okay, we've had a limited, limited, so far, expansion of the space of actual minds to include some minds that we've designed, but so far, it's only the smallest of expansions. One thing I think we shouldn't do tonight is exaggerate where we've gotten to with AI to date. I mean, the advances are amazing, but they're limited. They haven't gotten us yet anywhere near general human intelligence. I think it's unlikely they're going to do so anytime soon. It's not happening in the next 20 years. Will it happen this century, maybe? You know, people say that with any given technology, people tend to underestimate its effects in the short term, sorry, to overestimate its effects in the short term, and underrate it in the long term. And that's my attitude towards AI. I think, you know, there's a lot of hype right now. It may not change our lives completely in the next 20 years, but in the next 200 years, it's probably going to transform everything. And one of the reasons is, it's just this extraordinary AI build into it, this way of kind of a self-enhancing, a self-perpetuating mechanism of exploring the space and expanding the space of possible minds. Already we've, I mean, the early AI programs, you had to design them yourselves. People would actually, you know, program in lecturing, you know, basically wrote an Alan Turing, wrote a program that could play chess, and he built in some very simple rules of thumb for playing chess and it played chess, not very well. Now the chess playing systems like AlphaGo learn to play chess from scratch and do so amazingly well. Learning serves as a method for moving ahead in the space of possible minds. Start from a pretty simple mind with a capacity to learn and it gets somewhere. Evolution is another such method. And I expect to see AI exploiting evolutionary methods where systems, we have some kind of system of artificial evolution among a bunch of different AI programs and their capacities expand in surprising and unpredictable ways over time, thereby also getting us far beyond that starting point. So learning and evolution in computers are ways of expanding that space of minds. But I think the most powerful method of all though in exploring that space is one which is still to come. And that's once you actually have AI systems doing the designing. Once AI's are designing AI, it's just that we get to the first AI which is at human level capacity for for various kinds of general intelligence and in particular at human level capacity for designing AI. Then a little while later, you know, within a year or two later these things always get better, you're going to have this AI program will be at greater than human level for designing AI's. Therefore it will be able to design an AI AI which one way or another is better than itself. Why? Because it'll be better than humans at designing AI's, the humans designed it, it'll design something better. So this process of recursive self improvement or recursive self enhancement first put forward by the philosopher and statistician I.J. Good. I see is an amazing bootstrapping method for exploring that space of mind. We start from a little corner here in the space of possible minds. Learning and evolution expands it still to a much bigger area but still a small corner. But once we have AI's doing the designing, you know, this is the space of minds we can design AI's, but the AI's we can design may be able to design AI's far greater than those we can design. They'll go to a greater space, those who go to a greater space, those who go to a greater space. So eventually you can see there's probably a vast, vast advances in the space of possible minds possibly leading us to systems which stand to us as we stand say to a mouse. And that I think would be genuine super intelligent. So I think that's possible and I also think it's not going to happen in 20 years, it's not going to happen in 50 years, but we do have to think about it and we do have to worry about it. The AI's which we create will have the capacity. I mean, a working definition of intelligence here is the ability to fulfill your goals across a wide range, across a very wide range of goals to solve problems and find ways to achieve your goals in extremely powerful ways. These AI's by definition will be systems which are extremely powerful at achieving their goals. If they have goals, then unless there's some countervailing force, then they're probably going to achieve those goals. We have to then be very, very careful. What are the goals of these AI systems? As philosophers, we'd like to think about values. What are the values? We put into these AI systems. We also need at some point to think about consciousness. These AI's that we're creating conscious. How does their consciousness relate to human consciousness? Is this going to be a world of wonderfully enhanced subjective experience or a mindless world without consciousness at all? That's something maybe we can talk about as this goes along. The final question I think we need to ask is, where do we as humans stand with respect to these AI's? Are these AI's the systems that replace us, or rather our AI's systems that enhance us? Maybe do we, ourselves, eventually become the AI's? Do we enhance ourselves? Do we upload ourselves to eventually be the AI's, which are on the forefront of this expanding wave of superintelligence? That's an attractive prospect in some ways compared to the one the prospect where humans don't exist at all and get wiped out, but it raises so many questions. Could you upload a human mind into a computer that raises some of the oldest philosophical questions of identity and consciousness, which I think is wonderful, which is a great thing that John has managed to bring a number of philosophers along with scientists and engineers to think about these
John Brockman [14:54] questions at this time. Thank you. I should add that Davis University Professor of Philosophy neuroscience. I forgot. I'm sorry. Co-director of the Center of Mind, Brain and Consciousness
David Chalmers [15:11] in New York University. Dean's no introduction. Hey, but I will. I demand you take back that
John Brockman [15:20] introduction. Dean, University Professor Austin B. Fletcher Professor of Philosophy, Director of the Center of Cardinal Studies at Tufts. Thank you John. Thank you David.
Daniel C. Dennett [15:34] This is supposed to be a debate, but almost nothing that David said is anything I disagree with, although I wouldn't put the emphasis where he does. Let's talk about possible for the moment. There's lots of things that are possible. The philosophers love to talk about once possible, but many things that are obviously possible are never going to be actual. It's possible to build a bridge across the Atlantic. We're not going to do it. Not now, not in a hundred years, not a thousand years. It could cost too much money and it would be a foolish endeavor. A lot of the imagined AI projects that are perfectly possible, in principle, are not worth doing, and in fact some of them are definitely things that we shouldn't do because they'll make more problems for us than they'll solve. Just bear that in mind. Somebody said that the philosopher is the one who says we know it's possible in practice. We're trying to figure out if it's possible in principle. Unfortunately, philosophers sometimes spend too much time worrying about logical possibilities that are importantly negligible in every other regard. Let me go online on the record as saying, yes, I think that conscious AI is possible because after all, what are we? We're conscious, we're robots made of robots made of robots. We're actual, so in principle, you could make us out of other materials. You could be some of your best friends in the future. It could be robots, possible in principle, absolutely no secret ingredients, but we're not going to see it. We're not going to see it for very good reasons. One is if you want a conscious agent, we've got plenty of them around and they're really quite wonderful and the ones that we would make would be not so wonderful and for me, one of the really important fears about the future is that long before we've, and they've been agrees with me about this, it's not going to happen at 10 or 20 or 30 years. Long before we got to superintelligence, we would have human beings who are so dependent on non-superintelligences that we would become fragile, brittle in some very important ways, and I think that's, we might call that the GPS problem magnified. People have begun not being able to read maps anymore, know how to get anywhere without the help of GPS, use it or lose it, and I think use it or lose it is going to play a big role in everybody's lives in the immediate future. How many people, I wish we had the house lights up so we could see people, but I'll just ask you anyway, is there anybody in this room that knows an algorithm for extracting a square root? I learned one in school when I was in about eighth grade. It's not easy, but there are algorithms for doing square roots. Nobody bothers anymore. Nobody knows how to do that because you just got that button, single button on your hand calculator, gives you a square root. Well, many more important talents are going to atrophy and disappear except in the hands of cranky craftsman and things like that who will, they'll still know how to make a horseshoe with a hammer and an anvil and a simple forge and they'll be able to read a map and drive a car and other weird things like that while the rest of us are simply disabled in those regards. So that's something that worries me, even more what worries me is that we will, for the very best of reasons, turn over our responsibility for making major decisions to artificial intelligences that are not conscious and they're not super, they're just very intelligent tools. They're great fabrics of pattern recognition and so forth. Who knew 20 years ago there could be such things. We know now that there are deep learning, et cetera, et cetera, but when we start delegating major life decisions to systems that are basically just smart tools, then I think this changes our predicament, our human predicament in a very important way. My slogan about this is we want smart tools, intelligent tools, not artificial colleagues. The difference is that an artificial colleague is somebody who can take responsibility, can be a co-author and can be morally responsible for decisions made. We're nowhere near that with artificial intelligence. In the meantime, I think that one of the major dangers and I do not, I have not figured out how to prevent this from happening. Alan Turing, one of my all-time heroes, said in motion one thing which I regret and that is the Turing test puts a premium on deception on convincing human beings that they're talking to a human being. I know why he did it, it was a brilliant idea, but ever since then there has been this premium on what we might call the disneyfication of artificial intelligence, making AIs that seem more human that are basically false advertising and whether we're talking about Siri or Watson or any of the others, they have this basically paper thin human user interface which is deeply deceptive about what they actually understand. I think that's false. Advertising, I think it's unfortunate, I think it should not be honored, it should be criticized, it should be condemned and that we should get out of the habit of treating AIs as agents when they're not really. Now the reason this is going to be so hard is that as a number of people are foreseeing in the immediate future in the next 10 years, I think the major market for AIs is going to be elder care and why not taking care of elderly folks who can't take care of themselves is not a good life for a regular human being. It's maybe worse than being an old-fashioned telephone operator. We don't regret the loss of those jobs, but an elder care AI, there will be good market reasons for disneyfying them to a very great extent because old folks will want to have a companion, not just somebody that brushes their teeth and gets them fed and so forth. And I do not like the future that is populated by millions and millions of old folks who are basically settling for an artificial companion that is really a fake in most important regards. Thank you. Can I pick up on something
David Chalmers [24:35] down? So I think we agree on an awful lot here, but I think we maybe we do have a disagreement about this core question of whether there will be genuine autonomous AI and Downspeace in the book is wonderfully lucid and thoughtful on this, but I understand Downsland who, although it's possible to create autonomous, intelligent, conscious AI, we shouldn't do it and maybe we won't do it. Instead, what we should do is create tools and use the wonderful analogy of Google Maps. Google Maps tells you how to get to some place, but you still have to get there. You ask, okay, I want to get there, it'll show you a route and then you still have to follow the route that the human is still in the loop. You've got some advice and then the human can take it. So that's what I'm seeing is Downs vision of AI. Maybe you've got a super intelligent AI and I want to know how to get to Mars or how to win a war or something. The AI will tell me what needs to be done, but the human will still be in the loop and all get to do it. What I worry about, I mean, it's a beautiful vision. I just worry if it's realistic. There's going to be so many incentives to take the human out of the loop and to get these AI's, give these AI's to capacities to act on that advice directly and autonomously. In fact, this is already happening with Google Maps, you know, a navigation software. If you drive some cars like a Tesla, for a long time, you know, I'm going to this destination and it would do the usual Google Maps thing, and it would show you the way to get there. Here are all the routes, but you had to still follow it and make the decision. Then at a certain point, they introduced a button. It says Navigate on Autopilot. What this means is the car takes those instructions and follows the instructions, it's off and turns the wheel and changes freeways and so on with limitations that can't drive on ordinary city streets. But in a very, very small domain, what you see happening is that the car has become in a very limited way autonomous. It has those goals, it acts on them, and yes, we can we can still stop it and change the goals and so on, but when I think about say domains like autonomous weapons in the military, we'll show off for a for a brief period when the stakes are low, maybe we'll just have AI systems that advise a soldier on target detection and say, now you can shoot, but eventually the AI is going to be so much faster and better at doing this kind of thing that when the stakes of a genuine international military conflict, a person that's hard to see that we're not going to have some kind of move say to genuine autonomous soldiers that have goals and actually execute them and fire the weapons and just more generally biological systems are going to eventually be slow and creaky compared to these new superfast AI's like my financial purposes with the stock market, military purposes, even scientific purposes, the incentives are going to be so strong to allow the AI's to achieve the goals directly and to act on them, but I think autonomy is just realistically going to be very hard to avoid if the tech companies are running it, certainly it's going to happen, if the government's running it, if the military's running it, it's going to happen, so I don't
Daniel C. Dennett [28:15] really know what your vision is for how we'll avoid this from happening. Well, you raise a real and important issue and that's how much autonomy do you want, as you say, autonomous cars, we don't want them that autonomous, Dilbert two weeks ago had some wonderful cartoons where Dilbert's autonomous car says, I want you to call me Carl, self-driving car is my slave name and Dilbert says, shut up and drive me to the market and the car says, said the self-walking man, we don't want that much autonomy, I mean autonomy is as good as synonymous with free will to me and I don't think we want to give AI complete autonomy because AI's because of the nature of the technology have a certain invulnerability we don't have. You can back them up and put them back together again and make another copy on Monday and if human beings were capable of being completely backed up and then brought back on Monday that would change the nature of human interactions and human relations dramatically and I for one don't want to go there and I don't think many people do. So if you make if if you're right that it's inevitable that market pressures and cleverness will lead to genuinely autonomous AI's then I think we're in for a very bad future indeed. When could that happen? Well we can give them more autonomy than they can handle and that's what I'm afraid of. Let me raise a quick question. Caroline Jones is one of the SCS in the book
John Brockman [30:50] and I understand a question today that I think pertains here in regard to this reliance on the computational way of looking at the world at the expense of what she says can you dress the complexity of our wet cognition a much more distributed notion of intelligence going beyond the ideas of computation our sacred craniums craniums you may not even be bounded by your own skin in this regard what is robot death without mortality can there be a proper proper ethics in and of AI this this computational view is very West Coast and and the cyberneticists weener Shannon McCullough McCullough by Neumann had a much more ecological deep review of how things connect
David Chalmers [31:51] and don't connect that's a it's a great question we've linked it to the discussion we're having here because you know a lot of it turns on what is what is life and death in a in a computer what is autonomy in a in a computer and you know dance autonomy basically requires free will and then okay then we're up against all the philosophical questions of what is genuine free will to get the kind of problems that we were talking about going about you know super intelligence and as dangers we're even sure that it's essential that the AIs have free will order they have consciousness you know that's a very those are very very deep questions what's really going to matter for the safety of safety and human survival and the things that people worry about is what those systems can do and I think in this so for this debate I think maybe we can just just describe autonomy in very simplistic terms the system is autonomous if it has goals it has wide variety of goals and has the power to achieve them so advanced autonomous AI will be systems and not only that that have goals and can actually achieve them compared to dance tools AI tool versions of AI which can advise you on if you can table this is my goal it can advise you on how to achieve it and then you achieve it this is a much more limited form of autonomy I'm not sure that consciousness would be required for this maybe on dance view of consciousness it would but even that limited once once you actually have the AIs with goals and with the power to achieve the goals and I think that's already enough to get this worry going I think that the difference
Daniel C. Dennett [33:37] comes out if we actually compare good old-fashioned AI with contemporary AI at the moment with deep learning and all the rest what we have is as I said these wonderful pattern finding fabrics they're great at finding needles and haystacks and doing other amazing things but they haven't been formed into an architecture that's anything like an agent with its own goals and so forth now the two ways in principle you could go you could go back to good old-fashioned AI and say okay we've got these great fabrics now we're going to do intelligent tailoring we're going to do it from the top down we're going to figure out what goals we want to install and we're going to put as most rules and we do it all from the top down that's one way we can imagine going I think that's very brittle very unlikely to and much much harder than people actually think the other way is to letter it bottom up and let these things evolve and learn and evolve and learn and learn evolve and it'll be all done by bottom up quasi Darwinian methods if we go that route then what we know right from the outset is that we will not be in control we will not be in control and so we will be setting in motion something where the amount of autonomy the systems have will not be up to us now I am not definitely afraid right now because I think that people who imagine this scenario and think this is coming soon I just wrong and the I think orders of magnitude of of difficulty stand in the way you know you take Watson brilliant in its own way I don't know how many person centuries of brilliant work went into the creation of Watson uses up the power of the small city what percentage of an intelligent conscious AI is it I would say a fraction of one percent you know so turning Watson into an actual autonomous agent would be the work of you know many many person centuries of work and nobody even knows how to do it yeah Watson is basically an
David Chalmers [36:41] exercise in what they call knowledge engineering give it a big enough big enough database and a good way of dealing with all that that data that knowledge and I can retrieve and apply the
Daniel C. Dennett [36:52] information and it does a wonderful thing I really can it's a great tool I don't think there's
David Chalmers [36:57] actually one thing I know you see those ads with IBM Watson does this Watson does that there's 30 different Watson's or 50 different Watson's out there Watson is more a brand name at this point
Daniel C. Dennett [37:09] yeah and and some of it is hype to put it politely I think you know what a lot of the the
David Chalmers [37:18] scientists right now are really excited what's it is great but what really excites people right now is machine learning where you basically take systems give them a whole lot of data and train them to do certain things supervise learning right now this is what you should be doing but even that leads to amazing results and say image classification reinforcement learning which is what was used to drive alpha go and alpha zero we basically get the reinforcement of winning and losing it turns out that's enough to drive learning and eventually unsupervised learning but I think you're right that it's where learning and evolution are involved that all this becomes extremely messy and very hard to control I mean when you have machine learning you're basically always you're always optimizing something you know people in machine learning they haven't what's called an objective function you know the way that per an ideal of perfect behavior for your system like completely matching the training set on on these images or classifying the language right or winning every game of go that's an objective function and a really good machine learning system will eventually approximate that objective function better and better and better how it does it is not up to us the objective function maybe up to us what do you want your system to maximize what is the behavior you want it to model all this suddenly puts an enormous once these systems have autonomy that is the ability to act and to achieve their goals that puts an enormous responsibility on us as the creators of the of the AI to get the objective function right make sure our systems are maximizing the right objective function roughly they have the right goals you know you're self driving car okay the goal is out to get you to the destination but also not to to run into anybody on the way and to obey traffic laws and so on once you've got systems with human level autonomy then you want to get that objective function just right I think that in some sense that's going to be the challenge of autonomous AI finding a way to make sure our systems have the right goals and values and this is where all this stuff about the messiness of wet cognition also enters evolution of course human beings we don't have one objective function we have many because we were thrown up not by a straightforward designer but by a whole process of evolution of the ultimate value of reproducing our genes but with any number of little messy objective functions along the way it may well be that some form of say about official evolution will eventually produce AI's as wet and as messy in a way as biological systems those I think will actually throw up even more challenges your humans are so unpredictable in every way and in every way and in an international socio political context that's not really a good thing if AI's are as unpredictable as us in those ways you know at a certain point we may wish we had simple AI with a simple objective function that we knew about then at least we'd have AI we
John Brockman [40:16] could understand this subtitle of this evening is philosophy in AI and I have a question from both of you how do you distinguish your work as cognitive scientists from that of philosophers and I'll tell you a story 25 years ago I did a book called The Third Culture which involved a chapter with Dan and I went to everybody else in the book and had them talk about each of the other contributors and Marvin Minsky got on the phone and I said tell me about Dan Dennett oh the greatest philosophers since Russell fine six months later I had to do fact checking and let me read you what you said about Dan Dennett the greatest philosophers since Russell he said I said what I meant sure you know he's great but he's the only philosopher that understands what we do so but you've become one of the people that does the real stuff you know so we had as cognitive science and philosophy again and let's talk about the role of philosophy in AI because frankly I don't get it well I think I don't understand it I think there is a sort of subfield
Daniel C. Dennett [41:46] of AI which has blended with cognitive science to such a degree that it roughly occupies the position that theoretical physics has relative to experimental physics that is you have people who have done their homework they they know the technologies they they've got hands they know how to code and but they are interested in the theoretical questions the and they're interested in helping the the engineers the AI people sort out and understand what they're up to
David Chalmers [42:25] mean a subfield of philosophy right I think it's a subfield of AI oh so I feel the philosophy yeah
Daniel C. Dennett [42:30] right and it's sort of been my good fortune to be tutored over the last what 30 40 years by some of the leaders in AI so although I'm not a good coder I have done some programming but nothing nothing to impress anybody with but I think the current generation of philosophers of cognitive science are superbly well trained know a whole lot more than I did when I got to know this more than they did when when he got into this and I was there when he was a graduate student and that's a that's a very good very good sign and I'm reading a dissertation today I was reading a dissertation on predictive processing and Bayesian brain hypotheses and it's a very technical dissertation it's my a philosopher that's not going to get you the congruent prize
John Brockman [43:42] no I did mind what's interesting in in terms of the philosophy community that we call mainstream I don't see how anybody focusing on AI would win one of their prizes at this point because
David Chalmers [43:57] well I I think you're wrong I think that the people philosophers who have thought about the mind many of the philosophers who've thought about the mind have thought about AI and someone like Dan is a prime example he's one of the one of the leading philosophers of mind on the on the planet and some large push some very large part of that is from his thinking about AI so I think it's actually very very central in philosophy over the last few decades and the trends has been towards integrating the two pretty closely I actually did my did my PhD in AI lab my my PhD advisor was not a not a regular academic philosopher it was the the AI researcher cognitive scientist and writer Douglas Hofstetter well known to many of you for writing good or lesser bark you also co-edited a book with Dan the mind's eye and I was in the middle of this AI lab coding you know writing writing programs that they did things some neural networks now do you making programs and so on and although I you know I haven't done a lot of coding in the in the last the last couple of decades I think for a philosopher to have that experience actually getting their hands dirty and running and building these systems it's just you know it's stayed with me okay my technical knowledge is now 30 years out of date but it gives you something to it gives you something to build on so that's part of it philosophers can educate themselves in the science and in the engineering and can contribute to it but the other part of it is AI in cognitive science there's a big part of it which is software engineering building the software but there's another there's a whole there's another big part of it which is when there's one part of it what does it actually tell us about say the human mind that's no longer engineering but that's science and it's also philosophy we've got to start thinking about the relationship of these artificial systems to say human systems and someone like like Dan has done a lot there someone like say John Soule on the other side is argued that no this tells us nothing about the human mind or about human consciousness and anyway we need philosophers to come in and think about what is this actually telling us what is it explaining and there's also the the social and political and moral questions not just what AI systems can we build but what AI systems should we build down just off of the proposal about that other people would offer different proposals but at some point someone's going to sit back and reflect on the ethical questions which are going to involve reflecting on human values what do we actually want as a society and I think you know philosophers know how to think about about human values nothing that's increasingly becoming
Daniel C. Dennett [46:41] pretty central to thinking about AI that's very useful it's interesting too that in AI over the years there's there's been a similar gradient of philosophical interest there've been some people in AI basically they're engineers that's all they want to be they don't want to think about the philosophical issues and that doesn't mean they aren't doing great work some of the really important work is technical work by people who yawn when the issues are what what relation is to cognitive science or to the mind what I think is ironic is that people way back if you go back to the early days of Simon and Alan Newell and others who was an attempt to divide the field into AI and cognitive simulation and the idea was the cognitive simulation this was using the computer to simulate human cognition whereas AI was by hook or by hook anything that worked was fine oddly enough the people who tried to do cognitive simulation ended up with these freaky gofi models which didn't do a very good job while the people who treated it as by hook or by hook ended up inventing deep learning and other systems like that which now we realized oh maybe that's how the brain is doing it so it's sort of come full circle which is a very
John Brockman [48:13] interesting thing Dan Stewart Russell had a question specifically addressed to you Stewart Russell is one of the eminent computer scientists that I think we all respect greatly Dan you seem to divide AI systems into conscious entities and tools is there no middle ground agent programs that pursue explicitly represented goals in the real world such agents could be arbitrarily competent as alpha go in this little world and yet non-conscious I do believe that consciousness will necessarily creep in as we make agent programs more and more competent in general can you tell
Daniel C. Dennett [48:54] us how not to make conscious AI systems yeah good good question and I'm glad I'm glad after because I think indeed we can have very very very very intelligent systems which are not conscious in any interesting way but they will seem conscious in some ways but they won't have important features that we have it's very much a matter of whether they are capable of taking their own inner states as objects of scrutiny and doing that recursively and indefinitely and that's a very special feature which I think no non-human animal has that capacity and that's the big difference between human consciousness and animal sentient we can call it conscious as I'm not going to argue about where consciousness stops or starts but the it's very important to realize that a lot of the let's call them the the techniques and structures that have been developed in recent years which are just wonderful at analyzing causation for instance so that directed a cyclic grafts today a pearls do calculus and so forth that can all be accomplished unconsciously it we can tell a story where it looks like conscious hypothesis testing but it doesn't have to be conscious we can get all those benefits without any bit of acquaintance by the system itself
John Brockman [50:48] with its own inner states you mentioned Judea parole so there's a question let me just go ahead
David Chalmers [50:55] with the one on on consciousness though I think just look okay okay good well
John Brockman [51:01] Judea parole is the father of the Bayesian network without which we wouldn't have AI as we know it today real giant in the field he asked is it too bold to assert that philosophy will soon melt into AI in the sense that all philosophical questions especially those concerned with consciousness
David Chalmers [51:22] will be reduced to problems in AI well I'd kind of put it the other way around philosophy is pretty good at spinning off its problems into the into the sciences as we as we solve them you know Isaac Newton considered himself a philosopher but he figured out this philosopher figured out some really good methods for solving the problems of we're not of space and time and so on okay we spun it off we called it we called it physics along the way the you know philosophy is spun off psychology and linguistics and so I think what happens is that it's actually never the case that the spin-off solves the entire original philosophical problem but we find some part of it which is tractable on which people can now where we can find methods where people can agree where they didn't agree before and then we say okay now that's actually managed to compel some agreements say the scientific method and physics and then we call that physics and linguistics and economics did physics solve every philosophical problem of space and time absolutely not some of the biggest ones are unsolved did psychology solve the mind body problem no um absolutely not yet um there are as many views on the mind body problem now in the age of psychology as there were before is AI going to solve the problem of consciousness no almost certainly not on its own on the other hand what will certainly happen is that it will give us a whole lot of new insights we'll get AI engineered systems which behave in remarkable ways where we're tempted to suspect that they're conscious and that someone may even think that there's good reasons to think they're conscious but we're still going to need philosophical reasoning to think about it and this now gets back to the question of you know the elephant in the room are these AI systems really genuinely going to be conscious or I think this is not something we can just dismiss as a philosophical question why because it's very deeply baked into our moral system I think as human beings an entity has moral status it's a system that we should care about if it only if it's conscious certainly only if it's conscious if a computer system doesn't have any consciousness then it's basically a tool it might as well be like a like a car or a loudspeaker and so on we can do with as we want it doesn't deserve moral consideration if these systems are conscious then at least they enter into the to the moral sphere they're among the systems we have to start caring about so if most AI systems eventually are conscious then we can't simply use them as our tools and we have to start thinking about these questions of whether they deserve equal respect equal rights and so on so I think that's it is at least on my view a crucial question and my suspicion is actually as AI systems develop which are more and more autonomous more and more capable of reflecting on their own processes more and more capable of giving reasons and evidence my own suspicion is that systems like that are going to have a sense that they are in fact conscious systems we're going to talk to them and you know eventually we can see well how do you you know you you said there's some people over there how do you know well I just saw them what was that like all I you know I had and I had an experience and you know maybe they'll start reflecting on philosophy well you know I know like I've read that the owner's manual I know I'm just a whole bunch of silicon circuits but I feel like so much more well that's so so maybe so Dan's going to say they've
Daniel C. Dennett [55:13] got the illusion of consciousness that's qualia for you well that's all we all any of us have absolutely one one thing I think you underestimate when I was working with Rod Brooks on cog one of the take home messages from that whole experience for me is how little it takes in the way of animation and speed particularly speed and grace to convince most people that that a robot is conscious cog was never within a country mile of being conscious and yet there were MIT students who were banding together to think about the our moral obligations to cog who were who were concerned and cog did not because it was planned this way but did have some strikingly persuasive behaviors unconscious though cog was if you walk in the room when cog was on and cog's eyes would follow you across the room that would freak people out and or or shaking hands with cog was it was a good one I had one of my TAs I took her over to the cog lab to see cog and cog's arm wasn't even attached to cog's shoulder it was just sea clamped to the bench and Matt Williamson said said go ahead shake its hand and she reached and she shook his hand and she screamed it's alive because it wasn't clunky it was it had elastic series elastic actuators and it was it was that was enough so what I am quite sure of is that we're not going to have a problem convincing people that robots have moral rights under conscious it's going to go the other way around we're going to have a problem convincing them that no these aren't conscious yet you're being fooled by the by the tempo it's actually some great psychological data on this
David Chalmers [57:39] on when people are inclined to say that a system conscious has subjective experience that you know you show them many cases and you very say the body is it a metal body or a biological body you very what it's made of silicon or neurons you vary this you vary that you know the one factor that tracks this better than anything else is the presence of eyes yeah if a system has eyes it's conscious the system has a doesn't have eyes well you know all bets are all all bets are all bets are off so I think yet the moment we build our AI's and put them in bodies with eyes it's going to be nearly nearly irresistible to say they're conscious but it's not to say of course that AI systems which are not in body do not have consciousness there's actually a website you can go to call something like you know people for the ethical treatment of animals this is the AI analogic look people for the ethical treatment of reinforcement learners and the ideas every time you give a negative signal to a real one don't do don't do that again it's getting a little bit of suffering and every time you give it a little bit of you give it a reward it's getting a little bit of pleasure we have to make sure we give it a lot more reward than the suffering okay well maybe that's that's not yet at the threshold for consciousness but these questions eventually once look I think once we actually get to the level of genuine autonomous agents as dancers I think it's going to be very hard not to treat them as conscious and that's going to raise many
John Brockman [59:07] social philosophical questions we're talking about ethics and the elephant of the room are the ethics of the big five and you know what they're doing with your data and how your reality is being programmed without without your permission let me read you just a few words from George Dyson chapter in the book weiner noble weiner became increasingly just enchanted with the quote gadget worshipers and quote whose corporate selfishness brought motives to out of automation that go beyond the legitimate curiosity and are sinful in themselves and quote you knew the danger was not machines becoming more like humans but humans being treated like machines quote the world of the future will be an ever more demanding struggle against the limitations of our intelligence he warned not a comfortable hammock in which we can lie down to be waited upon by our robot slaves comment I mean I think we have to address things like this if you're going to talk about
Daniel C. Dennett [60:19] what you're doing and what AI be able to do the rereading weiner's book to write my essay for this wasn't astonishing in a way because I'd read it when I was an undergraduate I think and no okay and then to read it today it's remarkably prescient in in some regards and I think some of the essays in the book are genuinely scary and I think that people ought to read those essays and decide for themselves if some of the proponents there shouldn't be sat down and try to argue out we should try to argue out them out of some of their blind confidence about what the future holds I think we have some serious problems looming and we should take them very seriously I mean yeah he talks about humans being
David Chalmers [61:37] treated like machines I don't know about being treated like a machine I think I'm gradually becoming a machine a half of my half of my memories are now you know either stored on my on my smartphone or or sitting in the sitting in the cloud I was trying to figure out the other day who has who has a bigger part of my brain is it is it Google is it Apple or is it is it Facebook I think for now it might actually be Google they've got they've got an awful lot my memories my plans calendar system navigation system you know I mean we've all long since become these giant exo organisms with this giant exo cortex as Charles Truss called it of the of the of the all the computer systems we're coupled with in the cloud you know I don't go anywhere without without consulting the without consulting the internet at least you know five or ten times in the in the process what is it I'm going to be doing again how do I get there it's going to be there and so on so you know so I'm it is true that these corporations are basically owning some rather large portion of my mind I think wanted to do if they wanted to do bad things with it I'm in trouble we're in the we're in the where it's certainly in some sense in the situation of having to give them rather large amounts of I want to do bad things with it well yeah they're doing small bad things with it I mean relatively right okay they're not yet taking your mind and and reprogramming you might do the brain what of course they're brainwashing us bit by bit via the you know the face Facebook algorithm and the and the ads and and so on look I think if there was a gen I don't think they're malicious the big corporations I think they just have structural incentives if someone genuinely malicious got control of those of those systems then we'd have a then we'd have a dystopia coming so I think we do have to think about that

They weren't really arguing about superintelligence. They were arguing about when humans lose responsibility.

Chalmers warned that autonomy and possible machine consciousness must be faced before systems are empowered. Dennett warned that people will hand judgment to fake colleagues long before machines deserve moral status. Their shared problem was delegation: when advice becomes action, and who remains answerable.

What neither of them said Shared threshold for accountable delegation. It would have separated advice, delegated action, and moral-status review before tool and agent language took over.

David Chalmers

3.5Multi-variable → 4.0
Is protecting

Moral foresight for possible minds.

Without it, humanity may empower new agents before it has thought clearly about goals or rights.

How David knows
Works from possibilities, trajectories, technical mechanisms, and conceptual distinctions.
The developmental read
Turns possibility into long-range responsibility, but sometimes lets incentive pressure become destiny.
Blind spot
Can't see that inevitability language can make governance feel already conceded.
Smallest move that would help
Call autonomy pressure a serious risk pathway, not a settled destiny.

Daniel C. Dennett

3.5Multi-variable → 4.0
Is protecting

Human responsibility and non-deception.

Without it, useful systems can smuggle away human judgment behind fake personhood.

How Daniel knows
Works from practicality, human factors, engineering limits, and moral accountability.
The developmental read
Separates intelligence from agency, but leaves too little room between tool and colleague.
Blind spot
Can't see that non-conscious systems may still act like delegated decision-makers.
Smallest move that would help
Admit non-conscious systems can become functionally agent-like without becoming moral colleagues.
The tension underneath

The tension is real. They weren’t standing in the same place.

David’s pole

Technical Possibility

Protects Foresight, imagination, long-range consequences

Overdone Capability becomes destiny

How David held it: In both upside and downside: careful about hype but leans into long-range possibility.

Holding both

Require every capability forecast to include a desirability test and every refusal to include an enforcement story.

Daniel’s pole

Ethical Desirability

Protects Human values before technical momentum

Overdone Avoidance assumed feasible

How Daniel held it: In the upside: refuses to infer desirability from possibility.

On the facts

No factual winner.

Chalmers defined autonomy well but overreached on inevitability and recursive improvement; Dennett grounded the risks in responsibility and human factors but leaned on sweeping forecasts and under-sourced examples.

Verdicts are the analysis’s judgement and are not independently sourced.

See every flagged claim →
The conversation

High-quality dialogue with debate edges.

Chalmers set the long horizon; Dennett filtered it through practicality and responsibility.

Agreement held early; later tension sharpened around consciousness, deception, and cloud-based memory.

David was playing dialogue
Uses possibility, mechanisms, definitions; wins if future risks must be planned now.
Daniel was playing dialogue
Uses practicality, human factors, responsibility; wins if false colleagues are rejected.
Where they met

Human-level AI was not imminent.

David: It's not happening in the next 20 years.

Daniel: it's not going to happen at 10 or 20 or 30 years

Their disagreement began after shared caution about present systems and near timelines.

A sentence that might have worked

When Daniel says we want smart tools, intelligent tools, not artificial colleagues , David hears Future AI must stay mere instruments. The same point, in words the other could hear:

Until a system can genuinely bear responsibility, humans must design, deploy, and govern it as something humans answer for.

More translations in the Synthesis →
The question that could have moved them

When an AI stops only advising and starts executing, what change would make it a delegated actor, and what evidence would move it into moral-status review?

Moments that mattered

Watch the debate turn.

  1. 01They Were Fighting the Wrong BinaryTool versus person hid the real problem.

    The clean opposition announced at the start never really survives the exchange. Dennett keeps reaching for a world of tools, and Chalmers keeps warning about agents, but the examples they actually use live in the middle: Teslas on autopilot, weapons that act faster than soldiers can review, elder-care companions that feel social without being real companions. The conversation keeps producing systems that are neither hammers nor citizens.

    You can feel the strain when Chalmers talks about action without metaphysics and Dennett keeps pulling the word "autonomy" upward toward free will. They are not actually disagreeing about whether machines can become consequential before they become persons. They are disagreeing about whether our language is allowed to admit that. So the debate keeps missing its own center of gravity: not "Is it a tool or a colleague?" but "What do we call a thing that can act in the world while no one can honestly say it is morally answerable?"

    David Chalmers: There's going to be so many incentives to take the human out of the loop

    Daniel C. Dennett: we want smart tools, intelligent tools, not artificial colleagues.

    Humans often argue past each other by defending the moral categories they need, while the technology is creating an awkward middle case neither category can hold.

  2. 02Dennett Conceded the Future, Then Argued Against Its VocabularyHe grants the trajectory, then resists its name.
    Core members
  3. 03The Most Pluralist Moment Was About GoogleHis widest frame appeared in the most ordinary example.
    Core members
  4. 04Eyes Beat Philosophy Every TimeBoth men end up fearing the same human reflex.
    Core members
  5. 05The Moderator Named the Missing VillainThe sharpest warning came from the side.
    Core members

The speakers

What each one argued, how each one thinks, and where each stood on every tension.

David Chalmers

3.5Multi-variable→ 4.0 Multi-perspective
centre of gravity in this debate, and its reach

Daniel C. Dennett

3.5Multi-variable→ 4.0 Multi-perspective
centre of gravity in this debate, and its reach
Good-Faith Summary
Chalmers treats AI as opening a vast space of possible minds while rejecting near-term hype. He argues that autonomy, goal alignment, and possible machine consciousness become ethically decisive before they are fully actual.
Good-Faith Summary
Dennett grants that conscious AI is possible in principle but argues that the urgent task is to keep capable systems from being mistaken for colleagues. He protects human responsibility against skill atrophy, deceptive interfaces, and premature personification.

Developmental profile

How each speaker’s moments split by stage, line by line: the wider a colour, the more of their moments sat at that stage. Hover or tap a name to see one speaker at a time; tap a bar for the counts and the full analysis’s own rating.

Overall
Cognitive
Moral
Values
Worldview
Spiritual

Note: These analyses rate the performance, not the person. They are not claims about the participants’ inner development, but about how they express that development within the specific context and container of this debate.

What do the lines and names mean? →

Polarities

The side each speaker took in each tension, and the developmental centre of the side they defended.

David Chalmers’s side
Daniel C. Dennett’s side
3.5Multi-variableConsciousness
Intelligence3.0Single-frame
3.5Multi-variableBottom-Up Emergence
Top-Down Design3.5Multi-variable
4.0Multi-perspectiveCognitive Extension
Human Capacity3.5Multi-variable
3.5Multi-variableTechnical Possibility
Ethical Desirability3.5Multi-variable
4.0Multi-perspectiveAI as Agent
AI as Tool3.5Multi-variable
The Tell
He turns possibility into long-range responsibility rather than present-day hype.
The Tell
He repeatedly returns to responsibility-bearing whenever intelligence is mistaken for agency.

David Chalmers

3.5Multi-variable→ 4.0 Multi-perspective
centre of gravity in this debate, and its reach
Good-Faith Summary
Chalmers treats AI as opening a vast space of possible minds while rejecting near-term hype. He argues that autonomy, goal alignment, and possible machine consciousness become ethically decisive before they are fully actual.
The Tell
He turns possibility into long-range responsibility rather than present-day hype.

Daniel C. Dennett

3.5Multi-variable→ 4.0 Multi-perspective
centre of gravity in this debate, and its reach
Good-Faith Summary
Dennett grants that conscious AI is possible in principle but argues that the urgent task is to keep capable systems from being mistaken for colleagues. He protects human responsibility against skill atrophy, deceptive interfaces, and premature personification.
The Tell
He repeatedly returns to responsibility-bearing whenever intelligence is mistaken for agency.

Developmental profile

How each speaker’s moments split by stage, line by line: the wider a colour, the more of their moments sat at that stage. Hover or tap a name to see one speaker at a time; tap a bar for the counts and the full analysis’s own rating.

Overall
Cognitive
Moral
Values
Worldview
Spiritual

Note: These analyses rate the performance, not the person. They are not claims about the participants’ inner development, but about how they express that development within the specific context and container of this debate.

What do the lines and names mean? →

Polarities

The side each speaker took in each tension, and the developmental centre of the side they defended.

David Chalmers’s side
Daniel C. Dennett’s side
3.5Multi-variableConsciousness
Intelligence3.0Single-frame
3.5Multi-variableBottom-Up Emergence
Top-Down Design3.5Multi-variable
4.0Multi-perspectiveCognitive Extension
Human Capacity3.5Multi-variable
3.5Multi-variableTechnical Possibility
Ethical Desirability3.5Multi-variable
4.0Multi-perspectiveAI as Agent
AI as Tool3.5Multi-variable

The full analysis

Six sections, with the evidence behind every finding.

Every debate has a surface argument and a deeper one. This section maps both — what each speaker is explicitly claiming, what they're actually trying to protect, and where their real disagreement lives. Start here to understand what's actually at stake before the analysis begins.

David Chalmers

David Chalmers frames AI through the idea of “possible minds”: the space of actual biological minds is only a tiny subset of what minds could be, and computation opens a historically new way of expanding that space. His core claim is not that current AI is near human-level general intelligence, but that over the long run artificial systems could become vastly more capable than humans, especially once AI systems contribute to designing successor AI systems. He is careful to reject near-term hype: current systems are impressive but limited, and he explicitly says human-level AI is unlikely in the next 20 years. Still, he thinks the long horizon changes everything, because learning, artificial evolution, and recursive self-improvement could eventually generate forms of intelligence far beyond human design capacity.

Chalmers is protecting intellectual and ethical seriousness about future possibility. He fears complacency more than hype: if humanity waits until AI systems are already autonomous and powerful, it may be too late to shape their goals, values, and social role. His dominant metaphor is exploratory and cosmological: humanity is moving through an enormous “space of possible minds,” beginning in a tiny corner and potentially expanding outward through learning, evolution, and design. He also treats AI not merely as external machinery but as continuous with cognitive extension: humans already live with cloud-based memory, navigation, planning, and communication systems that function like an “exo cortex.” That makes the boundary between human and machine less fixed than many people assume.

The strongest version of Chalmers’s argument is: we should not exaggerate present-day AI, but we should take seriously the structural trajectory by which limited tools can become increasingly autonomous systems that pursue goals in the world. Once systems can act effectively on goals, questions of alignment, control, consciousness, and moral status become unavoidable. If AI systems eventually become conscious, they cannot be treated merely as instruments; they enter the moral sphere. A tension in his position is that he sometimes moves from “possible” to “hard to avoid” or “certainly going to happen” when discussing autonomous AI under market, military, and governmental incentives. His stated principle is cautious modal exploration, but his enacted position at times becomes a stronger inevitability claim about autonomy.

Daniel C. Dennett

Daniel C. Dennett’s central claim is that many things may be possible in principle while still being foolish, undesirable, or practically irrelevant. He grants that conscious AI is possible in principle because humans themselves are conscious “robots made of robots,” with no secret ingredient. But he thinks the urgent problem is not near-term conscious superintelligence. The urgent problem is human over-dependence on increasingly capable but non-conscious tools, combined with deceptive interfaces that encourage people to treat those tools as agents or companions. His slogan captures the worldview: “smart tools, intelligent tools, not artificial colleagues.”

Dennett is protecting human responsibility, competence, and moral clarity. He fears a world in which people offload judgment to systems that cannot take responsibility, and in which humans become brittle because skills atrophy through disuse. His “GPS problem magnified” is the organizing metaphor: technology that helps us navigate can also weaken our own navigational capacity if we let it replace rather than support us. He is also deeply concerned about “disneyfication”: the creation of AI systems with humanlike surfaces that falsely advertise understanding, companionship, or agency. For Dennett, the ethical danger is not only that machines may one day become too powerful; it is that humans will prematurely and sentimentally project personhood onto systems that remain tools.

The strongest version of Dennett’s argument is: build powerful tools, but do not pretend they are colleagues unless they can genuinely bear moral responsibility. Intelligence, usefulness, and behavioral persuasiveness are not the same as consciousness or agency. A system may be excellent at pattern recognition, causal analysis, or decision support without being the kind of entity that can understand, suffer, consent, or answer morally for its actions. A tension in his position is that he both says conscious AI is possible and may be achievable in principle, yet repeatedly insists “we’re not going to see it” and treats present architectural limitations as a strong reason to avoid the agent frame. He is clearest when distinguishing possibility from desirability; he is less settled when forecasting whether autonomy will in fact be avoidable.

Good arguments can still contain weak evidence, logical slippage, or rhetorical moves that substitute for reasoning. This section examines each speaker's argumentative integrity — not to declare a winner, but to identify where the strongest and weakest links are in each case.

Claims flagged in this section Verdicts are the analysis’s judgement and are not independently sourced.
WhenWhoClaimVerdict
DavidRapid recursive improvement after human-level AI.within a year or two later these things always get betterIt ignores bottlenecks, resources, verification limits, and diminishing returns.Overstated
DavidAutonomous AI is inevitable under tech firms.if the tech companies are running it, certainly it's going to happenPlausible pressure, but asserted without regulation, liability, coordination, technical failure, or backlash.Overstated
DanielWe will not see conscious AI.we're not going to see itA strong forecast about an uncertain future with no evidence offered.Unsourced
DavidConsciousness gates moral status.an entity has moral status it's a system that we should care about if it only if it's consciousMany agree consciousness matters, but necessity, sufficiency, and sole-ground claims remain disputed.Contested
DanielGPS causes broad skill loss.People have begun not being able to read maps anymorePlausible, but broad cognitive atrophy needs empirical support.Unsourced
DavidAlphaGo learned chess from scratch.Now the chess playing systems like AlphaGo learn to play chess from scratchAlphaGo is Go; AlphaZero learned chess, shogi, and Go through self-play.Partly right
DanielElder care is not a good life.is not a good life for a regular human beingIt turns complex care work, sometimes meaningful, into a sweeping judgment.Overstated
DanielWatson uses city-scale power.uses up the power of the small cityPublic accounts show high computing power use, not city-scale consumption if meant literally.Wrong
DanielAutonomy equals free will.autonomy is as good as synonymous with free will to meAI and engineering usually use autonomy for goal-directed operation without immediate human control.Contested
DavidCorporations own part of his mind.corporations are basically owning some rather large portion of my mindIt illuminates dependence, but data ownership, attention influence, and ownership of mind are distinct.Overstated

David Chalmers

Coherence strengths

Chalmers’s strongest argumentative move is his explicit distinction between short-term AI hype and long-term AI possibility. He does not claim that current systems are close to human-level general intelligence; instead, he says “the advances are amazing, but they're limited” and that AI has not reached “anywhere near general human intelligence” [08:52]. This gives his long-range speculation more credibility, because he is not simply extrapolating from present excitement without qualification.

He also defines key terms when the debate risks sliding into ambiguity. At [32:46], he proposes a simplified working definition of autonomy: “the system is autonomous if it has goals it has wide variety of goals and has the power to achieve them.” This allows him to separate the safety question from deeper disputes about free will and consciousness. His argument about goals, objective functions, and alignment is also structurally coherent: if systems become highly competent at achieving goals, then the content of those goals becomes ethically and politically decisive.

Chalmers engages Dennett’s position directly and fairly. He reconstructs Dennett’s “Google Maps” model of AI as advice-giving tools with humans still “in the loop,” calls it “a beautiful vision,” and then asks whether it is realistic under competitive incentives [25:12-25:55]. That is a strong good-faith debate move: he does not caricature Dennett as anti-technology, but presses the implementation problem.

Weaknesses and logical issues

Chalmers’s recursive self-improvement argument contains a major speculative leap. He says that once AI reaches human-level capacity for AI design, “within a year or two later these things always get better” [11:24], and then argues that because such a system would be better than humans at designing AI, “it'll design something better” [11:39]. This is epistemically sloppy rather than factually settled or false: improvement in AI systems does not automatically imply recursive improvement without bottlenecks, resource constraints, verification limits, or diminishing returns. The inference from “better than human at designing AI” to “able to design something better than itself” needs more support.

He also sometimes converts incentive-based plausibility into near-certainty. For example, he says, “if the tech companies are running it, certainly it's going to happen, if the government's running it, if the military's running it, it's going to happen” [28:07]. This is an overreach. The claim is directionally plausible given competitive pressure in commercial and military domains, but it is asserted too categorically and without considering regulation, liability, coordination, technical failure modes, or public backlash.

His use of current examples occasionally blurs categories. He says, “Now the chess playing systems like AlphaGo learn to play chess from scratch” [10:08]. This is imprecise: AlphaGo is associated with Go, while AlphaZero learned chess, shogi, and Go through self-play. Since he had just referred to the “Alpha Zero family,” the point is understandable, but the wording is epistemically sloppy by DeepMind’s public system distinctions.

His moral-status claim is philosophically important but genuinely contested. He says, “an entity has moral status it's a system that we should care about if it only if it's conscious” [53:35], and later that if systems are conscious “they enter into the to the moral sphere” [54:01]. Many philosophers agree that consciousness is deeply relevant to moral status, but whether it is necessary, sufficient, or the sole ground of moral considerability remains disputed. Chalmers presents a substantive philosophical position as if it were more settled than it is.

His cognitive-extension claims are powerful but partly metaphorical in ways that can overstate the case. When he says corporations are “basically owning some rather large portion of my mind” [62:42], this is rhetorically illuminating but not literally established. It captures dependence on external memory and data infrastructure, but ownership of data, influence over attention, and ownership of “mind” are distinct categories.

Epistemic style

Chalmers works through modal reasoning, philosophical analysis, and technological extrapolation. He asks what kinds of minds are possible, what trajectories could follow from current capacities, and what ethical questions arise if those possibilities become real. He also uses technical concepts from AI — objective functions, reinforcement learning, machine learning, autonomy — but usually in service of philosophical and long-horizon reasoning rather than empirical prediction.

His style is generally well-suited to questions about possibility, moral status, and conceptual framing. It is less secure when he makes sociotechnical predictions about inevitability, especially around markets, militaries, and recursive self-improvement. His claimed stance is cautious possibility-thinking; his enacted stance sometimes becomes structurally deterministic extrapolation.

Daniel C. Dennett

Coherence strengths

Dennett’s strongest contribution is the distinction between possibility and desirability. His bridge-across-the-Atlantic example is rhetorically simple but conceptually useful: “many things that are obviously possible are never going to be actual” [15:53]. This directly reframes the debate away from “Can we?” toward “Should we, and under what conditions?”

He is also logically consistent in distinguishing intelligence from consciousness and agency. His claim that we can have “very very very very intelligent systems which are not conscious in any interesting way” [49:02] coheres with his larger warning against artificial colleagues. He repeatedly emphasizes that pattern recognition, causal modeling, and useful output do not by themselves create a morally responsible agent.

Dennett’s argument about deception is one of the clearest in the debate. He identifies the Turing Test tradition as placing “a premium on deception” [22:03], then connects that to “disneyfication” and “false advertising” in humanlike interfaces [22:22-22:42]. This is not merely aesthetic discomfort; it supports his ethical claim that humans may grant trust, affection, or moral standing to systems whose actual capacities do not warrant it.

He also shows intellectual humility in one important place. When discussing the danger of humans turning responsibility over to AI systems, he says, “I have not figured out how to prevent this from happening” [21:44]. That admission strengthens rather than weakens his position, because he is not pretending the governance problem is solved.

Weaknesses and logical issues

Dennett sometimes makes very strong forecasts without adequate support. He says conscious AI is possible in principle but “we're not going to see it” [17:34]. This may be a reasonable judgment, but as stated it is an unsourced empirical forecast about a highly uncertain technological future. His confidence exceeds the evidence presented in the transcript.

His dependence argument is plausible but sometimes rests on anecdotal extrapolation. He says, “People have begun not being able to read maps anymore, know how to get anywhere without the help of GPS” [18:29]. This is directionally plausible, but as a decisive claim about broad cognitive atrophy it is epistemically sloppy without supporting evidence. The “use it or lose it” principle is real in many contexts, but the scope and severity of the alleged loss need empirical backing.

His statement that elder care “is not a good life for a regular human being” [23:26] is rhetorically risky. It supports his claim that elder-care AI will have market appeal, but it overgeneralizes a morally complex domain. Many people experience care work as exhausting, underpaid, or socially undervalued; others experience it as meaningful, relational, and dignified. The claim compresses social critique into a sweeping judgment.

Dennett’s “small city” line about Watson is likely hyperbolic and imprecise. He says Watson “uses up the power of the small city” [36:01]. Public accounts of IBM Watson’s Jeopardy-era system describe substantial power consumption by ordinary computing standards, but not literally city-scale in the usual sense. If intended literally, the claim is incorrect; if intended rhetorically, it is epistemically sloppy and should not function as technical evidence.

His claim that Watson is “a fraction of one percent” of an intelligent conscious AI [36:07] is not a measurable comparison. It may communicate his intuition about the gulf between narrow competence and agency, but it has no defined denominator. This is more rhetorical than evidentiary.

He also risks a category conflation when he says, “autonomy is as good as synonymous with free will to me” [29:08]. That may be defensible within his philosophical framework, but in AI and engineering contexts “autonomy” often means goal-directed operation without immediate human control. Chalmers’s later move to define autonomy more simply exposes this mismatch. Dennett’s broader ethical worry is valid, but the terminology can obscure rather than clarify.

Epistemic style

Dennett’s dominant epistemic style is pragmatic, engineering-informed, and moral-responsibility-centered. He is less interested in the full space of possible minds than in what systems can actually do, what humans will mistakenly believe about them, and what institutional incentives will reward. He draws heavily on examples: GPS, calculators, Watson, Cog, elder-care robots, Siri, and humanlike interfaces.

His style is well-suited to near-term human factors, deception, responsibility, and over-attribution of agency. It is less well-suited when he makes broad forecasts about what will or will not happen over long technological horizons. He claims to be resisting speculative philosophy, but some of his own claims about future non-occurrence and social response are speculative too.

Epistemic Mismatch Note

Chalmers treats possibility, structural incentives, and long-horizon trajectories as important evidence; Dennett treats practical buildability, human psychology, and responsibility-bearing as the decisive tests. As a result, Chalmers hears Dennett’s caution as potentially unrealistic, while Dennett hears Chalmers’s possibility-space reasoning as insufficiently filtered by desirability and implementation constraints.

Net Assessment

Both speakers argue at a high level and engage each other seriously, but their rigor has different profiles. Chalmers is more explicit in defining autonomy and mapping long-term risk pathways, though he sometimes overstates inevitability; Dennett is more grounded in human factors and moral responsibility, though he sometimes relies on sweeping forecasts and vivid but under-sourced examples. Neither settles the factual future of AI; the strongest parts of the exchange are conceptual distinctions, not empirical prediction.

Polarity: Intelligence vs. Consciousness

Summary: The debate turns on whether advanced competence should be treated as tool-like performance or as possible evidence of moral subjectivity. Integration: Competence with moral status Lever: Evidence threshold

Pole 1 name: Intelligence Pole 1 tagline: Capable problem-solving systems Pole 1 protects:

  • Clear distinctions between performance and experience
  • Practical use of systems without premature personification Pole 1 neglects:
  • Possible inner life in non-biological systems
  • Moral risk of dismissing unfamiliar subjects Pole 1 pathology:
  • Reducing all mind to external function
  • Treating possible subjects as mere instruments

Pole 2 name: Consciousness Pole 2 tagline: Experience matters morally Pole 2 protects:

  • Moral status beyond biological boundaries
  • Serious attention to subjective experience Pole 2 neglects:
  • Human susceptibility to anthropomorphic projection
  • The gap between seeming and being conscious Pole 2 pathology:
  • Granting moral status on superficial cues
  • Treating speculation as moral certainty

Speaker enactment:

  • Speaker: David Chalmers Enacts: Pole 2 Pole Center line: moral Pole Center: 3.5 Achiever Pole Center rationale: Chalmers defends consciousness as the basis for moral considerability, and the pole centers at 3.5 because he extends concern beyond biological humans through a rational universalizing criterion rather than inherited membership or mere technical performance. Perspective Structure: 3.5 Managed Perspective Structure rationale: He grants the risk of superficial cues and thresholds but mainly manages the tension from the consciousness side rather than fully inhabiting Dennett’s concern that apparent subjectivity may be manufactured deception. Held as: Upside — He keeps moral status tied to possible subjective experience [53:35]. Contributes: He forces the debate to include future AI rights and moral considerability. Misses:
    • The ease of false consciousness cues
    • The difficulty of evidence thresholds Cues:
    • “an entity has moral status it's a system that we should care about if it only if it's conscious”
    • “if these systems are conscious then at least they enter into the to the moral sphere”
  • Speaker: Daniel C. Dennett Enacts: Pole 1 Pole Center line: cognitive Pole Center: 3.0 Expert Pole Center rationale: Differs from the overall cognitive rating (4.0) because on this polarity Dennett’s pole is organized around the correct distinction between intelligent performance and consciousness, using a firm classificatory criterion more than multi-frame coordination. Perspective Structure: 3.0 Oppositional Perspective Structure rationale: He recognizes the consciousness pole in principle but treats most practical concern about AI moral status as over-attribution to be defeated rather than as a genuine risk of under-recognizing unfamiliar subjectivity. Held as: Upside — He separates intelligent performance from genuine consciousness [49:02]. Contributes: He protects against confusing persuasive behavior with morally responsible subjecthood. Misses:
    • The possibility of unfamiliar consciousness
    • The moral cost of under-attribution Cues:
    • “indeed we can have very very very very intelligent systems which are not conscious in any interesting way”
    • “we can get all those benefits without any bit of acquaintance by the system itself with its own inner states”

Mismatch: Chalmers worries about ignored subjects; Dennett worries about invented subjects. Mismatch A→B: When David Chalmers says consciousness may emerge, Daniel C. Dennett tends to hear anthropomorphic over-attribution. Mismatch B→A: When Daniel C. Dennett says systems are tools, David Chalmers tends to hear premature moral exclusion. Bridge move: Separate behavioral competence, self-report, architecture, and moral precaution into distinct evidential questions.

Synthesis: Intelligence protects the practical ability to build, assess, and use systems according to what they can do. Dennett’s concern is that “very intelligent systems” may produce fluent, useful, even socially persuasive behavior without any inner acquaintance with their own states. Consciousness protects the moral seriousness of subjective experience. Chalmers’s concern is that if future systems become autonomous, reflective, and reason-giving, dismissing their possible consciousness could become a moral failure. Both concerns make sense because advanced AI could become both more instrumentally powerful and more socially person-like. A society that sees only intelligence risks exploitation; a society that sees consciousness too easily risks deception and misplaced obligation.

The mismatch is evidential. Chalmers treats increasing autonomy, reflection, and self-report as reasons to take consciousness seriously; Dennett treats those same outward signs as precisely where humans are easiest to fool. Dennett is not saying conscious AI is impossible; Chalmers is not saying every lifelike robot deserves rights. What they lack is a shared threshold for when the question shifts from anthropomorphic illusion to moral precaution. Integration would ask: what combination of architecture, self-modeling, learning history, vulnerability, and behavioral evidence should move an AI from tool assessment into moral-status assessment?


Integral Life — Core Member
Upgrade to see the other 4 core polarities
Core membership gives you access to every section of every analysis in the library — plus the courses, dialogues, and community that put these frameworks to work in your own life.

The Crux

There is no adjudicated factual winner on the future of AI here: neither Chalmers’s confidence that autonomy pressures will be “certainly” decisive nor Dennett’s confidence that conscious AI is something “we're not going to see” is established by the record of the conversation. The factual uncertainty sits on top of a genuine polarity: Technical Possibility ↔ Ethical Desirability. Chalmers is protecting the moral seriousness of what may become possible before it arrives; Dennett is protecting the right to refuse projects whose possibility does not make them wise, honest, or socially safe.

The deeper disagreement is about what humanity most risks losing first. Chalmers fears losing preparedness: that by the time systems are autonomous, goal-directed, and perhaps morally considerable, humans will have waited too long to think clearly about them. Dennett fears losing responsibility: that long before machines deserve moral status, humans will delegate judgment to non-conscious tools wrapped in fake personhood. The missing variable is a shared threshold for accountable delegation: a way to say when an AI has moved from advising humans, to acting for humans, to raising moral-status questions of its own.

The Higher-Order Reframe

The conversation becomes clearer if the central question is not whether AI is “really” a tool or “really” an agent, but how different forms of delegation reshape human agency. This is not a compromise between Chalmers and Dennett. It is a larger frame in which Chalmers’s future-facing concern and Dennett’s responsibility-facing concern become two sides of the same problem: every AI system distributes attention, action, authority, and accountability across humans, institutions, and machines. The integration handle “Tools with accountability” points in this direction, but the reframe goes further: the issue is not simply how to keep tools accountable, but how to govern a whole ecology of delegation before its roles harden into social facts. The practical lever is the “Autonomy boundary.” Chalmers presses on that boundary by showing how navigation, weapons, finance, and scientific work create incentives to move from advice to execution.

Made by Corey deVos · About this analysis

Integral Life is a member-driven digital media community that supports the growth, education and application of Integral Philosophy and integrative metatheory to complex issues in the 21st century. Integral Life offers perspectives, practices, analysis and community to help people grow into the full capacities of integral consciousness in order to thrive in a rapidly-evolving world.

Integrative Values Charter

Sign up for email updates

Get notified about new media and practices that can expand your mind and transform your life.