Transcript of journalist and senior media executive Richard Sergay's interviews with Walter Sinnott-Armstrong, Jana Shaich Borg, and Vincent Conitzer for the “Stories of Impact” series.

Watch the video version of the interview.

RS =  Richard Sergay (interviewer)
WSA =  Walter Sinnott-Armstrong (interviewee)
JSB =  Jana Shaich Borg (interviewee)
VC =  Vincent Conitzer (interviewee)

WSA: I’m Walter Sinnott-Armstrong. I'm the Chauncy Stillman Professor of Practical Ethics and the Kenan Institute for Ethics in the philosophy department at Duke University. But I also work in the Duke Institute for Brain Science and the law school and in conjunction with the computer science program and the psychiatry department. 

RS: Tell me about the genesis of the Templeton project and what do you hope to achieve. 

WSA: Well the genesis is simply our concern about the role of artificial intelligence in our society today. It seems to be pervading many, many aspects of life, some that are quite surprising and raise some concerns. What we want to achieve is to figure out which of those uses are ethically acceptable to people and which we think are morally justified but also maybe if we're going to try to program morality into computers so the computers will be less likely to make decisions or perform actions that we find morally problematic.

RS: The title for the project?

WSA: We tend to just call it moral artificial intelligence. But it could go by a number of names.

WSA: One reason we live in a moral artificial intelligence is there's a certain ambiguity to it. It's mark-- it's artificial intelligence that we view as doing moral things, but it's also building morality into artificial intelligence so the system itself is making moral judgments. So that nice short title seems to capture both aspects of the project. 

RS: What's the need to imbue a machine with morality?

WSA: Well one reason is that humans aren't very good at it. They make a number of different mistakes. They for example overlook morally relevant features. They get confused by very complex problems and they have biases and they're not very quick. If you have an emergency situation in an automated vehicle, the computer can calculate quite quickly. If you have a sudden change of plans and an autonomous weapon, humans will, under the pressure of the situation, make mistakes. And our hope is that computers might be able to do it better.

RS: And you assume the computer will not make a mistake.

WSA: Absolutely not. Of course computers will make mistakes. Nothing's perfect. If your goal is perfection, you might as well give up now. What we want to do is reduce the number of mistakes and reduce in particular the kinds of errors that we have identi-- that I just identified. Ignorance, confusion, and bias.

RS: Define morality. 

WSA: Morality is impossible to define. I have a whole series of articles arguing this which I won't go into detail but the kind of morality that we're talking about, which is not what everybody considers to be morality, is composed of two elements. One, reducing harm to others and respecting other people's rights. So you want to reduce the amount of harm that people suffer in their lives. But you want to do it in a way that doesn't violate the rights of individuals along, along the path.

RS: Who gets to define morality, over the centuries it's clearly changed?

WSA: Moral beliefs have changed. I'm not sure that morality has changed. A hundred years ago when a husband raped his wife violently, they viewed that as ok in some places, because they thought the wife was just the property of the husband. That's what they believed. I think they were wrong. So I don't think the moral rights of women have changed at all. It's just that now we recognize those rights and before they didn't. 

RS: So if computers existed 100 years ago the morality that would have been imbued in this machine could be quite different from the one today. 

WSA: Absolutely, absolutely. And I suspect that if morality is put in a machine today it will be different from what happens 100 years from now, because we haven't got it all right yet. But what you have to do is do the best you can with the views that you have and admit that some of them might be wrong. Again there's no perfection, there's no guarantee, there's no certainty. The trick is to use machines to help us get a little bit better.

RS: Is there a single human morality? 

WSA: Again I want to distinguish moral beliefs from moral rights and moral actions. I think there is a single morality that applies to all humans including, to repeat my example, don't rape. But does that mean that everyone recognizes that. No, unfortunately not. 

RS: How do you think a moral machine fits in the latest chapter of evolution. 

WSA: Well machines are not part of the evolution of humans. They're something we create. Automobiles are not part of the evolution of humans. Evolution is a biological process that changes our bodies and our cells and the way they interact. But it changes the environment in which we use it. And then it indirectly affects our evolution. I remember this movie where it suggested that humans might evolve to the point where they only had one finger because all they have to do is push buttons. I don't think that's going to happen. We're not going to radically change in that way. But maybe our brains will get bigger and better because we're going to have to interact with machines that will require different types of cognitive apparatus than we had before. I wouldn't be surprised if there were some changes like that. So I think machines can affect human evolution but they're not really part of human evolution.

RS: And imbuing a machine with morality. How, how is that done and who are the decision makers in trying to architect that. 

WSA: Well different people do it different ways. I'll tell you about our project. We are not trying to take our favorite moral theory, build it into a machine, have it apply to problems and have everybody agree with us. That's not the goal at all. Our project looks at survey data about what a wide variety of people take to be morally relevant features and then develops an algorithm again based on experimentation with actual humans alive today, to determine how those different features interact. And to-- and then the algorithm predicts which judgments humans would make. Notice that they're not always going to agree. Sometimes 90 percent all agree, sometimes 70 percent, sometimes 50 percent. And then it's up to humans what you do with that information. But the machine can give you a better sense of which judgments humans would make if they considered all the morally relevant features, that is the features that they themselves take to be morally relevant, which are legitimate. They don't think those are illegitimate biases and they're not getting confused by the complexity of the situation. So our project is not taking our theory and applying it to those problems, but instead trying to figure out the kinds of computations that humans go through in order to reach the moral judgments that they reach. Correcting those for mistakes that those people themselves see as mistakes and then taking that algorithm and using it to predict and suggest a moral judgment in a given situation. 

RS: Example?

WSA: An example that we have focused on is kidney exchanges. Sometimes when you're distributing kidneys to potential recipients you have to decide which one gets it because you've got one kidney and lots of potential recipients. So should you give it to younger people rather than older people. What if the person has been on the waiting list longer than another person. What if this person's, they're responsible to a certain extent for their own kidney problems because they did something that helped cause those problems. What if they have large numbers of dependents at home. There are lots of different features that people might take to be morally relevant.And yet when people think about those they might forget one of those. They might misunderstand one of those. Why does it matter whether someone's younger or older. Is that just all about life expectancy? And they might get confused on a multitude of different morally relevant features and they might be biased. They just don't like this person who's going to receive the kidney but that shouldn't affect their judgment and they know it shouldn't but they can't stop themselves. So humans are going to make those decisions currently and they're going to be subject to kinds of mistakes that we think a properly programmed computer could help avoid. Then the product-- the suggestion would be, the human still has to make the decision but the computer can say this is the decision that you would make given your values and the things that you yourself take to be morally relevant, given them interacting in the way that you yourself take to be appropriate and getting rid of all those biases that you yourself take to be features that should not figure into your moral judgments. Then the computer can say this is a judgment you would reach if you corrected your own judgment in that way. And then if the judgment you think is right or wrong disagrees with the computer, now you've got a problem. But if it agrees you feel ok.  The computer has helped me confirm I'm going to be more confident. If it disagrees, then we have to do some research and we have to think about it more. So one thing computers are going to do is they're going to help us determine which are the problems you really have to focus on and figure out what's going on. And the rest of them  it's going to say yeah, that one works, that one works, that one works.

RS: Are you testing it now?

WSA: We're testing it now in very limited ways. I wish I could say we had made more progress but this is very slow work. And so we are testing it in the sense that we had a demo and a computer science conference where we brought people in and had them make judgments. We figured out an algorithm from each of them on the basis of former data and then tried to predict what they would do in a given circumstances. And it worked better than chance but not as well as we would have hoped.

RS: So right now a transplant board at a hospital will make a decision about who gets the kidney the way it currently works. (YES) Your hope would be in the future that you put the machine into operation and it will help the decision-makers understand the complexity of the universe of decisions that they could reach. 

WSA: Yes, in part. But also these committees are very busy. So let's suppose they have to make 10 different decisions in a meeting and they're 10 people. And they've got an hour. Well what are they going to do? They might in advance look at the case and say what we ought to do in case number one is this, what we argue in case number two is this. Number three, number four, number five. And then the computer says according to the values you've expressed in the past, we think you ought to do this, right. Given your values corrected for biases and ignorance and confusion. Then they can say oh of those 10 cases there are only two where we're disagreeing with the computer. Let's spend the hour on those two cases. Right. And so you can know which ones you need to focus on and pay more attention to those and figure out what's going on. Now notice at the end of the discussion there are two possibilities. You might say, the computer is wrong. At which point you enter that data into the computer and the computer can recalculate the algorithms and learn from that experience. So over the course of years it gets better at predicting what the committee agrees upon reflection. But you can also think oh, we were wrong, because we overlooked something that the computer took into consideration. I guess we got confused. I guess we were biased because we didn't like this person. Or because we did like this person, they're a friend of ours, a donor of a hospital,  something like that. And so there can be a number of different reactions. But the goal is to help the committee make the judgments according to the features that they themselves take to be morally relevant. 

RS: How do you define bias?

WSA: Well I think bias is when you make a moral judgment or make a decision on the basis of factors that you-- yourself think you should not be making them on the basis of. Not that you, that you should not be making. Now you recognize it as a bias when you yourself recognize that. So for example, if I walk into a room and there's an African-American and I sit five feet from the African-American, whereas if this person was European-American I would have sat one foot from them and started talking to them, then that's a bias. Now I don't want to be like that. And yet the data shows that most people are like that. They've got biases of that sort that are going to affect their daily lives even though they're not aware of them. And even though they think they're wrong. And so it's not unusual to think that these hospital administrators would have biases that are affecting their decisions that they're not aware of and that they themselves think are wrong. It happens to all of us. We're all human and the computer can help us figure out when it's happening and then correct for it.

RS: Could the computer differentiate bias that might be geographical in nature where folks from the South might have a different sense of bias to those for example from the West. How do you compensate on a general level for bias that might be culturally specific or geographically specific. 

WSA: Well there's no easy answer to that question. I think it will take time. Let me give you an example. Criminal record, if you have two people who both need a kidney and one of them is a murderer, maybe released from prison, maybe still in prison, and the other has a clean criminal record, no criminal record at all. Then should the fact that they have a criminal record, affect whether they get a kidney or not. Let's suppose in North Carolina they say yes it should. And in California they say no it shouldn't. And you can imagine arguments on both sides. And so what do we do? Well I don't want to come in and say you know, California, you're wrong or, North Carolina, you're wrong. Instead I think these are state-funded hospitals in that particular community. They should probably try for a while and see how it works out. And see what happens to the way the system operates and the way people adjust to those views. And it might turn out that the people in California realize that if you're giving kidneys to these murderers, either in prison or having been released, then there are other people dying who were good contributing members of the community. Whereas the people in North Carolina might realize oh my gosh, what that's doing is introducing racial bias, because most of the murderers in prison who were caught you know, are African-Americans. And then we don't want that. So they can learn from each other by seeing each other's experience over the years. Now notice the computer is then not solving the problem in an instant. This is not the kind of problem you can solve in an instance. This is something that the computer is going to inform both of their decisions. But you still need years of experience to work out that disagreement and maybe it'll never get worked out. I'm not sure that's so bad, maybe different communities have different values. That's what democracy is all about. Different states have different values and make somewhat different laws. I would be upset if there were a state that said we're never giving a kidney to an African-American. Of course that would be horrendous. But there are many decisions on the border where I think those decisions are reasonable. Even though I've got my favorite.

RS: Wouldn't you agree that there is a basic set of human values that a machine would need to be imbued with, that all could agree on? 

WSA: Absolutely. Here's one. You should not cause death to a person with moral rights unless you have an adequate justification or excuse, almost everybody is going to agree to that. Of course what they're going to disagree about in cases of euthanasia, what counts is death, in cases of omission, what counts as causing. Who actually has more rights, do prisoners have rights, do fetuses have rights, do animals have rights. What counts as a justification or an excuse. When as a justification or excuse adequate. So yes there are going to be universal principles that people can agree on, but that doesn't rule out individual variation about how those different variables get filled in, in a particular community. There's a myth that universal morality is in conflict with local variation. You can have both if you have a principles and parameters view of the way morality is structured. 

RS: What does that mean?

WSA: That there is a general principle like you should not cause death to organisms with moral rights unless you have an adequate justification or excuse. That's a principle. One parameter is what counts as causation. Another parameter is what counts as death. Another parameter is what counts as a justification. What counts as an excuse. Another parameter is when are those adequate. Those are the different parameters that get filled in by the local morality. None of that rules out an abstract, universal principle that everyone is subscribing to.

RS: You referenced earlier that the transplant board could go back to the computer and the computer potentially learns from a possible mistake or a miscalculation. Talking a little bit about that, how does that work? You're actually imbuing a machine with thinking skills or learning skills.

WSA: Learning skills, exactly, they call it machine learning or deep learning. And what happened in say the game of Go is the machine just plain millions of games of Go against itself and learn which moves worked. Same thing's going on here. So we take people on the Internet or face-to-face and we have them do say 30, maybe 50, of these choices. Who gets a kidney a or b. Who gets a kidney C or D who gets a kidney E or F. They do 30 to 50 of those, and then the computer can develop an algorithm that's supposed to reflect the computations that they are making in that training set as it's called. And then we have a test set where we give ten new ones, and not the ones they were trained on, but 10 new cases. And the computer is supposed to predict what they're going to say about those 10 cases. Let's suppose the computer gets eight of those right. But what about the two that went wrong? Now it has to figure out how to adjust the algorithm in light of those mistakes. So that it'll be a better predictor for the next set of ten, and even better for the next set of ten, and even better than the next set of ten. So the computer is learning as it gets more and more experience with people's moral judgments. It can learn to get better and better at understanding human morality.

RS: My thought bubble when we talked about this at St. Andrews is HAL. Where a computer takes on a mind of its own and makes decisions that it thinks are correct despite what the human may think and takes action. Are you building a HAL? 

WSA: No we're not building a HAL. Notice that HAL went against what, I forget the character's name. Dave. No, we're not building a HAL. Notice that HAL went against what Dave thought was the right thing. Now we've got to ask, wait a minute, why do we trust Dave. We have certain empathy for Dave because they're both humans but humans make mistakes too. So I don't remember the movie well enough, so let's say there are two Daves. One where Dave is making a moral judgment about what ought to be done. That almost all other humans would agree with. Well then we don't want the computer to go against it. But if Dave is making a decision that's going to be good for Dave but contrary to everybody else, then maybe we want HAL to stop him. So what's the decision there, it's, we want a HAL that is going to reflect what most or almost all humans would think is the morally right thing to do. Which may or may not agree with the individual Dave who might be out for his self-interest. We want the computer to reflect human morality in general, rather than a particular individual. And that's what our program allows us to do because we can take the data from large numbers of humans, we can develop an algorithm that predicts what most humans would say, and in the situation where the computer is out for its own interest it doesn't want to get turned off, right. And contrary to human interests, then we want the computer to be programmed so it'll do what's in human interests, not in the machine's interest.

RS: Human interest can vary though, there could be one Dave and another Dave. So how do you define his interest. 

WSA: Remember what we're doing is we're asking people what do you think are the morally relevant features. We're asking people how they interact. We're asking people which one should we get rid of because they're illegitimate biases. That's what we want built into the computer. That's what we're defining you know are the human interests or the human judgments that we want the computer to conform to. But you're absolutely right. They're going to be, you know, Dave's that do different things. We don't want the computer to do the same thing in every circumstance. We want it to be sensitive to those particular variations and that's where the morally relevant features come in. When the features change the judgment ought to change. And that's what we want the computer to do, not simply to follow a simple rule in every circumstance. That would be a mistake.

RS: So tell me the upsides of imbuing a machine with morality are?

WSA: One upside is that it's going to improve human moral judgment. Because the hospital ethics committee now is making these decisions about who gets a kidney, they've got no check on them. How do they know whether they got it right? There's-- there's nothing to tell them. I've been on these hospital ethics committees before and you reach a consensus but you still think, I don't know. It was a tough case. The computer can either agree or disagree and when it disagrees you have to think about it more carefully. And what that means is that you might have to recognize, I forgot something in that case. I got confused in that case. Oh no way, I had a bias in that case that I don't want to have. And so it can train me to be a better version of myself by reflecting the judgments that I would make if I were not ignorant, confused, and biased. 

RS: Are there upsides? 

WSA: Other upsides is people get kidneys when they ought to get kidneys. If there are fewer mistakes then the people who need the kidneys and who deserve the kidneys are going to be more likely to get them. You're also going to have people who, well, maybe they don't deserve it as much now but they've got a special condition which makes it very unlikely they'll get matched in the future. Well that's something that a human might not be aware of but the computer could figure out quite easily by looking at the data of large numbers of kidney donations. One of the amazing feats of Watson was in diagnosing a leukemia patient in Japan. That doctors could not figure out. Watson came in and figured out what the person's problem was, treated the person and saved their life. The way Watson did that, is it read, I kid you not 20 million papers on leukemia. Now doctors can't do that. Doctors have to be seeing patients. When's the last time you sat down and read 20 million papers? Nobody has the time for that. But the machine can do that. So the machine can actually lead to better decisions in a medical way as well as getting rid of bias in a more moral way.

RS: Do you see any downsides of imbuing machines with morality?

WSA: I see downsides of imbuing machines with the wrong kind of morality and doing it in the wrong way. 

RS: What does that mean?

WSA: So for example, if-- if we programmed the computer, I've got a-- I've got a great idea. Let's just minimize harm, how we are going minimize harm in the world. The answer is kill all humans. Because if we weren't around there'd be a lot less harm. Well that's not what we want. So you put the wrong kind of morality in, you get disaster. Isaac Asimov wrote about this a long time ago with his laws of robotics. But if we put the right kind of morality into computers so that it reflects what intelligent and unbiased humans would want to be done, and would think it's the right thing to be done, then I'm not sure what the downsides are.

RS: When we say the right kind of morality, how is that defined, whose is right?

WSA: Corrected views of the group. Remember North Carolina might be different from California. Right. So the right common morality in North Carolina is the corrected moral judgments of the people of North Carolina. 

RS: Corrected?

WSA: No ignorance, bias, or, or confusion. That's why I keep listing those three errors. No ignorance, confusion, or bias. That's what needs to be corrected. You've avoided those problems. Because computers don't forget morally relevant features. Computers don't get confused by complex situations. Computers don't have biases. If you don't enter the person's race into the dataset, they can't be biased. But humans see the patient and they know what race they are. You can't, you can't avoid it.

RS: Why do this now? 

WSA: Now because we're finally able to do it in a way that's more likely to work with the developments in machine learning and deep learning. Computers have new capabilities. So that's reason number one. Reason number two is that this is coming soon. Computers are getting used in more and more and more and more different areas. And so we want to get a little bit ahead of the game so that when computers start getting used in questionable areas, we'd be more likely to do it right. But the third thing I would add is to reject your question. I apologize for the impoliteness but we're not doing it now. I want to make it absolutely clear, the program that we're doing is a research program. It is nowhere near ready for real practice. And so we're not doing it now. We're starting to think about it now because it'll take a long time to think about it. And then the actual implementation will be decades away.

RS: Because, why decades?

WSA: Because it's a very complex problem. Some problems are simpler than others. I think the kidney exchange problem, we picked because it's relatively simple. But notice you can get a judgment from the kidney exchange computer and the hospital ethics committee looks at it and thinks about it, because they got plenty of time. You start saying well let's program it into autonomous weapons in the military that have to make split second decisions about who to shoot or autonomous vehicles that have to make split second decisions about which way to turn. That's a much more difficult problem. You don't have somebody listing all the relevant features of the people on the sidewalk for the car. You have to pick it up visually with a-- with some type of camera or radar or LIDAR and so that's a much, much, much, more difficult problem. So you'll start getting some of this in, in something like kidney exchanges fairly soon because it's relatively simple. The other programs are going to come later. That's what's decades away. Because we haven't solved that problem of how do you take a camera and know whether that person on the sidewalk that your car needs to avoid is a criminal or a saint. You know, is elderly or a child. How do you tell? 

RS: Not enough just to be human for a decision to be made?

WSA: So certainly being human carries a lot of weight, but if you have to decide which person to run into and which person is going to die, then it matters whether they have five years left of life or 50. 

RS: So those sorts of calculations are going to a self driving car in terms of making a decision on terms of safety in an accident.

WSA: They very well might, remember our method is to ask people what do you think is morally relevant, which things do you think are not morally relevant, and which one should you use. But the moral machine group at MIT has done this for cars the way we've done it for kidney exchanges and they have found that age is a factor that many people take to be morally relevant. Because you want to maximize the number of years of a certain quality. So you want to maximize QALY's quality adjusted life years. That's used in lots of policy making in many, many different areas. If it transfers over to machines then yeah, other things being equal, you know, if you have-- if one pedestrian is 85 and the other is 5 or 25 then some preference would be given because they are more QALYs lost if you kill the younger person. Some people object to this idea of using quality adjusted life years and directing the care into the 85 year old instead of the 25 year old, because they're saying each life is equally valuable. but, and that the use of this makes one life more valuable than another. But another way to think about it is what's equally valuable, is one year of your life is exactly as valuable as one year of my life, one year of the 85 year old's life is just as valuable as one year of the 25 year old's life. It's just that the 25 year old has more of them, and the 85 year old has already had all of the ones that the 25 year old has to look forward to. So it's not clear that people are getting treated in an unequal and partial or biased way. instead each year counts the same for everybody. but there still will be objections and this might be one of those issues that North Carolina disagrees with California about. Then you have to see how it works out over years of practice. 

RS: Is part of the aim to provide a check or enhance or both, human decision-making?

WSA: Absolutely. A check in the sense of preventing the most common errors. Ignorance, confusion, and bias. And enhance, absolutely. Because I don't want to be ignorant, I don't want to get confused. I don't want to have biases that I myself see as inappropriate. So if the machine can point those out when I'm doing it, I'm going to learn to avoid them, and my judgments will be better. That's the sense in which my judgment will be enhanced. I have to make the decision in the end, but I want to make the best decision and the machine can help me do that.

RS: How do you ensure the machines are learning what you want them to learn? They may learn on their own and make their own decisions?

WSA: Yes, they could learn on their own. They could develop it. But notice whenever you have a machine learning or a deep learning algorithm, you have to have a certain goal. If you want to teach a machine-- a computer, via machine learning or deep learning to play Go, you have to tell them what it is to win at Go, and then it develops a strategy for getting there. So if we build in that the goal is to mimic human judgment, then it's unlikely that it's going to go off and do something radically different from human judgment. But even if it does, if somehow it did, then we would just tell, we can constantly check these machines to make sure that their judgments that they're using are still in line with what humans would say, by continuing to ask humans what they think. Do we need further surveys, and making sure that those line up with what the machine is doing. The nice thing about computers is, you can test and see whether they're working properly. Humans you can't do that. How do you tell whether the hospital ethics community is messing up? You know, the machine's not going to be perfect. But it's going to be easier to regulate and control because you're going to know what's going on much easier than a hospital ethics committee composed of humans, they tell you what they're doing and why they're doing it, you don't know whether that really reflects what they're doing and why they're doing it, because they can have all kinds of implicit biases, they can have confusions they're not aware of, they can overlook things that they don't know they're overlooking. With a computer it's going to be much more transparent and easier to fix. 

RS: Could these machines one day develop consciousness?

WSA: I don't know. I mean that's a very interesting question, it's going to depend on a number of different factors. What is consciousness, you'll have to explain your question to me by distinguishing the 17 different notions of consciousness philosophers have distinguished. you know, secondly, can they develop it. Well that's going to depend on developments in computers. Right. And it's very hard to predict. If you had asked me to predict ten years ago where computers were going to be today, I would have been way off base. And so I'm not sure that I have an answer to your question. It's an interesting question, we can speculate about it. I can give you reasons for and against, but I would never claim to know that about the future.

RS: Do you think it could develop its own machine morality, separate from human morality?

WSA: In theory it could, it depends on how you program it. But if you're using machine learning and deep learning techniques, and you specified as its goal that it should develop an algorithm that best mimics human morality, then that makes it unlikely to all of a sudden start making up its own morality. Think about a program that has learned to play chess. You've defined what checkmate is, and its goal is to checkmate the other side. It's very unlikely a program that tries to maximize the chances of checkmating the other side, well all of a sudden switch and want to checkmate its own side. That's just not the way machines work. If you put in the goal of checkmating the other side, it's going to find a way to do that. So if you put in the goal of mimicking human morality, it's not going to all of a sudden start doing something different. That's just not the way machines work. And that's not the way deep learning and machine learning work. 

RS: In the future, does it in any way diminish the human capacity for understanding morality in a way that 100 years ago we didn’t... kids are so glued to technology that they've lost a capacity for reading and books, other sorts of things. Is there a possibility we will hand over the keys for morality to a machine?

WSA: First of all I want to defend the current generation of children from this charge that they no longer read books. The Harry Potter series was read by many, many, more kids than ever read a book in 1900, because most kids in 1900 didn't know how to read. My kids read books all the time, and by my kids I mean the kids in my classes as well as my own biological children. So I don't think it's really true that kids don't read books anymore. but none-the-less, there's some kids that don't. And your question is really, will people stop thinking about morality because they've got the computers doing it for them. And I want to say first of all, the answer to that is no. Because the machine will not be replacing the human. The machine will be operating as a check on the human. If you simply handed over the decision about kidney exchanges to the machines, you wouldn't need the hospital ethics committee anymore, but that is not at all what we're proposing. What we're proposing is that the committee use this machine to correct them, to enhance their judgment and to learn from. Then they're not going to stop making judgments themselves. Ok. So I think this fear that we're not going to understand morality or even be able to make moral judgments, is misplaced. In addition, I want to say that these programs can tell us a lot about our own morality. Because I do moral psychology and have done it for many years now, a couple of decades. And nobody has been able to figure out the algorithms, the computations, behind moral judgments. The way people have made a lot more progress, understanding for example the computations behind fission(?), but the computations behind moral judgments are much more complex. We have not been able to figure that out. I think these computer programs can help us figure out that very deep problem in moral psychology, and thus understand our own moral judgments in a much more profound way than we've ever been able to do before. So I actually think much to the contrary, that these programs will not reduce our understanding of morality or our tendency to make moral judgments, instead it will improve and enhance our understanding of morality, and also enhance the judgments that we make.

RS: Where are we in the arc of history in terms of reality?

WSA: I think we're much better than years ago, at least some of us, most of us. It's always a gradual thing that goes up and down and the percentages change and so on. But when I was growing up, people thought it was immoral for a black person, a white person, to kiss. I-- today you tell students that and they go, how old are you again? They think you must be hundreds of years old. But no, in my lifetime, that has changed. Gays are accepted now much more. I think that's moral progress, so I think we're doing a better job and that's just within the last 50 years. 200 years ago we had slavery. You know, 400 years before that, you know, women were just treated like chow, and poor people were shipped off to a country they didn't want to live in, like Australia because they had, you know, stolen a loaf of bread to feed their families you know those kinds of things aren't done anymore. They're seen as inappropriate and I think that's progress. 

RS: In the future, what you're trying to build, sketch for me what it looks like.

WSA: So, it's hard to predict what's going to happen, ten years from now, twenty years from now, fifty years from now, I'm not sure. We're working on the kidney exchange problem because it's relatively simple. See how that goes. If that goes well, maybe we can also do it with regard to bail, and parole, and sentencing, in a judicial context, where you also have plenty of time. Now we've got to ask, will it go into the military. How far will it go, which programs will we be able to develop, I don't know. but each one, if done properly, can be an aid in that area. And if it's not aiding in that area, we shouldn't do it. So my hope, my vision for the future is that we will be testing to see where this will help, where it will not help, and we'll use it where it helps and we'll avoid it where it doesn't help. That's my vision for the future, you want anything more concrete, you'll have to wait. 

RS: You're an optimist?

WSA: I am an optimist, absolutely. I'm an optimist because look at all the things that computers have been able to do in the last you know, ten years, twenty years. I first programmed computers in 1971. They've come a long way since then, an amazingly long way. And so yes, I'm an optimist about what they'll be able to accomplish in years to come. I'm also a pessimist about them accomplishing everything. I think there are going to be limits. What those limits are, I don't think we know yet, that's what makes it exciting, is that we don't know. And that's why I think this field is very interesting and important to explore.

JSB: I’m Jana Shaich Borg, Assistant Research Professor and Director of the Master and Interdisciplinary Data Science here at Duke University.

RS: The genesis of this project - machines with morality?

JSB: You know I think it came organically out of many different aspects. Both Walter and I have been studying moral judgment from different perspectives for a long time. Walter from the perspective of philosophy and then neuroscience and me from the perspective of neuroscience and then deeper neuroscience. And so we've always for-- we've always been thinking about first of all how we make our moral judgments. But then second of all how do we make them better. And especially for both of us but especially for me, what gets me out of bed in the morning every day is how do we make judgments, how do we treat each other better, and honestly how do we not be jerks to each other, especially when it comes to violence. So we've been thinking for a long time about how one makes a moral judgment. So as technology started to develop and it started to become possible to start thinking about something else making a moral judgment, it was a very natural switch to think well first of all, how would you design one, then second of all how could we use the need for developing technology to make us make better moral judgments. And for me, it was also natural to move from thinking about how the brain makes these computations to how would you make something else make these computations. And we are lucky enough here at Duke to have Vince Conitzer, who's the AI expert in the world and actually Walter and Vince started talking before I got here and then when I got here to it Duke was a natural collaboration. And so we've been working ever since and it's been wonderful. 

RS: How do we make moral judgments?

JSB: Most of the time, despite what we'd like to think, we make our moral judgments based on intuition and emotion. We often then postdoc rationalize that, thinking that we have some principles that we're using, and sometimes we do. It's not all the time but most of the time when we make our moral judgments it's an instinct and we go with it.

RS: Is there a better way of doing it?

JSB: Sometimes. Depends on the context. So the reason why we do that, many of us think, is that you can make computational and more qualitative models about it. It helps you be efficient. You can't think about absolutely every little detail about every decision you're making in the world. It's the same concept as every decision you make takes energy. So that's why some people wear the exact same thing every day. It's the same thing with moral judgment. So you don't want to be taking into consideration every single detail of every single moral judgment all the time. And that's why instinct can be good but then sometimes it gets you into trouble because sometimes you need to be taking into consideration things that you are.

RS: How do you take into account the evolution of notions of morality?

JSB: That's a great question. So I think there are, there are a couple of different ways to respond. And on one end, you can say, you can ask, is there something that unifies that. And I would answer that there is. It is fuzzy. It's hard to get access to, it can be very abstract, but there are at least some notions that seem to persist throughout, at least written over time when and when humans have been able to articulate what they're thinking about. And then there are other things that change kind of around that. And so one way is to think about the things that are consistently true where everyone's intuition seemed to align and then to go to the other end and try to figure out how to fill in the blanks. And the other is to focus on the differences and try to figure out why those differences are there. And so we try to do both. So how would you handle things changing over time, in our approach you handle both. So you focus on the things that have been the same and are consistent and you can hold onto those and then you also make sure that you're monitoring and you're checking and thinking about the things that are, are changing appropriately.

RS: Example?

JSB: Of something that's consistent over time? I will purposely be a little vague about this but I think that across time people feel that unjust harm is wrong. So now the question is what is unjust and that can be both vague and it can also change over time but still unjust harm is wrong. 

RS: Is there a universal definition of morality? 

JSB: It's a great question and I'm curious what my colleague, Walter, would say. My response is no, I don't think there is. But it doesn't bother me because everyone has some sense of what morality is for them. So experimentally, for example, when I ask questions I don't ask about universal morality, I ask about what's moral to you. And everyone has some sense of what is moral to them and that's what we go after. Is that the exact same feeling, the exact same concept? I don't think so. Probably not but that's ok.

RS: Who defines morality? How does that sense of  morality get imbued into a machine?

JSB: So, it's a good question and I think that part of what you're getting at is this idea of when we're building a moral artificial intelligence is it the same as a human moral artificial intelligence. So I'll answer both of those things. So, who defines what morality is, that was your question. And I would say it's you who define it. Morality is a very personal thing. And so what feels moral to you is the important thing, kind of similar to what I had said before. So that's different from what would be moral for an artificial intelligence because we are building a, our project is dedicated to developing a moral artificial intelligence with a small eye rather than a big eye. So we're not trying to create something that is equal to humans. We're trying to create something that can help solve problems. And so in that case, what we are putting in there is what humans think is morality, not what the machine thinks is morality. And then whatever the machine thinks is morality is what we define for it. 

RS: Why not leave it to humans?

JSB: So the main reason to not leave moral judgment documents is because humans are fallible. So we get tired. We get tired and we lose cognitive resources and emotional resources. So by the end of a day when you're tired, when you're sick, when you've just made too many decisions, you often can't make the decisions as well as when you started. And so what we're trying to do is just build some help.

RS: Your aim is to enhance decision making by what otherwise would be considered flawed humans. 

JSB: That's exactly right. That's exactly right. 

RS: Any downsides, worries about creating a hell?

JSB: I think the downsides are, are more unanticipated consequences rather than if we achieve our ultimate goal. I don't think there are downsides. But inevitably there will be some unintended consequences and those consequences are the type of consequences that come a lot in type of data science or machine learning or artificial intelligence work. When you are training an algorithm or creating an algorithm and you're using data, often that data is biased or flawed in a way that you didn't anticipate. And so then that can get built into your algorithms. And so we want to-- that-- that could happen in our project and that could have some, some bad downsides. Built into our project is monitoring those things and testing those things out. But you can imagine that it's possible that there could be some downsides to that. 

RS: Transplant and kidney issue -- could ignore computers completely?

JSB: So there are two different pieces there. So you had asked about if there was a downside. I think the downside would actually be the opposite of what you said, that if they didn't rely on artificial intelligence all the time then they might kind of lose their own moral strength, stop practicing their own moral decision maker, internal moral decision maker. So you might imagine that could be a concern. If they rely too much on the machine and don't rely enough on their own decision making to put checks, especially if the artificial intelligence told them some-- a mistake itself or said something that was really inconsistent with their values because wouldn't be perfect. We wouldn't want them to just give up their own moral compass. But that you are bringing up another part that's very in part-- or very important part of our project which is that it's not just about making an algorithm that predicts something well, it's about figuring out a way to communicate the information in that algorithm to decision makers in a way that they will actually incorporate it into their decision making. And that's a big part of this that's not easy.

RS: Because?

JSB: Because we don't know the answers yet. We don't know the best way to present either data or outcomes of artificial intelligence to a human intelligence in a way that feels right to us. So it'll be a lot of trial and error. And there's some theory out there but the truth is we don't know yet.

RS: So you bring that perspective when you are a scientist. Tell me what the challenges are from your perspective.

JSB: I think one of the main challenges is what we were just talking about which is that if you have a strong moral conviction or strong emotion associated with the judgment that you intend to make, and all of a sudden there is something else, this artificial intelligence, that's telling you that's not the decision you should make, or the decision you feel so strongly about is actually inconsistent with your values. That's-- will cause a lot of cognitive dissonance. Both emotional dissonance and cognitive dissonance. It will make it very hard I think for the decision maker to actually receive the information that's being given. So figuring out how to overcome that I think is going to be one of the biggest challenges. And from a neuroscience point of view I know how strong those circuits are that bring the emotion about and what happens when those emotions and those circuits are so activated they short circuit in some ways. Other types of circuits that you need to make very rational or objective decision making.

RS: How difficult it can be to hear criticism from a colleague -- let alone a machine that has been programmed to check your moral values.

JSB: That's exactly right. Yes. So I do think there are individual differences, but for some people having any type of negative feedback or what feels like negative feedback or is interpreted as negative feedback, is very hard to hear. And it makes it very, you know, it makes you inhibit your rational cognitive processes. Now for some people, even if that's normally the case, being in what feels to you like an objective environment, at least this is an objective decision maker it's not a person, so whatever it says it doesn't really matter. That might give, be able to lower the anxiety when-- when being given feedback and maybe will make it easier. For others, it might be even more frustrating. Because now you have yet another thing that's supposed to be right. And now it's telling you you're definitely wrong and so it might feel even more harsh. And for those people I think it's going to be very hard to hear that feedback and actually incorporate it in a meaningful way.

RS: Even geographical differences. So for example a decision making board in North Carolina hospital versus a decision made board in San Francisco on the same issues might reach very different conclusions on culture, bias, and geography. Agree, disagree.?

JSB: Absolutely. There will be many local differences that go with some of the universal similarities.

RS: How do you account for that?

JSB: Yes. So there are a couple of different things but one of the main ways that we're designing this project is to give you advice about consistency within your own moral judgment. So in that application it doesn't matter what other people are saying, it matters what you say, what you think is right. So we would train based on your own judgments and we're telling you, well the judgment you're making now is inconsistent with what you've made before or what you've said is important to you. So you might want to reconsider based on the things you've said are important to you.

What happens if your moral judgment is inherently biased? 

JSB: That's a great question. So, there need to be multiple different ways that the AI can give you feedback. So one should be, are you being consistent with your own choices, your own behavior, and the other should be, are you being consistent with your local group or with certain groups that you might care about. And one might be for a local group and another might be your profession. Another might be the population as a whole. And so you're given those different pieces then those can be used to help you realize even if you're being consistent. If your consistency is not consistent with others then that should make you think about whether there is, perhaps you have a motivation that you might want to reconsider.

RS: Why this project now?

JSB: I think it's not AI as much as how easy it is or the different ways that are been conceptualized now for collecting data. So part of what we need is a lot of data about how people make judgments and what their judgments are. And in the past we've always had to do that by inviting people into a lab and asking them some questions. And then in the past 10 years there started to be a new approach which is you could ask some people online but it was through Amazon Turk. So there was a pool of about 6000 people and maybe you could pay through some other types of agencies or resources to get maybe 10,000 people but that's a very select sample and you certainly don't get the entire world. But now there's been a lot of both work done but it's kind of just new creativity around the ideas of how you could collect data from everyone, not just from the people that are associated with these areas.

RS: Crowdsourcing?

JSB: Crowdsourcing, that's right. So part of the project is to develop a website where people from anywhere in the world can come and tell us about their moral judgments. 

RS: Part of Templeton?

JSB: Yes. And so that knowing that, that is something that can be done, there is a little bit of precedent for this but it's, what we've seen in the past couple of years is that you can crowdsourced science. And so with that development we then started to realize well now we can get the data we need to actually train a moral AI. 

RS: Could this have been done before? Why now?

JSB: Yeah I think it's the openness of society to new things that makes a big difference. Now four years ago the idea of self driving cars still sounded a little nuts and now everyone accepts it as an inevitability. And so I think before the idea of moral artificial intelligence sounded nuts, even if on the technical side you knew it was possible, and now people I think will be more open to it and I think that's what made us think now is the time to actually explore this.

RS: How does it fit into decision-making?

JSB: The techniques that we develop for the kidney exchange scenario, we believe should be easily generalizable to self driving cars scenario. 

RS: Any other areas?

JSB: I think another obvious one is drones. But there are an increasing number of places where AI is being implemented in society alongside human decision makers. So there is also  in the judicial realm, AI is being used to decide how much punishment someone should receive or how likely someone is to recidivate, and how that should be incorporated into the decision. So I think any of these developing areas are places where what we are working on can be useful.

RS: Architecture around morality, rule-based. Decisions will be locally made? Through communities?

JSB: I think there are two different applications. or two different ways we'll be using the AI, that's kind of related to something I was saying before. So in one case we want to tell you about your morality. So that would be local. So in that case we just need to tell you, are you being consistent with the things you told us you cared about, your decision in the past and the principles you've told us that you want to implement. So in that case we just need to know yours. If we're going to give you feedback, which is something we're going to do in the other application. If we're going to give you feedback about whether your choices and your motivations are similar or different from others, then we need to take into account the community. And so in that case we need to, on the technical side, need to have a good way of summarizing, integrating the community's collective sense of morality.

RS: Your research into this arena focuses on what areas?

JSB: My most recent is on empathy. So the story behind this is that when I started studying moral judgment, especially at that time, everyone kind of believed that it was a very cognitive process. And now people laugh at that. But we still really did. And so we thought well if I wanted to change moral judgment then I should understand how we make conscious moral decisions. And as I started to do more and more research it became more and more clear that only a very small subset of our moral judgments are actually based on things that are at the forefront of our mind. Often it's based on what we feel in our gut responses and certainly our moral behavior is much more related to what we feel than what we say. And that's a methodological issue that actually has a big impact. So sometimes people will ask, is it wrong to do something or would you do something and that sounds like there isn't a big difference. There's a huge difference and what people say is wrong to do is not necessarily what they would do. And it turns out that your behavior is much more related to your feelings, your emotions, and so that's why I moved from studying moral principles which I still do, and it's still important, but to actually try and understand the motivations that correlate more with actual behavior rather than just what you say and empathy is one of the things that seems to correlate most with your behavior, your moral behavior.

RS: How do begin to understand a gut instinct versus the neuroscience of a cognitive decision around morality, explain that to me.

JSB: That one actually ended up being easier than we thought. The details are much harder than we thought but there are some parts of the brain that are evolutionarily very ancient and species, all species have them. And those tend to be the regions that are involved more with affect or emotion. And so these are the brain regions that help you make fast, instinctual decisions, actions, emotions, and the newest parts of the brain are the parts that tend to be more cognitive, that tend to require some type of conscious intentional deliberation or at least that's part of what they're involved in. And so what we learned pretty early on in the field actually is that many of our instincts seem to be represented or in some way related to these kind of old evolutionary regions and some of them were deliberation or deliberate parts of our cognition happened in these kind of newer parts of the brain that are on the outside.

RS: Do you put a value judgment around one versus the other?

JSB: I don't, actually. So some do. And in the field there are some people that say you should not be making emotional judgments. And so they would say that the right moral judgments are the ones that you only use the cognitive parts of your brain for. Others would say actually what matters is that you have the right emotions and that your emotions match the actions that you're performing. So for example if you're helping someone but you're just doing it in this purely cognitive way that's actually not something that you should be very happy about. You should actually want to help them. And so for me I'm more interested in trying to understand the relationships between what happens and what we actually do.

RS: The neuroscience of understanding instinct and gut reaction -- how do you do that? 

JSB: That's a great question. So there are, you have to take many different approaches and the approach I took was very, very unconventional. So the way I ended up going about it was well, if these things are truly instinctual and they should exist in non-humans. If the parts of the brain that are actually involved are deep and the brain are concerned? then we should be able, first of all, to be involved in other species and to be able to see evidence of it in other species. So in graduate school I spent a long time trying to find what other species could I look at that showed some evidence of And I thought I was going to use primates and I ended up using rats. And so a good chunk of my research is about well, first of all I ended up learning after many years in a basement testing many different things, a rat will give up something it cares about in order to prevent another rat from getting shocked. And the very surprising thing is that the brain regions that encode that decision are the exact same that encode it in humans and indeed it is the same brain regions that are deep in the brain that are involved in instinct, also involved in moral judgment.

RS: How do you define diverse intelligence?

JSB: I would turn the same thing back to Templeton, probably. What I would say, intelligence is something that solves problems. And so I don't put capital I on it when I'm thinking about it. That can be an interesting exercise to put the capital I, but I think it's equally interesting and important to think about intelligence with the lowercase I. So I wouldn't ever say that what a rat is doing in my experiments is the same as what a human is doing. But that doesn't mean that they can't inform each other and that there aren't some similarities and things to learn from them. So what is it? A diverse intelligence, diverse intelligence are different ways, different species, different approaches, of solving problems.

RS: When did the switch happen from cognitive?

JSB: It happened, and I shouldn't say that it was, it's not a 100 percent switch, it's a switch of focus. It happened around 2005, actually I can know exactly what it was and exactly some of the experiments because one of the big motivations for it was I started working with clinical psychopaths. And we know that some of the biggest deficits and clinical psychopaths are their emotions. So they are unregulated, they have emotions when they shouldn't, and they don't have emotions when they say so they don't have empathy, for example. That's one of the hallmark characteristics. And so the assumption should be that they should also have some impaired moral judgment but they don't. So they can tell you in almost every situation the right thing to do. And I spent a lot of time working with them and getting a lot of negative results that I could never publish because everything we tried they can always find some way to tell you what the right thing to do was. But they never did it. So what that brought to me was the, which helped highlight for me that we really had to start thinking about the difference between what you say is right and what you actually do. And for me because the reason I get out of bed in the morning is to prevent violence and help us treat each other better. I care about the behavior more than the judgment. I also care about the judgment. The behavior is the most important thing and if I care about that it is clear that the cognition wasn't the best way to get there. 

RS: If a psychopath knows what is morally right, is that genetically encoded in all of us?

JSB: That's interesting-- well you're asking a biologist or a neuroscientist I would say everything is biology encoded. But the answer you're probably looking for is not necessarily. I mean they learn through, through society in many of the same ways that non-psychopaths do. So they can learn from others what they should say. And so the reigning theory right now is they say what they know they should in order to maximize their goals. And they're always trying to use you for instrumental reasons, and so if they say the right thing they're likely to get some type of benefit from you.

RS: Will these machines that would be imbued with human morality potentially create a machine morality that's different from what we understand morality to be?

JSB: You'd have to define what you mean by machine morality. So because we are making an artificial intelligence with a small eye, we're training this intelligence, so it's always going to have the same goal which is, based on one of our applications to predict what your judgment would be. So that goal is never going to change. So it's not going to suddenly have a morality that has a different goal where things might get a little interesting. the way it gets there might be different than how we make our moral judgments. And so if, you might say that that's a different type of intelligence than ours. But it's not going to be different types of moral intelligence that tries to achieve a different type of goal.

RS: The goal at this point is quite specific?

JSB: That's right, it's quite specific and we define it. So the goal of our algorithm is to predict what you would say is right or wrong and it will never change that. 

RS: In terms of machine learning, could this take on a life of its own, a consciousness because of unintended consequences?

JSB: At least in what we are building, no. Is it possible to build one, that could be an interesting theoretical question, but the one we're building, no it can't take on its own consciousness.

RS: Creating a machine that would reflect the most rational, impartial person that understands morality in its broadest form to make a decision, the "right" decision. Is that accurate?

JSB: Yeah. So we have these two different types of information we want to give people. One is the most impartial version of themselves and the other is the most impartial version of the collective around them. 

RS: What's wrong with the kidney transplant hospital board today that you're solving for. 

JSB: In many ways our projects kind of started with that question. So right now the way kidney exchanges are implemented is purely based on medical and practical considerations. And one might think that that is the right way to do it. So we ask people, we ask people, are there things-- other factors that you think should be taken into consideration. And it turned out that there are many, yes, at least on Amazon Turk. And there were some that almost everyone agreed should be taken into consideration. Like age, how old you are. Another one that came up was how much you had drank alcohol before, and whether you had some type of medical disorder or a medical disease that would impact how long you might live. So people say yes, so it's an empirical question for us. It's not  it's not a normative one. People want these things to be taken into consideration. So now for us it's figuring out how to do that.

RS: The inference being that the hospital board was not those into consideration. 

JSB: Right. And explicitly they would tell you they are not. The only way that it could be taken into consideration is that, always, and this would still be the case, even if we succeed with our project, the surgeons can always make a final decision. So they can always say, that was nice, thanks for that recommendation. But I'm going to do something else. 

RS: What are the parameters? 

JSB: So it-- different exchanges have different rules but some of the most common ones are a blood type for example, and how far away the donor would be from the recipient. Because you have to actually transport the organs to the recipients and so those are very practical, medical considerations. 

RS: No sense of need or how sick you are?

JSB: That's right. 

RS: Sounds quite clinical.

JSB: That's right, it does. And so some people of course up to this point, how many people think that's the way it should have been. And I think you can make a convincing argument, it just turns out most people don't agree with that argument.

RS: How receptive do you think they will be to something like this?

JSB: I think it will be different in different exchanges. So for example, I'm going to go talk to a group in Europe for a couple of months. And I think that they might be much more open in some ways and much more closed in other ways than the U.S.. And an interesting thing about the U.S. system is that there are different exchanges. So you could have one exchange that's open to incorporating this information. And another one that isn't, and they could both function at the same time. So that will be an interesting practical consideration that we have to think about. 

RS: The neuroscience side, in terms of impact, what have we missed? The contributions you're making to helping understand how to imbue morality in a machine.

JSB: I think one of the biggest contributions is not just in how to imbue morality in a machine, but in the second step, how do we use that information to impact our moral judgment. I think the past 15 years that I've been trying to understand how we make our own decisions is really impacting how we think about, well how should the information that comes from the AI, be presented to humans to actually influence their behavior in a way that they find will be effective and useful. And that's really in some part, one part of my life, I think about things in a way of what would tickle in the brain to change a moral judgment. And so this is a similar type of question, I'm just not tickling something physically in the brain.

RS: You're concerned about how receptive humans will be to this?

JSB: Absolutely. Because we haven't done a good job with our goals, if we-- we can create a perfect artificial intelligence. But if humans aren't receptive to it then we're not improving anyone's judgment. And so we haven't achieved our goal. So a very big part of the project is figuring out that second step of how we translate the AI to humans in a way that the two intelligences can work together and together be better because there's a big risk that the artificial intelligence will not be integrated into human decision making at all.

RS: Your theory on how to do that is?

JSB: The theory is that it's going to have to be very individual specific. So we're going to have to learn something about certain individuals' decision-making, moral decision-making, and then present information in a tailored way to them in a way that allows them to make the decision impartially, without anxiety, without feeling like they're being attacked. And I think different people will need the information presented to them in different ways. 

RS: If it works for a hospital board does that mean it could work anywhere?

JSB: With enough work on the AI, if it works at a hospital board then in theory we should be able to find ways to design an AI that could be generalized to other situations as well. 

RS: What's the timeline?

JSB: We'll be able to tell you, you just check in every couple of months. I think some parts of it we can achieve quickly. I think other parts of it are going to be harder. So I think we're going to be able to have a proof of concept relatively quickly. We already have a pretty good proof of concept but to get to the perfect moral AI, that might be a forever long project because, especially because everyone's mortality may change in the ways you described before. And so we're going to need to make sure that we have a way to adapt to that always and at least right now the state of our AI is that it doesn't generalize well. So chances are we're going to have to keep working on it. 

RS: How hard a challenge is this?

JSB: It's a very big challenge and I don't think it would be possible without a very interdisciplinary approach. Certainly I couldn't do it without Walter and Vince, and I don't think they could do it either without someone who was really thinking about how we make moral judgments. 

RS: And your work on empathy helps-- helps us get to understand how we make moral judgments?

JSB: Yeah, because usually when we make, it's not moral judgments as much as the behavior that we then perform, usually that's based on empathy more than other things. And so the more we can either tap into empathy or understand the influence empathy will have, the more we will be able to understand the behavior you will likely follow. 

RS: Why empathy?

JSB: I don't know why humans were designed that way or that...

RS: The connection between the issue of empathy and moral judgment?

JSB: The main connection is that a lot of moral judgment, not all of it, is harm based. It's based on trying to avoid harm. And there are many types of judgment that don't fit in that, or at least don't fit in physical harm. Things like lying. But usually there's some part of us that is turning the moral scenario into something about thinking about how someone else would feel, what kind of pain it would cause them, what kind of distress it would cause them. And we as humans and it turns out other species too are wired up to avoid others distress. And so if you can understand how that ends up impacting pretty much every moral judgment we have, you can end up understanding quite a lot of what we will do.

RS: Sounds like therapy.

JSB: A little bit, sometimes it does feel that way.

RS: What have we missed? You're an optimist?

JSB: I am. There are some things that I think are going to be harder than others. I think one of the hardest things is going to be trying to figure out the best way to represent everyone's moral sentiments. So if we're trying to predict what your, just your moral judgments are that'll be a little bit easier. Except it's harder to get enough data from you to train our AI well. In the other case where we're trying to tell you about how your judgment is going to relate to other people's judgments. It's much easier to get the data you need. But it's going to be much harder to figure out a good principled way we all feel comfortable with, that everyone's views are being represented in the way that they should be. And that's a technical problem but it's equally a theoretical problem. And so that one, that's the one I'm most nervous about figuring out how to do in a way that we all feel satisfied with. 

RS: Integrating it all into a real life situation.

JSB: I think for the kidney exchange situation it's a little easier to imagine how it would be integrated into a real life situation than something like self-driving cars. Self driving cars scenario, it's so unconstrained and you have people in all different types of situations and it happens very quickly. Whereas in a kidney exchange at least, there's a certain set of people who are doing it in a pretty constrained context. And so that's part of the reason we took this as our first context to work with because you can imagine if you had an AI, how it might actually be implemented in real life. So that part in this particular context I think is not quite as hard to imagine as the actual, how do you get everyone's opinion incorporated well, problem.

RS: We're going to rely on that to help us make a decision...

JSB: That's exactly right. So that's where I feel like, so neuroscience isn't just about what's happening in the brain, it's how we behave and what our behavior is. And so I agree, I think that part of, half of the project is figuring out how to get this information into those people's heads in a way that they'll be receptive to. 

VC: I’m Vincent Conitzer. I am the Kimberly J. Jenkins University professor of new technologies, also a professor of computer science of economics and philosophy here at Duke University. 

RS: Tell me about the genesis project and why it's important and why now?

VC: So I've worked in AI for many years now. I think one thing that we're seeing now is that as AI is being broadly deployed in the world, the kinds of problems that we are dealing with are changing, right? So as an example I like to give this example of a problem in AI known as a reinforcement learning problem where there is a little card that's on a railroad track and there's a pole on top of it that's connected with a hinge. And the goal is to get this car to move back and forth in such a way that the poles keep standing upright. And this is a very nice little problem and it teaches you a lot about artificial intelligence. But there's no difficulty whatsoever in specifying what the goal is, what the objective is. The objective is for the pole to stay upright. So that's very simple in that regard. As we're deploying a lot of these techniques in the world today, that's changing, that we really have to think about what the objective really is. For example, if you're working on speech recognition, and there have been amazing advances in speech recognition, I can dictate an email to my phone now and it's much faster than typing it. But as you're developing this technology in the lab you might have a very simple objective. For example, you might have recordings of a lot of people saying various things. And your goal might be just to minimize the number of errors that you make in your dataset overall. Then when you deploy this in the world what you might find is that it works really well for the majority dialect in the population but very poorly for a minority dialect. And so even though your overall performance is good you might worry that this is not the right thing to do, that this way you're putting a certain group of people at a disadvantage and so you have to reconsider what the objective really is. You might have to think about fairness across populations for example. And that's really changing how we're thinking about a lot of these different problems.

RS: Why imbue a machine with morality?

VC: So there are a couple of different reasons why we can't always outsource moral decisions to human beings. Sometimes an AI system will have to make a decision very quickly. An example might be a self-driving car that has to make a sudden decision whether to switch lanes or not switch lanes thereby putting different people at risk. In principle you might think that the AI system could then call on a human either the driver in the car or somebody else to make that decision. But they won't be paying attention to the situation and will not be capable of making that decision well in a short period of time. So in that case the car really has to make its decision on its own. So that's a reason of speed. In the context of kidney exchanges, which maybe we'll talk about a little bit later, the reason is a little bit different. So here, the reason that we need AI is that we need to optimize and figure out what the best option is in a very large space of possible alternatives. And it's just not feasible for a human to look at all the different possibilities of how you might match people together. And so instead we have a computer search through all those possibilities but that computer, the algorithm needs to be guided in how it performs that search. And so that's a different reason why you might have the algorithm itself make those decisions instead of calling on a person.

RS: Is the inference that the transplant boards at hospitals do not have all the relevant information or are potentially making decisions and therefore an AI machine can help them do this? 

VC: So it's one possibility that the board is biased and that so for that reason it might be nice to have a clear policy. In a lot of circumstances there is actually a clear policy. In general, certainly in those kinds of contexts, the board would have the same information available to it as the algorithm. It may not understand all the consequences of the data that it has. Right. So one feature of a patient, the board might think that this is predictive that this patient will die soon and in fact if you look at the data and then in combination with the other features that the person has maybe it's actually not that predictive. Right. And in that case an algorithm might detect that better. But I think the primary reason in the context of kidney exchanges is not so much that the algorithm has different data available to it, but that it can search through a larger space of possibilities. If you think about what kind of options there are in this context, right. So these are contexts where we have patients and we have people who are willing to donate kidneys to those patients but they're medically maybe not compatible. For example, the patient might have blood type A, the donor might have blood type B, and so the donor can no longer give to the patients. In that context it might be the case that there is another pair of people in the same situation except there the patient has blood type B and the donor has blood type A. And in that case they can swap. So that's a fairly simple arrangement. You can come up with more complicated arrangements, for example, you might have donor 1 give to Patient 2, donor 2 to patient 3, and donor 3 back to patient 1. So that's what's known as a 3 cycle. So that's a little bit more complicated. And you can imagine that if you're running an exchange, you get all of this information from all these different people, what their medical data is, who might be matched with who. And now you have to sort through this enormous number of people and figure out what is the best way to match them all together. And if you think about it for a while, this quickly becomes very overwhelming. You might find something reasonable but how would you know that you found the best way to match them all. Even if what best means is just maximizing the number of people that get a kidney, that already becomes a hard problem. And there is a sense in which it's actually formal-- formally computationally hard. So this is where algorithms come in because they don't mind searching through a large base of possible alternatives, right. Whereas you or I would get bored of this and overwhelmed and might start making mistakes. Computers are very good at systematically searching through a space of alternatives. So that's I think the primary reason that AI is getting involved in these kinds of things.

RS: Enhancing the human decision-making process?

VC: That's right. So I think with the current algorithms, just about any reasonable policy that you would give them, that you would give the algorithm, it could actually handle, right. So it can deal with an objective that is just to maximize the number of patients that get a kidney. It can also deal with a somewhat different objective that says we value younger patients twice as much as older patients. I'm not saying that that should be the policy but if that's what you wanted your policy to be, it could accommodate that just as well. So in some sense we still need human beings to decide what the relevant features are and how much they should be weighed. And because one thing about AI today is that even though we have made tremendous progress, one thing that it does not yet have is a broad understanding of the world. So we humans are very good at having a broad understanding of our world, especially our social world. It's very broad, it's very flexible, it's very integrated, and that is something that we have not yet replicated in AI systems. 

RS: Is the inference that the current process is flawed?

VC: So here specifically we're looking at the kidney exchanges more than who gets kidneys off of the donor list. Right, so there's, there's two ways that you might obtain a kidney transplant. One is that an organ donor dies and their kidneys can be transplanted to somebody. Unfortunately there are not enough donors like that and also outcomes tend to be better with a live donation. So we're focusing here on kidney exchanges where we have live donors who are not necessarily compatible with their patients. And these are usually handled in batches, so to a certain point in time we try to figure out what the best way is to match people together. And there I think we can really improve on just the person making those decisions and certain real kidney exchanges already use algorithms in this way just because the algorithm is, is patient and can continually search through all the possibilities, whereas a human being would get tired and wouldn't work fast enough to be able to go through all the possible options, would make mistakes.

RS: You still have to put all this data into the machine? 

VC: That is correct. So somebody needs to get that data about the patient's blood types and other medical information. And that is being done by people, right. And moreover, how those inputs are judged. So now if you want to go away from this model, we try to maximize the number of people that get a transplant and you might like to prioritize certain transplants over others. For example because you feel that this one is very urgent that this person has a long life left to live and for some reason we might want to prioritize them, if you want to set those kind of priorities then it's important for a person to do that, because today's kidney exchange algorithms, while they're very sophisticated in searching through this large space of possible ways of matching people, they have no concept whatsoever of what it's like to be a person, what even a kidney is, to them it's just zeros and ones, that they are interpreting in a particular way just for the purpose of this matching problem. But they have no concept of what it means, right? So somewhere at this point humans still need to come in and make those value judgments. But they need to be somehow formally encoded in the algorithm. Right. So whereas when a person is doing this, you might have some general guidelines and say well you know generally try to prioritize people who have more years of life left or something like that. That is a very soft criteria, right. It doesn't tell you exactly how much that, to value that versus something else. Unfortunately when you, or maybe fortunately when you code up an algorithm you have to be precise, right. Whatever algorithm you design, whether you think about this implicitly or explicitly, it is going to make decisions in a particular way and you will have set a policy for matching people in a particular way. So I think it's better to think about it explicitly. The difficult thing is that you have to be very precise in that case, right? And what we typically learn from ethics courses is that you know, it's never really quite that simple and usually you don't come away from an ethics course learning that this is the formula according to which everybody should make decisions and that's that, right. Instead what you get away as you get an understanding of the different kinds of things that people take into account that might be relevant, what intuitions you might have, why you might have them, why they may or may not be correct. And that is a very valuable thing but to somebody designing this AI algorithm it's not exactly what they need. It would be much easier if you had a very specific formula that you could just code up and you would have done your job. 

RS: Who gets to decide what's moral?

VC: That's a great question. So in this particular context, people's intuitions obviously disagree. Some people would for example say that you should take into account whether a patient has dependents like small children that they're taking care of and other people would say that you should not take that into account. Right, that's a controversial feature as to whether it should be taken into account. It seems unlikely that we will get to a point that everybody agrees on this. And so we need some kind of methodology for, from the opinions of different people that disagree with each other, arriving at a single policy. Right. You can think of this as a kind of voting problem. There is a theory called Social Choice Theory that deals precisely with this topic of how you make decisions based on the preferences of multiple people that are potentially in conflict. And so this is a great area for trying to apply and develop some new social choice theory.

RS: Why do this now?

VC: Right. I think one of the main things here is that AI is now starting to be so broadly deployed, right. In the earlier days of AI, AI researchers just worked in the lab and again, thinking about the objectives and their problems wasn't that important, right. You only needed to have a sensible objective that was useful for, from the perspective of making progress in research. Right, that's how AI researchers proceeded. They came up with a problem and then they tried to figure out how to make progress according to some measure. So they needed to have some measure that was a reasonable measure of how well the system was performing.
But it wasn't important that this was the measure that was really important to solve in the real world, necessarily. Right, you could always change a measure. 

RS: But now?

VC: And now as these systems are being broadly deployed, we actually do care about the objectives that they pursue. Again, all kinds of issues of fairness might come up as another example. If you're one of the big tech companies you might be concerned that some of the accounts that are made on your site don't correspond to real people, and so you might try to predict which of the accounts correspond to real people and which ones do not. 

RS: Let's stick with morality. Why now? 

VC: I think first of all, imbuing the machine with morality, it's still in a very limited way that we encode morality into the machine, right. Because even after we set these kinds of priorities to the algorithm, it still has no real understanding of what it is like to be a person in need of a kidney, right? So you can certainly argue that it still misses some of the aspects that to us are important in morality. So we are still quite limited in that regard. That being said, we're increasingly finding the situations where AI systems have to make these decisions that in our eyes have a significant moral component. And so that, because of that we really need to think about how to encode those priorities in the algorithm itself. 

RS: Will these machines learn on their own after you've programmed it with a broad set of what moral choices could be?

VC: The way we've designed them so far, we just encode the preferences and that's that. In principle they could continue to learn, but they need some kind of input for continuing to learn. And they need some kind of objective again, for what they're trying to learn. If they get more data over time as to what that human beings value, then they could learn based on that, right? And that way even if society's preferences about these things shift or society's perception of what is valuable and what isn't shifts. The system could keep track of that and change its priorities and principles as well. But it would need to be supplied with this data. Other things it could also learn, right. So you might give it a high level objective. For example you might say why you should be doing it if you should be trying to maximize the number of years of life saved. And then over time it might actually learn what is effective as it gets more data of which patients survived for certain numbers of years. But I think it wouldn't really learn more about morality per se unless it has some additional inputs. 

RS: Any fear of creating a hell here?

VC: So I think that's a somewhat different issue from what we're currently working on. So this is certainly an issue of concern to many people that one day maybe we end up building systems that are vastly more intelligent than we are, and that we will not be able to shut off and that will take over the world from us. I think it's fine for some people to be concerned about that. And I think it's worth thinking about. But the issues that we're working on currently are still in a more near term view of AI. I think most AI researchers agree that super intelligent AI isn't exactly on our doorstep. There's disagreement about when we might expect that or not. But this is not quite the focus of our research at this point in time.

RS: You referenced the limitations of AI's understanding of the world right now, explain that to me, what does that mean?

VC: So there are still many things that human brains can do that we don't know how to replicate in AI systems. And this ranges from things like consciousness, the fact that they're, there is something it is like to be me right now, that is something that I think we don't even have a very good conceptual handle on. More practically there is this issue of having a broader understanding of the world. So most AI systems today, even the very successful ones, are still very narrow in that they focus on a very particular domain. You might think about Alpha Go, Alpha Go is exceptionally good at playing Go, but it cannot play chess at all, right. The basic AlphaGo setup. You might modify it to be able to play chess as well. Even then there are other things, right, you might imagine as a person you might at some point come up with the idea that wouldn't it be fun if we added some red stones to the game of Go, and play the game in a particular way with those red stones. There is absolutely no way that those systems are designed to solve that problem right now. And so that's kind of what I mean by the fact that we have kind of a very broad understanding of our world and the different kinds of objectives. We can switch levels in how we think about things, right? We can think inside the game and we can also jump out of the box and think about the purpose of the game is to have fun or present the challenging problem to ourselves and we might actually modify the game in order to achieve that. That's not to say that AI systems, even today's AI systems, can't be creative. They actually display some kinds of creativity that within the domain that they're working in, they might find new solutions that none of us have ever thought about. But usually they're restricted to think within a particular domain. And that is something that we haven't quite figured out how to give AI systems this kind of broad and flexible understanding of our world. So that they are not just good at performing one very specific task but can easily be redeployed to other settings. 

RS: How far away are we from that sort of world?

VC: That is a very difficult question to answer. There are people working on what seem to be key components to that. So there is something known as transfer learning, where you learn in one domain and somehow what you've learned, you're able to use in an entirely different domain. That is a very hard problem in general in what is known as the field of machine learning. But people do try to make progress on that. I think these predictions of how far away we are from super intelligent AI or various other AI milestones, are generally very difficult to make. Technological Forecasting is always notoriously difficult. And this is no exception. Certainly I have often been surprised by progress that's been made. For example, when Ken Jennings and Brad Rutter were defeated by IBM's Watson, that was very surprising to me. I didn't think that that problem would be solved so quickly. There is this interesting issue that AI researchers often complain that whenever they solve something then people stop considering it AI. And I think this was also the case with Jeopardy to some extent. And typically the reason for that is that, once we have solved one of these grand challenges, we understand a little bit better what was important in that challenge. And we also realize that even though Watson is extremely good at Jeopardy, it's not going to replace us in very many other things. 

RS: Will artificial intelligence replace the human brain at some point?

VC: So, if you go far enough into the future, again, that becomes a very difficult prediction to make. Certainly people worry about these kinds of existential risk scenarios. Now that may not even require super intelligent AI, right. An AI system that doesn't even understand the world as broadly as we do, might for some reason still be able to displace us if we're not careful. 

RS: Will these machines, do you think one day these machines could develop consciousness?

VC: So I don't see a fundamental reason why not, at the same time I think consciousness is still something that we have such a poor handle on that it's difficult to make that kind of assessment. If you ask me right now to build a system, an AI system that is conscious, I wouldn't even know where to begin, right. Whereas with some of the other questions, if you asked me to build an AI system that was funny, that's hard to do. But at least I would have some sense of what it would mean to succeed, how I would evaluate that, I might come up with some initial plan for how to do that. If you asked me to design an AI system that is conscious, it's very hard to know even when I would have succeeded, where I would get started. There are theories of what gives rise to consciousness. So presumably that would be the place to get started. But it's not clear that those theories really explain everything that we mean when we talk about consciousness.

RS: The other issue is whether consciousness is a human trait.

VC: Or an exclusively human trait. (CORRECT) And so it seems, again there's many things that you might mean by consciousness. I don't see that we have a very strong reason to think that animals would not be conscious. Especially if you think about great apes and dolphins, I think probably the default position would have to be that they are in some sense conscious like us. 

RS: Explain game theory in its connection to the basis of programming in terms of AI.

VC: So game theory concerns how to make decisions in environments where there are multiple actors that have potentially different objectives in mind. And this is generally a very tricky problem because what is optimal for one agent in this situation to do generally depends on what the others do. But what's optimal for them to do in turn depends on what this first agent does. And so it seems potentially very circular and game theory is exactly about how you deal with this potential circularity and still arrive at some notion of what it means to make the right decision. And also had some notion of how we can predict what others are likely to do when they're acting strategically. Because game theory is about settings where you have multiple actors that have different objectives in mind that care about different things, it resembles very closely many moral dilemmas where also typically we end up trading off the welfare of one agent versus another agent. And so for that reason it seems sensible to think that game theory might somehow inform moral reasoning, and that's some of the examples that we see in game theory can be helpful for elucidating our own moral reasoning as well. There's a nice example of a game called the trust game. So in the Trust Game, there are two participants in the experiment. Let's say you and I. I get ten dollars and I can give any amount of that to you, that gets tripled. So if I give you five dollars you actually receive fifteen dollars and I still have the other five. That's step 1. Step 2, you give back whatever you like. This time it doesn't get tripled. So you can give me back nothing and then you still have the 15 dollars or you might give me back 10 and I have 15 and you have only five. So this is what's known as the trust game and if you take a very cynical perspective on this set up, and if you believe that all people want to do is maximize the amount of money that they themselves walk away with, then you would never give anything back because why would you give anything back, that would just reduce your amount of money. And I can anticipate that, I can anticipate that you won't give me anything. So I will give you nothing. And so we end up in the beginning state where I have ten dollars and you have nothing. But we both could have been better off if I had given you five and you had given me that let's say seven back. Now we both would have been better off. And indeed when you have people play the trust game, they do tend to give money and they do tend to give money back. And so why is that? So this brings us into what's known as behavioral game theory. So whereas most game theory studies how idealized agents would act in a world where they are infinitely rational and optimally pursue whatever their objective is. Behavioral game theory in contrast is about how people play games. And sometimes the issue is that the game is too complicated for people to find the mathematical equilibrium of that game. But it doesn't seem to me that that is the case in the trust game. The trust game as I just explained it to you is a very simple game. And so it seems that people can understand this game and they still choose to give money and to give money back. And I think there we're really going into ethics. If I give you some money and if you can imagine now me explaining to you that look, I'm giving you this money so that you will give something back and I trust you. I know you can walk away with this money but I trust you, I trust you to do the right thing and give something back to me. Then it simply seems wrong for you not to give anything back. And I think that is what's driving a large, to a large extent the behavior of people in the trust game.

RS: As you peek into the future of AI and morality, what are the challenges?

VC: I think there are a lot of challenges in the-- so in the somewhat shorter run. We have to deal with this problem of how to set policies or objectives for algorithms based on the opinions of multiple people. Which are to people that should make those decisions and what kind of information should they have access to. In the context of kidney exchange, you might think that the best people to make these kinds of calls are people with medical training. And there seems to be some truth to that. But on the other hand, for determining whether it's relevant that a person has dependents or not, it doesn't seem that medical training particularly qualifies them. And so maybe what you really want is kind of a jury of your peers, randomly selected members of the population, that nevertheless have access to expertise that allows them to understand which of these features are relevant, what we need to think about in the context of kidney donation. So you can think of a kind of expert witness model, where in courts you might bring in an expert witness to inform the jury. That kind of model might make sense. But I think this is still a very open question and I think that's one that really as a society we need to be thinking about. What the right protocol for making those decisions is. 

RS: And down the road further?

VC: Down the road, I think as AI systems get more sophisticated, they will be acting in many different domains. And there I think this issue of how do you take insights that you've achieved in morality in one domain and extend them to a different domain, as these systems become better and more flexible and able to switch contexts more easily. They also need to be able to extend their moral reasoning accordingly. And so they might have to make moral decisions in context for which they have no data about how people would make those decisions. And still we would like them to somehow do something that is in accordance with our own moral perception of the world. And that I think is a challenging problem because in those cases they will not have seen examples of humans making decisions in that context. And still they have to somehow infer what a person would have done. Or perhaps what the person would have liked to have done in an ideal situation.

RS: You're an optimist?

VC: I am an optimist. I think there's a lot of opportunity here to do good for the world. I think AI systems will make human lives better. We have to be very careful and make sure that they don't make human lives worse. I think there are real things to worry about. We can worry about technological unemployment, autonomous weapons systems, large scale surveillance, these are all issues tied to AI and that AI could be, it could be abused to create a worse world for ourselves. I think in large part that it is up to us as humanity to determine how we're going to use this technology, what uses we will permit and which ones we won't permit. How we design the systems that we do allow out there. I think this is exactly the right time for society to be thinking about these issues as AI systems are starting to get deployed. [15:13:57.16] But there are still many things that are the prerogative of human beings that the AI systems cannot yet compete with humans on. But there will be more and more aspects of human life, where AI will start to play a role. And the kinds of decisions that we make now and maybe more generally the frameworks that we come up with for making these kind of decisions, for setting these kind of policies, I think will have a long-term effect and will shape how we make those decisions in the future as well, for applications that maybe right now we cannot even imagine yet.