Well thank you very much for joining us.

First could you please introduce yourself and your

position.

Adrian Weller, Program Director for AI at the Turing Institute and I work in Machine Learning at
University of Cambridge.

Excellent. Our first question is just biographical. How did your work in AI begin?

I’ve always been interested as long as I can remember been interested in intelligence, in
understanding human intelligence and artificial intelligence. My dad is a psychiatrist and I would
grow up with my brother going to meet my dad for lunch in mental hospitals and he would say
he doesn’t feel at home unless he’s in a mental hospital. I think he was joking. He would
sometimes have patients come to our home and my brother and I would sometimes be
entertaining mental patients before they were going to see our dad. So, we early on had an
interesting perspective on the ways that the human mind can perform in different kinds of
directions. But always been very interested in how, what a human intelligence really means. I
think that’s one of the very deepest questions. And then I think another question that can
possibly rival that in terms of depth is what does intelligence mean more broadly? So, long been
interested. My path into where I am today is maybe a little bit different from some other folks
you’ve been speaking to in that I, I spent quite a few years in, working in finance and trading in
the markets. But there was a natural pause in what I was doing there, and I had some time off
where I had a sort of non-compete period. And I thought that if I didn’t try to get back into this
sort of thing, if I didn’t do it then it really would be too late. So, I went back to into an academic
world to do a PhD, but later on it was saying I was really excited to be able to come back to the
UK and to engage at Cambridge and to engage here.

What was your PhD in?

In Computer Science. Machine Learning.

So, either early in your interest in intelligence or later on when you dived into the PhD, was your
interest influenced by popular culture such as fiction writings or movies and such?

Sure. I think all of us are very heavily influenced by those around us and by all the aspects of
culture which touch upon us so, I’m sure, yes. But are you asking were there any very specific
things that I can remember?

Yeah, I’m curious if there was something specific.

I doubt anything unusual. The usual sorts of things. Asimov’s books, 2001, all the usual sorts of
things I guess. Let me just think if there was anything unusual. So, one of the earliest stories,
this is not about fiction, one of the earliest stories that I hear from my parents about my behavior
was that when I was maybe four or so, I was looking at a fly and observed to my dad that
they’re really just machines aren’t they. So, so...

So there’s a little bit of Turing in you back then.

I don’t know about that. I didn’t know what Turing thought about flies. But I think human beings
are amazing, creatures are amazing, but it’s also amazing sometimes to try and think about the
ways that they work.
Excellent. Well that’s, it really actually colorful and interesting. Let’s dive into the present and
your present position at Cambridge for instance. What’s your present research area and focus
in AI?

So, I’m broadly interested in quite a range of things. Initially I was very interested in graphical
models and inference in graphical models and ways to try to perform inference in learning that
scale efficiently. Some of those questions are related to sort of the limits of what could be solved
in polynomial time. Oops. You see we come across all kinds of technical challenges and it’s
hard to anticipate them all. So, inference in graphical models. But then I’ve become very
interested, I’m broadly interested across a wide range of things. So, at Turing I’m involved in all
kinds of different areas but the main areas of my research focus on various aspects of what I
would describe as technical approaches to safe and ethical AI. [...] We’re seeing significant
increases in capabilities of AI systems and that’s leading to deployment broadly across society
in ways that already have impact on many of our lives through areas such as hiring decisions or
even more directly criminal justice. But also, even more broadly areas which perhaps seem a bit
less severe but I think still are very important for example in social media. So, the things which
we’re recommended to, to look at. The way that we see the world and way filtered through a
digital lens controlled by companies who are motivated to get us to click on things in the short
run, so that may not be in our long-term interest. So, I think the sorts of challenges around that
are pressing, very interesting and very important. They reveal things about us and what we want
for our lives. And they raise lots of really deep challenges, that are highly interdisciplinary. So,
as I said they relate to ethics and social science, but also to a lot of really deep interesting
technical questions that tie into that. So, one of my other affiliations actually that I didn’t mention
at the beginning is I lead a project at the Leverhulme Center for Future Intelligence which is
based in Cambridge but also has other branches, other spokes around the world including with
Murray Shanahan who I think you’re going to be seeing later. So, that group focuses broadly on
societal implications of AI, and care a lot about those. So, my project there is on trust and
transparency which ties in with this. We also have a project here which I coordinate on fairness,
transparency and privacy. But, more broadly, I would suggest that as these algorithmic systems
are deployed across society, one way to have an overarching perspective on at least how I think
about them is that we’d like to feel that we’re able to trust these technologies and it’s dangerous
to trust things too much. When you trust something, you’re exposing a vulnerability and you
want to make sure that you’re not going to be abused. So, following what some other folks have
said, I don’t know if you’ve seen the work of Baroness Onora O’Neill, so she says, “No we need
to have measures of trustworthiness,” and she’s talking about dealing with people when she
talks about reliability, honesty and competence. I suggest that when we’re talking about AI
systems I’m going to give you a laundry list of things which I think are important to focus on.
Each of these has really important areas of technical research and all of them it’s important to
connect with social scientists and philosophers, etc. but I’m very interested in the technical
aspects of all of these. So, I would make my list as fairness, so that certainly includes
measuring and mitigating bias against an individual or any particular subgroups, transparency
which is very broad but certainly includes things like understanding where data comes from
that’s used to train algorithms, how it’s been processed, how it was decided where to get that
data, and then all these issues around interpretability, so how can we try to understand how
algorithms are really working and we could speak about that for a long time. I think there are
actually a lot of different kinds of transparency depending on to whom you’re trying to explain
and what the context is. I’m happy to speak a lot about that. Privacy and security. So how can
we ensure appropriate protection of individuals privacy? There are many applications where we
believe we could learn all kinds of amazing things if only we could gather together lots of
people’s data. But of course, people are rightfully worried about concerns over their privacy. So,
healthcare would be a very important example. Related to privacy is there’s security, so how
can we also ensure that systems are robust against cyberattack, not only traditional forms of
cyberattack, if that’s the right way to describe them, but also more recent forms of attack,
specifically on machine learning or AI systems. So, so-called data poisoning attacks. So
fairness, transparency, privacy, security, along with security I talk about resilience, so even if
part of a system goes down how can we try to ensure that a system remains resilient in being
able to operate, like an ant colony can work well even if part of it goes down. Robustness, by
which I mean something a little bit different, so we’re used to a machine learning paradigm
where we learn a model in the lab and we expect that the data we’ll see later on will be of the
same kind, will be i.i.d apart from of course that’s often not the case in the real world so how can
we try to ensure that an algorithm will reliably perform well even when it comes across settings
that are very different than those in which it was trained. So that’s a very broad field, it includes
things like modeling uncertainty, so if we know that an algorithm is uncertain about what to do it
might want to switch to a safe fallback mechanism and raise some kind of alarm that uncertainty
obviously may also help guide it towards knowing where it should next look in order to learn
best. And it includes a whole host of interesting areas including for example, robustness against
adversarial [missed @ 9:36], in some sense that the worst possible kind of thing. And some
other areas I’d mention are controls, so how can we ensure that as systems become
increasingly autonomous, they do the things we would like them to do, and influence, by which I
mean broadly this area which we’ve already touched on which is we know that algorithms are
influencing us, nudging us in various ways, how can we try to quantify that and control it so that
we can attempt to ensure, in a transparent way, that we know that we’re being nudged in a
direction that we want to be nudged.

Excellent. This is a fantastic list. When I look at that list though, I juxtapose it with something
you said earlier, which is that people are starting to field systems already into society.

Starting to field.

You said that people are already beginning to field certain AI systems into society.

Yes.

So, things are being birthed into the wild.

Yes.

And yet, the list that you went through just now are all items that arguably there’s a great deal of
research yet to do on these.

That’s right.

For us to feel really assured that we’re designing systems in a way that is subject to these in a
good way.

That’s right.

So that takes me to the public.

Yes.
And my next series of questions are a little bit about how we communicate about AI to the
public. The first one is, can you think of examples where AI has been accurately or appropriately
communicated to the public?

Good. That’s a fantastic question. But if I may, can I go back on your previous comment? In
terms of public concerns over, over these technologies being deployed, I had two more
thoughts. One is that of course another super important area is the impact on the workforce. So,
disruption to jobs. But I’d suggest that there’s less that I can immediately do from a technical
perspective to try to improve AI systems in that way. There are, there clearly are quantitative
analyses you can do that people like [missed @ 11:26] might have been involved with, which
are really interesting and important. But in terms of public concerns, the other thing I’d say when
you talk about systems have already been deployed, I think an important point to make is that
as you well know, if you’re an academic and you want to try to release some system to learn
information about what the public do, you have all kinds of important ethical hurdles you need to
overcome before you’re allowed to do that. You have ERP Boards, or other kinds of ethics
boards you need to get through. And establishes a very different set of criteria for what’s
necessary compared to the mentality for startup which may have the mindset of move fast and
break things.

Direct quote I believe from somebody in startup land.

That’s right. And you can understand why they feel that way. So, we’re in a world where. Well,
it’s always been the case if you can establish mark of dominance and set up a moat around
your business that’s just a great thing to do from a commercial perspective. Arguably, it’s even
more important and there are even more opportunities to do that today in the digital realm. So
therefore, you really want to be the first guy and you want to establish that moat, so you really
do need to move fast and break things in order to get there quickly. To hone your product and
get there.

[missed @ 12:41] mentality.

Not only mentality, but it’s a fact, it’s arguably an empirical observation in many companies that
we see today in social media or transport applications, various things like that. And so, given
that there’s a great incentive for people to move fast and break things and I think we need to
pause and think about what that means for society. So, are we okay with people adopting that
kind of behavior or do we need to think about some possible kind of regulation around that? And
when we do that, I don’t at all suggest that there’s an obvious answer, and clearly there are
advantages to developing these things. They can be really good for mankind. We need to, we
need to weigh out the pros and cons. And I should mention the one thing that I think the UK is
doing well is that, I think that they’re being thoughtful about these kinds of issues and they’ve
just set up something called the Center for Data Ethics and Innovation, I don’t know if you know
about that, but that’s a sort of stewardship body to think medium to long term about these kinds
of issues.

Well, so let me push that back on the public in a different way. Which is what is our
responsibility as a community in terms of communicating with the public?

Good, good. That was your question.

What kinds of influences does the public need to have about AI?
I think it’s really, really important that, if I say we in the technical community, we have a great
responsibility to communicate effectively with the public and with a really broad audience so that
we can try to have an informed discussion and debate about some of these topics. The issues
which I mentioned are that there are properties that I think many people would agree we’d like
systems to have. But that’s a bit like saying for our economy we would like to have no taxes and
we would like to have a huge social safety net. There are often tradeoffs between these different
things. And in order to have a proper discussion and debate about the tradeoffs, we need an
informed public. So, I think we have an obligation to tell our public about what’s going on, how it
works, how it’s affecting them, so that we can try to have this kind of discussion. Now it’s not
easy. If I were to ask you when’s the last time you think we’ve had an informed discussion about
anything, it’s challenging, but it’s really important that we try. And we can talk about how to try to
do that in a second. Clearly, it’s also important as we think about these issues to be interacting
with and working with policymakers, social scientists, economists, ethicists, etc. So, we really
need to work closely with them. But we do need to inform the general public and to help them
understand what’s happening and to work with them. And then to try to talk about how can we
best do that, I don’t claim to be a great expert, but there are some things we can try to do, it
would be nice work in this country where DeepMind partnered up with the RSA and Turing’s
getting involved in some of that work to have kind of citizen juries to ask them how they feel
about different issues. We’ve partnered with the BBC recently to try and help inform the public. I
think that’s a really good channel. Actually, I think the BBC is interesting. Of course, you could
have any media partner to help communicate, and that might work well. BBC is particularly
interesting because they have an explicit mandate not to try to make money but to inform the
public and to be a responsible operator that tries to present balanced perspectives. And that’s
also a separate issue which we’re talking about is interesting which is what should responsible
media company do in this age?

Excellent. I have another set of questions about, about labor. Not labor economics, but about
the kind of the philosophical parts of labor. And the first question is if you think about AI systems
up until now, how have they changed the way people work?

Do you mean how has it affected the jobs that people do?

Yeah, how has it affected what people do from beginning to end of the day? So, the jobs they
do or the way they work, or how they derive efficacy from their work?

Well, it’s a great question, and I’m sure that there are great experts on this and I think it’s
important to get reliable data on this. Cause if we’re trying to think forwards about the
techniques which we’re likely see, it’s really important to be able to look historically and see
what has happened. So, I don’t claim to have expertise, I’ll tell you some things which occurred
to me after with my head. Clearly, it’s had a massive impact. And I think you asked specifically
about what impact AI has had.

Yeah on the way we work.

But can I?

But you can broaden it to computation with technology.

Or can I broaden just to technology?

Yeah.
Cause a lot of these issues I think are really more about technology than specifically about AI.
Although AI does also have some particular issues associated with it. But clearly the whole
world has been transformed. Even I mean if, I don’t know how far back we can go, but far, also
into the ways that human live together has been transformed by our ability to produce food more
effectively and to be able to live together in closer harmony. Things like email or the various
ways we can communicate very efficiently means that it’s perhaps less necessary to actually
work together in person, although I think, still think it’s by far the best way to work and to be
face-to-face. There’s some good examples from Alan Turing’s life. As we sit here, I’m sure you
know that many of the computations used to be carried out by what are called computers which
were people who individuals often women who are actually doing computations. Even what we
call a computer has changed dramatically just because automated computational capacity has
changed, so clearly that’s had an enormous impact. But broadly, anything to do with those kinds
of calculations or computations has become enormously easier. We’ve recognized for decades
computers are much better than us at doing those sorts of things. So, adding numbers together,
or doing those kinds of calculations. Or having spreadsheets. And it’s interesting to see how this
isn’t quite your question, but the way that we think about what real intelligence is and our, as
humans we often like to think about ourselves as having something that’s very special, as we
see technology enable computers to do things that we thought we were the best at doing, we
shift those into a category where we think that those perhaps are not so important in terms of
intelligence. But for a long time, we’ve had computers do things that are better than us in
particular ways and those ways keep expanding, and clearly as machines get better than us at
doing those things, we shift to doing other things. And for the most part I suggest that that is a
good thing. It enables us to focus on areas which for the most part, so far, have been more
interesting for us. And have enabled us to be much more productive in total, but of course it
does raise questions about how that will impact things over time, and will there be big disruptive
effects which we need to think about?

As that shifts us, are there tensions you might expect in regards to human dignity when we
shift? Are there cases where in fact it may be a bit of a challenge to human dignity when AI
comes in and does some of the work that we do and comes into shift?

Yeah. So, I think there are many aspects of that, and we just talked about two of them. So, one
of them is the direct and potential impact through inequality. So, the economy is a complex
system and you get complicated, unexpected consequences of things, but I would suggest that
for the most part, most of the time, technology is a kind of multiplier that enables people with
greater ability to use it to expand the degree to which they can contribute more than people who
are not so well capable of using it. And therefore, that leads to greater inequality, if you don’t
really think about it.

Is it between the technologically fluent and those who don’t have that fluency?

That’s right. For the most part of course there are all kinds of issues around that. But for the
most part, I would suggest that technology does tend to do that. I mean it doesn’t always, so as
an example, it will, as we were saying before it enables more efficient communication, so it can
enable people say in a developing country to perform a job which they wouldn’t have been able
to do before because you can link up to them directly, and that is improving their capabilities.
Enable them to earn more money than they would before, so raising their standard of living. And
overall, we should keep in mind that people like Steven Pinker point out that the number of
people who live in poverty has been reducing in absolute terms at quite a rapid clip. So that’s all
great. There’s a lot of great stuff going on. On the other hand, there are also increases in
inequality. And I think that it’s right to think of that as one issue about dignity. I think that as
human beings we’re social animals and when we lift necks to we live near or we see some sort
of [missed @ 20:49] even if it’s through social media or on television, we’re constantly reminded
that people who live much better lives than us. That is an arguably an issue about dignity. So
that is one aspect. Another aspect I’ll touch on that I think people often think about is things like
being cared for, particularly for the elderly. So, the world population is generally aging
significantly. There’s a significant demographic shift. And there are strong differences in opinion
of whether people think it’s a good thing or a bad thing that the elderly may be increasingly
looked after by machines. Some people think that’s an assault on their dignity and is a terrible
thing. And it’s easy to see where people come from thinking that but at the same time there are
plenty of people getting older who have no help whatsoever and surely, they should have the
right at least to be taken care of by these robots, machines, and what if they actually prefer it to
being looked after by people. How do we feel about that? I think there are a lot of issues that are
interesting to explore and well to some extent we’ll see how they play out.

Excellent. Alright thank you. Our next set of questions is about power and power negotiations.
First of all, can you think of a, an AI tool, whether an existing one or, or kind of conceptual one,
in which power is transferred from a human user to the AI system, and how that affects human
power relationships?

I think that’s a complex question. So, you used the phrase something like “power is transferred
from a human to an AI system”, is that what you said?

That’s right.

And I think that there are important things to discuss about whether that ever happens. So, is
the AI system ever actually really in control, or is it ultimately the user of that system who has
the power? And there are aspects of the answer to that which go in both ways. Certainly, in
terms of accountability and responsibility, I think it’s really important at least for the foreseeable
future, that we hold accountable the operator of an AI system, as we would hold accountable
the operator of any technological system if something goes wrong. So, we shouldn’t allow
anyone to say: “No it wasn’t me, it was my algorithm.” And therefore, to avoid culpability. So, it’s
unclear whether the algorithm actually has any direct power. On the other hand, though, let me
give some examples where it certainly feels as if it does. And then lastly perhaps talk, I’ll
describe situations where it might seem as if it doesn’t but in practice maybe it does.

And you mentioned the criminal case earlier, the crime.

Yeah.

So that might be one you want to throw it in there too. Cause that’s maybe another example.

Yeah, good, good. Remind me about that if I don’t come back to it. But there are already
settings where I suggest that algorithms are operating in the wild at speeds that mean its
impossible for a human really to intervene meaningfully. And so, I think it would be hard to
argue that they are not operating by themselves in some aspect. And a great example would be
trading algorithms, so high frequency trading algorithms, where they are competing at the
certainly the millisecond basis, maybe even microsecond basis, against each other in a very
high stakes game, and they have real world impact in ways which are sometimes difficult to
foresee, which is an interesting thing for us to consider and think about. So, typically these
systems in normal circumstances add liquidity to the markets, help the markets function more
efficiently, but there are concerns that when things go wrong, potentially things can go wrong in
worse ways. We’ve already seen things like flash crashes. And it shows that even if we can
understand exactly how one system operates when it’s added in the world and operating
against a whole set of other agents, it can be really quite difficult to predict the sorts of
consequences that can happen if various things go wrong.

It’s like emergent phenomena.

Emergent phenomena. That’s right.

What about the opposite case? What about the complementary case? Can you describe an AI
system, hypothetical or real, that empowers a group or subgroup of people?

Sure. Do you want me to do that right now? Or do you want me to come back to the second part
of the previous question?

Oh, you’re right. Let’s go back to the second part, please.

Okay. So, situations where it may not seem as if they have power, but I suggest they would
have power, and maybe we’ll use an example from criminal justice, which I think is an extremely
important set of use cases, which are where we start to see increasing use particularly in the
States. So, there’s been a lot of attention, I’m sure lots of people have talked about this around,
a particular algorithm called COMPAS, which attempts to predict recidivism risk. And it’s used to
help decide whether or not to grant people parole, or even to help decide how long people
should be locked up for. And obviously it’s a very important issue, if an algorithm tells me I
should be locked up for six years. The first thing I would say is, is there an important
requirements about transparencies. So, I really want to understand did it come to that
conclusion. Check to make sure it follows due process, correct process, in a way that would
allow me meaningfully to challenge it. So, I think it’s actually one really good, clear use case
where you do really need to have transparency. There are cases where perhaps you don’t need
to. But you really do need certain kinds of transparency. The kind of transparency that enables
meaningful challenge. The concerns that were raised that were interesting were not so much
about transparency. And were also interesting not so much about accountability which is a big
deal. Just to comment on that I’d say that the judges that are using this system, I think,
absolutely need to be held accountable if it doesn’t work correctly. And that’s an important
subtopic. But the way in which they’re attacked was about fairness, what was it treating different
groups fairly? And that’s another whole big topic. Roughly speaking, there’s a lot to think and
say about what’s it mean for an algorithm to be fair. And there are various ways you might try to
define fairness. And in technical community, what we typically like to do, is we like to be told
exactly what fairness means as a formula, so then we can say “Okay, thank you. That’s what it
means. Now I can optimize my algorithm subject to this constraint and if I can do that efficiently,
I can write a paper too and that will be really great.” And then as I’m sure you know, there’s this
whole field where we got about twenty different definitions of fairness and there are lots of
different papers talking about how you can optimize with respect to these different definitions.
And as you might expect, it turns out you can’t simultaneously enforce many of these
constraints at the same time, cause it turns out to be impossible to do that. The ethicists and the
lawyers on the other hand say you guys are crazy, you don’t understand, you haven’t got the
correct notions about judgement and appreciation of context that are really important to really
understand what fairness means. And both sides have a point. I would suggest that its actually
really healthy that we’re getting this interaction cause it's helping us to sharpen what we think
we really want to have as a fairness measure. But going back to actually what you asked about,
sorry I’ve drifted a bit, I think that if an algorithm makes a recommendation for how long
someone should go to jail and that goes in front of a judge, the judge might or might not take it
on board. The first thing to say is that rather than myopically focusing on whether what the
algorithms outputs is a fair output, we should be looking at the combined system. The human
and the machine together. And is the machine helping the human produce a sort of combined
output which is more fair, more reasonable or not. Cause you could imagine, you could have a
very biased judge who chooses to go along with the algorithm when it says the same thing as
him, but who chooses to overrule or goes the other way, and arguably you could have a fair
algorithm but it’s being used to increase and exacerbate unfairness. So that’s one important
point. But again, coming back to your question, and its perhaps not so realistic in this specific
setting, but let’s imagine you had a setting where the algorithm was making its recommendation,
and the judge would agree with that say 98% of the time. Over time, people get a bit tired of
looking, people are naturally a bit lazy. And actually, it’s a good thing in many cases, because it
gets them to try to be smarter about doing things. But, if as a human you’re meant to be making
your decision but actually the machine is getting it right 98% percent of the time, you’re unlikely
to start really paying as much attention over time. This is an example of what is sometimes
called technological enfeeblement. If some technology does something for you, we as humans
become less good at doing that. Over time, that hasn’t been such a bad thing, if we’ve got
worse at remembered large volumes of information because we’ve developed writing, but that’s
been okay. But it’s unclear how that will necessarily work out going forwards, and specifically in
this kind of case, the human can eventually just really become a rubber-stamping entity, or
observing machine that’s just saying “Yes, I agree,” “Yes I agree.” And then it’s unclear whether
the human does really have any control, even though it might look like a human is in the loop, to
use the expression. And this is really important so that there’s legislation like the GDPR that I’m
sure you’re familiar with, that has been issued in Europe recently, which has different rules
depending on whether or not a human is in the loop. But sometimes whether formally human is
in the loop doesn’t necessarily imply they are really engaged in the process.

Excellent. These are all outstanding issues for students to look into, so I, I love the fact that
you’re privileged in these issues. The last major category we have is autonomy. The whole idea
of autonomy. So, to get, to begin philosophically, what do you perceive is valuable in the
prospect of machine autonomy?

I think we’ve already covered quite a lot of these issues. So, the issues we talked about before
about when a machine is actually taking the decision for itself I think is extremely close, really in
spirit, to what we mean by autonomy. Taking actions or predictions by itself.

So, I’ll take you, I’ll push on that a little bit. If we consider machines that can make predictions
by themselves or make decisions,

I misspoke. Yeah, predictions is okay. Actions or decisions by themselves.

Actions or decisions by themselves. Something that you have said though is the idea that we
need to hold accountable somebody who’s using that system.

Yeah.

If we look forward a little bit and think about for instance, a self-driving systems, self-driving car
system, that is adapting in some way and learning some way and therefore its behavior is a
mixture of what the person in the car does with it and its sort of genetic precedent of how it
started. How do we think about ascribing responsibility to the decisions it makes and the harm
that it might do?

That’s a great question. Just before I get to that, can I suggest that one interesting thing to look
at which is not immediately obvious as an analogy for artificial intelligence, is something that
David Runciman at Cambridge talked about. Is that when we think about these topics, don’t just
think about artificial intelligence, think about artificial agency. So, it turns out that we have a long
tradition, hundreds of years, of setting up artificial agents, that is agents who aren’t real persons,
and we even call these things persons. And those are companies. Sometimes countries. But
certainly, limited liability corporations. And I think it’s interesting to reflect on why those were set
up and what were the benefits and disadvantages of doing it. Clearly one disadvantage is
exactly along the lines that I said, and maybe you’re raising. It is if I’m worried that my algorithm
is going to do something that’s wrong, I was just arguing that well it shouldn’t be okay for me to
release it in the world but say if it goes wrong it’s its fault. I should remain liable. And yet, having
a limited liability company ran something does exactly enable that. I can say, well I set this
company. The company has released this product. If the product goes wrong, the company
goes bust, but I’m protected as this limited liability. So, I think it’s really interesting to look at that
again and think about these issues. You think.

In fact, the EU has a preamble now on the cost of considering personhood for certain kind of...

They do, and I don’t think that’ll be taken very seriously in the near future. I mean it was taken
somewhat seriously, I think we perhaps even debated in European Parliament. But I think it’s
unlikely to happen soon for the reasons that I gave, but I do think it’s important also to compare
to these issues about limited liability companies. But, you also, I think you were touching on a
very set of really important, interesting issues which are really a big section of artificial
intelligence and the law in some ways. So, how do you know which person including companies
here, is liable if something goes wrong? And I’m not a lawyer, I’m very interesting in working
with lawyers actually, and this is one of the questions that I think is really an important one. How
should you ascribe liability in these kinds of cases? First, we should say that it’s not a new thing
that you have a complex technology and it can be difficult to figure out who’s at fault. That’s
been going on for a long time. I think there was one great example that I heard about where
maybe Toyota had some problem with the brakes in the States maybe twenty or thirty years
ago. And NASA got involved an expert witness to help opine on who should, who should be
held liable. And it was a whole complicated thing. We need to have practical ways, we can’t
always bring NASA in to opine on these kinds of situations. We need to figure out how to have
some scheme that enables us to assign liability in some reasonably efficient fashion, and we
want whatever mechanism we come up with to be good in the sense that it encourages the right
kinds of behavior in society. So that’s a, a bit of a puzzle. But again this isn’t the first time this is
being raised, but I think that there are issues, specifically with machine learning systems, that
make this complicated. So at least in UK law and you can tell me right away there’s probably
similar issue in US about foreseeability of harm. So, in general if I do something and it was
reasonable for me to foresee that it would cause harm, then I can be held liable. On the other
hand, if I do something and it causes something totally unexpected that no one reasonably
would have foreseen, then it’s less reasonable to assign that liability to me. Now with machine
learning systems, by their essence, they’re learning from data, typically in order to find patterns
which are very difficult for humans to see, so very likely they’re going to be doing something
which is hard to foresee, so how are we going to deal with that? Is I think, raises interesting
questions. Also raises interesting questions about the way these systems interact in complex
ways, as we talked about before.
And this harkens back to the word transparency that you gave earlier as one of the major
considerations.

Yeah, but let speak to that. So, I’ve argued for transparency, but let me raise some potential
arguments from the other side against why sometimes transparency is not the best thing. First,
there are often tensions, alright. I mentioned that there are often tensions between all these
different properties. One immediate tension that’s easy to see is between transparency and
privacy. So, if I wanted to make everything transparent about my algorithm, particularly if its a
machine learning algorithm, its predictions, its behavior is typically going to be very dependent
on the data on which it was trained. So, if I really want to be totally transparent I’ll show you all
the data on which it was trained. But that data might have personal information, you know, there
are clearly privacy constraints against being completely transparent. Also concerns about the
IP, maybe I’ve got private IP, I don’t want to share this information with you. There are lots of
interesting ideas coming from areas like homomorphic encryptions and secure multi-party
computation that might allow nevertheless for aspects of the system to be checked without
revealing the exact information, we have a paper about that, there was [missed @ 13:53] so
there are interesting questions around here. But that’s an interesting example of one tension.
Let me give an example of another tension that’s perhaps a little bit less obvious. I think
transparency is often a means to an end. And not actually the end that we really want, so as an
example, people will say of course if autonomous vehicles are going to be riding around on the
street, we need to understand exactly how they’re operating. Otherwise how can we possibly
allow them to be on the streets. But ultimately, perhaps what we care about is having cars which
drive around safely and don’t kill people. Right now, humans kill a lot of people. Would you
rather have autonomous vehicles which operate in a fully transparent way but, but kill a million
people a year, or cars which we don’t really understand but they only kill a hundred people a
year? And I think people would go for the ones which empirically don’t kill very many people.
And that’s actually somewhat similar to the way that we deal with aviation. We have a whole
load of, a whole battery of tests that they need to satisfy. And if they satisfy those tests, you
could imagine gradually letting cars out in the real world. Seeing how they work. As long as
they’re not killing people and working well. Maybe that’s okay. So sometimes we care more
about some of those other things I mentioned before, reliability, robustness, etc. than
specifically about transparency.

Let me push back on that and ask you, because you suggested a, a dichotomy between cars
that are opaque and killed fewer people or transparent and killed more people.

Yes.

But if we’re designers and we get to decide how to design AI and learning systems in the future.

Good.

Can’t we decide that we want to make systems that transparent and kill few people? Or is there
some fundamental flaw in that direction because the transparency itself barriers us from killing
few people?

Some people talk about transparency as a barrier and there necessarily being a tradeoff
between transparency and performance. We don’t know exactly where the frontier is of that. It
may be that we can have transparency and performance. It’ll depend on many factors. But what
there certainly is, is there’s a question of opportunity cost. So, we only have limited resources,
only certain number of people. And when you have certain resources to deploy, should we
spend all those resources on transparency, or should we split them and maybe doing have
some of them on transparency and some on those other areas. And perhaps working directly on
safety, for example, might be more important.

So, let me ask one final question on that topic, which is if you think about a metaphor of this or
an analogy of this is the Google search engine system. People demand transparency so they
want to understand why the search engine at Google rates them poorly. Yet, with transparency
into Google’s search engine comes gaming and the fact that people can then.

Yeah.

Manipulate the results because they understand deterministic system...

That’s right.

That defines the result.

That’s right.

Even tension we face with things like self-driving cars as well.

It’s one of the drawbacks, one of the potential drawbacks of transparency. I’ll speak to that and
then I’ll tell you some more if you like. So, I have a paper on challenges for transparency that
addresses why transparency can be bad in various contexts. But that’s a general phenomena
and it relates to, so whenever you release the walls of a system, people can try to game it.
Sometimes that’s actually good. But usually it’s bad. It’s usually good if you have complete
knowledge about the way the world works, and you’ve set the parameters exactly to what you
really mean, so I’ve set up the walls and I am so sure that these are the right walls that I’m
happy for everyone to know them, because if they game them that’s only going to be good.
Arguably that’s the way the attack system works, arguably, or something like that. But perhaps
not, but otherwise you’re subject to what’s sometimes called Goodhart’s Law. So, Goodhart was
one of the members of the Monetary Policy Committee which is like Fed here in the UK. And he
observed that when you’re dealing with economy, you typically have certain macro measures of
the economy which would be good general indicators of economic performance. But if you then
say, as a policy matter, you’re going to focus on those, often you don’t end up getting the
benefits you hope because you’re pulling levers to try to optimize those specific things which
typically would be correlated with good things in the economy. But those relationships can break
down when you focus exclusively on those.

Like Moore’s Law for processing speed. Becoming an edict rather than a description of what
actually happens.

I’m not sure if it’s quite the same thing. So, tell me what you mean by exactly by Moore’s Law?

Well, in Intel corporation case, that became the specific milestone in other words we have to
create new benchmark algorithms that will perform at Moore’s Law on a new processor rather
than we’re going to make processors as fast as we can, and oh look, they happen to follow
Moore’s Law.

That’s right. Or...

Like tail-wagging the dog.

That’s right. Or perhaps we should operate in a way that will maximize long-term shareholder
value, or something like that. But it’s tricky because having some robust, having some pretty
robust, very specific measures can give people focus. And as long as they work reasonably
well, that could be a good thing, but sometimes they don’t work reasonably well. Can I point out
one other issue with transparency which is not obvious? So sometimes if you make information
transparently available it can actually lead to a system becoming less efficient. And there’s a
nice example of something called Braess Paradox. Are you familiar with Braess Paradox? So,
it’s easy to see with a picture, so I’ll just try to describe it briefly. And you could look at this paper
if you’re interested in it. And this specific example is due to Frank Kelly, who’s a great professor
at Cambridge. So, if you imagine we’ve got a road system, with cars coming in here and they’re
going out here and they have to decide where to go in the middle. The typical model of how cars
will behave is that they will optimize their route they follow until they can’t go any faster. So, if
everybody does that, the system will settle down to what’s called a Wardrop Equilibrium where
no one can do any better by taking different routes, or everyone has the same delay going
through the road system. A reasonable assumption is that the more cars go down any particular
road, the more delay everyone will suffer going down that road. And that’s what an example of
what’s called an externality in economics. So, what I choose to do will have a cost which you’ll
have to bear, as well as on me. And it’s known in economics that when you have those kinds of
situations if everybody is selfishly optimizing their best interest, you often won’t get the best
outcome for everybody. So anyways, we have this system and everyone’s doing the best and
we have a Wardrop Equilibrium, now let’s say actually the whole time there was an extra road
which existed somewhere in this network, but no one knew about it. And we’re now going to
make this information transparently available. So, we’re providing transparency to everybody.
Typically, an engineer thinks, “Oh gosh, we’ve added some capacity, this must be really good”.
But you can show in certain examples that actually perhaps it makes more sense to think about
this as you have enabled each of these individual agents, you’ve enabled them to more
powerfully try to optimize their own self-interests and that actually leads to everyone doing
worse. So, the new equilibrium actually has everyone going more slowly. So, by transparently
revealing more information, the system has behaved worse. And we should keep that in mind
this is maybe a different kind of transparency to what we were typically talking about before, but
this does come up sometimes. So as an example, in Europe there’s something called Method
Two, which is a new form of regulation for financial securities. And I don’t quite know the details,
and I’m not saying that this is going on here. But one thing it does it that it makes certain
requirements about transparency about what each agent is doing in the system because that’s
thought to be a good thing to make the system work more efficiently. But I’m just noting that
sometimes it can actually lead to worse outcomes.

Interesting nuance. This is the end of our main questions. We have one final question we’ve
been asking everybody which is what your thoughts are on the development of artificial general
intelligence?

So, would you like to define that? Would you like me to define that?

I would love for you to define that in your words.

Clearly, first, everybody...

It’s in the news.

Sure, it’s in the news, not less than artificial intelligence, and no one knows quite how to define
that either. But broadly speaking if we say that AI is something like trying to get machines to do
things which behave in a way which would be considered intelligent, according to some
definition. Cause what we think is intelligent keeps changing too. If AGI is something like getting
machines which are able to think in a broad, general way, a bit more like what humans do, that
is we can learn from one domain, learn concepts and apply them in another domain. If we
interpret it that way which is, I would suggest, a somewhat narrow definition compared to the
way some people define it, then I think that we are getting close to that, so with techniques like
transfer learning we are starting to find ways to learn from one domain and apply them in
different domain. We don’t yet manage to do that in a way where we in any really meaningful
way I think can extract concepts which make sense to us and can apply them in a different
domain, I think that’s really exciting line of research, cause if we could do that it would help us in
all sorts of ways. It would likely make performance more robust. It would likely make it more
interpretable. All sorts of things. And it would add significant, you know, tremendous capability
to be able to do this kind of general learning. But sometimes what people mean is even, much
broader than that. It is can we make machines which will be able to do everything that a human
can do and if we could do that, wouldn’t it very quickly then do lots of things which are beyond
what humans can do and then we get some intelligence explosion is I. J. Good, called and
where would we be, and would they take over, and what would this mean for mankind. So, I
don’t dismiss those concerns, and some people poopoo that, but I think if we did do anything
like that, it really I think would be one of the most significant things in the history of the planet.
Maybe even the most significant thing in the universe as far as we know it. So, I think it does
make sense to think about it. And it does make sense to suggest that we should maybe be
concerned about that. I don’t think it’s ridiculous to have these concerns. I do think that it’s
difficult to imagine what it would be and how it would behave. So, I like the framework of
focusing on near-term issues, near-term sort of safe and ethical issues as I describe. Not only,
but now you ask, not only because I think they’re really pressing important issues now, but also
because they help give us a toolkit which will we hope be useful in the long term if we do
something moving in that direction.

Alright.

And I shall, I guess I should also add of course if we could develop these generally intelligent
machines, we shouldn’t only think of it as a threat, I mean it creates tremendous opportunities
for mankind too. But we should have some eye to the potential dangers.

Excellent. Anything that we haven’t covered that you’d like to add to the conversation?

I’m good. Thank you.

Alright thank you for the conversation.

Thanks so much for your time.