Autonomous Systems : Social, Legal and Ethical Issues

Cover and back spread:Cover and back spread 19/08/2009 15:17 Page 1
The Royal Academy
of Engineering
As Britain’s national academy for engineering, we bring together the country’s
Autonomous Systems:
most eminent engineers from all disciplines to promote excellence in the
science, art and practice of engineering. Our strategic priorities are to
Social, Legal and Ethical Issues
enhance the UK’s engineering capabilities, to celebrate excellence and
inspire the next generation, and to lead debate by guiding informed
thinking and influencing public policy.
The Academy’s work programmes are driven by three strategic priorities, each
of which provides a key contribution to a strong and vibrant engineering
sector and to the health and wealth of society.
Enhancing national
Recognising excellence
Leading debate
capabilities
and inspiring the next
generation
As a priority, we encourage,
Excellence breeds excellence. We
Using the leadership and expertise
support and facilitate links
celebrate engineering excellence
of our Fellowship, we guide
between academia and industry.
and use it to inspire, support and
informed thinking, influence
Through targeted national and
challenge tomorrow’s engineering
public policy making, provide a
Please r
international programmes, we
leaders. We focus our initiatives to
forum for the mutual exchange of
enhance – and reflect abroad –
develop excellence and, through
ideas, and pursue effective
e
c
y
cle this br
the UK’s performance in the
creative and collaborative activity,
engagement with society on
application of science, technology
we demonstrate to the young, and
matters within our competence.
transfer, and the promotion and
those who influence them, the
The Academy advocates
ochur
exploitation of innovation. We
relevance of engineering to
progressive, forward-looking
e (the c
support high quality engineering
society.
solutions based on impartial
o
research, encourage an
advice and quality foundations,
v
er is tr
interdisciplinary ethos, facilitate
and works to enhance
eat
international exchange and
appreciation of the positive role of
ed with a r
provide a means of determining
engineering and its contribution to
and disseminating best practice. In
the economic strength of the
e
particular, our activities focus on
nation.
c
y
clable laminat
complex and multidisciplinary
areas of rapid development.
e
)
The Royal Academy of Engineering promotes
The Royal Academy of Engineering
excellence in the science, art and practice
3 Carlton House Terrace, London SW1Y 5DG
of engineering.
Tel: 020 7766 0600 Fax: 020 7930 1549
Registered charity number 293074
www.raeng.org.uk

---

Autonomous Systems:
Social, Legal and Ethical Issues
© The Royal Academy of Engineering
ISBN 1-903496-48-9
August 2009
Published by
The Royal Academy of Engineering
3 Carlton House Terrace
London
SW1Y 5DG
Tel 020 7766 0600
Fax 020 7930 1549
Copies of this report are available online at:
www.raeng.org.uk/autonomoussystems
Registered Charity Number: 293074

---

Contents
1. Introduction
1.1 What is an autonomous system?
1.2 The ethical, legal and social implications of autonomous systems
2. Autonomous road vehicles
2.1 Technologies – from GPS and car-to-car communication to centrally
controlled autonomous highways
2.2 Timescales and transformation
2.3 Barriers: ethical, legal and social
2.4 Recommended actions
3. Artificial companions and smart homes
3.1 Technologies – from blood pressure monitoring to Second Life
3.2 Timescales and transformation
3.3 Barriers: ethical, legal and social
3.4 Recommended actions
4. Conclusions
4.1 Communication and engagement
4.2 Regulation and governance
4.3 Ethical considerations
4.4 Looking for applications
4.5 The wider landscape
5. Appendices
5.1 Working group and acknowledgement
5.2 Statement of Ethical Principles

---

Introduction
1. Introduction
This is a report of the discussion at a roundtable meeting held at The Royal
Academy of Engineering on the social, legal and ethical issues surrounding the
development and use of autonomous systems. The meeting involved
stakeholders from a range of areas, including medicine and healthcare,
transport, defence, systems engineering, computer science, financial systems,
public engagement and policy development. A full list of participants and
other contributors is given in 5.1.
Autonomous systems are likely to emerge in a number of areas over the
coming decades. From unmanned vehicles and robots on the battlefield, to
autonomous robotic surgery devices, applications for technologies that can
operate without human control, learn as they function and ostensively make
decisions, are growing. These technologies can promise great benefits,
replacing humans in tasks that are mundane, dangerous and dirty, or detailed
and precise. They also have potential in allowing the remote performance of
various functions, from defusing bombs to monitoring the ill or housebound.
However such technologies raise a number of social, legal and ethical issues,
just some of which will be explored in this short document. The focus will be
on two emerging areas of technology – transport, in terms of autonomous
road vehicles; and personal care and support, in the form of artificial
companions and smart homes. These examples are at different stages of
development and will have quite different impacts on people’s lives, but are
both areas where autonomous systems will be quite visible to the
general public.
Two main kinds of issues arise in relation to these classes of technology. First,
are autonomous systems different from other complex controlled systems?
Should they be regarded either as ‘robotic people’ – in which case they might
be blamed for faults; or machines – in which case accidents would be just like
accidents due to other kinds of mechanical failure. Secondly, autonomous
systems allow for detailed recording of usage. Currently, road accidents, even
fatal ones, attract only cursory investigation compared with air or rail accidents.
But as recording quality improves and costs go down will all accidents be
carefully analysed? This raises legal and privacy issues: what would happen if
most road accidents, currently insurance-classified as ‘accidental’, were reliably
blame-assigned? And how would we feel about our movements in our
automated apartments being recorded and kept by a third party?
Early attention to the issues raised by the introduction of these technologies is
important to ensure that their introduction has the public interest in mind and
has appropriate support. Public engagement efforts on autonomous systems
and debate about their impact is valuable in order to understand and address
the genuine expectations and concerns of public stakeholders, and to take
these into account in the development and implementation of the
technologies. There are significant benefits to be gained from the development
of these technologies.
Autonomous Systems: Social, Legal and Ethical Issues 1

---

1.1 What is an autonomous system?: The spectrum from user controlled,
to automated to autonomous systems
Automated or automatic systems are commonplace, and cause little concern
or interest. Lifts work automatically without human controllers (though they
did have operators in their early days). What makes the difference between an
automated system and an autonomous system?
There is a range of levels of control or involvement that human operators can
have in a system. The following sets out the different grades of control:
• controlled systems: where humans have full or partial control, such as an
ordinary car
• supervised systems: which do what an operator has instructed, such as a
programmed lathe or other industrial machinery
• automatic systems: that carry out fixed functions without the intervention of
an operator, such as an elevator
• autonomous systems that are adaptive, learn and can make ‘decisions’.
These levels of control lie on a continuum from controlled to fully autonomous
systems with no fixed distinction between them. The spectrum can be
correlated with a proportional lessening of the degree of human intervention
or interaction. The degree of autonomy employed needs to be appropriate to
the task.
For many of the systems that lie on this spectrum there is a human agent with
direct responsibility for how they function, whether that be the operator of the
system or the designer of an automatic system. However, with fully
autonomous systems it is more difficult to discern where the human
responsibility lies for its functioning or malfunctioning. There will always be
humans in the chain, but it is unclear in the case of injury which human in the
chain bears responsibility – the designer, manufacturer, programmer, or user?
Do we expect autonomous systems to simulate humans, learning from
experience and making decisions? Does the famous ‘Turing Test’ for a thinking
machine apply here? In some circumstances, an autonomous system may be
more predictable and reliable than a human. An autonomous system that
‘makes decisions’ will to a large extent be determined by input and past
operation, and would not be ‘impetuous’ in the way that a human could be.
This is what makes them so potentially valuable for tasks that require quick
decisions in potentially dangerous circumstances. However, in highly complex
situations, the breadth of human experience could give rise to better
judgements than those made by a system ‘programmed’ by a narrow range of
previous behaviour.
2 The Royal Academy of Engineering

---

Introduction
1.2 The ethical, legal and social implications of autonomous systems
The very nature of autonomous systems raises social, legal and ethical issues.
People tend to be more accepting of a technology if they can choose whether
or not to adopt it and have some control over its use. Public perception of risk
is very sensitive to the distinction between imposed risk and risks that
individuals feel they can control. However, autonomous systems are inherently
outside of users’ or the public’s control, and therefore it is possible that people’s
reactions to such technologies will be negative. Attitudes to autonomous
systems are likely to vary according to the area of application, and it is not
necessarily the case that acceptance of autonomous systems used in
dangerous military settings transfers to the use of autonomous systems in
every day life.
Autonomous systems require people to give up some of their own choice and
agency. Are people happy to do this in certain contexts, or is this inherently
objectionable? This question will be of particular relevance in relation to
autonomous road vehicles, as discussed below.
Some general ethical questions can be asked about the acceptability of
autonomous systems. One question is: should systems be designed that
cannot be shut down by a human operator? It might be thought that there is
always need for human intervention, but sometimes autonomous systems are
needed where humans might make bad choices as a result of panic –
especially in stressful situations – and therefore the human override would be
problematic. Human operators are not always right nor do they always have
the best intentions. Could autonomous systems be trusted more than human
operators in some situations?
All technologies are liable to failure, and autonomous systems will be no
exception (which is pertinent to the issue of whether autonomous systems
should ever be created without manual override). Dealing with the outcomes
of such failures will raise legal issues. If a person is killed by an autonomous
system – say, an unmanned vehicle – who is responsible for that death? Does
the law require that someone be held responsible? Legal and regulatory
models based on systems with human operators may not transfer well to the
governance of autonomous systems. In addition, the law currently
distinguishes between human operators and technical systems and requires a
human agent to be responsible for an automated or autonomous system.
However, technologies which are used to extend human capabilities or
compensate for cognitive or motor impairment may give rise to hybrid agents
– humans with autonomous prostheses which support their physical or
cognitive functioning. Without a legal framework for autonomous technologies,
there is a risk that such essentially human agents could not be held legally
responsible for their actions – so who should be responsible?
The concern with potential failures of autonomous systems could mean that
such technologies are held back until they are believed perfect. But is this too
strong a requirement? Autonomous systems tend to be compared with the
paradigm of perfection, and not with the systems that they will replace.
Reaction to failures of autonomous systems is somewhat different to reaction
to failures of manned systems. For example, a failure of a surgical robot that has
Autonomous Systems: Social, Legal and Ethical Issues 3

---

genuinely learnt from past use cannot be attributed to any one person. Is it
therefore the case that no one can be held responsible? This has been a barrier
to introducing adaptive surgical robots since in the health sector it is usually the
surgeon who has responsibility when things go wrong during a procedure. As
well as the concern about the legal framework covering autonomous systems,
without an understanding of responsibility for the functioning of autonomous
systems and a formal process for their certification, it may prove difficult to
secure insurance for autonomous systems. These issues must be addressed, as it
is almost inevitable that autonomous systems will experience failures, but if
those failures are less frequent than for human-operated systems it would be
counterproductive if they were a barrier to their development and use.
These kinds of social, legal and ethical questions need to be debated so that
autonomous systems can be developed in a way that is legally and socially
acceptable. Some general questions for debate are:
What are the consequences of giving up human choice and independence?
When is an autonomous system good enough? Must there be higher
standards for autonomous systems than those with human operators? How
much higher?
Can the law keep up with technology? How can the law be changed to
accommodate the range of autonomous systems?
What are the criteria for assigning responsibility, or degree of responsibility, for
the failure of an autonomous system and the harm it may cause?
Who will be responsible for certification of autonomous systems?
4 The Royal Academy of Engineering

---

Autonomous road vehicles
2. Autonomous road vehicles
2.1 Technologies – from GPS and car-to-car communication to centrally
controlled autonomous highways
There is a spectrum of technologies between driver-operated vehicles and
autonomous road vehicles. The following sets out the likely timeline for the
introduction of autonomous vehicles:
Driver information systems, including driver warning systems
These provide warnings and alerts to drivers and leave drivers with a choice as
to how to act on those warnings
?
Advanced driver assistance
Some automation of responses to warnings
Self-parking systems
?
Cooperative vehicle highway systems
Allows information to be relayed between vehicles as well as from operators to
vehicles, for instance when cars are too close together, to prevent collisions
?
Automated highway
The ultimate vision, with the highways and the vehicles within them being
managed and controlled as a system, instead of individual drivers making
decisions about speeds, stopping distances, routes and so on
2.2 Timescales and transformation – what can this technology offer?
When will it appear?
Automated and autonomous controls in vehicles can help to prevent human
error as a result of tiredness and lack of concentration. Road accidents are more
likely to be due to a driver not reacting appropriately or quickly enough than as
a result of vehicle failure. With automated vehicles that can select their own
routes, congestion can be controlled and driving made more efficient, leading
to reductions in pollution.
Robotic vehicles are technologically closer than people might think. The DARPA
grand challenge in the US, a competition to build a driverless car, has been
held three times since 2004. A number of vehicles passed the challenges of
navigating complex courses over long distances within a set period of time,
with no driver control. Such vehicles are still some way from the mainstream,
but there are vehicles available with intelligent sensor driven cruise control.
Vehicles with a greater degree of autonomy are technologically possible, but
their development is hampered by the lack of a legal framework for them – at
present, they would be impossible to insure.
While completely driverless cars may not be a significant proportion of the
traffic for some time, convoys of heavy trucks that are controlled like aircraft are
potentially nearer the horizon. So, while the ultimate vision of the fully
automated, driverless highway is still in the distance, the steps for getting there
are being addressed. A great deal of the technology needed is already
available, and under test. A model currently being investigated is that of long
lines of regulated vehicles over which the individual drivers have limited
control. The technology to achieve this is potentially close to market, however,
it is of limited value unless there wide adoption. Highway operators need to be
Autonomous Systems: Social, Legal and Ethical Issues 5

---

aware of when they are likely to reach mass market, so that systems can be
implemented to make the most of them. However, there is a chicken-and-egg
relationship between the uptake of such vehicles and the implementation of a
centralised highway-control system.
However, a top-down model of an automated highway, with ultimate control
of vehicles left to a centralised control system, may not be the only option. It
might be that the move to such a system is prompted by individual drivers
seeing the benefit of collision avoidance and driver warning systems, so that
the system emerges from the use of such devices in individuals’ cars. For
example, GPS systems in cars became popular very quickly and if driver
warning systems and similar devices are marketed in a similar way they could
be taken up very swiftly. The possibility of this is a reason why Government has
chosen not to legislate but to let the market dictate the growth of the
automated highway.
The latter approach would mean that the automated highway is achieved by
degrees. A first step could be technologies for allowing vehicles to
communicate with each other. Once such systems are in the majority of cars
then Government can take control and demand those vehicles comply with set
standards and can form a unified system.
But the order in which technologies will appear is not fixed. More important for
Government is the timescale in which they will appear and the point at which
changes need to be made in the way that the highway system is managed.
This point may not be so far away with the prevalence of GPS and the growth
of car-to-car communication. Therefore, the time to establish a common
approach to vehicle automation may be near. This would be by a process of
‘smoothing’ the system – perhaps starting with control of speed since there is a
good case for this.
2.3 Barriers: ethical, legal and social – administrative structures, enforced
compliance and conformity
It is worth considering the steps that have already been made toward
autonomous vehicles. Most vehicles are controlled to a significant extent by
engine management systems, drivers are reminded to close doors and put on
seat belts, and many people give over responsibility for their route to GPS.
Driving is a quite different experience now to what it was 20 or 30 years ago.
So, are we just separated from autonomous vehicles by a matter of degree, and
is the acceptance of such vehicles therefore likely?
There are a number of ethical, legal and social issues surrounding autonomous
vehicles. First, there is the potential for exclusion of those who do not want to
be part of the system. With an automated highway with centralised control,
some routes will simply be shut to those who do not buy in. Is that fair? Even
with co-operative control, drivers in vehicles without such systems are likely to
be treated quite differently, especially by insurers or in the case of an accident
where culpability is under question.
This raises the issue of the impact of failure of such a system. How will
accidents be dealt with, and how will insurance models have to be changed?
Will such a system be able to deal with different vehicles that do not all work
perfectly, or that host different versions of the technology? The technology
needs to be kept up to date for the system to work well, which is a challenge
when it takes up to 10 years to turn the fleet around.
6 The Royal Academy of Engineering

---

Autonomous road vehicles
Regulatory models will also struggle to fit this system, as can be seen from the
analogous case of automatic trains. Automated trains are most easy to
implement on metros with only one operator. They become more complex in
situations as in the UK where the infrastructure is owned by a different
company to the freight owners and there are a number of different operators.
This suggests that the situation for road transport with individually owned cars
will be even more challenging, as individual drivers are likely to be reluctant to
give up their autonomy.
Governments will have to play a bigger part in controlling the highways system
if autonomous vehicles are to be introduced. Currently car manufacturers play
an important role and Government does not yet feel ready to dictate the
market – the market is currently the dominant force. Government cannot take
ultimate control whilst it remains a relatively open system – how and when will
the change to a regulated system occur? Another obstacle is the fact that there
would have to be retrospective legislation for older cars, or the result will be
exclusion of their drivers from the roads.
In summary, the main issues relating to autonomous road vehicles are:
Will the development and uptake of autonomous vehicles marginalise road users in
older vehicles?
How can the autonomous highway be regulated? Who manages the road system,
and how can autonomous vehicles from a variety of manufacturers and with
individual owners be regulated?
How will the insurance industry deal with responsibility for failures and accidents
involving autonomous vehicles?
2.4 Recommended actions – for engagement and policy development
The area of autonomous road vehicles is particularly interesting because,
assuming a top-down model of highway control, it involves so many people at
once ceding control to an external operator. However, is this so different to the
case of passengers relinquishing control when they travel on trains or planes?
Will car drivers be reluctant to give away control in this way or will they see it as
a convenience? Would drivers have a different attitude to a centralised system
or a cooperative system that emerges out of the technologies that drivers
choose to have in their cars, so that each individual simply gives over control to
their own vehicle?
Public engagement on this issue seems to be needed in order to inform
government policy on this issue, especially before infrastructure is invested in.
The use of autonomous vehicles links up to and raises a lot of other issues
about road user charging, since such systems are likely to be put in place
alongside charging systems. This is an area where public feeling runs high and
hence there is value in engaging with the public to build a model that will be
considered acceptable.
Autonomous road vehicles, whether controlled centrally or cooperatively, are
also likely to generate a lot of data about where a car has been, how it was
driven and so on. Therefore, there are potential ethical issues surrounding the
Autonomous Systems: Social, Legal and Ethical Issues 7

---