Real-world Ethics for Self-Driving Cars

Presentation for the Virtual ICSE Poster Session

Cars over the last decades

Complex driving tasks are successively replaced by
advanced driving assistant systems.

With Self-Driving Cars into the Future

No Steering-Wheel or other primary driving controls,
the former driver becomes solely passenger.

The (Trolley-)Problem

The Trolley Problem is an unsolvable thought experiment and often presented as a key issue for self-driving cars.

Source: The New York Times; Illustration by Frank O’Connell

The question often posed in context of self-driving cars is:
whom will the Self-Driving Car kill, when it has to decide?.

Irrelevance of the Trolley Problem - Aspect 1

Currently, there are no fully autonomous cars and humans are still responsible for the decision making. The process of increasing automation goes via step-wise improved driving performance of a car based on machine learning.

Two recent studies that compare crash experiences of automated vs. conventional vehicles show that automated vehicles perform better.

The behaviour of the car is based on learning from experience and not on programming predefined hypothetical scenarios.

Irrelevance of the Trolley Problem - Aspect 2

The assumption made in the Trolley Problem about the deterministic nature of all the involved processes is wrong.

It means that all the objects have perfectly known positions from which only one perfectly calculable consequence will follow.

In the real world, we have a complex system under uncertainty that is not possible to predict exactly in real-time, and the way those phenomena are handled is by machine learning.

Learning from real-world driving experiences leads to improved data, which constantly improve the capabilities of the self-driving cars.

Irrelevance of the Trolley Problem - Aspect 3

The Moral Machine experiment, asking people all over the world about what they would do in a Trolley Problem situation, is often mentioned in connection to autonomous cars, in spite of the fact that it is about people and not about self-driving cars.

It is not a way to understand what cars should do, as humans are known to be the main cause of car accidents. According to The National Highway Traffic Safety Administration, 94% of serious crashes are due to human errors.

Irrelevance of the Trolley Problem - Conclusion

Trolley Problem scenarios explore differences among people in what they believe they would do in certain traffic situations.

But, the relevance for the development of Self-Driving Cars lies in the techno-social and ethical aspects of real-world engineering.

Our Approach

Instead of pointing towards the unsolvable Trolley Problem,
we use a hybrid interdisciplinary methodology to identify
relevant ethical and societal challenges for the development of self-driving cars.

Combining literature on value-based design and regulations, guidelines and standards, with the technical characteristics of present-day automated cars and their anticipated developments, allows to extract a list of most important topics.

Our Results

We identified ...

technical challenges and their manifestations grouped by requirements: safety, security, privacy, trust, transparency, reliability, responsibility, and accountability, quality assurance, and sustainability.

social challenges and their manifestations grouped by requirements: social challenges of disruptive technology, stakeholders/general public interests, and legislation, norms, policies, and standards.

The list and given examples can be used for discussion with experts, stakeholders and for further validation studies.

Developing Self-Driving Cars is an Iterative Process

Example of a Technical Challenge

Sensors and Recognition software, aim to detect objects (cars, buildings, etc.) and living beings (cyclists, pedestrians, etc.).

There are different stages of recognition:

In the last stage, if it were technically possible, there is a privacy problem.
This specific problem is for example solved in Europe by the GDPR.

Example of a Technical Challenge

By analyzing photos of pedestrians, for example,
a neural network can learn to identify a pedestrian.

How safe, reliable, or precise must a recognition be? In other words, how good is good enough?

Conclusion

It is of key importance to include ethical thinking and reasoning into the design and development process in every phase from requirements, till testing, maintenance, and evolution.

Architectural and design decisions should be taken through a process that includes ethics,
for example as a required non-functional requirement.

Transparency will be key to be able to observe and evaluate processes and software independently.

More Information and Resources

You can find a list of references and papers
on our project page: https://ethics.se