How to prevent robots cooking the cat

It is easy to imagine a scenario in which a care robot assesses a pet cat as a nutritious meal. Photo / 123RF

How can we ensure that computers do what we want them to do when they are increasingly doing it for themselves?

That may sound like an abstract philosophical question, but it is also an urgent practical challenge, according to Stuart Russell, professor of computer science at the University of California, Berkeley, and one of the world's leading thinkers on artificial intelligence.

It is all too easy to imagine scenarios in which increasingly powerful autonomous computer systems cause terrible real-world damage, either through thoughtless misuse or deliberate abuse, he says. Suppose, for example, in the not-too-distant future that a care robot is looking after your children. You are running late and ask the robot to prepare a meal. The robot opens the fridge, finds no food, calculates the nutritional value of your cat and serves up a feline fricassee.

Or take a more horrifying example of abuse that is technologically possible today. A terrorist group launches a swarm of bomb-carrying drones in a city and uses image recognition technology to kill everyone in a police uniform.

As Prof Russell argues in his latest book, Human Compatible, we need better ways of controlling what computers do to prevent them acting in anti-human ways, by default or by design. Although it may be many years, if not decades, away, we must also start thinking seriously about what happens if we ever achieve superhuman AI.

Getting that ultimate control problem right could usher in a golden age of abundance. Getting it wrong could result in humanity's extinction. Prof Russell fears it may take a Chernobyl-scale tragedy in AI to alert us to the vital importance of ensuring control.
For the moment, the professor is something of an outlier in the AI community in sounding such alarms. Although he co-wrote a previous textbook on AI that is used by most universities around the world, Prof Russell is critical of what he calls the standard model of AI and the "denialism" of many in the industry.

He is now working with some other researchers and tech companies to design what he calls "provably beneficial" AI systems. The idea is to create more deferential programs that would periodically check in with humans to ensure that they are doing the right thing, rather than blindly following one fixed objective. The goal should be constantly to optimise human preferences.

Stating the challenge so simply is to open a wonder-box of complexity, though. Since the dawn of thought, philosophers have been disputing what makes us happy. Who decides those preferences? How do we weigh the views of the old against the young — and the unborn? What happens if, like King Midas, we make a dumb choice and then change our minds?

Experiments with "wireheading" — artificial brain stimulation — also suggest our preferences can be manipulated. In the wrong conditions, wired-up humans would probably behave just like lab rats do, maniacally pressing a dopamine-releasing pleasure button until we die.

Of course, the challenges of optimising human preferences are not unique to AI. That lofty aim is, in theory, what drives democratic governments every day. It is also supposed to explain the workings of the market. But Prof Russell argues that shareholder capitalism is a good example of a system with a bad objective function, or target. We have instructed corporations to maximise shareholder returns, ignoring other societal preferences, such as preservation of the environment.

"Think of the fossil fuel industry as an AI system. We set it up with this fixed objective of maximising its profit and it has won," Prof Russell says. "It is a machine that has human components but functions as a machine. It has outwitted the rest of the human race because it is a sort of a superintelligent entity."

Just as environmental economists model how to price in negative externalities — by imposing carbon taxes, for example — so computer scientists will have to design more flexible AI programs that reflect a range of societal preferences.

To that end, Prof Russell is launching a computer science course at Berkeley with input from the philosophy and economics departments. The new Centre for Artificial Intelligence at University College London is also adopting a cross-disciplinary approach.
In writing the instruction manual for our future, software engineers will have to create a constant and unbreakable dialogue between machines and humans. AI is too important to be left solely to the computer scientists.

Written by: John Thornhill

© Financial Times