It's a race between the growing power of the technology and the growing wisdom we need to manage it.
It begins to convert all kinds of ordinary materials into paper clips. Eventually it decides to turn everything on Earth - including the human race - into paper clips.
"You could have a superintelligence whose only goal is to make as many paper clips as possible, and you get this bubble of paper clips spreading through the universe," Bostrom calmly told an audience in Santa Fe, New Mexico, last year.
Bostrom's underlying concerns about machine intelligence, unintended consequences and potentially malevolent computers have gone mainstream.
You can't attend a technology conference these days without someone bringing up the AI anxiety.
People will tell you that even Stephen Hawking is worried about it. And Bill Gates. And that Elon Musk gave US$10 million for research on how to keep machine intelligence under control. All that is true.
How this came about is as much a story about media relations as it is about technological change.
The machines are not on the verge of taking over. This is a topic rife with speculation and perhaps a whiff of hysteria.
But the discussion reflects a broader truth: we live in an age in which machine intelligence has become a part of daily life.
Computers fly planes and soon will drive cars. Computer algorithms anticipate our needs and decide which advertisements to show us.
Machines create news stories without human intervention. Machines can recognise your face in a crowd.
New technologies - including genetic engineering and nanotechnology - are cascading upon one another and converging.
We don't know how this will play out. But some of the most serious thinkers on Earth worry about potential hazards - and wonder whether we remain fully in control of our inventions.
Many people talk about something called "the Singularity". The idea dates to at least 1965, when British mathematician and code-breaker I.J.
Good wrote, "An ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind." In 1993, science fiction author Vernor Vinge used the term "the Singularity" to describe such a moment.
Inventor and writer Ray Kurzweil ran with the idea, cranking out a series of books predicting the age of intelligent, spiritual machines.
Kurzweil, now a director of engineering at Google, embraces such a future; he is perhaps the most famous of the techno-utopians, believing technological progress will culminate in a merger of human and machine intelligence. We will all become "transhuman".
Whether any of this will happen is the subject of robust debate. Bostrom supports the research but worries that sufficient safeguards are not in place.
Imagine, Bostrom says, that human engineers programmed machines to never harm humans. But the machines might decide that the best way to obey the harm-no-humans command would be to prevent any humans from ever being born.
He isn't saying this will happen. These are thought experiments. His big-picture idea is that we've seen astonishing changes in the human population and economic prosperity.
In Bostrom's view, our modern existence is an anomaly - one created largely by technology. Our tools have suddenly overwhelmed the restrictions of nature.
We're in charge now, or seem to be for the moment.
But what if the technology bites back?
There is a second Swede in this story, and even more than Bostrom, he's the person driving the conversation.
His name is Max Tegmark. He's a charismatic 48-year-old professor in the physics.
I actually think it would be a huge tragedy if machine superintelligence were never developed. Nick Bostrom department at the Massachusetts Institute of Technology. He's also a founder of something called the Future of Life Institute, which has been doling out Elon Musk's money for research on making AI safer.
"The future is ours to shape. I feel we are in a race that we need to win," says Tegmark. "It's a race between the growing power of the technology and the growing wisdom we need to manage it. Right now, almost all the resources tend to go into growing the power of the tech."
In 2014, 33 people gathered in Tegmark's home to discuss existential threats from technology.
They decided to form the Future of Life Institute. Tegmark persuaded numerous luminaries in the worlds of science, technology and entertainment to add their names to the cause. Skype founder Jaan Tallinn signed on as a co-founder.
Actors Morgan Freeman and Alan Alda joined the governing board.
Tegmark put together an article about the potential dangers of machine intelligence, lining up three illustrious co-authors: Nobel laureate physicist Frank Wilczek, artificial intelligence researcher Stuart Russell and the biggest name in science, Stephen Hawking.
The piece, which ran on the Huffington Post and in the Independent in Britain, included a tutorial on the idea of the Singularity and a dismayed conclusion that experts weren't taking the threat of runaway AI seriously. Artificial Intelligence, the authors wrote, is "potentially the best or worst thing ever to happen to humanity."
"Stephen Hawking Says AI Could Be Our 'Worst Mistake In History,'" reported one online science news site.
And CNBC declared: "Artificial intelligence could end mankind: Hawking." So that got everyone's attention.
Tegmark's next move was to organise an off-the-record conference of big thinkers to discuss AI. About 70 scientists and academics, led by Tegmark, convened in Puerto Rico to discuss the existential threat of machine intelligence.
Musk, the founder of Tesla and SpaceX, joined the group in Puerto Rico. On the final night of the conference, he pledged US$10 million for research on lowering the threat from AI.
"With artificial intelligence, we are summoning the demon," Musk had said earlier, a line that sent Twitter into a tizzy.
In the months that followed, 300 teams of researchers sent proposals for ways to lower the AI threat. Tegmark says the institute has awarded 37 grants worth US$7 million.
But it's time for a reality check. More than half a century of research on artificial intelligence has yet to produce anything resembling a conscious, wilful machine. We still control this technology. We can unplug it.
Close to Tegmark's office is MIT's Computer Science and Artificial Intelligence Laboratory, where robots are aplenty. Daniela Rus, the director, is an inventor who just nabbed US$25 million in funding from Toyota to develop a car that will never be involved in a collision.
Is she worried about the Singularity?
"It rarely comes up," Rus says. "It's just not something I think about." With a few exceptions, most full-time AI researchers think the Bostrom-Tegmark fears are premature.
A widely repeated observation is that this is like worrying about overpopulation on Mars.
Rus points out that robots are better than humans at crunching numbers and lifting heavy loads, but humans are still better at fine, agile motions, not to mention creative, abstract thinking.
"The progress has not been as steady as people say, and the machine skills are really far from being ready to match our skills," she says. "There are tasks that are very easy for humans - clearing your dinner table, loading the dishwasher, cleaning up your house - that are surprisingly difficult for machines."
Rus makes a point about self-driving cars: they can't drive just anywhere. They need precise maps and relatively predictable situations.
Self-driving cars struggle with heavy traffic, she says, and even rain and snow are a problem. So imagine trying to understand hand gestures from road crews and other drivers.
The future is implacably murky when it comes to technology; the smartest people on the planet fail to see what's coming. For example, many of the great sages of the modern era didn't anticipate that computers would get smaller rather than bigger.
Anyone looking for something to worry about in the near future might want to consider the opposite of super-intelligence: super stupidity.
In our increasingly technological society, we rely on complex systems that are vulnerable to failure in complex and unpredictable ways.
Deepwater oil wells can blow out and take months to be resealed. Nuclear power reactors can melt down. Rockets can explode. How might intelligent machines fail - and how catastrophic might those failures be?
Often there is no one person who understands exactly how these systems work or are operating at any given moment. Things can go wrong quickly and disastrously.
Such was the case with the "flash crash" in the US stock market in 2010, when, in part because of automated, ultra-fast trading programs, the Dow Jones industrial average dropped almost 1000 points within minutes before rebounding.
"What we're doing every day today is producing super stupid entities that make mistakes," argues Boris Katz, another artificial intelligence researcher at MIT.
"Machines are dangerous because we are giving them too much power, and we give them power to act in response to sensory input. But these rules are not fully thought through, and then sometimes the machine will act in the wrong way," he says.
"But not because it wants to kill you."
A living legend of the AI field and the MIT faculty is Marvin Minsky, 88, who helped found the field in the 1950s. It was his generation that put us on this road to the age of smart machines.
Minsky, granting an interview in the living room of his home a few miles from campus, flashes an impish smile when asked about the dangers of intelligent machines.
"I suppose you could write a book about how they'll save us," he says. "It just depends upon what dangers appear."
The AI debate is likely to remain tangled in uncertainties and speculation. In theory, as that original Huffington Post article stated, there is no theoretical limit to machine intelligence - "no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains." But the academic and scientific establishment is not convinced that AI is an imminent threat.
Tegmark and his Future of Life allies decided this (northern) summer to take on a related but more urgent issue: the threat of autonomous weaponised machines.
Tegmark teamed with Stuart Russell, an artificial intelligence researcher, on an open letter calling for a ban on such weapons.
Once again, they got Hawking to sign it, along with Musk, Bostrom, and about 14,000 other scientists and engineers. Last July, they formally presented the letter at an AI conference in Buenos Aires.
Russell says it took him five minutes of internet searching to figure out how a very small robot - a microbot - could use a shaped charge to "blow holes in people's heads." A microrifle, he says, could be used to "shoot their eyes out."
"You'd have large delivery ships that would dump millions of small flying vehicles, probably even insect-sized, the smallest you could get away with, and still kill a human being," Russell says.
After Bostrom's lecture in Santa Fe, he went to a book-signing event. His book is a meticulously reasoned, rather dense tract titled, Superintelligence: Paths, Dangers, Strategies. Bostrom held forth amid a small cluster of party guests.
Then he sat down for an hour-long interview. Reserved, intensely focused on his ideas, the 42-year-old Bostrom seemed apprehensive about whether his ideas could be fully grasped by someone who is not an academic philosopher.
Asked if there was something he now wishes he had done differently with his book, he says he should have been clear that he supports the creation of superintelligence. Unsurprisingly, most readers missed that key point.
"I actually think it would be a huge tragedy if machine superintelligence were never developed," he says. "That would be a failure mode for our Earth-originating intelligent civilisation."
In his view, we have a chance to go galactic - or even intergalactic - with our intelligence.
Bostrom, like Tegmark, is keenly aware that human intelligence occupies a minuscule space in the grand scheme of things.
The Earth is a small rock orbiting an ordinary star on one of the spiral arms of a galaxy with hundreds of billions of stars. And there are at least tens of billions of galaxies twirl across the known universe.
Artificial intelligence, Bostrom says, "is the technology that unlocks this much larger space of possibilities, of capabilities, that enables unlimited space colonisation, that enables uploading of human minds into computers, that enables intergalactic civilisations with planetary-size minds living for billions of years."
There's a bizarre wrinkle in Bostrom's thinking. He thinks a superior civilisation would possess essentially infinite computing power.
These superintelligent machines could do almost anything, including create simulated universes that include programs that precisely mimic human consciousness, replete with memories of a person's history - even though all this would be entirely manufactured by software, with no real-world, physical manifestation.
Bostrom goes so far as to say that unless we rule out the possibility that a machine could create a simulation of a human existence, we should assume that it is overwhelmingly likely that we are living in such a simulation.
"I'm not sure that I'm not already in a machine," he says calmly.
