Their thinking is that once AIs are cleverer than humans - which is an 80 per cent possibility in the next 100 years, according to FHI estimations - we could face a future in which the interests of a machine do not include human wellbeing.
Take the example of an AI whose instruction is to make as many paperclips as possible.
If it has the self-improvement capacity to follow this goal to its logical extent, it will harvest every iron atom on the planet as it seeks to maximise the number of paperclips in existence. The sensationalism-averse researchers of the FHI will sigh at the inclusion of this point, but in this scenario the countless atom irons in human bodies are fair game: this is a much more likely scenario than the luridly anthropomorphised Terminator stereotype.
With such scenarios being bandied about, the FHI has now advertised among scientists and programmers in the hope of finding three research fellows to create a Strategic Artificial Intelligence Research Centre. The trio will join the thinkers of the FHI, a small but growing offshoot of the University of Oxford which, since 2005, has made its main purpose the prediction and prevention of large?scale risks to human civilisation.
If we are right that [AI] is the most important area to be working in, then this would be an almost unique place in the world to be working, because you'd be working on the most important problem in the biggest group working on these issues.
If there were such a thing as a poster boy for research into the risk of computers surpassing humans, it would be the institute's founding director, Prof Nick Bostrom. The inscrutable Swedish professor of philosophy earlier this year addressed a UN committee on the risks posed by AI and other man-made threats, and has twice been named in Foreign Policy magazine's list of the Top 100 Global Thinkers.
"We've been scouring the world for hidden talent in this area," says Bostrom, a strong-jawed, grey-eyed man of 42 who is standing in the institute's tiny kitchen as he assembles his daily vegetable smoothie.
The FHI itself feels like the School of Athens taking place in an IT support office in an unlovely Oxford back street. It has 11 full-time researchers, experts in everything from AI to nanotechnology and population ethics - a team of big brains who are in the market for those three more specialists. "Our last three lunch conversations have been on issues ranging from the distribution of energy in the universe to totalitarianism and inverse reinforcement learning," says Dr Niel Bowerman, the institute's boyish, cheerful assistant director, whose ebullience belies a formidable CV that includes undertaking climate change research for the World Bank when he was still a Masters student.
The most important thing is brainpower and an ability to engage with questions, even when there is no clear predefined recipe or method for how you go about doing that.
While Bostrom tends to attract the limelight, behind every door there is an influential thinker of some stripe. Take, for example, Dr Toby Ord, the moral philosopher who attracted attention in 2009 when he began giving away everything he earned over pounds 20,000 - to statistically effective charities. His day job for the FHI is to investigate the "deep future" - what humanity might look like thousands or millions of years from today, and how the universe could sustain a vast and advanced vision of humanity.
Back to the kitchen, where Bostrom has left his cabbage, cauliflower, carrot and lime on the sideboard. "If we are right that [AI] is the most important area to be working in, then this would be an almost unique place in the world to be working, because you'd be working on the most important problem in the biggest group working on these issues."
However, the future may not hold only bad news: the researchers will also be examining the benefits of AI, or the reasons why we shouldn't cast everything carrying so much as a microchip into the flames yet.
"Today, benefits are evident in the technology of self-driving cars, or improvements in supply chains, which drive down prices," says Dr Bowerman. "In the future, these artificial intelligence algorithms have the potential to create breakthroughs driven by improved understandings in fields as diverse as genetics, the environment and macroeconomics."
So what will Prof Bostrom be looking for in the FHI's new hires? Applicants are being asked to submit a writing sample and research proposal and those shortlisted will be interviewed by an FHI panel led by the professor.
"The most important thing is brainpower and an ability to engage with questions, even when there is no clear predefined recipe or method for how you go about doing that.
"To some extent," he says, discouragingly, "you know it when you see it. These people are hard to find."
