Borrowing from terminology coined by Stanford University emeritus professor James March, who specialised in organisational study, Kay says the processes of "exploiting" and "exploring" create a conflicting dynamic for innovation within organisations.
Exploitation involves people making things more efficient, removing the volatility from the production process "and all those things that basically lead to profit, which is very important, but for those things to be working really well, they're actually the opposite to what you need to innovate in many cases because that's a process of exploration where you have a high appetite for uncertainty and risk and you accept that things are ad hoc and you don't really know what's going to happen", Kay says.
He says the problem in large organisations is that the two things tend to get confused - approaching innovation by putting some "nice solid exploit" measures in place "but it's pretty difficult to have a return on investment outcome if no one has ever done this before".
"All the variables of that equation are not yet known so the problem is when you apply those measures to new innovation, and if the new idea has to come up through the same budget processes as, say, sacking 50 people or something like that, sacking 50 people is always going to win because it is a certain outcome."
Kay faced that tension in his job as head of strategic innovation at Westpac, but is diplomatic about what that first-hand experience felt like.
He says you see choices made and wonder why anyone would make those decisions, given that they will kill off any innovation.
"If you think about it from an 'explore' and 'exploit' point of view, actually the decisions are very rational."
Kay says there is no point getting upset that people aren't being innovative - they're just doing what they've been hired to do and if that has the unintended consequence of reducing innovation, it is not their fault, it's just a general lack of understanding of how innovation works.
The balance between exploit and explore is unique to each organisation, says Kay, but he suggests it's typically about 15 per cent explore, with a businesses such as Google being much higher on the explore scale. He bases that assessment on a study in which chief executives were questioned on what they characterised as resilience. The majority described it in purely reactive terms: being able to change and adapt.
About 10 per cent thought of resilience in the context of shaping the environment and the renewing of the organisation.
When he questioned the organisations on how much time they were spending on resilience - the things that are not business-as-usual - most came in around 15 per cent, but those "shapers" were up around 30 per cent or more.
"How you conceptualise your own resilience and how you conceptualise what innovation is will have a direct effect in terms of how much you explore and how much you exploit."
Kay is quick to stress that "shaping" isn't necessarily better than "changing and adapting" - "it's a horses for courses thing again".
He gives the example of a water company providing an infrastructure service and not necessarily competing in the same way as an internet service company. "Your purpose is to ensure the water stays healthy and you don't kill people.
"I don't really want them going out and reinventing what water is.
"I want them to be ensuring that the water is safe." He says there are many psychological reasons why people are not going to explore - everything from team dynamics to their personal biology - that make it harder to step away from their normal way of perceiving things.
There are ways to overcome this, through creating innovation units or reading up on innovation, but Kay says these tend not to be sustainable.
"But if the organisation has a very clear notion of what its goal is, what its finish line is, that becomes a point in the sand around which you can orient yourself.
"So when you're having to make a call between allocation of resources to 'explore' activity or to 'exploit' activity there's a line in the sand that says 'well, this is what cannot be compromised'."
Details
Robert Kay is in New Zealand as a keynote speaker at the Institute of Directors leadership conference, April 21-22 at The Langham, Auckland
More information:
iod.org.nz/conference
