Statisticians well-placed in job market as demand for data keeps growing.
David Scott doesn't take offence when he hears the words "lies, damned lies and statistics". That might be a bit surprising since number crunching is his job - he is associate professor of statistics at the University of Auckland. "It's just something we have to live with," says Scott. "But to our mind, statistics has got the runs on the board. It's proved itself to be useful in all sorts of areas."
Sport is a case in point. Quite apart from the endless variety of statistics recited by cricket commentators when there's no action on the pitch, sports betting agencies are a growth area for statisticians' jobs.
"Probability started from gambling and there's quite an interest these days in gambling applications," Scott says.
He has dabbled himself, developing a Super 14 rugby results prediction method based on form and home advantage.
"It runs pretty well compared to most of the pundits, yet it's a purely statistical thing with no sport input at all."
The statistics on statistics are heading upwards, Scott says. "Never in my life have I seen so many ads for statisticians as in the past little while."
Pharmaceutical companies, governments, market researchers and outfits involved in business analytics are all hiring number crunchers.
Analysis of customer data is a particular growth area, but a double-edged sword. On the one hand it can help organisations do a better job of giving customers what they want, but on the other it can lead to ever more intrusive forms of marketing.
"Amazon sends me messages from time to time saying I should buy a particular book or CD based on other purchases I've made, and I'm not always sure I want that," says Scott. He concedes that such statistically based services can be helpful - such as when he wanted to find a follow-up to a book he had read some time ago, which he did by seeing what other people who had bought the same book subsequently bought.
Yet there's a sinister aspect to the slicing and dicing of vast amounts of customer data. Scott says that while government statisticians go to lengths to ensure population data is anonymous, he's not sure the same can be said of commercial data hoarders such as Google.
"It's definitely an area of concern. I think there's potential for identifying individuals, drilling down to the individual level, based on intersecting enough things.
"If you start looking at people who are in a particular age group of a particular sex living in a particular area, who work in a particular industry, you start to get down to a particular person.
"I would think the information held by Google might be adequate to start identifying a person and to start identifying aspects of behaviour that they might not like to be revealed."
The "incredibly informative" data sets stored by credit card and mobile phone companies potentially have even more colourful stories to tell about their customers.
Computing power is at the heart of modern statistical analysis, and that's a relatively recent phenomenon. Scott's first calculation machine as a statistics student in the late 1960s was an electro-mechanical device with wheels that went round and round.
"The day we got calculators that could divide instantly, I thought it was wonderful." Scott's work today is completely bound up with computers, and he teaches and writes applications in R, an open-source statistical programming language developed 17 years ago by his colleagues Ross Ihaka and Robert Gentleman.
"R is used all around the world. It does everything. It's like Apple's model of apps for the iPhone. You have a basic program and people around the world developing apps for it, so the likelihood is that pretty much any new development in statistics will be programmed in R."
A key feature of R is that it is open source, so the program's code is accessible to anyone. In that way it supports the scientific method of reproducibility of results.
"If someone produces a result in a lab without showing how it's been done, how can anyone tell whether it's right? If you have something in an open-source language, the data is there, and the code's there.
"Everyone can check it, they can reproduce it. To my mind it's a fundamental aspect of research to have open-source software."
R's success in the world of statistics has seen it nominated in the education category of the New Zealand Open Source Awards, to be held in Wellington on November 9.
Scott says another notable achievement by colleagues was their presentation of a paper at a meeting of the Royal Statistical Society last week in London, marking World Statistics Day.
"That's a pretty signal honour. In the statistical world that's right up there."
Anthony Doesburg is an Auckland technology journalist
