The trend has spawned a new occupation: data scientist. The first summit for data scientists was held in Las Vegas in May and if you're thinking of retraining, or looking for a new career, you could do a lot worse. A recent report from the McKinsey Global Institute (MGI) forecasts that by 2018 the United States alone could face a shortage of up to 190,000 data scientists and up to 1.5 million data-savvy managers and analysts.
MGI describes "big data" as the next frontier for innovation, competition and productivity, and suggests huge potential savings in sectors such as electronics, finance and insurance, and government.
Not that there is anything particularly new about big data. Newspapers, after all, have been capturing a plethora of information and making sense of it for their readers for more than a century. And business intelligence specialists have been around for a while.
But the falling cost of data storage, and the development of some very clever software that enables huge collections of data to be condensed into much more manageable forms, means the exponential growth of big data has now reached truly astonishing levels.
Research company IDC believes that between now and 2020, the amount of information in the digital universe will grow by an inconceivable 35 trillion gigabytes as all major forms of media - voice, TV, radio and print - complete the journey from analogue to digital. It has also been estimated that this year alone, the amount of digital information created and replicated surpassed 1.8 trillion gigabytes - growing by a factor of nine in just five years. That's nearly as many bits of information as there are stars in the physical universe.
As always with The Next Big Thing, big data's promoters are probably over-hyping its potential, and under-estimating the practical realities of public and private inertia. But it's still bad news if you're paranoid, or just value your privacy.
Until now, access to the information recorded by computers, security cameras, ATMs, ID cards and cellphones has mostly been limited to law enforcement. But as smartphones and tablets grow in popularity, and GPS chips and radio-frequency identification tags become embedded in more and more everyday objects (cars, for example), some people believe we are reaching a tipping point where more data is being created by machines than humans.
The upshot: we are rapidly approaching the point where almost everything we do is capable of being tracked and analysed.
So should we be worried that Big Data could turn into Big Brother? Andy Lark acknowledges it's a concern.
After several years as head of global marketing for Dell's public and large enterprise group, Lark has recently returned closer to home as head of marketing for Commonwealth Bank of Australia, owner of ASB.
"These issues of identity and privacy are going to be major, major issues going forward," he agrees.
Although banks already have large business intelligence teams, they are so far only scratching the surface of big data, says Lark. He expects that to change as data mining software becomes increasingly sophisticated.
"We're putting huge amounts into it. We're looking at all kinds of next generation technologies. There's huge amounts of innovation in this space."
Banks can already monitor customers' transactions in real time, enabling them to detect potential fraud and other risks, he notes. If you buy petrol, sunglasses and jeans in a single day, for example, you could get a call from the bank, says Lark - those three items happen to be the most common purchases by credit card fraudsters.
"There's this fine line you've got to walk between the group of customers who love the fact that you reach out proactively based on data, and the other group who find it slightly creepy."
Banks are also keen to branch out into other digital storage services. Lark confirms CBA sees "lots and lots of tangential opportunities" it could potentially pursue. But in the meantime, banks already know a lot more about their customers than they probably realise, he notes.
"We look after your money and your wealth, and we know everything you bought and we know where your dollars have gone. We really do know what is happening in your life at any given point in time and that's an important, precious relationship. So as much as this is about the growth of big datasets, it's also about being respectful with the dataset we do have, because it's sensitive."
Meanwhile, companies such as Facebook and LinkedIn are exploiting the big data revolution for all they are worth - which happens to be quite a lot. Between them the two social networks already have nearly 1 billion members.
At this year's South By Southwest festival, and at the Web 2.0 Expo, LinkedIn founder Reid Hoffman declared that Web 3.0 would be all about data and its analysis. And its chief data scientist, Peter Skomoroch, was recently quoted as saying that all companies should be thinking about leveraging data at all times.
"When you have to make a business decision, you should back it up with data. The data that is flowing through your systems can be used to help your users," he suggested.
Some governments are embracing a trend towards open data. Many governments, including those of Britain, America and Spain, publish a vast array of documents online for the public to scrutinise. New Zealand has its own site, where featured datasets include the expense claims of dozens of public sector chief executives.
Lark is convinced corporate expenses claims can be controlled in the same way. "You might think twice about having the merlot or the pinot at lunch, if you knew everyone was going to see that choice," he suggests.
Other uses of big data include interactive 3D maps that track crime trends in specific neighbourhoods, "smart grid" software that enables energy companies to better manage loads on their networks, the tracking of vehicles to allow better design of urban environments, and networks that harness big data to prevent global pandemics before they start.
Another ambitious project is the Broad Institute, a collaboration between MIT and Harvard University, which is trying to identify the cellular mutations that cause cancer, and establish the molecular basis of viruses, bacteria and other pathogens that cause infectious disease.
The institute aims to capitalise on the extraordinary achievements of the Human Genome Project, which in 2003 completed a 13-year mission to sequence the 25,000 or so genes and the 3 billion chemical base pairs that make up human DNA. That project proved we still have much to learn, and further genome research is currently one of the biggest challenges for big data.
The Broad Institute's data footprint grew to eight petabytes in the past year (that's equivalent to 20 billion documents, or more than 2000 times the amount of information digested by IBM's Watson), and is doubling every five months.
On a much more basic level, online shopping recommendations, the monitoring of embedded sensors in bridges and buildings to detect real-time stresses, and basing car insurance premiums on where people drive, are examples of how big data is already affecting our lives.
