This is the fifth edition in an eight-week series, made possible by MYOB, looking at how technology is changing the way New Zealand businesses operate.
Humans have always strived to become more knowledgeable by gathering information to learn from.
But collecting, storing and processing information used to be a manual and laborious process.
Until now.
Thanks to rapidly developing and essentially limitless computing capability, we can now digitise vast amounts of information, store it, transmit it, and process what's been gleaned.
From internet of things (IoT) sensor networks to close-circuit cameras in public, from web and even social media users, everything and everyone generates digitised information on a massive scale.
This is the era of Big Data and it has well and truly begun.
Greg Sharp, managing director of Auckland IT services firm Base 2 expects that over the next five years more data will be created than what was generated over the past 55 years.
"The amount of data that is being collected is doubling every six months," Sharp says.
We're only starting to dip our toes into ever-increasing big data pools, but Sharp believes that used correctly, they can give small business forecastability and help them make good decisions, which is most definitely a competitive advantage.
Big Data is often compared to oil because of the possibilities it creates. That comparison doesn't go far enough because unlike oil, data can be re-used any amount of times as long as it remains valid and relevant.
What's more, data from a certain application can be compared to and combined with information gathered from other sources, creating new insights that weren't possible to obtain before.
Large data sets allow artificial intelligence to become more accurate and human-like than in the past.
For instance, the Elon Musk-backed OpenAI scientists trained the GPT-2 model with a dataset that was 15 times the size of that used for earlier, state-of-the-art AI.
Thanks to the bigger and broader dataset training, GPT-2 is considered to be so realistic (and potentially dangerous) that the researchers have only released parts of it as open source.
"The ability to create future trends and predictions is on a hockey-stick curve," Sharp says.
Knowledge is power, and analyst firm Forrester estimated in 2017 that companies armed with artificial intelligence, big data and IoT will grab a staggering US$1.2 trillion from their less-informed peers by 2020.
There are many ways to obtain the data that produces knowledge and insights.
Wellington data analytics company DOT Loves Data sifts through large amounts of social media posts to help their clients learn about what makes customers tick.
Cloud-based veterinary practice management provider ezyVet is geocoding customer data to find clinical and financial trends by geographic region. The company believes it has the biggest repository of veterinary medical data of any cloud provider in the world now.
Modern farming requires strict adherence to resource consents and permits for water, effluents and fertiliser use.
Big Data can help here too. Wellington-based Regen Technology uses specialised automated data collection hardware to measure soil conditions, and collects 50 million markers annually from these.
Large amounts of past and present digitised information on environmental factors collected and processed with Regen Technology helps farmers accurately understand when and how much to irrigate and fertilise. This takes the guesswork out of farming, and Regen promises that the data-driven approach will return $5 for each dollar spent, being more efficient and economical as well as better for the environment.
Similarly, Base 2 works with a milk processing plant that collects large amounts of data from dairy farms.
The data collected includes the type of grass cows feed on, what temperature the milk was when collected, how it was treated and looked after before going into the tanker and tracked throughout the dairy plant until it becomes a ready product.
That Big Data scenario allows the milk processing plant to understand the quality of the final product, meaning they can charge a premium for what is sold to exacting international customers, Base 2 says.
Closer to home, Base 2's Sharp points to networked smart fridges that are able to monitor how much milk and cheese is left, and can automatically place orders with Pak n Save and Countdown when needed.
Before jumping into the Big Data lake, it's worth noting the dangers of indiscriminately collecting information, especially when it involves people.
Privacy watchdogs around the world expect Big Data to respect people's right to confidentiality and if they wish, to be forgotten.
Furthermore, data that could be construed as sensitive in any way must be stored securely, or deleted after use. It is valuable information that should be protected from unauthorised access and potential misuse.
Judging by how many wideopen databases security researchers find with ease on the internet, this often doesn't happen and it can lead to massive privacy breach fines for their owners.
Knowledge is power, and in the wrong hands even simple things like knowing where large groups of people are during the day could carry the potential for misuse.
However, there's no doubt that Big Data will benefit business and society as a whole, and create new opportunities that previously wouldn't have been possible. To fully reap the benefits of Big Data, people must be able to trust and feel safe with the technology collecting information on them.
Signs are that the trust is eroding, with Sharp admitting that he's lately become wary of the digital footprint he leaves when using the internet - knowing that data is being collected and never deleted.
"I'm conscious of that so I probably do a little less than I used to," Sharp says.
Awareness around how Big Data can be misused has lead to strict new laws such as the European Union's General Data Protection Regulation appearing. These seek to rebalance the privacy equation in favour of those most affected by information collection, namely individuals like you and I.
Ignoring the GDPR will be costly for data collectors and processors, as EU watchdogs have already started handing out hefty penalties. Recently, Google copped a €50 million fine for not properly obtaining users' consent for data collection. Other companies are facing billions in GDPR sanctions for potential privacy violations as well.
The tide is turning, even Facebook founder Mark Zuckerberg now admits that the social network hasn't done well as a steward of the Big Data gleaned from hundreds of millions of users and will try to do better to keep their information confidential.
This might be mission impossible for Facebook considering users and the data they generate are the revenue earners for the social network.
But it does point to a need for clear ethical guidelines for data collection and processing. Local Big Data shops would do well to bear that in mind when they look to the immediate future.
Midday Wednesday, the Herald will run a live panel in which experts will discuss how this technology is changing small business in New Zealand. Tune in to participate in the live chat.
Stats NZ has revealed its latest Retail Trade Survey for the September quarter.