In 2015 there will be the equivalent of eight Great Walls of China full of data; emerging from structured corporate information systems, from social networks, emails, GPS coordinates from mobile phones, eftpos transactions, sensors attached to everything from cows (true!) to electricity meters.

By next year the combined data storage of the planet will be almost eight zettabytes. To get a feel for how big that is—if the takeaway coffee cup on your desk is equivalent to a gigabyte of information, a zettabyte is equivalent to the Great Wall of China according to communications company Cisco.

By 2020 there will be more things connected to the internet than there are people on the planet. Best estimates suggest that as many as 50 billion devices will be connected by then. It's a lot of data waiting to be explored and exploited.

Wearable technology such as Google Glass, smart-watches and digital fitness bands has the potential to add an order of magnitude to the range of information available for analysis. How much would your health insurer like to know your heart rate before it priced your premium?

Commonwealth Bank Managing Director Ian Narev recently described big data as having the power to completely disrupt Australia's financial services sector.

It's conceivable that banks could harness big data techniques to monitor their customers' accounts, track their spending and saving patterns, keep an eye on social media posts to know when people are looking to move out of a unit and into a house, and send a text spruiking a special mortgage offer just as the couple is walking toward an auction for a three bedroom home near good schools. Clearly the banks that do this well will have a huge competitive advantage.

What is driving big data are the torrents of information being produced and consumed each day.

This is big data—characterised not only by its volume, but by the variety of data involved and the velocity at which it is created. And it has big business very excited.

If it all sounds too futuristic then have a look at www.peoplelikeu.com.au, which allows you to compare your spending behaviour with that of other people like you living all around Australia. Established by the NAB's online-only bank UBank, it's a fairly simple example of the sort of profiling that is now possible thanks to big data analysis.

It's not just the banks either—ever wondered how internet advertisers seem to know when you've been searching for a hotel room in Berlin, and suddenly all the advertisements on websites you visit suggest different Berlin hotels? In that case your search history is being analysed, and as you click on a website the advertisement space on the website is being auctioned off in real time to the highest bidder who wants to get their product or service in front of you right now.

Clearly there are huge advantages from being able to understand the patterns emerging in data and act on them in real time. Banks can develop more targetted products, they can reduce risk by predicting who is about to default on their mortgage allowing them to intervene and offer a new payment plan. Retailers can support shoppers with timely offers delivered to their mobile phone as they walk down the supermarket aisle based on past preference, and they can streamline their supply chains by being able to predict customer behaviour based on weather patterns. Mining companies can attach sensors to their equipment to detect patterns that might predict a looming problem, allowing preventive maintenance crews to be dispatched to avoid downtime.

The trick for organisations that want to exploit big data is to extract value for themselves and add value for the customer without being inappropriate. A US retailer attracted criticism for using predictive analytics to spot when a customer was pregnant through their buying patterns, to send out discount coupons for baby products. All well and good until a package of coupons landed for a teenage girl who hadn't told her parents of her predicament.

Ian Narev and the Commonwealth Bank plan to avoid being inappropriate by allowing customers to opt-in to big data initiatives, and any organisation planning to use personal data will in any case have to carefully navigate the new Australian Privacy Principles, which come into force this month.

Corporations and governments (which are equally keen on big data—the Tax Office in particular) also need to tread the fine line between big data analysis and surveillance.

And it is a fine line. In February this year Motorola demonstrated what it calls the police car of the future—able to perform what it described as "predictive policing". Elements of the technology are already being trialed by police forces in Victoria and the Northern Territory. The company's vision is that as the car patrols the streets, a camera would automatically scan and capture car registration plates, use facial recognition systems to identify passers-by, and provide that insight to the police in the car and to a central command centre to direct resources where they are most needed.

There is no doubt that big data, handled carefully, promises significant benefits to consumers, citizens, corporations and government—but it needs careful consideration and proper scrutiny. Are you prepared to opt in to the big data era?

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.