Every challenge represents a new set of opportunities, big challenges bring forth big opportunities and big data is a big challenge. It challenges the way we have been viewing, storing, analyzing and interpreting our data. The term “Big Data” is kind of a misnomer since it implies that the only issue with today’s data is its sheer size. There are lots more to Big Data than just its size. Big Data applies to the set of data that can’t be processed or gained insight into using the traditional tools and techniques. The amount of data in our world has been exploding. We capture trillions of bytes of information about customers, suppliers, and operations, and millions of networked sensors are being embedded in the physical world in devices such as mobile phones and automobiles, sensing, creating, and communicating data. Individuals with smartphones and on social network sites continue to fuel exponential growth. Big data—large pools of data that can be captured, communicated, aggregated, stored, and analyzed—is now part of every sector and function of the global economy. Let’s look at some of the facts how the data is proliferating – There are 5 billion phones in world right now that are generating data by seconds, just the Facebook alone is generating 30 billion pieces of content every month, the Library of US congress has around 235 terabytes of data, twitter alone generates around 7 terabytes of data every month, The Large Hadron Collider experiments represent about 150 million sensors delivering data 40 million times per second, Wal-Mart handles more than 1 million customer transactions every hour. These are just the tip of the iceberg and it demonstrates how the data is becoming massive by seconds. Digital data is everywhere – in every sector, in every company, in every economy. Today we store everything – Environmental data, financial data, medical data, and surveillance data and the list goes on and on. According to an MGI estimate, in 2010 the corporations stored more than 7 Exabyte’s of data on their hard drives and the individual consumers stored more than 6 Exabyte’s of data. Google’s executive chairman Eric Schmidt brings it to a point: “From the dawn of civilization until 2003, humankind generated five Exabyte’s of data. Now we produce five Exabyte’s every two days…and the pace is accelerating.”
The possibilities and opportunities presented by the proliferation of big data are constantly evolving, driven by innovation in technologies, platforms and analytical capabilities. The Big Data can be defined by three of its main characteristics – Volume, Variety and Velocity of data. Till now we have just discussed about the size or the volume of data. With the proliferation of sensors, smart devices, social collaboration tools, enterprises today are faced with not only the traditional relational data but also data in a raw form, semi structured or unstructured form. The sheer variety of data that are getting captured today presents a unique set of problems to our traditional tools and techniques in storing and analyzing them. As the volume and variety of data that gets captured or stored today has changed so has the sheer velocity at which this data is getting generated. With the emergence of RFID sensors everywhere and other information streams, the data is getting generated at such a pace that has made impossible for our traditional tools to handle. For many applications, the speed of data creation is even more important than the volume. Real-time or nearly real-time information makes it possible for a company to be much more agile than its competitors.
The opportunities presented by Big Data are huge. According to various researches, there is a potential value addition to the tune of 300 billion dollars just in the US healthcare, there is a potential to generate 250 million pounds in the public sectors in Europe, there is a potential to generate 600 billion dollars in consumer sectors just if we start analyzing the location centric data, in coming 5-7 years the big data is going to generate 1.5 to 1.8 million jobs just in US alone in the field of deep data analytics, big data mangers and engineers. The opportunities presented by the big data is immense if we just keep pace by evolving our technologies and tools to keep with the volume and velocity with which we are generating the data.