April 18, 2016
Information Insight

Big Data and the Canary in the Coal Mine

I have never been a big follower of fads. While I’m from Indiana, I might as well be from Missouri. True to the “Show Me State,” you have to show me the evidence before I’m going to believe it. This approach has saved me from following many of the greatest passing fads: Crocs, the Atkins Diet, myspace, Flash Mobs, Beanie Babies, the Macarena, and Koosh Balls. Therefore, I hope you appreciate my perspective and evaluation criteria when I tell you this: Big Data is not a passing fad.

The biggest challenge organizations face with regards to Big Data is that it is not well understood. When most people hear of Big Data, they think of Facebook, Twitter, and Yelp. They think of mining unstructured social media data in order to gauge customer sentiment. While these things are an aspect of Big Data, using a maritime metaphor, these would only be the minor swells that precede the tsunami. Big Data is evolving beyond these modest roots. The concept of Big Data goes beyond the use of unstructured data. It also refers to the rapid growth of data and how that data is then used to improve processes, create new opportunities, manage issues, and mitigate risks. Big Data is not new. Prior iterations of the evolution of data included: mainframe computing, client server applications, and the Internet. Each of these exponentially increased the amount of data in existence along with the methods in which it could be captured, stored, and utilized.

“Big Data is like teen sex. Everybody is talking about it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” – Dan Ariely, Center for Advanced Hindsight at Duke University

The majority of the data that exists today was manually produced by humans. News stories, research papers, blogs, Facebook posts, and tweets were all manually written by a human being. Even the majority of the supposedly automated transactional data that exists today had heavy human involvement. A significant proportion of the data was entered manually. Point of sale data captured at a cash register or stock trades entered through a brokerage website are good examples. However, in the very near future the majority of data in existence will not be created by humans. This evolution will be highly disruptive, and the exponential growth of data will be like nothing we’ve ever seen before due to the adaptation of some new technologies. This is the coming Big Data tsunami.

The Internet of Things. If you haven’t heard this term before, I strongly suggest you head over to Google and do some reading. To summarize, the Internet of Things is the unique identification and connectivity of all the objects in the world. Technologies such as radio-frequency identification (RFID), near field communication, and advanced sensors have enabled what was first discussed as science fiction in the early 1990s to become reality. One of the simpler examples of the Internet of Things is smart warehouses. Smart warehouses contain products tagged with RFID tags. The inventory and location of all products is always known because of sensors that locate the objects by RFID tag. Additional products can be automatically ordered to replenish inventory, all without any human involvement. This innovation has led to discussions around smart homes where refrigerators could automatically build grocery lists and tell you what recipes you could cook for dinner based on having all the necessary ingredients. The ability to create the data and exchange it between tens of billions of new devices spanning the Internet, all without direct human involvement, is the tsunami. Don’t believe in Big Data yet? Consider General Electric’s new jet engines that analyze 5,000 data points every second. A single six-hour transcontinental flight will produce 240 terabytes of data that detail every aspect of the jet engine’s performance and health. Then consider that there are approximately 20,000 commercial aircrafts in operation today (CIO Magazine, June 13, 2013). Wake me up when you’ve completed your analysis on all flights in 2013.

The arrival of these initial swells to shore is the canary-in-the-coal-mine moment for your business, they are early warning signs that something big is on the way. While preparing your business for Big Data doesn’t mean that you need to start mining Facebook and Twitter data this moment, it does mean that you need to start preparing your organization by considering the following:

  • How is Big Data going to change your business processes and existing technology? Where do the opportunities lie?
  • Data mining, analytics, and data driven decision making will no longer be competitive advantages for businesses. Instead, they will be operational mandates, just as email or a general ledger have become. If you haven’t invested properly in these capabilities to date, how much do you need to invest to catch up?
  • Data governance will be thrust to the forefront. Is your data governance program (policies, procedures, and people) ready to handle the exponential growth of data?
  • What is your organization’s analytical culture? Do you have the human capital with the analytical skills to drive your business in a new, data-centric world? Do you have the right human capital to manage those skills? Do you know how to recruit and develop those skills?
  • Business units will need to own Big Data projects. These projects will be no different than projects of the past— they should only be undertaken if their Return on Investment (ROI) is deemed significant.
  • How well do your business and IT units work together? The demands of Big Data are going to force business and IT to collaborate and work closer together than ever before. The lines between business and IT will need to be blurred, allowing for better understanding of needs and capabilities.
  • Does your organization have a tendency to pick a single tool for any given capability? In order to survive in the world of Big Data, multiple tools will be needed in the toolbox. One tool will not solve all.

So, how is your canary doing? For an assessment of your company’s data strategy, contact us at

How Can We Help?