Buzzwords have varying levels of actual significance. It doesn’t require significant meaning to a lot of people. If social media “share” buttons cost money or had a limit, then maybe it would be easier to distinguish the buzzwords we should pay attention to from the ones that don’t represent an actionable trend. But as long as they are free, buzz will exist for anything that is amusing or interesting on any level, and the general public can be exposed to “the grumpy cat” as much as a business buzzword such as Big Data.
So since I’m writing this blog it’s clear I think Big Data is going to have a longer shelf life than our furry friend. But why? I will get to that, but first let me give my one paragraph take on what Big Data is, since it has different meanings to different people.
Gartner defines Big Data as “high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.” One consistent term you will hear is the “3 V’s” deriving from volume, velocity and variety as you see above. Those are the important drivers of Big Data, but when does “a lot of data” become Big Data? To me, it is at the point when using traditional enterprise SQL based relational data structures no longer meet the analytical demands of the organization. Existing systems either begin to feel the strain and begin to perform poorly or just can’t take on additional information within the current design .
That is a very broad definition, since there are many different approaches once you have reached that particular tipping point. Future blogs will dive into these different approaches and provide technical facts and techniques. But the point of this blog is to let you know what the drivers are that will drive most businesses to this point. And the most important thing to consider is once you get there, you’re already behind. A Big Data strategy takes years to plan for and execute. If any of the descriptions below fit your company, you should be thinking about life after traditional data warehousing:
- You have a customer reach that is at least national in scope
- You care about what customers are saying to people in their social network
- You could benefit from a more detailed profile of your customer
- Your customers have many interactions with your company and you could benefit from storing not just transaction detail but other conditions that could be a factor
- You believe that the majority of people who will be your customer in 2-10 years are NOT you’re customer today
- You are a company that might need to defend itself from legal issues where a greater volume of available data could be helpful in litigation
So if you’re a medium size or above company, it is very likely multiple scenarios above describe you’re company. I attended the Gartner BI and Analytics Conference in March in North Texas, and to me the most resounding point made at the conference was that the future of analytics involves data that is not managed by your company. Companies aren’t just gaining advantages from data generated by their CRM, Accounting, HRIS, POS and ERP systems any more. There is sentiment analysis, social analytics, online customer profiles, competitive data, geospatial data and legal statutes at all levels of government. These are coming from 3rd party providers, social media API’s and public web feeds, not from your systems. Whatever business you are in, your competition is utilizing or planning to utilize this data to craft better strategies and make better decisions. And if you’re not, they have gained a competitive advantage.
And as I mentioned above, putting a Big Data strategy in flight requires a long runway. While there are several Big Data methodologies out there including providers such as Teradata and Neteeza all the way to MapReduce technologies such as Hadoop, the market is broad, layered, diverse and still maturing. You won’t find the kind of standards, best practices and accelerators that have emerged in traditional BI stacks.
Some estimates put the total of internet content on the web at over one yottabyte, which is 1024 bytes. And yes, while maybe more of that is “the Grumpy Cat” then should be, a lot of it does impact your business and will help lead to a better corporate roadmap if available to your organization.
If you agree and you haven’t started down the road of a Big Data strategy, my recommendation is to form a Big Data focus group. This group should consist of leaders in all departments. Big Data is not an IT issue. One of the sessions I attended at Garner spoke about the need of a Chief Data Officer, and an office of data competency that is independent of IT. While technology will definitely be part of the solution, it is very much a business problem. Operations, Finance, Human Resources, Marketing and Legal will all play an important role. While I’m not suggesting everyone hire a Chief Data Officer, it’s important that Big Data is a company initiative and not just an IT initiative, and a Big Data focus group is a great place to start.