How to Set Up and Find Success in Your Big Data analytics Project – A Primer

How to Set Up and Find Success in Your Big Data analytics Project – A Primer

We at Express Analytics often get this question – how do we (a business) go about setting up our Big Data analytics project?

Typically, we get this question from two types of businesses – those that are modernizing their data architectures, and those that have never had traditional BI and data warehousing systems but want to jump straight on to the Big Data bandwagon.

Both situations pose certain challenges; some unique, some common to both categories. So we thought, why not spell out things in black and white by way of a blog post which will work as a primer to help out both these categories.

What is Big Data?

Going ahead with any analytics project, one must at least have a rudimentary understanding of the definition of Big Data.

That’s the right start point, especially for those enterprises that want to get into the game straight away.

To be candid, the word “Big Data” may have been coined recently but the act of collecting and analyzing data is decades old.

Simply put, Big Data denotes the large volumes of structured and unstructured data that flows into your business every hour, and every day.

The now-industry accepted definition is the three Vs:

Volume: Data today flows into an organization from numerous sources: websites, social media, mobile computing devices, sensors, and so on.

In earlier times, storing this humungous volume was a problem, but new tech like Hadoop has now resolved that problem.

Velocity: Denotes the speed at which data flows in, and has to be dealt with.

With the advent of the Internet of Things (IoT), for example, data from sensors, RFID tags, or from the Edge necessitates businesses to deal with data in almost real-time. Other businesses continue to deal with data coming through in batches.

Variety: Of course, like humans, not all data was made the same. Comes in different formats, size, formatted, unstructured, and so on.

The massive size of Big Data brings with it challenges such as data storage, scalability over time, noise accumulation, false data correlation, and measurement errors, all of which require new computational and statistical models not available in the traditional form of BI.

Now, having grasped the basic definition of Big Data, businesses have to understand that there’s but only one fundamental thought running the capturing of that massive volume of data – what do you do with it?

Collecting data is one part of the operation, but the story does not end there. In fact, some may say it starts there.

Big Data Analytics has to be analyzed for insights, it has to be checked for patterns. Why? Because that leads to better decision-making within your organization.

All of your enterprises’ decision-making is now on a scientific, evidence-based platform, not some random gut feeling or superficial market research.

Big data analysis offers interaction with data that was not possible with the earlier, traditional enterprise business intelligence systems.

But here’s the thing about Big Data – it cannot just be said of it that it will provide 360 degree on-screen enterprise-wide high-level insights; that would be over-simplifying.

According to this bunch of researchers, big data analytics projects have no single canonic use case or structure.

Instead, the applications that could benefit from analyzing Big Data cut across industries and involve a wide variety of data sources.

What does Big Data Analysis mean?

I shall simplify it – Big Data analysis does not throw up a single set of results to be implemented across the enterprise.

Also, it does not have one form throughout the organization.

Big Data analysis can have many forms and many outcomes within various departments within the same organization.

Outputs may result in either revenue gains or cost reduction, or even risk mitigation.

It can be used in marketing and sales, Customer Relationship Management (CRM) initiatives, and brand monitoring, to name a few departments.

That’s why before launching a Big Data Analytics project in your enterprise, certain steps have to be taken.

Because of its very complexity arising from the variety and volume of data, and its sophistication, project plans need to dovetail into the intended business outcomes.

One of the biggest challenges in this kind of project is to avoid working in silos ‘cause that’s one of the main reasons for failure.

The fundamental difference between the traditional ways in Business Intelligence (BI), and Big Data analytics is that in the latter, an analyst or trained employee finds patterns from analyzing data, and then gets answers to questions that were, in fact, never asked in the first place.

This may sound strange but that’s how Big Data analysis differs from the traditional BI methods where answers to questions were sought, against structured “predefined” data sets.

We will pause here but in part 2 of this post, we will discuss how best to get started with a Big Data project, and also the common pitfalls to avoid. So stay tuned.