Long Live Small, Fast, Actionable Insights!
Big data is everywhere. Organizations are being advised to hoard it, and to do everything they can to derive
actionable insights from it. This article will argue that this approach puts the cart before the horse.
First, let’s start with a revelation. Despite the hype and despite everything you’ve been told about big data as
a necessary precursor to achieving valuable business insights, it has by and large failed to deliver on its
All the data in the world means nothing, if you can’t make sense of it. And that’s the hard part. Even Gartner
predicts that through 2017, 60% of big data projects will fail to go beyond piloting and
experimentation, and ultimately will be abandoned.
High Risk and Cost of Failure in Data Analytics
Part of the reason for this failure is that organizations rush to accumulate and analyze as much data as
possible, all at once, and without much of a plan. This usually ends terribly. It’s not unusual anymore for
organizations to house multiple petabytes of data. These compulsive data hoarders pathologically store any data
they can get their hands on, thinking that the more data they have, the better off they will be, the more
intelligent they will be, and the higher the quality and number of insights they can glean from it.
But starting big can lead to an overwhelmed team, not to mention huge costs. These organizations won’t be able to
see the insights for the data, so to speak. With more data and more capabilities to rationalize data across
silos, it can feel like there are more opportunities than there are resources to exploit them.
The problem is exacerbated by the fact that traditional approaches often fall short. Heavy duty enterprise
business intelligence (BI) tools, central data warehouses, etc., designed to analyze big data sets, take years
and cost millions to implement (something your BI service provider won’t tell you at the outset).
These heavy data warehouses can pay off. But when they don’t, the cost of failure is high. In a survey of more than 500 executives by McKinsey Analytics, 86% of
respondents reported that their organizations were only somewhat effective at meeting the goals they set for
their data and analytics initiatives.
The moral of the story? Hoarding big data is not a prerequisite for smart insights.
A Shorter, Nimbler Path to Insight: Follow the Breadcrumbs!
The truth is, for many organizations, starting small is a much more fruitful (and lower cost) exercise than
bringing in heavyweight tools and consultants. Time and again we have seen some of the most valuable business
insights derived from surprisingly small data sets. It’s time to focus, not on big data, but on
the right data
at the right time, and find a way to ask the right questions of that data.
It’s an approach to data analytics that we replicated from our best practices in software product development.
Focus on delivering an MVP as fast as you can, and iterate from there. In this case, MVP does not stand for
Minimal Viable Product, but Minimal Viable PREDICTION.
The MVP approach challenges traditional perceptions and established norms of big data and data analytics. The
notion of having to assemble data, at any cost, is an extreme one. Instead of eating the entire elephant at once
(your mouth is too small and you’ll choke), we recommend a low-risk approach to data analytics that a successful
start-up might take. It is grounded in best-in-class software development methodologies. Here’s how it breaks
Start with focusing only on the #1 problem you want to solve -
Instead of assembling all your data, disregard the noise and assemble only the data that correlates with your
number one problem.
Once you’ve established your primary problem, work backwards from there identifying other related data, and
follow the data breadcrumbs that lead to actionable outcomes. For example, you may start with understanding and
predicting customer complaints - something that can have a very real impact on your revenue and profits. So,
you’d assemble complaints data. Your next data breadcrumb might be customer service representative data, a
potential reason for complaints. Another set of data could be transaction size by customer. Easy enough so far,
and guess what? You may already be halfway there or more! Perhaps a single customer representative evokes
unusually high numbers of customer complaints. But you can further query the data to understand whether
transaction size shows correlation and predictability of customer complaints in association with that customer
You have just reached MVP status – you’re a Most Valuable Player having created a Minimal Viable Prediction! This
sounds straightforward, but it can be tempting to jump into the data lake in front of you instead of just
following small, relevant, iteratively assembled data breadcrumbs.
This approach also goes back to the earlier point about not being a data hoarder. Data becomes dated – fast. Make
a point of spring-cleaning your data on a regular basis so that you avoid a glut of irrelevant data.
Iterate, constantly – As with software product
development, don’t waste time trying to find the ultimate product (or in our case, the Answer to the Ultimate
Question of Revenue, Profit, and Everything). Predictive analytics is a process of trial and error. There is a
tendency in traditional data analytics to keep hoarding and trying to analyze all that data at the same time.
Instead, focus, focus, focus, and iterate, iterate, iterate.
Start with prioritizing all the problems you might be able to solve with data analytics. Rank them by a)
potential business impact and b) ease of assembling the first set of data. Then pick the highest-ranked problem,
and follow the breadcrumbs. If you hit a dead end, retrace your steps and follow the next set of data
breadcrumbs. Do it deliberately and with focus. Don’t get distracted. Just because you could easily assemble
another set of data doesn’t mean that you should. Don’t muddle your quest to predict the solution to business
problem number one. The more self-discipline you can demonstrate in pruning your data tree and avoid analyzing
too much at once, the easier it is to generate insights and achieve a better outcome. As you continue to
iterate, include more data, where and when it’s relevant, and only then.
Keep refining – Once you’ve solved your number one
problem and have your first MVP, refine it again and again so you have an ongoing stream of actionable,
affordable predictions. Here’s an example in which you operate a trucking company. Your number one headache is
customer complaints. So your first data set includes all your complaint records. Your second tier of data may
include a list of drivers and their HR records, shift schedules, truck inventory, trips taken, weather data,
road condition data, etc. You dig into that data and find that weather and road conditions have no impact on
complaints. However, surprise, you find that marital status has the highest impact! So, you decide not to hire
married drivers or adjust your training methods. This is just an example and by no means a recommendation that
you violate anti-discrimination laws! But you get the picture. The point is you are on the way to
establishing a methodical process that iteratively solves your number one headache. You can now focus on the
business problem ranked second and iterate, again and again.
This means investing time and changing mindsets inside and outside the organization to dismantle ingrained dogma
that data analytics is an “all-or-nothing” big budget investment and encouraging a leap of faith to go another
By adopting the principles used in software development – starting small with a self-imposed incremental budget
(why spend so much money, if you don’t have to?) and growing from there – you’ll realize results faster,
cheaper, and without the risk of throwing millions of dollars and thousands of man hours at a heavyweight
solution, where the probability of success is low.
Look to the Cloud as an Enabler
Cloud-based pay-as-you-go tools, such as Microsoft’s Power BI, are ideal companions for this approach, at least
as a starting point. Today, data management, analysis, and visualization tools – all in convenient
out-of-the-box, cloud-based packages – are making it more feasible for organizations, small and large, to become
data savvy. And they’re doing it without the need to have multiple specialists on-hand and at a much lower
price point. As you grow you may need to add-on additional functionality or move to more powerful tools
– just don’t start there.
Follow the Data Breadcrumbs to Success
What does success look like? Once you have your first MVP, you are already more successful than 50% of other big
data projects out there. Using our approach, we’ve seen clients reach that point in month two or three, not year
two or three. By only focusing on the relevant data breadcrumbs and starting small, it becomes much easier to
gauge whether you’re on the right track and correct any mistakes relatively cheaply.
Have the Courage to Go Against the Flow
Ok, so we have flipped the way organizations approach data analytics on its head. But having the courage to go
against the flow by adopting such an agile, nimble approach to big data analysis can pay dividends. With
self-discipline, focus, and a readiness to throw irrelevant data, no matter how “big”, out the window, the
opportunity to make smarter, data-driven decisions and even monetize your data has a much higher probability of