And how to make your next data analytics project succeed
What does data analytics really mean and how can it help you today? Search Google for the term “data analytics”
and you’ll be confronted with more than 39 million results. Good luck with that.
To put it simply, data analytics can help you predict the future and make decisions with relative certainty.
Though it may provide a competitive advantage today, analytics-based decisions are increasingly becoming the
norm across industries. Whether in the private sector or a government agency or nonprofit, everyone is jumping
on the bandwagon. Yet, analytics has its challenges and, done wrong, can quickly drain your resources without
leading to actionable results.
Data analytics and predictive modeling is notoriously hard. Just look at the obstacles:
- Too many data systems and tools – In the business intelligence (BI) arena there are dozens
of comparable yet distinct tools to evaluate. How do you know which are right for your industry or business?
- User competence – Any team embarking on data mining, predictive analytics, and predictive
modeling must be very well versed in math and statistics.
- Dirty data – Getting to a point where you have data that you can work with can suck the
life out of your analytics project. Many organizations spend 80% of their time cleaning up data.
- Misplaced priorities – It’s imperative, but often overlooked, that analysis is done early
on to reconcile what business leaders think they want, versus what they need, versus what they can achieve.
Typically, this happens too late during the discovery and project phase.
- Analytics is boring – The C-suite has a short attention span. To keep them engaged, you
must continuously provide them with insights to ensure their full support throughout the project.
To better understand why analytics projects fail, we’ve broken these challenges down and proposed some best
practices to ensure the success of your next analytics project.
Data Analytics: Challenges and Misconceptions
There are several challenges and misconceptions that lead data analytics projects to fail. Here are eight of the
most common ones we encounter:
1. Data Siloes
Any vendor who’s trying to sell you a BI tool will invariably start their pitch talking about your in-house data
challenges, most of which have their genesis in disparate data sources and data siloes.
Here at EastBanc Technologies, we see customers with tens of thousands of internal databases that don’t talk to
each other. Many lack the robust API layer needed to get programmatic access to data. Instead, they gravitate
towards the concept of the data warehouse, where big data is collected and stored in one place before it’s used
for analysis. Although they have a role to play, data warehouses are overrated. Not only do they take years and
cost millions to implement (something your BI service provider won’t tell you at the outset), they also add to
your data glut and delay actionable insights.
Data warehouses can also up your storage costs and make the data analytics process more laborious. Because
warehouses extract data into data cubes each time you need to use it, duplicate data is created (one terabyte
can quickly become 50 terabytes). Plus, the data cube data isn’t updated automatically, potentially leaving you
with stale data that creates noise in your system and making the data cleaning process even more laborious.
Tip: An API eliminates this problem by automating data sharing so that you can start
showing results with the data sampling that you have, rather than wait for all your data to become available.
2. Dirty Data
The second data analytics challenge facing organizations is dirty or inconsistent data. Common examples of dirty
data include basic hygiene issues like misspellings and empty data fields which can cloud your insights. But the
problems can be more systemic. For example, a company’s CRM system may define the northeast U.S. in a way that
includes Washington, D.C. However, the HR department may not consider Washington, D.C. to be a part of the
northeast. This creates inconsistencies. If you merge CRM data with HR data, you have a problem. While tools
exist that can pull data out of disparate systems, they don’t always flag data conflicts.
3. Lack of a Plan
To get to predictive analytics nirvana and rein in data sprawl, you need a plan – an analytics roadmap. Since
internal teams rarely have the expertise for this task, organizations, quite rightly, look to outside experts.
It’s very hard to generate the needed experience (amount and diversity of projects) that outside experts can
4. Fear and Fiefdom
Which comes to the next challenge – fear and fiefdom. Data owners may fear they’ll lose relevance or control of
data if they are forced to share data sets with other departments, agencies, or external expertise is brought
Tip: Counteract fiefdom through evangelism. It behooves your organization to prove how
sharing data creates value across the board.
5. The Old School Mindset
Moving up the internal organizational hierarchy, the leadership team also has a role to play in why analytics
projects fail. Many business leaders have a “green bar report” approach, meaning they trust the old way of doing
things. Instead of basing their decisions on actionable results, they want to review each bar, a time-consuming
In addition, there’s a lack of continuous involvement or sponsorship by executives in the analytics process. It’s
the very antithesis of agile
Consider a typical analytics project. During the kickoff meeting, everyone is in the room. Yet, a meeting with
leadership involvement may not reconvene for another three months. They ask for an update and find that the
outcomes are not what they were looking for or priorities have shifted, but the analytics team doesn’t know
because no one bothered to tell them.
Tip: The solution to this problem is to adopt a lightweight, agile
approach. Your business sponsor doesn’t need to be involved all the time, but you do need periodic, short
feedback loops. Feed your sponsor miniscule pieces of fast progress. This will ensure they see immediate,
incremental results, get exactly what they need, and drive greater engagement.
6. The Big Bang Approach
If you lack the internal resources to support your data analytics project and decide to bring in an external
firm, that’s fine. No one ever got fired for choosing Accenture or Deloitte. But, these industry leaders often
promote the concept of an all-in, big bang, multi-million-dollar approach. The approach may work, but it’s not
the only way to go.
There is another lower risk, fast results route that embraces agile and focuses on achieving Minimal
Viable Predictions. After all, an analytics tool, like a piece of software, is a product and
product/software development best practices still apply. Most companies that provide analytics services, don’t
have a product mentality. Instead, they focus on maximizing billable hours which stymies the transfer of
knowledge and intellectual property to the product owner who is going to run the tool, aka the customer.
This approach also plays on the common misconception that analytics is a one-time project. That you set-up
everything and reap the benefits. It doesn’t work that way. It’s also not a black box effort. Everything must be
transparent with immediate, disciplined, and regular feedback loops.
7. Focusing on Big Data, Rather than the Right Data
Another misconception is that you need to start big. Starting small is a much more fruitful (and lower cost)
exercise than bringing in heavyweight tools and consultants. Time and again we have seen some of the most
valuable business insights derived from surprisingly small data sets. It’s time to focus not on big data
but on the right data at the right time, and find a way to ask the right questions of that
data. Starting small also minimizes risk.
Is there a tool for this? Again, that’s another misconception. You can’t just buy a tool, install it, and hope it
will magically predict the future. As your business needs evolve and digital transformation matures your choice
of tools will also change.
8. Expecting Certainty
Finally, don’t expect 100% certainty in your predictive analytics. It’s simply not possible. In some cases, we’ve
seen margins of error of 30-40%, and that’s ok.
Predictive Analytics Done Right – Breaking Through the Obstacles
How can you overcome these obstacles and misconceptions and ensure data analytics success?
Start by addressing the number one challenge facing internal teams – the data silo problem. How do you start
analyzing data when you can’t even integrate it cleanly?
EastBanc Technologies came up with a concept to break out of these data silos without data owners fearing they’ll
lose control of their data. By packing all relevant technology into a container and giving each department
access to the data that’s relevant to them (or that the data owner feels comfortable sharing) via an API – silos
are easily overcome. We call it an “API-in-a-box” and it can be spun up in minutes, eliminating the
time-consuming data integration problem. Plus, after data errors are found and one department’s data is merged
with another, actionable insights start to emerge and the barriers of fear and fiefdom start to break down.
But a best practice approach to predictive analytics done right doesn’t just hinge on technology. How do you
focus on the right data? How do you start small and see actionable results in weeks or months not years?
Part of the reason that many projects fail is that organizations rush to accumulate and analyze as much data as
possible, all at once, and without much of a plan. This usually ends terribly. It’s not unusual anymore for
organizations to house multiple petabytes of data. But starting big can lead to an overwhelmed team, not to
mention huge costs. These organizations won’t be able to see the insights for the data, so to speak. With more
data and more capabilities to rationalize data across silos, it can feel like there are more opportunities than
there are resources to exploit them.
Our best practices approach to this challenge borrows from agile software development. We call it the Minimal
Viable Prediction (MVP) approach.
Instead of eating the entire elephant at once (your mouth is too small and you’ll choke), the MVP is a low-risk
approach to data analytics that a successful start-up might take. Disregard the noise and assemble only the data
that correlates with your number one problem, as fast as you can, and iterate from there. It also ensures your
business sponsor will see the kind of actionable results that drive greater engagement and ultimately, ensures
the long-term success of your predictive analytics projects. Read more about how
to implement MVP.
Another best practice approach to better data management is to rely less on your own internal data. There is a
tendency to want to own everything in-house (no matter how dirty). Instead, try to create and maintain as little
data as is necessary and always look for external sources. Data can be readily leased or purchased. For example,
weather data is readily
available for purchase and is much cheaper than collecting your own.
This monetization of data is increasingly commonplace. Another example is an EastBanc Technologies client who
assembled sports data from national leagues, packaged it into an API and licensed it. All the data was created
and maintained by the leagues, the company simply utilized that data and provided it to others for a fee.