7 reasons big data analytics initiatives fail

By Prakash Kini

Every business worth its multi-million-dollar tagline wants to understand Big Data Analytics and leverage it. What I have realised is that it takes not only skill but finesse as well to understand and appreciate Big Data in all its beauty and derive true and timely business value from it.

Of the many reasons that Big Data Initiatives could fail, I have picked out seven. Bear in mind that these are purely from my own experience in the field. For better context, I have put the reasons under three separate themes, organizational quagmire, old habits dying hard and new approaches creating new problems.

Organizational Quagmire

1. Queue of Unabated Requests
An investment bank elevated a lead technologist to Chief Data Officer. On taking up office, she was inundated with requests.

Here’s a look at the top five requests:

The CRO wanted to address regulatory and risk management gaps using Big Data solutions. The CMO wanted deeper customer understanding to inform marketing spends and drive revenue through personalization.

The COO wanted to benchmark operations against competition and build dashboards to optimize and improve efficiencies. Trading Managers sought solutions around portfolio optimization, automated research and similar information. The CIO wanted to reduce costs by leveraging artificial intelligence for production monitoring and other technology operations.

Big Data can address a range of problems but the creation of a steering group to funnel, prioritize, filter and sequence requests becomes critical. Otherwise, there’s a risk that the Big Data initiative becomes the ‘panacea’ to all the organization's problems.

2. Lacking Synergy and Accountability
A retail company's IT team was structured horizontally with sub-teams for program management, business analysis, user experience, deployment and database management. Each team had their own lead and sub-culture. A little down the line, awareness and interest in Big Data led the company to add three more sub-teams, namely, data science, data engineering and data visualization.

Soon, a high profile Big Data project came along and people from each of the IT sub-teams were assembled to collaborate and deliver a solution. However, communication between the individual groups was poor. With each transfer, a lot was lost in translation.

A lean startup delivery mode coupled with collective responsibility to the product could have mitigated this scenario. It also helps to ensure strong vertical synergy between the Data Scientists, Data Engineers, Data Visualisers and the Business Analysts on the project.

3. Poor Business User Analysis and Engagement
The IT arm of a retail organization sponsored a prestigious Data Analytics group. The group quickly gained free rein within the organization because business viewed it as an ‘IT project’ and did not intervene. Six months went into building sophisticated models for demand forecasting and inventory management.

When a demo was arranged with the business users, a reasonably expected response ensued - ‘How does this model compare with our however archaic, older model?’ and ‘The new model is not really necessary. The old model does what we want, at least for now.’

But the IT lead having already made the sizable investment decided to proceed with productionizing the proposed model. A year later when the new model had to be incorporated into regular business user flow, the tide turned. Business users refused to use it.

There is no such thing as an ‘IT project’. Technology is an enabler that helps actuate business results. And it will always be important to develop an early strategy around continuous business user engagement.

4. Careless Positioning Statements
The aforementioned retail company could also have suffered from misinterpreted internal PR.

Quite often Big Data Analytics tools are meant to provide inferences such as trading recommendations, process and asset recommendations and more. This crosses paths with what senior business analytics and functional experts also work on, therefore it’s important to be prudent when positioning these tools for the larger organization.

People should see Data Analytics as a force multiplier rather than a threat. This helps create the right human-machine symbiosis and interactions.

Old Habits Die Hard

5. Falling for Big Data, Unprepared
A product company’s CTO placing Big Data at the fulcrum of his discretionary investment secured a sizeable budget for Data initiatives through sheer passion. The CTO invited popular Big Data firms to present demos and the company, blown away, bought the best looking algorithm for 50% of their hardware budget.

A year down the line, the company brought on board ten new customers and data volume increased a hundredfold. But the product company’s current infrastructure only supported 10% of their use cases and the CTO was caught between a rock and a hard place.

Often technology choices are made too soon, without keeping these two things in mind: realistic business use cases and time for sufficient experimentation.

The Big Data space is quite fragmented. It incorporates tools and platforms that might serve the source company but may not be the solution to all industry scenarios. Moreover, a lot of kinks and bugs need to be ironed out before some of the tools or products are made production ready.

New approaches create New Problems

6. Expanding Without Foresight
A heavy equipment manufacturer wanted to use predictive analytics for effective lead generation. It planned to aggregate wear and tear telematics data from on-field heavy equipment along with machinery specifications and customer service schedules. The information was going to predict service visits and recommend preventive maintenance interventions to customers.

The manufacturer then decided to use brute force to integrate the accumulated data into the legacy Data Warehousing and Business Intelligence infrastructure. They wanted to build a stored procedure engine for preventive predictions.

The result was an engine seize.

The reason was that it’s never going to be enough to re-label a company’s DW/BI investments to Big Data Analytics. Big Data is a specific set of tools that is meant to solve a specific set of problems. There is a deeper data mindset that needs to be introduced into the organization when expanding its offerings to include the likes of predictive analytics.

7. Building Organizational Trust in Data products and algorithms
An oil major wanted Big Data Analytics to provide insights and optimize their oil pipeline and storage assets. One of their quantitative experts developed an ingenious algorithm that pulled in all pipeline and storage capacity and consumption details plus demand and supply scenarios. Using this information, the algorithm presented recommendations.

For the leads to trust the algorithm and the results it promised however, the Data team had to develop detailed and creative data visualizations. This requires excessive pre-planning and proper instrumentation of code and pipeline to record the right intermediate snapshots.

Initial time should be invested in getting the leadership and key teams to understand the scope and potential, investments and running costs of Big Data products. This will ensure less time being defensive and more time being innovative.

In conclusion, Big Data Analytics is a high impact area. There are risks that can be smartly mitigated as businesses embark on their Big Data journeys. Clients need a partner who understands their business, the opportunity to innovate and how to avoid pitfalls. All this, while innovating for the future.

Prakash Kini is a Big Data Analytics Practice Leader at ThoughtWorks

Published Date: Feb 26, 2016 08:12 am | Updated Date: Feb 26, 2016 08:12 am