Editor’s note: This is an edited version of a post that original appeared on Consultancy.uk, a U.K.-based online platform for the consulting industry, under the title, “70 percent of big data projects in U.K. fail to realize full potential.”
Big data has rapidly ascended as a technique for companies to capitalize on data’s potential. According to a new study, nearly half of U.K. organizations now apply big data for insights; however, the track record isn’t great, as just 30 percent of organizations that use big data are extracting enough value. Shortage in skills and too little knowledge about the topic are the main bottlenecks for successful implementation.
By definition, big data entails the underlying need to process a voluminous and complex amount of data in the area of terabytes if not petabytes (in other words, a large volume of data). Its use has been touted as a potential game-changer for businesses as the capture, storage and analysis of such data could reveal vital statistics relating to everything from corporate fraud and changing consumer behavior to supply chain inefficiencies.
Big data is regarded as one of the most important drivers of digital transformation, however, tapping into that potential is easier said than done. Many U.K. companies are still finding their way around big data, testing technologies, use cases, tools and new operational models.
According to a survey from digital consultancy firm Whishworks, which polled 100 big data strategists, architects and users, the sector leading the way in big data is media and entertainment, with over 75 percent of the sector’s data being processed by a big data solution. Media and entertainment is followed by telecommunications and financial services, which have both been using big data successfully for over three years. Further findings in Whishworks’ The State of Big Data in the U.K. 2018 report revealed that big data also presents significant challenges, from costs to integration into a business model.
Implementing big data
Despite the relative youth of the big data concept, nearly half of U.K. organizations have big data projects and initiatives. Of the organizations that have implemented big data, only 19 percent are processing the entirety of their data. Those leveraging solutions to analyze vast data sets are mostly early entrants.
While smaller, more agile companies may be getting the most from big data, the allure of the concept to big business is evident. The majority of organizations that have implemented big data initiatives, tentatively or full scale, come from the 500+ employees’ category. Scale alone suggests a high probability of a large volume of different types of data understandably believe they have a great deal to gain from big data, even if the vast majority have not yet developed solutions which work at the high speed required to process their information.
As an organization becomes more confident with using big data, it is often the case that new and/or larger-scale initiatives (like developing new revenue streams and driving innovation) come into play. Most respondents expect to get the most value out of big data with regards to improving their current operations. Those quizzed by Whishworks singled out product marketing strategy, revenue and processes as the areas which will likely benefit them the most.
Data investments are also intrinsically tied to return on investment as companies at the beginning of their big data journey hope to see faster returns via optimizations of their existing operation, reducing risk in the process.
Whether a company’s projects actually deliver on the potential of big data remains to be seen. Success is by no means a certainty. Gig data is a fairly new concept and the distributed processing technologies leveraged to analyze it are still rapidly evolving, and with them, the way organizations operate and compete with one another.
It is easy to misjudge the launch of a big data project, or to be outflanked by nimble competitors. The U.K. is facing a talent shortage due to tightening border controls and an ageing population. This is inflamed by a recent prediction from IDC that the big data technology and services market will see record growth reaching $58.9 billion by 2020, provoking a surge in demand for data processing and analysis talent. Many companies are desperately waiting for a new generation of graduates to join the workforce.
As a result, organizations are rarely completely ready to undertake the big data challenge. Thirty percent find their big data operations to be an outright success, while seven in every 10 respondents of Wishworks’ poll said that their initiatives had been only somewhat successful so far. Despite the challenges, organizations considering a new big data initiative can take heart from the fact not one participant in the survey was willing to describe their efforts as “not successful.”
Room for improvement
Respondents identified a number of key areas which could lead to an improved implementation of big data. As previously mentioned, these were dominated by lack of skill set, at 71 percent. Firms also said their current crop of staff tended to suffer from a skills gap. Sixty-five percent of companies said that limited understanding of big data across their organization hampered their efforts. Cost and data integration were also factors regarded as chief challenges.
More than half of respondents noted that cost was a key concern. It is difficult for firms to effectively estimate the cost of a big data project, making it easy to run out of budget. Acquiring new hardware or paying a cloud provider could all lead to a budgeting shortfall, while a company with an on-premise solution will also need to consider the entire cost of ownership including cost of training, license renewal, maintenance, upgrades and expansion.
The pressure on firms to acquire top talent also plays into this. Big data projects require sophisticated teams of developers, architects, data scientists and analysts who are well-heeled in a number of technologies, including Apache Hadoop, Apache Spark, NoSQL Database, data science, analytics and data visualization. Obtaining these skill sets does not come cheap amid a tightening market.
Forty-seven percent of respondents said the integration of big data into their business model remains an area of difficulty, partially due to the lack of an adequately trained workforce at most companies. Relating to this, 47 percent of respondents also noted that there was no shared data strategy or vision at their firm – which is likely at the root of their concerns around integration.
An area which has remained a perennial concern for years is security and governance of big data, currently rated at 35 percent. The EU’s landmark privacy legislation – General Data Protection Regulation – could see corporate entities fined as much as €20 million, or 4 percent of global revenues, should they be found to abuse the data of their customers or fail to adequately guard it from hackers. In terms of business-related challenges, 41 percent of respondents named changing business requirements as a top issue, while a further 35 percent said a lack of data governance was a major worry.
Benefiting from big data
Commenting on what may come next for companies looking to leverage big data, Edward Davies, sales director at Whishworks, said, “As companies gain more understanding of how they can benefit from big data, the rate of adoption is going to increase exponentially. In 2018, we expect to see companies to process more of their data with big data technologies and start implementing more complex and/or impactful use cases. It is also going to be interesting to see how different industries will evolve taking advantage of the new opportunities big data presents, and what new disruptors will surface.”