A vast majority of the average enterprise’s data — and possibly a majority of their most valuable data — currently sits in the dark, according to the popular industry parlance.
Unlike structured data, which fits neatly into spreadsheets and databases that can be easily searched and analyzed, a majority of enterprise data currently sits in emails, texts, contracts, invoices, PDFs and Word Documents, which is far more difficult to search and extract insights from.
Thus the data is considered in the dark, and it is estimated that about 80 per cent of the average enterprises’ data currently exists in this state — but not for long.
The extraction of dark data and its conversion into usable information is considered one of the greatest business opportunities of 2018, inspiring companies of all shapes and sizes to go chasing the light, typically using artificial intelligence tools.
Here’s what you need to know about dark analytics.
Data extraction is worth billions
Big data is big money, for both the individual corporations that currently sit on these mountains of data and for the industry helping them extract it.
According to the International Data Corporation (IDC), the big data and business analytics industry reached $150.8 billion in 2017, up 12.4 per cent from 2016. IDC predicts that the industry will continue to see an annual growth rate of approximately 12 per cent through the year 2020, when revenues will reach over $210 billion.
For the typical Fortune 1000 company, just a 10 per cent increase in data accessibility will translate into more than $65 million of additional net income, according to Forbes, which also predicts that by the year 2020 there will be 1.7 megabytes of new information created for every human on earth, each and every second of the day.
The opportunity is so massive that the world’s biggest company made a deep investment in getting ahead of the curve. In May of 2017, Apple acquired AI company Lattice Data, a specialist in extracting unstructured data, for a reported $200 million.
How to bring dark data into the light
With such a high price tag, the gold rush has started to develop artificial intelligence tools that can shed new light on previously unusable data. These tools can help pour through documents, messages and contracts to help extract information and sort it into structured formats that can be used to gather insights.
According to CIO magazine, modularity and interoperability are vital to the success of an AI-based conversation process. “To get the most benefits on this IA [intelligent automation] to AI spectrum, data analytics, AI and a set of core digital technologies need to work seamlessly together and easily interconnect with the enterprise infrastructure,” wrote contributing author and Genpact chief digital officer Sanjay Srivastava.
“This interoperability is the second key to success, as it allows companies to easily add data analytics and artificial intelligence to the technology foundation they have previously built. Interoperability is critical for enterprises to harness all the data that already exists.”
Srivastava adds that an integrated command and control center that can allow for traceability in AI decisions is also key to successful data extraction.
Data analytics forecasts
According to Forrester research, this will be a pivotal year for AI and dark data analytics. They predict that by the end of 2018, one in five enterprises will deploy AI to help make real-time decisions.
They also predict that half of enterprise companies will adopt a cloud-first strategy for their big data analytics efforts and that the insights-as-a-service market will double, with 80 per cent of firms relying on service providers for a portion of their insights capabilities.
Overall, we are in the midst of a pivotal year for bringing dark data into the light.