Using a sports metaphor, business intelligence and data discovery are akin to blocking and tackling in the sport of American football. You need to be able to block and tackle well to stay in the game – these actions are the bare minimum requirements to not get blown away by your competition during the game. One team that blocked and tackled well not only on the field but also off the field was the New England Patriots, which helped them win several Super Bowls in the last decade.
The Patriots organization mined data to successfully help them prepare for games against opposition. They used disparate sets of data like stats on players and competitors; statistical probabilities of various game outcomes; weather impact on players and teams; and reviews and feedback on social media when scouting for talent. In a nutshell, they used a plethora of structured and unstructured data and connected seemingly disconnected sets of data to make decisions.
You are probably wondering “Are business intelligence and data discovery the same thing?” The right answer is they are two sides of the same coin; in short, data discovery solutions focus on discovering correlation amongst disparate elements, while BI solutions offer a platform to report and analyze on causal patterns in data. Business intelligence and data discovery have the same goal, however – to help end users make better decisions. To understand the two concepts in a little more detail, let’s now look at some of the specifics of these two concepts.
Business intelligence is primarily a way of analyzing the transactional (historical) data of an organization through data mining and online analytical processing. Data discovery is an extension of data mining with the intent to discover data patterns. Business intelligence focuses on identifying the data, extracting the data and validating the data (aka: finding causation in the data). Data discovery focuses on finding correlations in any available data set without the need to validate it via IT. Data discovery allows you to explore the data and discover new questions that you did not think of, or to discover answers to questions that you may not have asked.
In today’s world, there has been a tremendous increase in the volume and types of data. Add this to the proliferation of social networks; businesses now need to track the pulse of their markets from new angles as well. Data does not come from within the supply chain alone (product, customers, suppliers, etc.). Data discovery efforts allow businesses to see correlations in various types of data outside transactional supply chain data sources.
Data discovery is a more interactive and iterative process than the analysis and reporting performed in a business intelligence platform. There is no need to start with a question during the analysis/discovery phase, to source the data, and then to report and summarize the data like in a typical BI solution. With data discovery solutions, you can instead use any source of data, immerse yourself in an iterative discovery process, and arrive at new questions or answers.
With traditional BI applications, data governance processes typically ensure the cleanliness of the data. With data discovery, business users can discover patterns in any data source, even if this data is located on their local desktop as a text or an Excel file. Data that is found in obscure sources has historically been mined sparingly for a myriad of reasons. This data usually gets lost and never found. Data discovery allows for users to mine this data and detect patterns without relying heavily on IT resources. However, this also poses a challenge for data discovery applications as the data has to be self-governed.
Failures of business intelligence efforts due to reasons like heavy reliance on IT and capital and resource intensive projects (to name a few) led to the entry of data discovery into the business intelligence landscape. Data discovery is not a new concept and it has existed for some time. It historically took a lot of technical expertise from the individual with help and reliance on technology solutions to facilitate data discovery efforts. The ones who could mine and discover patterns and were successful at it used it sparingly as the business needs kept changing, and institutionalizing the process was an arduous task. Discovering data patterns quickly and easily without a heavy reliance on technology became the need of the hour. Connecting the dots and simplifying the analytics in a disconnected world is a prime requisite for most successful organizations today. Today, a product like Oracle’s Endeca Information Discovery (OEID) fits the above data discovery needs.
Data discovery empowers all end users – not just the data/information hoarders. Business people like the fact that they do not have to rely on IT teams to find data correlations and to make faster business decisions. With a data discovery solution, business users can focus more on analyzing the data and making a story out of the data available. This enables a quicker turnaround when analyzing near real-time data than BI solutions. Business people typically have short memories – so it’s great to create a story around data that is not only relevant but also timely.
One thing that will probably never change is that business decision makers will never be satisfied with the systems and data in any BI or data discovery solution. Change is the only constant. Therefore, data discovery will not replace BI. The two concepts will coexist at least into the foreseeable future to satisfy the ever-growing data needs of a business community that increasingly favors self-service and faster decision-making cycles.
Author: Sreekanth Kumar, Performance Architects