The Future of BI

source: http://tinyurl.com/64rv9f

At a recent summit of Business Intelligence experts, hosted by Dashboard Insight, one of the questions posed to the panel I was sitting on was "Whither BI, what is the future of BI?" This is a question that has been steadily plaguing me over the past several years, and one of the reasons why I had consciously started to focus my attention on the dashboard side of the business. After engaging in many well-intentioned and executed BI projects that yielded mediocre results at best, I started to wonder "Is BI broken? Is there a better way?" At the time I saw a focus on dashboards solving many of the inherent problems that I had recognized in the traditional approach to BI, because a greater emphasis was put on solving user problems versus data ones. Over the course of delivering nearly a hundred unique dashboard solutions I came to adopt what I refer to as a user-centric approach to business intelligence design. Some people also refer to this as a top-down approach, which lies in stark contrast to the traditional bottom-up or data-centric approach used today in most significant business intelligence projects.
How we got to where we are:
At this point it might be helpful to talk a bit about the history of BI and the shifting user base it has been designed to target. The bottom-up (data-centric) approach to BI started when the primary goal of business intelligence was solving the engineering and architectural challenges of integrating and reporting against a companies internal data that was often siloed in a few large data repositories that were generated by companies’ various critical business software such as the accounting, sales, and inventory systems. The focus of these efforts was creating data structures that data analysts could do rudimentary reporting on and analytics against, so when business users needed answers to questions like "what are our top 10 selling products versus our top 10 most profitable products" the data analysts could design reports that tied together the accounting data with sales and inventory data. The primary users of these systems were trained data analysts that served as a human bridge between the business users and the specialized tools designed to access the back end BI systems.
In an effort to create reporting systems that gave a small specialized group of data analysts the ability to generate a myriad of reports for a diverse set of business needs, the focus was on designing highly flexibly systems that anticipated a wide array of abstract business requirements. The BI industry responded accordingly by developing the necessary tooling to extract, transform, process, organize, and analyze these abstract data structures, and thus we saw the emergence of ETL tools, Data Warehouses, OLAP cubes, as well as many other specialized technologies. Up to this point, the innovation in BI was really focused on how we managed and manipulated data. Technologies were invented and methodologies developed and espoused, in some cases with an almost religious fervor. But in this early period of BI, during the emergence and development of BI as a discipline, one key ingredient was conspicuously missing: the end-user who was the ultimate beneficiary of this "intelligence."
Where we are today:
Cut to today, and we find ourselves in a surprisingly different environment. Business and technology have been radically altered in many unexpected and significant ways through humanity's adoption of the internet during the mid to late 1990's. We now find ourselves in a highly dynamic and connected environment where business moves at a much faster pace, requiring that decisions be made faster and with more accuracy. We also are faced with exponential growth in the volume of data we produce, collect, analyze and are forced to interact with. Not only has the amount of data grown significantly, it is also far more distributed and heterogeneous than it ever was. Companies no longer have their critical business data stored in just a few large systems, but they also receive important business data from many ever-changing outside sources that the company may have little or no control over.
As BI has evolved over the past twenty years and has tried to keep pace with these ever more complex set of business and technology conditions, there has been more and more attention on enabling business users with direct access to these business intelligence tools. The first of these end user tools came in the form of static or canned reports that users could access directly, followed by the invention of "ad hoc" reports that gave users WYSIWYG tools to create their own reports against pre-determined data structures, and we now have progressed to easy-to-understand highly visual dynamic dashboard displays. The phrase "BI for the masses" has come into vogue over the past couple years, but unfortunately BI as an industry is still correctly perceived by the business community as having very little success in achieving this vision. BI tools are still considered too hard to use, too long to implement, and costing too much. Why is this?
Obviously there are real technology and business process challenges that we must overcome to accommodate the volume and pace of data generated by our internet enabled global economy and business conditions, but I believe the primary challenge we are facing is the BI industry itself. At the risk of being lambasted by the cadre of established industry gurus I would like to posit the thesis that the large BI companies and the recognized "experts" are actually hindering the very innovation and processes that would most benefit business and increase the efficacy of these tools. I make this statement because I believe too much of the focus is still being placed on collecting, manipulating and managing data when it really should be put on how users interact with the data, and what business conditions they are trying to improve via this interaction.
Does the BI Industry have it all wrong?
Big BI (as I define the small group of large but influential BI technology companies that we all know) has painted itself into a corner that is very difficult to escape from. As a result they are slowing down innovation and, more importantly, the processes used to implement true business intelligence. The mature and established BI players have built their fortunes on developing and selling large data-centric tools, applications, and services to the enterprise. In lock step with Big BI the professional service providers and resulting industry experts have all developed methodologies and approaches they espouse and use to implement these technologies. Both the BI companies and the solution providers have a vested economic interest in continuing to see these technologies and approaches purchased and implemented. So what happens when we start to question how effective these technologies and approaches truly are to actually helping the business and, even worse, what if we conclude that they are no longer (or never were) optimal solutions to truly address the business problems we were trying to solve?
Let me illustrate this problem with a concrete example that has become painfully obvious with the popularity of dashboard technologies. Dashboards are probably the first true end-user tool that can effectively deliver business intelligence data to every user within the organization with very little friction and low to no training costs. Done correctly, dashboards can be very effective business intelligence tools. Done incorrectly they can be little more than superficial window dressing. But the epiphany I came to a few years ago is that you can build a highly effective and maintainable dashboard solution with a very simple BI infrastructure. You don't need a data warehouse, you don't need an OLAP cube or a sophisticated ETL process. This isn't to say that these technologies are not immensely helpful, and in some cases even necessary, but in almost every case they are the exact wrong place to start for a company or department pursuing its first BI initiative. But I can almost guarantee you will never hear this from any person trying to sell you a BI solution. Why is that?
If the bread-and-butter of your revenue stream and expertise is selling large and complex BI systems and services that focus on all the inherently complex issues with the integration and dissemination of data, do you start telling your prospects that all of that stuff isn't really the right place to start, especially if your product line blindly assumes these pieces are necessary? This is exactly what is happening today, although I do think it is starting to change.
If you try to purchase any of the top-tier dashboard solutions from any of the major BI vendors you will be told you must first implement the rest of their "stack" which will consist of myriad back-end data-centric tools that will enable the dashboard. This is because even though the BI vendors know that you might not need all or even a fraction of the functionality in the rest of the back-end stack, they can't tell you that. If they did, what would they tell all of their current customers who were told these significant investments were required in order to produce an effective dashboard? What would the impact be to their revenue stream when they all of sudden tell prospects that they no longer need their most expensive and profitable products and instead could purchase what was being sold as the sizzle as opposed to the steak? This same conundrum holds true for most of the industry experts who have built their careers on developing and evangelizing rigorous disciplines and techniques to build these data Taj Mahals. What do they do now? Say "Oops, sorry we were wrong, all that stuff we told you to do for the past 15 years is really not the most effective approach, we got it backwards?"
Are Dashboards pointing the way to the future of BI?
In coming to the realization that we really need to start top down and make the focus about the user and not the data, what has become very clear to me is that dashboards, more than anything else in BI, are the most effective tools for validating your user's needs. Unlike written specifications, data diagrams, and other typical requirement artifacts dashboards provide users something they can touch and play with and let you know immediately if you are on or off track in meeting their needs. When executed properly, dashboards become the meeting ground between company strategy and company execution. Because (good) dashboards are designed to be intuitive and highly focused they force a lot of critical issues to the surface and require the business to ask questions like "why is this important to us, what problems does it help us solve?" Additionally, on the back end, dashboards help to clearly answer the engineering questions of what data is really needed to support users, where is the data coming from and at what frequency, and what data structures and formats really do need to be supported.
For these reasons I strongly advocate that clients embarking on any business intelligence project either large or small start with their dashboard and/or report design first. Before worrying about what data exists and what format it is in (and even worse having your design influenced by what data does NOT exist) define your project in terms of the user interface that will be used to meet your business requirements for the people actually using the solution, as opposed to the engineers building it. With smaller sets of relevant sample data and user requirements focused on solving the true needs of the business community you can use your initial dashboard prototypes as a tool to not only provide some immediate user utility, but also as a set of very specific business and technology requirements for larger back-end BI initiatives. One of the reasons that this technique is so effective, is that it allows you to iterate and test assumptions with the user community with significantly less cost and time than it takes using a bottom up approach. It takes days or, worst case, weeks to prototype a dashboard, while it takes months or even years to create an integrated data warehouse.
Where are we headed?
So where does that leave us today, and what does this all mean for the future of BI? I think dashboards represent just the first step for the next major phase in BI both from a technology and a methodology perspective. For the lack of a better term I will label this next phase the "BI user experience" as represented by user interfaces that information workers and business executives interact with to "experience" their data. Notice how I use the term "experience" versus analyze or view. Think of the difference of looking at a picture of an airplane versus reading a written description of one. In the first instance you have an instantaneous recognition of the symbol and its meaning, in the second instance it may take several seconds or more for you to understand what is being represented through written language. Processing of visual imagery leverages a completely different part of the human brain than reading the written word. Unlike the written word, which is processed through our pre-frontal cortex and language centers in a linear fashion, visual images are processed in a much deeper and more powerful part of our brain in parallel. This is an extremely important point and cannot be understated. Take a look at a long column or row of sales numbers in Excel and then look at those same numbers in a bar chart. Your ability to process that information and the inherent relationships within that data is exponentially higher and faster with the bar chart. This is one area where the human brain still far exceeds the power of technology-driven computation in its ability to recognize and process patterns composed of large volumes of information.
As an example of where I see the potential future of BI I want to draw a parallel to what is becoming a ubiquitous and re-evolutionary product, the iPhone. The iPhone doesn't provide any functionality that dozens of other smart phones weren't offering before it, and in many cases it provides less. But talk to any iPhone user and you will sense an almost evangelistic zeal in their voice when they describe it to you. What the iPhone does amazingly well is take all of that smart phone functionality, which on many other devices is perceived as too complex or burdensome to use, and literally puts it at your fingertips. By creating a feedback loop between two parallel processed sensory mechanisms (vision and touch) combined with a very well thought out design, Apple has made what was once a complex user interaction incredibly simple and intuitive. I believe the same can be done for business intelligence as well as data analysis, where deep and complex data sets can be presented and interacted with in ways that are tailored to specific business functions and conditions and provide end users powerful and intuitive interfaces. At a minimum, with a little innovation we should be able to take data visualization and user interaction techniques that are being used in other current technologies (such as video games) and combine them in a way that allows us to see the same data we are looking at today in more intuitive and relevant ways.
I see dashboards, especially interactive dashboards, and the technology they are built upon as the first step in this next phase of BI. While we still have many challenges to solve on the back end with the exponential growth in volume, diversity, and distribution of data sources, I see the real innovation occurring on the front end in both user interface design and user experience through innovation of user interaction techniques and business methodologies. I think the sooner the BI industry is able to re-align its focus (a.k.a. revenue streams) on these new areas of innovation instead of clinging to the solutions and methodologies that were designed for conditions and technologies that existed 5, 10, and even 15 years ago the more the BI industry and the business it serves will prosper.
About the Author
Mr. Gonzalez is the founder and Managing Director of BrightPoint Consulting, Inc. Tom's primary role at BrightPoint is leading the design and development of customized business intelligence solutions that incorporate his visionary approach to software design, resulting in immersive, interactive, and intuitive end-user applications.
For over 20 years Tom has followed his passion in developing innovative and value driven software applications both large and small, with proof-of-concept projects lasting days to multi-million dollar, multi-year development efforts. During his career, working for such notable clients as Apple, Adobe, Business Objects, BP-Arco, Kimberly Clark, Northrop-Grumman, Symantec, as well as many others, Tom has developed a multi-disciplinary approach to developing data-centric applications by uniquely combining his experience in software architecture, data warehousing, data visualization, business process, application development, user-centric design, and visual arts.
Tom graduated with honors from the University of California, San Diego, where he studied a unique combination of Computer Science, Visual Arts, and Economics. He currently resides in Carlsbad, California, with his wife and two sons.