Agile Business Intelligence

[The first part of this post was produced for MultiChannel Merchant magazine in conjunction with its Operations Summit, Memphis, May 2-3, 2012.]

It has never been easy getting reporting and analysis tools into multichannel environments that satisfy all the potential users. Some of this is a data issue, some of it reflects varying users’ needs, but most of it, especially recently, is a reflection of how fast the business landscape keeps changing.

Data Challenges
The data issues are easily described though difficult to resolve. There are at least three major types of data that need to be accounted for: transaction data (related to sales and orders), marketing data (related to offers and campaigns), and accounting and financial data (related to payments, discounts, refunds, credits, and so on). And of course this data comes from multiple channels.

Within each of these categories are subcategories. The transaction data, for instance, includes data on inventory, which can actually be a subset in itself, related not only to SKUs but vendors, items on-order, returned items, etc. Marketing data includes promotion codes and source codes, and is also directly related to accounting data because the prices paid by customers using offers will affect revenues in an obvious way.

Taking into account how all of these various types of data need to be coordinated for reporting and analysis purposes is a job that demands interdisciplinary rigor and a clear idea of what each constituency of users wants and needs.

Understanding the User
Which brings us to the second challenge. Marketer evaluating campaigns need to look at response rates, but they also often need to look at items ordered. So they need both transaction data and “marketing” data. And they will want to look at item returns, as well, which is more transaction data, but a subset of it. And if they are looking at the impact of free shipping and handling, they are going to want to see financial data.

The complexity of doing this probably obvious. And let’s not forget that even among marketers, what one wants to see isn’t necessarily what another really wants.

In addition, between user sets there can be incompatible frameworks, such as a 12-month marketing calendar but a 13-month retail calendar.

For reasons such as these, most organizations decided quite a while ago to just dump data into spreadsheets and let each user do their own thing.

This works fairly well if the organization is small. But when you have more than a half dozen or so people doing their own thing, you quickly run into apples and oranges comparisons, which everyone knows can be misleading and dangerous.

“Dashboards” used to be a popular alternative, having a unified data set that let everyone pick and choose their own versions of how to see the data. But these are limited, as is standard “OLAP,” which is a drill-down type of analysis that relies on pre-set data parameters, which is also somewhat limiting.

Getting Agile
For reasons like these, the new hot trend is something called “agile analysis,” based on the concept of “agile programming,” which says that the best way to develop a system is to work closely with its users to get feedback at multiple and frequent checkpoints.

Agile Business Intelligence, or Agile BI, is a reflection of what David Weinberger, a senior researcher at Harvard’s Berkman Center, calls the “changing shape of knowledge.” The impact of the Internet in driving information, and the effects of the ever-evolving Social Web, mean that the Internet is no longer an environment of linked pages but more like a mash-up of rich applications. And the impact of Social Media will only continue to add velocity, density, and volume to the data analysis mix.

Agile BI must reflect these changes, allowing decision-specific applications to be joined together to form flexible and evolutionary BI platforms, some of which may even be embedded in transactional or other applications. And as requirements change, transactional or analytical applications can be added, removed, or updated quickly because they don’t require an assessment of their impact on a universal “single version of the truth” data model that prevailed in the pre-Web dark ages.

Most of the agile BI solutions are or will be Web-based, but that’s not the critical element. What is essential is the ability to rapidly source information, connect it to other information in both a tightly and loosely integrated fashion, and quickly connect BI applications and platforms together. This is the combination that will be critical in meeting rapidly changing reporting and analysis requirements in the ever more fluid multichannel environment.


Agile BI Solutions
"Agile BI Out Of The Box" by Boris Evelson, a Forrester report produced for for Business Process Professionals, references and profiles the following Agile BI solution providers, which I am listing here for your to follow up on:

Balanced Insight
BIReady
Composite Software
Oracle Endeca
IBM Cognos AAF
Kalido
SAP NetWeaver BW
WhereScape

This is a very mixed bag of recommendations, but the report offers a good comparison chart to help you narrow down which solutions are likely to be best for your situation.

Mr. Evelson recommends not taking a headlong plunge into Agile BI, but to test the waters first with a project or two to get the hang of it and provide a "Proof of Concept" for your organization. Excellent advice!