CITO Research

Tell Us a Question.

JUNE 2010

A CITO Research Explainer:

In-Memory Intelligence for the Way We Live and Work

Sponsored by QlikView

Contents
Introduction
The Planned Tyranny of Business Intelligence

1
1

The QlikView Hypothesis The Problem with Queries and Cubes

Big BI and Little BI Cubes: Rigid Precalculation The Stack Argument: How Much Is Enough? The BI Chasm

2 3
4 4 5 5

QlikViews Associative In-Memory Architecture

Step 1: Load Step 2: Search Step 3: Visualize

5
7 7 7

Common Questions Business Implications of QlikViews Associative In-Memory Architecture

Data becomes more valuable Living Reports

8 11
11 11

Examples
Fraud Shipping Customer Profitability Business Discovery Day

12
12 12 13 13

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

Introduction
QlikViews goal is simple: to provide tools for exploring and visualizing data that enable you to personally find answers and drive innovation. To understand the value of what QlikView offers, its important to first take a close look and whats right and wrong with the way we currently use information. In business, the practice of using information to run a business more effectively is called business intelligence (BI). This paper aims to explain how QlikViews Associative In-Memory Architecture brings BI to a new level of power that is under the direct control of the person asking the questions. This paper should appeal to anyone wondering what QlikView can do for them and to users wanting deeper insights into how the application works in order to make the most of it. It also explains how QlikView is fundamentally different from traditional BI across a number of dimensions.

The Planned Tyranny of Business Intelligence

Problems are almost by definition unexpected. We rarely wake up in a cold sweat at 4:00 AM knowing that therell be a crisis later that afternoon and that wed better start researching now to be ready. When we start asking and answering questions, the pathway of analysis is rarely predictable. One question always begets others. Offloading the task of finding and understanding information to someone else isnt very productive, either. Theres not much point in waiting while they sift through search results and return with their findings, only to send them back again with another idea once the results come up short. Putting others between us and the information we need means missing out on ideas that occur to us during the search itself. And how many iterations of this could you stand to go through in a day? This research process summarizes the problems with most modern BI systems:
n n

The questions must be planned in advance

 he search is frustrated; instead of knowing exactly what we want or being T open to new ideas, the process is dragged out and hindered by query mechanisms every step of the way

Of course, many questionswhat are my revenues? my profits? my margins?can, in fact, be answered ahead of time. Queries do work. But typically the action is moving too fast, and BI systems are too monolithic to change and keep up with day-to-day needs. We want to recalculate profit by region today, by product tomorrow. Most of

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

the time, BI cannot keep with our changing needs or provide an answer in a split second. The result most often is that data is not used to help answer key questions. When the questions must be answered, the typical BI process results in a large bottleneck facing anyone trying to answer new questions and meet new challenges.

The QlikView Hypothesis

QlikView takes an alternative view of analysis that can be summarized in three steps:
n 

Consolidate: Identify related data sets, map the associations between them, and load it all into a QlikView file resident in memory Search: Explore the data using list boxes that display the unique values in each field and can be highlighted and aggregated. Both information included and excluded from the selection criteria is displayed and updated instantly  isualize: Maps, charts and assorted graphics can be created and instantly V updated

n 

QlikView aims to increase your chances of making genuine discoveries and eliminate much of the grind:
n  n

No queries, no middleman: You click to select data and click again to deselect it  atching and non-matching data is displayed: Its not just what you see, M but what you dont seewhich data was excluded and why? What happens when you mix it in? No waiting: The answers are right in front of you

n 

The idea behind QlikView is that closing the loop on asking questions and encouraging individual exploration leads to better answers, insights, and innovations. Examples can be seen at demo.QlikView.com. Why this is true might best be explained by Eric Von Hippels theory of user-driven innovation, which argues in favor of providing tools directly to end-users. Whats more difficult to grasp is how, exactly, QlikViews software manages to do whats just been described. That is the focus of this paper. The following sections explain QlikViews In-Memory Associative Architecture and its implications for your business.

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

User-Driven Innovation
Why should companies pretend to know what customers want? Instead, why not give customers the tools to create or improve their own solutions? In a nutshell, this is the argument of MIT Sloan School of Management professor Eric von Hippel. Decades of research has found that innovations frequently trickle up from usersone of his studies found that 82% of new capabilities for scientific instruments like electron microscopes developed this way. Why? Because users came up with changes the manufacturers never considered. Von Hippel believes that users can help companies innovate more quickly and inexpensively than companies can internally. The key is giving users the tools. This has proven relatively easy to do in software and on the Internet, where open source breakthroughs like Linux are old hat. As Moores Law and economies of scale conspire to drive down costs and complexity, it becomes easier than ever to place powerful instruments in the hands of the people, an untapped force von Hippel once described as the dark matter of innovation.1

The Problem with Queries and Cubes

Database queriesthe bedrock of traditional BIare powerful, but theyve always faced limitations. For one thing, theyre often incredibly complex and difficult to construct, requiring subqueries of data subsets that can boggle the mind. (Entire books have been written on how to build better queries.) Figure 1 shows an abstract view of what an SQL query looks like.
Select Field 1, Field 2 Sum (Field 3) where Key 1= Key 2 and Field 3 > 0
Select List of Fields Aggregate Functions Where Join Criteria Selection Criteria

Figure 1: An Abstract View of an SQL Query

1 http://www.nytimes.com/2007/03/25/business/yourmoney/25Proto.html

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

Queries have an ontological problem as well: they can tell you known unknowns common, dependable metrics such as sales and customers by regionbut they cant tell you unknown unknowns, the questions you havent thought to ask. The former is the province of traditional BI; the latter is where insights are discovered. A query returning a list of every widget sold to every customer in North America might be useful, but a list of who didnt buy and what didnt sell is equally if not more useful. QlikView always shows you both at the same time: what matches the selection criteria and what does not. Why are queries so rigid? Because BI was developed 25 years ago at a time when storage, memory, and computing resources were scarce. It was impossible to process millions, if not billions, of records in a timely manner. As a necessary shortcut, the relevant data was culled and mapped to precomputed parts of answers called cubes to be queried later. It was an elegant solution to hardware constraints that no longer exist, but decisionmakers are still living with the consequences of the earlier and more limited design.

Big BI and Little BI

The current way information is used in most organizations might best be described as Big BImonolithic enterprise applications that are infinitely scalable but require full-time care and feeding from IT departments to create an endless array of queries, cubes, tools, and dashboards requested by users. This inevitably creates a bottleneck when simple requests for a dashboard are waitlisted for nine months or queued with a dozen similar dashboards until implementing them all at once justifies the resources. Big BI is devoted to making as many people as happy as possible with a solution for the lowest common denominator; if youre one of the unlucky few, your only other option is Little BIbreaking that dashboard into ad hoc queries and spreadsheets and getting by with what you have.

Cubes: Rigid Precalculation

Big BI is in the business of building precalculated cubes: simply tell them ahead of time every possible dimension you could want to analyze, and they will calculate a metric across all of those dimensions and store it somewherein a cube. When you ask it for that metricsay, profitabilityit instantly returns the answer. The upside is speedtheres no waiting. The downside is that you can only ask questions that you already knew you wanted to ask.

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

The Stack Argument: How Much Is Enough?

Big BI is also in the business of selling you features, like a car salesman would. The most important question when choosing a vehicle is: will it get you where you want to go? But sometimes the decision gets bogged down in choosing features rather than a solution. Big BI is a large commercial utility vehicleit bristles with features and has the brute strength to do just about anything. Its also expensive, temperamental, and not very maneuverable. You cant just take it out for a spin, or hand the keys to just anyone. In fact you may need to hire drivers with a special license. You dont need a tank when a bicycle will do.

The BI Chasm
Between Big BI and Little BI is the BI chasm the missing middle ground between queries and spreadsheets, where enough of Big BIs capabilities can be brought to bear powerfully, with enough speed and ease-of-use that users can grasp its value immediately. What do you need to bridge that gap?

QlikViews Associative In-Memory Architecture

QlikView aims to bridge the BI chasm by replacing queries and cubes with its associative in-memory architecture, shown in Figure 2. Rather than precalculating answers, the software loads data sets into memory and maps the associations between them. Because its associative, there are no predetermined paths and no precalculations you can layer on as many metrics and ask as many questions as you like. Because its all done in-memory, the answers are returned instantly and updated continuously. Your BI is no longer as good as your IT departments last cubeits as good as the questions you ask.

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

Figure 2: Associative In-Memory Architecture Explained in 3 Steps

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

The Associative In-Memory Architecture is put to work every time you use QlikView through the three basic steps explained next.

Step 1: Load
QlikView uses load scripts to load data from different repositories into a single large table where associations are automatically made. The load scripts do the job of the join criteria of an SQL statement. Experts might think of what is in memory as the result set of an outer join of all data loaded. Everyday users can think of the data in memory as being loaded into a massive in-memory table, like a spreadsheet, that has columns of all loaded data and where each row has all the data for all rows linked with the join criteria expressed in the load script. Loading the data into a QlikView file sets the stage for analysis.

Step 2: Search
QlikView doesnt show you a view of the raw in-memory table full of data. Instead, you see list boxes, one per column. Each box shows all the unique values in the field. Doing the work of the selection criteria in an SQL statement is as simple as clicking specific values in each list box: green items are the ones youve selected. In other list boxes, values included in your selection are white and those excluded are gray. The relationships are easy to see. QlikView uses the in-memory table to keep track of which data is associated with the selection criteria. Using list boxes to search through and understand data is more like a conversation than a query. Which products were sold in North America? Who are my gold customers? And how do they overlap? In this manner, you can construct complex queries on the fly while never losing sight of which data has been included or excluded often its the latter where crucial insights are found. What didnt sell in North America? And who didnt buy those products? Maybe you never thought to ask that before. Or maybe you couldnt.

Step 3: Visualize
Data is visualized in two ways: through metrics that summarize the data and graphics that display the data that is selected along with graphical forms of the summaries.

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

Aggregate metrics allow functions to be calculated for all selected data. Aggregate metrics are defined by choosing from more than 200 standard mathematical and statistical functions (standard deviation, mean, median, count, and so on). They can be combined to create custom metrics for your business. These metrics are shown in charts based on what has been selected by list boxes and are updated instantaneously whenever selection criteria are changed. QlikViews graphic visualizations render field and aggregate values in any number of charts and representations, updated continuously by the underlying data. The columns and sizes in a bar chart change instantly as the selection criteria are updated. While the details of the associative in-memory architecture could fill a book, the feeling that you get when using QlikView is that the data set comes alive. You see how aggregate metrics and graphics are transformed as different sets of data are selected. You do this yourself, with no waiting, so the trip from a question to an answer is short and direct.

Common Questions
Here are some of the common questions that weve heard about QlikView.

What Do Load Scripts Do?

Load scripts and QlikViews ability to make associations between data sources replicate the job of the join criteria in an SQL query. A QlikView file may consist of data from many sources. Unlike a spreadsheet where you may have one worksheet with one type of data and another worksheet with unrelated data, all data in a QlikView file must be related to the other data in the file. The job of the load scripts is to bring the data in from external sources and explain the way that the data is related. This is what is meant by associative. Each field and row is connected to the others in the file.

What Does Associative Mean?

Associative means that each field and each row is connected to every other field and row. The load scripts tell QlikView how they are related. The in-memory data structure can then be used for rapid recalculation. Figure 3 shows the Table Viewer in QlikView that keeps track of how each data source is connected to the others.

QlikView
In-Memory Intelligence

CITO Research
Tell Us a Question.

Figure 3: QlikViews Table Viewer Shows the Associations Between Data Sources

QlikView automatically discovers associations based on common field names, or the associations can be set in a load script after the data is loaded.

What Is a List Box?

At the heart of QlikView are list boxesroughly the equivalent of a column in a spreadsheet or a database with one important difference. List boxes display every unique value in the fieldduplicate values are not displayed. The other large difference is that list boxes control what data is selected. By clicking on a row or range of rows in a list box, all data elements with those values in that field are selected. You see all selected data turn white in the other list boxes. Lets say you have list boxes for products sold and the regions they were sold in. There might be 200 products and five regions. Each region and product is a list box. Select a region and QlikView displays the products sold there; select a product and it will show the regions it was sold in. These relationships are mapped automatically. In this way, QlikView acts as an association engine, offering end users a simple way to see and understand the relationships between data.

QlikView
In-Memory Intelligence

10

CITO Research
Tell Us a Question.

What Is an Aggregate?
Aggregates are mathematical and statistical functions combined to create metrics. An example might be the average price paid for a product in a given regionor any of more than 200 built-in functions. Aggregates are custom metrics that are relevant your business.

What Do You Mean by Visualization?

In QlikView, visualization means more than it does in other software. Charts (and any number of graphic representations) are updated continuously by the data underneath them. This in itself is not unique to QlikView. What sets QlikView apart is being able to visualize the associations between the data, which is even more useful than dynamically updating charts.

Why Does In-Memory Matter?

Historically, BI made the best of a bad situation by coaxing insights from tremendous volumes of data beyond the ability of most hardware to handle. It discovered a shortcut by pre-processing queries and cubes ahead of time. While computing has been transformed since, BI hasnt. Today, BI itself is the bottleneck. QlikViews Associative In-Memory Architecture is an attempt to reconceive BI for the modern computing environmentone in which the tools for analysis and discovery are placed in users hands. Big BI was born into a computing universe of scarcity. But 25 years and a dozen cycles of Moores Law have made memory cheap and hardware exponentially more powerful. Someone like Wal-Mart might not be able to store its trillions of transactions in memory yet, but it will. Many businesses already can. QlikView was the first to handle all data in memory; other vendors are starting to follow suit.

What Database Does QlikView Run On?

QlikView does not run on a database at all, but can load data into memory from as many different databases as you may have in your landscape.

QlikView
In-Memory Intelligence

11

CITO Research
Tell Us a Question.

Business Implications of QlikViews Associative In-Memory Architecture

The business value of QlikViews architecture is commonly revealed in the following ways.

Data becomes more valuable

The structure of traditional BI reflects the structure of the traditional corporation, where power was centralized, consolidated, and controlled. Vertical organizations long ago gave way to horizontal ones, but BI has struggled to keep up. The great advantage of QlikViews easy-to-use footprint and modern architecture means that the data youve been studiously collecting throughout your organization can be pushed to where its needed and put to use directly by the people who need it to make decisions. By placing the tools of analysis in the hands of users, youre potentially unlocking a resource that no BI system has ever been able to harness the intelligence of your colleagues. Senior managersthe traditional constituents of BIfind QlikView to be a powerful tool for accelerating their own analysis and decision-making. Those most accustomed to the limitations of BI will appreciate associative search more than anyone.

Living Reports
BI systems produce reports telling you about the pastwhat happened last year, last quarter, last week, yesterday? The analysis is out-of-date the moment its published. QlikViews graphic visualizations are drawn in real-time from the underlying data as it changes. In this way, QlikView provides a living, breathing real-time view of your business.

End-User Creativity Is Unleashed

Traditionally, BI applications produce one report and copies are forwarded to everyone who needs them. Putting QlikView in the hands of dozens, hundreds, or even thousands of users has the potential to transform how BI is used by taking it out of the hands of IT and placing it in the hands of end users. While this can potentially place incredible strain on the underlying hardware, QlikView customers report that the benefits of democratizing business discovery are transformative.

QlikView
In-Memory Intelligence

12

CITO Research
Tell Us a Question.

All Levels of Users Are Supported

QlikView is one of the rare technologies that is popular with the entire range of the end-user population in a company, from the most advanced business analysts and hands-on users who want the answers in just few clicks. Early adopters tend to have PhDs, programming experience, and the initiative to create 50 QlikView-powered applications that they then seed to business analysts. These superusers tend to make the case internally for QlikViews worth and broader adoption. Less technically skilled but eager to accelerate the process without IT, business analysts look to QlikView to quickly create apps and explore relationships between databases, often building them from top-to-bottom. They might need some guidance to create load scripts and integrate data sources, but after that, theyre self-sufficient. End users need the most help of all, requiring analysts and data experts to create models and handle the heavy lifting, but once the interface is in their hands, theyre also to dive deeply into the data.

Examples
Here are a few examples of how QlikView is being used today.

Fraud
A number of retailers using QlikView have discovered instances of fraud that had eluded their best efforts. One company uncovered previously invisible patterns in ordering, shipping, and returns that pointed to millions of dollars in fraudulent transactions accumulated a few hundred dollars at a time. It discovered the losses after using QlikView to combine and analyze formerly separate databasesits inability to do so had created the loophole in the first place. The fraud was the result of an inside job, perpetrated by employees familiar with the limitations of its systems.

Shipping
While hunting down fraud, the same retailer discovered that it had never taken a close look at the relationship between order patterns and shipping container logistics. Simply by rearranging how it packed containersand by offering extra incentives on especially efficient ordersit saved $400,000 a year.

QlikView
In-Memory Intelligence

13

CITO Research
Tell Us a Question.

Customer Profitability
Customers are using QlikView to make many areas more efficient. Sales and marketing teams have embraced the tool to identify high-potential clients and campaigns in an effort to maximize scarce resources. Sales teams are not usually among early adopters, but the value of the data is compelling for them and directly impacts the bottom line.

Business Discovery Day

Another customer has instituted Business Discovery Fridays, days on which developers are encouraged to put aside regular duties and follow the data where it leads them. This has a two-fold purpose: one is training developers to think associatively and to improve their pattern recognition skills; the other is to discover new projects to implement in QlikView.

A CITO Explainer
CITO Research is a source of news, analysis, research, and knowledge for CIOs, CTOs, and other IT and business professionals. CITO Research engages in a dialogue with its audience to capture technology trends that are harvested, analyzed and communicated in a sophisticated way to help practitioners solve difficult business problems. This document is a CITO Research Explainer, a form of content intended to explain a topic that is of potential importance to CIOs and CTOs. This explainer was sponsored by QlikView in order to create a clear explanation of its products. To find out more about how explainers are created, go to www.CITOResearch.com.