An Oracle Endeca Whitepaper January 2010

Reinvent Your Business with Agile BI Infrastructure

Table of Contents
EXECUTIVE SUMMARY TRADITIONAL BI SYSTEMS ARE NOT DESIGNED FOR AGILITY BUSINESS SPEED AND COMPLEXITY REQUIRE NEW BI CAPABILITIES MAKING EXISTING INFRASTRUCTURE GO FASTER IS JUST NOT ENOUGH A NEW ARCHITECTURE FOR BI AGILITY ENDECA LATITUDE ORGANIZATIONS EXPERIENCING THE BENEFITS OF BI AGILITY 2 3

5 6 7 8 12 12

SUMMARY ABOUT ENDECA

Reinvent Your Business with Agile BI Infrastructure

Executive Summary Over the past 20 years, the speed of business and the nature of the information businesses collect, share, and analyze have changed dramatically. During that same period of time, the approach to business intelligence (BI) has essentially been fixed and has largely focused on efficiently generating reports for a few select individuals within the company. While this reporting is highly valuable and must continue, it is no longer sufficient. The traditional approach to BI, with its heavy reliance on data modeling and its appeal to a few, highly sophisticated users, just is not capable of responding to new business requirements such as the need to make midterm course corrections, to push information to a wider audience, and to consider a wider variety of data types, including both structured data and unstructured content. Efforts to employ agile development processes and to replace existing architecture components with faster versions of the same components simply do not go far enough. To date these solutions have had little effect on addressing the backlog of BI requests that builds as demand for modifications and new analyses increase. In addition, just making the existing infrastructure faster does not address the need to incorporate new data types or provide the ease of access that the broader user-base demands. To adapt to the new business requirements, organizations should augment their traditional BI reporting infrastructures with technology designed to handle the new data integration and presentation requirements. By adding a specialized database optimized for exploration, integration tools for unstructured content, advanced search, and visualization capabilities, IT organizations can both provide their business decision makers with new insight into critical business problems and dramatically reduce the BI backlog that has plagued them.

Reinvent Your Business with Agile BI Infrastructure

The value of agility lies in the creation of multiple scenario casting, a form of intellectual property. This levels sole concern is with the future tense or what will happen, driving an anticipatory, opportunistic type of decision making as one or more scenarios come into view.
BUSINESS INTELLIGENCE AND DECISION IMPACT GARTNER, INC. MAY 2010

Traditional BI Systems are Not Designed for Agility

The traditional approach to business intelligence (BI) has reached its limits. Over the past 20 years we have developed a set of procedures and technologies that allow us to aggregate, cleanse, prepare, and report on enterprise data. While there have been incremental improvements in efficiency and speed over that time, the fundamental approach to BI has not changed much. During that same period, however, businesses and their information needs have changed dramatically. Enterprises have transformed themselves from rather isolated entities, often with an inward focus on operational efficiency, to players in geographically dispersed, multi-partied ecosystems that need to have as much understanding of their customers interests and partners operations as they do their own operational efficiency. The information they collect, transport, and analyze has changed accordingly and, more importantly, so has the speed with which they need to act. Today agility is the goal of most organizations regardless of their industry and it is a top goal of business leaders whether they are in sales, marketing, manufacturing, engineering, customer care, or service delivery. Businesses are competing not just on efficiency, but on their ability to sense market conditions and quickly respond. That is exactly where BI and the business needs have diverged. In many organizations, BI systems are used almost exclusively to generate standard monthly or quarterly reports. These reports deliver great value certainly most organizations could not survive without them. But traditional BI systems require an expensive and time-consuming process to identify all the answers the business users ultimately want, build a data model that captures that information and unify data from disparate sources into that data model. Often, multiple cycles of this process are required before the needs of the business users are fully addressed. More than anything, a traditional BI approach is designed to structure data, ensure consistency across systems, and efficiently handle large amounts of structured data goals that seemingly run counter to the need for agility.

 Business Speed and Complexity Require New BI Capabilities

The demands on BI systems are changing signicantly. Technology innovations, such as the Internet, email, video conferencing, and mobile technologies, have accelerated business communication across the enterprise and between the enterprise and their customers and partners. With business moving at a faster pace, the amount of business data generated, the number of data sources, and the variety of data types has also increased. To make informed decisions, people often must consider a broader set of data from a variety of systems (see Figure 1). But because this data is so diverse, it dees conventional modeling. In addition, the new speed of business has caused consumers, partners, and employees alike to expect instant access to the information they need any time that they need it. Traditional BI systems are not designed to meet these new demands. In particular, traditional BI has difficulty supporting three key trends that represent the future of BI and enterprise information:

Reinvent Your Business with Agile BI Infrastructure

Diverse Information Sources Create Jagged Data

To make informed decisions, people often must consider a broader set of data from a variety of systems. But because this data is so diverse, it defies conventional modeling

Increased Importance of Unstructured Content  Traditional BI processes have almost entirely ignored unstructured content, such as word processing documents, email messages, and even text-based elds on structured forms. The reason the structured data of relational databases was the low hanging fruit. It was easy to model and the content from unstructured documents was not. We convinced ourselves that we could live without knowing what information was captured in the text. But the exponential growth of the Internet has both increased the importance and the volume of unstructured text. Especially in consumer-oriented businesses, consumer sentiment as shown in blogs and Web content, including RSS-based news feeds and social media sites such as Twitter and Facebook, can no longer be ignored. Yet, traditional BI platforms, with their focus on structured data and heavy emphasis on modeling have had a difficult time extracting value from that data and combining it with structured data. The Need to Address Emerging Challenges with Midcourse Corrections The competitive battle is increasingly being waged on how organizations address new questions in the moment. Agile operations mean that businesses can make rapid adjustments to new information and new conditions. Adjustments may be slight or major and new questions can arise at any time, whether it is mid quarter, mid-month, or even intra-day. Typically these involve new data sources that are being introduced every day, and existing sources that are undergoing constant change. The heavy modeling burden of traditional BI complicates the task of unifying data, often removes its distinctions, and prevents decision makers from having a clear view of the most up-to-date information.
4

Reinvent Your Business with Agile BI Infrastructure

If architected and designed properly, traditional BI applications provide deep insights into past and present events, as well as help identify patterns in past sales, marketing, supply chain, and financial performance. But in this new age of future-facing business analytics, any BI user without strong advanced analytics capabilities is seriously deficient and at a competitive disadvantage. Traditional BI tools effectively answer pastfacing questions such as, What happened? and Why did it happen? But they do not offer much assistance with proactive concerns such as, What will happen?, What may happen next?, and What could possibly happen if I take action X versus action Y?
BUSINESS INTELLIGENCE (BI) POLISHES ITS CRYSTAL BALL FORRESTER RESEARCH SEPTEMBER 2009

Decision Making and Enterprise Information Extended to More and Less Technical Users  An agile organization is one that can sense trends and changing conditions and react quickly. That requires all organizations within the enterprise and all levels of employees, regardless of their technical skill, to be an information receptor and to either be able to feed that information back to the central part of the organization or, ideally, respond appropriately on their own. Analytic tools associated with traditional BI have generally been designed for sophisticated power users, which has limited the use of those tools to only a small fraction of the decision-making population in a company.

Making Existing Infrastructure Go Faster Is Just Not Enough

Some organizations have responded to the need for greater agility by improving the scale and speed of their data warehouses, databases, and other elements of their BI infrastructure. Still others have responded by implementing agile software development processes, such as a Scrum process, to help their IT groups respond more quickly to business requests. While these changes have resulted in BI systems that can process more data than ever before and software development teams that can release new reports more quickly, they simply do not go far enough. Neither improvement, for example, directly improves decision making agility. Analytic tools remain too difficult to use for a broad audience of less-technical users and unstructured content is still ignored. So while the speed of the traditional infrastructure has improved, it has not improved the decision making capability of the organization because we have not enriched the data that can be considered for decisions and we have not expanded the number of people that can access and use the data. Increasingly, this makes the IT group appear as if they are out of touch with the needs of the business. Similarly, neither of these improvements provides a good solution for reducing the data modeling burden, which is the primary bottleneck restricting the throughput of the IT organization. In fact, according to some experts, even an agile software development approach does not address this issue because it calls either for ignoring modeling altogether, which seems unrealistic, or it calls for very rapid modeling, which, if left as a people-driven process, is easier said than done. Without a reliable, streamlined data modeling procedure, IT will continue to have difficulty keeping pace with the requests from business users. Every report and dashboard that is created spurs additional questions. Is this normal? Why did it change? Questions that seem simple turn into signicant IT projects if the questions have not already been anticipated and modeled. Because new questions are created faster than they can be modeled, a backlog of BI requests builds. The BI backlog has a number of undesirable side effects. For decision makers, it can limit their ability to make rapid, mid-course corrections. For IT managers, it can make them look unresponsive to the needs of the business. It also causes unwanted headaches for IT because it drives the business users to develop their own reporting capability outside of ITs data governance and security models.

Reinvent Your Business with Agile BI Infrastructure

Even the most powerful, advanced, and wellarchitected BI applications have one serious weakness: They are only as good as the underlying data model. Whether relational, multidimensional, or hierarchical, data models that form the basis for all BI applications have to be designed based on all potential questions that might be asked of the BI application in the future.
BUSINESS INTELLIGENCE (BI) POLISHES ITS CRYSTAL BALL FORRESTER RESEARCH SEPTEMBER 2009

A New Architecture for BI Agility

Since traditional systems and processes cannot be easily modied to address the need for agility and the new BI requirements, a new agile architecture is needed to extend the traditional BI infrastructure. This architecture should be designed for ease of use, so that it can help a broad audience, and exibility, so that it can support a broad set of analytic tools and easily incorporate new, varied, and changing data sources. This new infrastructure should not replace the legacy BI infrastructure. In fact, it is best to leverage the traditional BI infrastructure for its strength historical reporting and compliment it with the agile architecture. Over time, the agile infrastructure may prove that it can also handle the historical reporting task as well, but given the enterprises reliance on historical reporting and the value that is still being derived from the traditional BI infrastructure, a prudent strategy and one that reduces the risks for the IT staff is to postpone that decision to later. The agile infrastructure should include the following components: A Specialized Database Designed For Exploration The core component of the new architecture is a database that is specically optimized for exploration, not reporting. The database should accommodate any type of information, including structured data and unstructured content. The resulting combination will be extremely jagged and sparse, necessitating a database that does not rely on a pre-determined data model, but instead derives a data model based on the data presented to it. Finally, to support on-going exploration from users, the database must be able to process and respond to rapid-re queries in sub-second intervals and should be able to accommodate additions, subtractions, and changes in the data sources with minimal rework. Specialized databases of this type reduce the need for data modeling and support the broad exploration and analysis needs of the business users. High-Level Tools For Integrating Unstructured and Structured Data The system should have comprehensive ability to turn structured text into searchable data. That usually comes by way of advanced text processing and analytics capability that includes Web crawlers for identifying text-based data on the Web and in corporate data stores, natural language parsers for identifying important words, phrases and concepts within those documents, and a metadata model for tagging information so that it can be quickly found and incorporated into an analysis involving either structured data or additional unstructured content. In addition, the system should have the ability to present all data types in a consistent and unied interface to allow business users to consider the impact of all types of data on their analysis. Advanced Search The system should have search capabilities that are intuitive to a non-technical business user and even an occasional user of the system. Advanced search capabilities in BI today are far from a simple search box to nd dimensions. Instead let the user search both the data model and the data simultaneously. In addition, they have features such as auto-correct, spell-check, and faceted search that provide context-sensitive guidance to the user throughout their search experience. Multiple Visualization Options While some analyses are best supported by text-based responses, others are best addressed through a variety of graphical forms. To answer the broadest array of questions for the broadest number of users, the agile infrastructure should support many different forms of visualization and should allow the user to seamlessly navigate between them.
6

Reinvent Your Business with Agile BI Infrastructure

A Data Governance and Security Model That is Consistent with Existing IT Controls The infrastructure should provide the ability to extend existing IT policies, especially around data governance and data security, to the agile infrastructure without requiring substantial additional IT overhead. Adherence to IT standards, such as Web services and SOA, is, therefore, important to simplify maintenance and allow for rapid integration of distinct data systems. Rapid Provisioning Environment To enable rapid creation of search applications, the infrastructure should provide high-level tools for designing and modifying Webbased interfaces. These tools should, at a minimum, eliminate the need for scripting by providing graphical drag and drop interfaces. Ideally they would provide interfaces that are designed for end users or power users and, therefore, if desired, would allow IT to offload the interface development or maintenance work to less-technical people within the organization. With this architecture, IT is able to quickly set up applications where business users can fully explore varied data and can answer questions they may not anticipate at the outset of their exploration. This brings agility to decision making and also reduces the BI backlog because it reduces the number of BI requests sent to IT.
In the case of Endeca, based on XML architecture, data, metadata, and applications (reports and dashboards) are actually one and the same.
AGILE BI OUT OF THE BOX FORRESTER RESEARCH APRIL 2010

Endeca Latitude
 ndeca solves the new BI requirements around agility with Endeca Latitude a new E product that combines the simplicity of search with the power of BI. Endeca Latitude is used to create BI applications that present users with wide-ranging data sources and data types and provide users with discovery capability an ability to ask new questions that lead to better outcomes. Because of their breadth and exibility, a single application designed with Endeca Latitude can take the place of literally hundreds of traditional BI reports. Just as importantly, applications developed with Endeca Latitude embody the best search technology so business users can productively use them with essentially zero training. IT organizations also benet from Endeca Latitude. Endeca Latitude reduces the time, effort, and cost of creating BI applications by enabling IT to quickly and easily combine data and content of any type, format or source without complex modeling, integration or app development, even as the data changes. In addition, Endeca Latitude can enhance a traditional data warehouse and reporting infrastructure by offering a cost-effective way to deliver applications that answer unanticipated, in-the-moment questions. Endeca Latitude includes the following major components: MDEX Engine The core of Endeca Latitude, the MDEX Engine, is a hybrid searchanalytical database that allows users to explore data and content in an unconstrained and impromptu manner and to rapidly address new questions that inevitably follow every new insight. In the engine, there is no overarching schema for the data. Instead every record, every document has its own schema. The MDEX Engine has the ability to index specic data elements in great detail, including the content within unstructured documents, and can use that information to automatically identify relationships between records and documents based on shared attributes. In addition, the MDEX Engine also supports the rapid inclusion of new data sources or changing data because
7

Reinvent Your Business with Agile BI Infrastructure

those changes update the relationship between data elements, but do not require the entire model to be reconstructed. These capabilities allow Endeca Latitude to bring together diverse datasets even structured data and unstructured content without complex modeling and integration. The engine then conducts search, navigation, and analytics on this jagged data. In addition, the MDEX Engine supports IT-managed data governance and data security, such as promoting IT policies and support for security at the application, view, record, and property levels. With support for industry standards such as XQuery, XML, and SOA, the MDEX Engine is designed to t easily into existing enterprise architectures. Discovery Framework Provides a library of components and visualization tools that embody advanced search capabilities, such as Guided Navigation and multifaceted exploration, and enable the rapid conguration of Discovery applications. The Discovery Framework also provides a set of comprehensive management tools. With the Discovery Framework, you can respond to the requirements of a changing business by re-designing reports and dashboards in minutes, incorporating diverse data sources in days, and deploying entirely new applications in just a few weeks. Data Integration A collection of Web content crawlers, and le system crawlers, and connectors to best of breed ETL tools such as Informatica. These tools enable rapid integration of both structured data and unstructured content into the MDEX Engine and allow for rapid data updates to ensure that the MDEX Engine accurately reects the source data.

Organizations Experiencing the Benefits of BI Agility

Numerous clients are using Endeca technology to add agility to their operations and decision making. By combining unstructured and structured content in a single view into their decision-making process, and using Endecas Guided Navigation user experience and multiple visualization options, these organizations are discovering new relationships in the data and supporting a broad range of analyses. Endecas customer base represents many different industries each utilizing the technology to satisfy unique business intelligence challenges, including: A large financial services firm that needed to quickly model investment options and increase the number of trade opportunities per day. A high-tech manufacturer searching for a way to combine data from multiple repositories after a large-scale acquisition in order to streamline component selection for their products. A fortune 100 consumer packaged goods seeking to obtain an integrated global view of sales information to make the most effective product placement and promotion decisions possible. A national police force which required a zero-training user interface for all of their officers so they could quickly interact with criminal event data to discover unknown relationships to help identify perpetrators, prevent crime and protect their citizens.

Reinvent Your Business with Agile BI Infrastructure

 Wall Street Financial Services Firm Increases Amount of Trade Opportunities Per Day This nancial services organization, specializing in investments, realized it had a signicant barrier to productivity and sales. The dramatic increase in the amount of information available on a daily basis with regard to market changes and investment options made it difficult for brokers to keep up with in time to help their clients make informed investment decisions quickly. In addition, each client holds different investment interests, so pairing up potential good investment options with the right client demographic was a cumbersome manual process. Finally, because this process was so difficult, brokers tended to focus on their larger clients which left many investment opportunities with smaller clients off the table. The organization realized there was a need for a new information system that enabled brokers to quickly see important changes in the market and match those changes up with the client who, based on their investment prole, would be most interested in that information. Endeca Latitude was the only technology capable of combining structured information such as stock tickers with unstructured information such as news reports, and providing one navigable view of that data. Today, through a combination of Guided Navigation and visualization options, brokers are able to quickly identify information that is likely to trigger an investment from specic customers. By adopting an agile approach to analytics and reporting, this organization has made hundreds of millions of dollars in incremental sales with smaller clients that were previously off the table. High-Tech Manufacturer Cuts Millions of Dollars in Supplier Costs On the heels of a large-scale acquisition, this high-tech manufacturers supply chain organization was given a mandate to locate one billion dollars in savings for the organization. The product line they acquired was similar to their own, and they needed to continue manufacturing the acquired product line, but streamline the set of suppliers they used. They knew this was possible as the components that made up these machines were similar in specication, but they did not have a way to combine all of the data across the organizations to gure out what could be rationalized. To accomplish the necessary reduction in costs, this manufacture needed an innovative process to enable its staff to analyze product characteristics at the engineering attribute level in order to identify opportunities to standardize similar components and optimize design. Endeca Latitude was able to provide a unied view of data and analytics across a multitude of data sources that include large scale ERP, engineering and BI systems. With innovative visualization tools, such as bubble charts and scatter plots, the organization could identify similar components and appropriate substitutions. This organization has already been able to achieve millions of dollars in savings through things such as 60% reduction in amount of steel coils used, to 74% reduction in amount of valves needed, to 77% reduction in number of brand badges needed.

Reinvent Your Business with Agile BI Infrastructure

Combining Search and BI Finds Component Supplier Savings

The Endeca Latitude application showcased above is a generic example of what an organization looking to streamline design and sourcing costs would utilize. It is showcasing ways for analyzing total spend by a variety of facets such as part type, specific supplier or products.

 CPG Manufacturer Achieves Single View of Channel, Territory and Brand Performance This leading CPG manufacturer sells many different products and brands globally, which has resulted in numerous information sources located in a variety of systems. The organization had not been able to achieve a single view of all product line sales that also showcased which channel, territory and brand those sales came from. They were also trying to tap into sources outside of their organization such as consumer blogs to help bring that information into their decision making process, but had no way to incorporate that type of content into their analysis. Furthermore, the audience that needed to access this information was every day business users marketing, sales and brand managers who were not highly technical and required a user-friendly interface for exploring all of this information. With Endeca Latitude this organization was able to provide 13,000 users with a zerotraining interface that utilizes the ease of search through Guided Navigation and helpful visualizations, such as heat maps, for easily analyzing regional sales volumes and promotion performance. They now have clear visibility into sales performance by brand, product, distributor, campaign and even dynamic product groupings to analyze sales performance. This has enhanced understanding of why certain promotional programs are more effective so that those efforts can be repeated.

10

Reinvent Your Business with Agile BI Infrastructure

 National Police Force Provides Officers with User-Friendly Search Experience to Catch Criminals This police force had the challenge of anticipating and preventing crime within its jurisdiction with minimal budget increases. They needed a powerful method for correlating data from multiple data sources to enable their staff to respond more effectively to incidents, and also proactively discover emerging crime trends. In addition to their every day crime prevention needs, there was an international event being hosted in one of their largest cities in the near term. This made it essential that they nd a technology that would be deployable in a short timeframe to help their force protect the inux of people for that event. By implementing Endeca Latitude, this customer has a single search interface across multiple operating systems that includes structured and unstructured content from sources such as the local newspapers. The application enables proactive discovery of emerging trends and threats through correlating diverse data to showcase patterns in people, objects, locations and events. Police officers using this application on a daily basis are able to discover the important correlations in data that help identify and apprehend perpetrators, prevent crime, and protect citizens.
A Single Search Interface for Criminal Activity Helps Anticipate Future Incidents
The Endeca Latitude application shown here is a generic example of what a defense organization may use to analyze criminal activity. It is showcasing ways for analyzing specific public transportation locations by amount of crimes as well as station usage.

11

Reinvent Your Business with Agile BI Infrastructure

Summary
Today most organizations seek to have more agility in their operations and their decision making. These goals are driving new requirements of the BI infrastructure, such as the need to expand the number and nature of BI users and the need to easily integrate a greater volume and variety of enterprise data. Traditional BI has struggled to support the level of agility required. Simply improving the speed of the IT organization or the existing BI infrastructure is not enough to address the new requirements. To improve the agility of the BI infrastructure, organizations must add new technology to their existing BI infrastructure, such as a specialized database that can lessen the data modeling effort and search tools that can be used without any training. The agile BI infrastructure promises to reduce the BI backlog and simultaneously provide users with greater analysis and exploration capability to improve countless decisions made daily across an enterprise.

About Endeca
Endeca is a search and business intelligence software company that improves daily decisions for employees and purchase decisions for customers. Endecas unique innovation, the hybrid search-analytical database, enables IT organizations to deliver solutions with consumer ease-of-use on diverse and changing information. Endeca solutions drive hundreds of millions of dollars in reduced costs and increased revenues in the worlds most demanding decision-making environments in organizations like Boeing, Cox Newspapers, the U.S. Defense Intelligence Agency, Ford Motor Company, Hyatt, IBM, the Library of Congress, Texas Instruments, and Walmart.com. Headquartered in Cambridge, Mass., Endeca has operations in North America, Europe, and Asia.

12

ENDECA 101 Main Street, Cambridge, MA 02142 USA | T 617.674.6000 | F 617.674.6001 | info@endeca.com | endeca.com
FOLLOW US ON TWITTER: TWITTER.COM/ENDECA

READ OUR BLOG: ENDECA.COM/BLOGS

2010 Endeca Technologies, Inc. All rights reserved. Endeca, Endeca Latitude, Guided Navigation, InFront and MDEX Engine are trademarks or registered trademarks of Endeca Technologies, Inc. in the United States and other countries. All other product and service names mentioned herein are or may be registered trademarks or trademarks of their respective companies or organizations. Copyright 2010, 2012, Oracle and/or its affiliates. All rights reserved. Oracle Corporation Worldwide Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A.