Beruflich Dokumente
Kultur Dokumente
An Comprehensive Overview
IT 4713/6713 BI
Jack G. Zheng
Fall 2017
http://jackzheng.net/teaching/it6713/
2
Types of Information Processing
Transactional Processing
Focus on individual data item
processing: data insertion,
modification, deletion, and Analytical Processing
transmission Focus on reporting, analysis,
transformation, and decision
support
http://www.slideshare.net/fmhyudin/oltp-vs-olap-23317601
3
DIKW
Data: raw value elements or facts
Information: the result of collecting and organizing data that provides context
and meaning
Knowledge: the concept of understanding information that provides insight to
information, thus useful and actionable
http://en.wikipedia.org/wiki/DIKW_Pyramid
Analytical
Processing
Transactional
Processing
4
Examples of Analysis
How do advertising
activities affect sales of
different products bought
by different type of
customers, in different
regions? (synthesizing)
Figure from Database Processing, 13th Edition, by David Kroenke and David Auer
5
What is Business Intelligence?
Business Intelligence is a set of methods,
processes, architectures, applications, and
technologies that gather and transform raw
data into meaningful and useful information
used to enable more effective strategic,
tactical, and operational insights and decision-
making (to drive business performance).
Adapted from Forrester Report
Topic Overview: Business Intelligence, 2008
https://www.forrester.com/report/Topic+Overview+Business+Intelligence/-/E-RES39218
More BI from Forrester
https://www.forrester.com/business-intelligence
6
Additional Notes about BI
BI is the an umbrella term for a set of methods, processes,
applications, and technologies used to
gather, provide access to, analyze, and report data and information
support understanding and decision making
Broadly speaking, intelligence, or knowledge, comes from human
experience and tacit knowledge, in various format like text,
image, video, etc.
In this sense, BI is also related to knowledge management (either BI
under KM or vice versa)
http://capstone.geoffreyanderson.net/export/19/trunk/proposal/resea
rch/Knowledge_management.pdf
Narrowly speaking, intelligence comes from data (facts).
Traditional BI normally does not directly address other content
types and formats (which usually falls under artificial
intelligence).
In this sense, BI focuses on analytical data processing.
7
Data
Types of data
Numeric vs. textual
Structured vs. unstructured
Standard format vs. proprietary format
Internal vs. external data, system stored vs. file based data
Raw fact data vs. simulated/forecast/estimated data
Simple fact data vs. calculated metrics data
8
Decision Making
Decisions can be made based on
Facts, or data
Simulation (models)
Intuition, perception, sense
Group negotiation
Problem
A gap between data and knowledge (useful information leading to a
decision).
Management/operation by intuition
Lack of effective feedback and alignment systems, no improvement cycles
Need good analytical processing and models
9
BI: A General Process
The process involves analytical Results are presented and
The organization and components, such as
transformation of data delivered in different human
dimensional analysis, statistical comprehendible formats, to
into clean and common analysis, business analytics, and
models and formats. data mining, to extract support decisions. It also includes
information and knowledge. data exploration and reporting.
Data Preparation
Queries can also directly
The collection of raw present results to users
data from different The refined data will be modeled
sources by different and stored in a particular data without intensive
means, and in different management systems for quality analysis. This is usually
formats. management, easy and fast used for data exploration
access, and data profiling. and descriptive reports.
10
BI in the Decision Process
Another view in the business decision perspective
http://www.slideshare.net/junesungpark/business-process-based-analytics
11
The Evolution of BI Semantics
http://www.b-eye-network.com/blogs/eckerson/archives/2011/02/whats_in_a_word.php
12
Evolution of BI
The search for the perfect business insight system, from Performance
Dashboard, by Wayne Eckerson,
http://download.101com.com/pub/tdwi/files/performancedashboards.pdf
1980s Executive information systems (EIS), decision support systems (DSS)
2010+?? Analytics, big data, mobile BI, personal BI, in-memory, data science,
13
BI Capabilities
This is consistent with the
general BI or analytics
process but more from an
information behavior angle.
Figure from: Business Intelligence, Rajiv Sabherwal, Irma Becerra-Fernandez, John Wiley & Sons, 2011
http://books.google.com/books?id=T-JvPdEcm0oC
14
(Enterprise) BI Systems
A BI system is a computer information
system that implements (part or whole) BI
capabilities and processes
The value of BI Systems
Provide an integrated data (analytical)
processing platform
Enable easy and fast access of data and
information at all levels (raw data, analysis
results, metrics, etc.)
Streamline a controlled and managed process of
data driven decision making
15
BI System (Components) at a Glance
Data Storage Performance management
and Benchmarking
Management Presentation Applications Market research
CRM
Strategic management
Web site analytics
Website
Relational database Query Reports Reporting server
Data warehouse OLAP Information visualization Application server Users with applications
Data modeling Business analytics Dashboard BI server (browser, desktop app,
Data governance Statistics Scorecards Portal mobile app, email, etc.)
Data integration Data mining Strategy map Excel services and devices (computer,
ETL Text mining
tablet, phone, print-
Data quality Visual analytics
outs, etc.)
Metadata
Master Data
Data virtualization Management
Analytical and Delivery
Processing
16
14 Critical Capabilities of a BI and Analytics Platform
Gartner Magic Quanrant Report 2016
Infrastructure
BI Platform Administration. Capabilities that enable scaling the platform, optimizing performance and ensuring high availability and disaster recovery.
Cloud BI. Platform-as-a-service and analytic-application-as-a-service capabilities for building, deploying and managing analytics and analytic
applications in the cloud, based on data both in the cloud and on-premises.
Security and User Administration. Capabilities that enable platform security, administering users, and auditing platform access and utilization.
Data Source Connectivity. Capabilities that allow users to connect to the data contained within various types of storage platforms.
Data Management
Governance and Metadata Management. Tools for enabling users to share the same systems-of-record semantic model and metadata. These should
provide a robust and centralized way for administrators to search, capture, store, reuse and publish metadata objects, such as dimensions, hierarchies,
measures, performance metrics/key performance indicators (KPIs) and report layout objects, parameters and so on.
Self-Contained Extraction, Transformation and Loading (ETL) and Data Storage. Platform capabilities for accessing, integrating, transforming and
loading data into a self-contained storage layer, with the ability to index data and manage data loads and refresh scheduling.
Self-Service Data Preparation. The drag-and-drop, user-driven data combination of different sources, and the creation of analytic models such as
user-defined measures, sets, groups and hierarchies. Advanced capabilities include semantic autodiscovery, intelligent joins, intelligent profiling,
hierarchy generation, data lineage and data blending on varied data sources, including multistructured data.
Sharing of Findings
Embedding Analytic Content. Capabilities including a software developer's kit with APIs and support for open standards for creating and modifying
analytic content, visualizations and applications, embedding them into a business process, and/or an application or portal. These capabilities can reside
outside the application (reusing the analytic infrastructure), but must be easily and seamlessly accessible from inside the application without forcing
users to switch between systems.
Publishing Analytic Content. Capabilities that allow users to publish, deploy and operationalize analytic content through various output types and
distribution methods, with support for content search, storytelling, scheduling and alerts.
Collaboration and Social BI. Enables users to share and discuss information, analysis, and decisions via collaboration and social systems.
17
A Practical Architecture based on MSBI
18
Data Management
A special database system called data warehouse or
data mart is often used to store enterprise data
The purpose of a data warehouse is to organize lots of stable
data for ease of analysis and retrieval.
Traditional (operational) relational databases facilitate
data management and transaction processing. They
have two limitations for data analysis and decision
support
Performance
They are transaction oriented (data insert, update, move, etc.)
Not optimized for complex data analysis
Usually do not hold historical data
Heterogeneity
Individual databases usually manage data in very different ways,
even in the same organization (not to mention external data
sources which may be dramatically different).
19
Data Gathering and Integration
Enterprise level data are coming from multiple different sources, but
need to be combined and associated
Operational databases
Spreadsheets
Text, CSV
PDF, Paper
20
Analysis Tools
Descriptive reporting
Structured and fixed format reports
Based on simple and direct queries
Usually involves simple descriptive analysis and transformation of data,
such as calculating, sorting, filtering, grouping, and formatting
Ad hoc query and reporting
Business analytics
Business analytics (BA) is the practice of iterative, methodical exploration of
an organizations data with emphasis on statistical analysis.
Data mining
Data mining techniques are a blend of statistics and mathematics, and
artificial intelligence and machine-learning.
21
OLAP
Multi-dimensional queries
A dimension is a particular way (or an attribute) of describing and
categorizing data
Such queries are usually arithmetic aggregation operations (sum,
average, etc.) on records grouped by multiple dimensions
(attributes) at different aggregation levels.
OLAP is a function/operation that is optimized to answer queries
that are multi-dimensional in nature
OLAP solutions traditionally heavily rely on backend processing and
dedicated IT personnel
Descriptive and
operational report
Example analysis
"What is the total sales amount grouped by product line (dimension
1), location (dimension 2), time (dimension 3) and (other
dimensions)?"
"Which segment of business provides the most revenue growth?"
More open and
exploratory analysis
22
OLAP vs. Transaction Report
23
Basic Techniques in Business Analytics
Regression
Reasoning, estimating the relationships among
variables
Forecasting
Trend analysis, based on extrapolation of historical
data
Correlation
Relationship discovery between factors (but not
causal relationship)
Factor analysis
Determine impacting variables and their variability
24
Data Mining
Data mining (or, knowledge discovery in database,
KDD)
Processes and techniques for seeking knowledge
(relationship, trends, patterns, etc.) from a large amount
of data
Non-trivial, non-obvious, and implicit knowledge
Extremely large datasets
Data mining applications use sophisticated statistical
and mathematical techniques to find patterns and
relationships among data
Classification, clustering, association, estimation,
prediction, trending, pattern, etc.
Common techniques
Neural network, genetic algorithm, machine learning
25
Levels of Analytical Processing
A
27
Data Visualization - Charts
Dashboard
A dashboard is a visual display of the most important information needed to achieve one or more
objectives; consolidated and arranged on a single screen so the information can be monitored at a
glance. - Dashboard Confusion, Stephen Few,
http://www.perceptualedge.com/articles/ie/dashboard_confusion.pdf
A set of visualization or presentation of data views organized in a single screen/page
Scorecard
A scorecard is special type of dashboard typically with a tabular visualization of measures and their
respective targets with visual indicators to see how each measure is performing against their targets at a
glance.
http://www.dashboardinsight.com/articles/digital-dashboards/fundamentals/what-is-a-dashboard.aspx
29
Reports vs. Dashboard
http://prezi.com/qvhyfup5z7yz/dashboard-design-making-reports-pop/
Reporting and Delivery
BI reporting is about managing and
delivering analysis results to users
Live example:
https://discoverer.gsu.edu
https://sasweb1.kennesaw.edu/
SASPortal/public
Live example:
https://www.itdashboard.gov/
Figure
31 from Database Processing13th Edition, by David Kroenke and David Auer
BI Users
Super/power users
vs.
Casual users
http://eckerson.com/articles/part-iv-seven-keys-to-a-united-bi-environment
https://vimeo.com/68143902
33
Tools and Users Fit
Gartner Report,
Select the Right Business Intelligence and Analytics Tool for the Right User
Published: 23 May 2016 Analyst(s): Cindi Howson
34
BI Application Areas
BI can be applied in all businesses (industries,
functional areas, or domains) to drive business
performance in both private and public sector
Private
Retail, manufacture, real-estate, financial, sports, media,
entertainment, publication, etc.
Public (non-profit)
Education, government, healthcare, association, etc.
35
Sample BI Applications
Business management IT management
Strategic planning Web analytics
Performance management Usage analytics
Process intelligence Security management
Competitive intelligence Supply chain
Benchmarking Logistics
Marketing and sales Supplier and vendor management
CRM Shipping and inventory control
Customer behavior analysis Healthcare management
Targeted marketing and sales
strategies Insurance
Customer profiling City planning
Campaign management Traffic management
Inventory management Urban Analytics
Pricing Power usage
Financial management Education
Human resource/capital Learning analytics
HR analytics Institutional effectiveness
Talent management Internet and web
Project and program management Social analytics
Power and energy management Sports and games analytics
36
Figure from http://www.forbes.com/sites/louiscolumbus/2014/06/24/roundup-of-
BI Market analytics-big-data-business-intelligence-forecasts-and-market-estimates-2014/
Major vendors Worldwide Business Analytics Software Market Shares, 2015, IDC July 2016
https://www.sas.com/content/dam/SAS/en_us/doc/analystreport/idc-business-
analytics-software-market-shares-108014.pdf
Mega Vendors
Microsoft: SQL Server, Power BI,SharePoint, Excel
https://www.microsoft.com/en-us/cloud-platform/bi-analytics
SAP: SAP BusinessObjects BI
https://www.sapbi.com
IBM: Cognos, Watson
http://www.ibm.com/analytics/us/en/technology/business-
intelligence/
Oracle: Oracle BI 12c
https://www.oracle.com/solutions/business-analytics/business-
intelligence/index.html
SAS: SAS Enterprise BI
http://www.sas.com/en_us/software/business-intelligence.html
Other top BI tools
http://www.capterra.com/business-intelligence-software/#infographic
38
Vendor Positioning
Gartner Magic Quadrant for Business Intelligence and
The Forrester Wave Enterprise BI Platforms, Q1 2015 Analytics Platforms 2017
http://www.sas.com/content/dam/SAS/en_us/doc/analystrep http://www.jenunderwood.com/2017/02/22/2017-gartner-bi-
ort/the-forrester-wave-enterprise-bi-platforms-106893.pdf magic-quadrant-results/
More: https://www.g2crowd.com/grid_report/documents/business-intelligence-winter-2016-grid-report-final
39
BI Trends
http://www.b-eye-network.com/blogs/eckerson/archives/2011/03/bi_market_evolu.php
40
BI and the New Terms
Big data
Big Data is not a system; it is simply a way to say that you have a lot of data. https://www.linkedin.com/pulse/big-data-silver-
bullet-tomas-kratky
Big data covers non-structure and various data formats including text, blob, multimedia, etc.
Analytics
A more systematical, automated, and flexible process of data analysis for revealing insights and decision support
(Business) Analytics http://pestleanalysis.com/differences-between-business-analytics-and-business-analysis/
Data science
An interdisciplinary field about processes and systems to extract knowledge or insights from data in various forms
Focus on analysis and presentation models and methods
A good data scientist = data hacker + programmer+ analyst+ coach+ story teller+ artist
(http://analyticsindiamag.com/data-science-the-most-desirable-job-in-the-21st-century/)
All these new terms try to differentiate them from the (traditional) BI. However, if one considers BI is a dynamic
and evolving field, then all these new terms are just extensions/expansions of BI; they all still fall under the
umbrella of the general BI.
In its more comprehensive usage, BI is all of the systems, platforms, software, technology, and techniques that are
essential for the collection, storage, retrieval, and analysis of data assets within a given organization. Dataversity 2015
Report on BI vs Data Science
More perspectives
http://www.dataversity.net/distinguishing-analytics-business-intelligence-data-science/
https://www.betterbuys.com/bi/business-intelligence-vs-business-analytics/
41
Notable trends
Modern BI, or Personal (self service) BI
The business intelligence and analytics platform market's shift from IT-led reporting to
modern business-led analytics is now mainstream. http://bit.ly/2017GartnerMQbi
http://eckerson.com/articles/part-2-one-size-does-not-fit-all-customizing-self-service-
analytics-for-business-users
Embedded BI
Use of reporting and analytic capabilities directly in transactional business applications
(http://www.gartner.com/it-glossary/embedded-analytics/)
Integrating charts, reports, dashboards and self-service tools, into existing enterprise
applications through APIs
Visual BI - http://www.perceptualedge.com
Search based BI
http://www.thoughtspot.com/what-is-search-driven-analytics
https://www.youtube.com/watch?v=868-pR-cxZo
42
Personal BI
Traditional BI vs personal BI
Shifting focus from IT back to user
Personal level BI focus on self-service and ad hoc
analytic needs
More interactive and explorative
It still has fundamental BI components and provides BI
capabilities, but they are more integrated than separated
Very often work with enterprise systems
In-memory processing
Easy and efficient data models
Quick query and analysis
Flexible data manipulation
Simple delivery and sharing
43
Differences Between Traditional and
Modern BI Platforms
Analytic Workflow Traditional BI Platform Modern BI Platform
Component
Data source Upfront dimensional Upfront modeling not
modeling required (IT-built required (flat files/flat
star schemas) tables)
Data ingestion and IT-produced IT-enabled
preparation
Content authoring Primarily IT staff, but also Business users
some power users
Analysis Predefined, ad hoc Free-form exploration
reporting, based on
predefined model
Insight delivery Distribution and Sharing and collaboration,
notifications via scheduled storytelling, open APIs
reports or portal
Technology Insight for Modern Business Intelligence and Analytics Platforms
Gartner Report, October 2015
44
A Changing BI Platform
45
Embedded Analytics
46
BI Careers
Typical BI positions
BI solution architects and integration specialists
Business and BI analysts
BI application developers and testers
Data warehouse specialists
Database analysts, developers and testers
Database support specialists
BI jobs in Atlanta
http://seeker.dice.com/jobsearch/servlet/JobSearch?op=300&LOCA
TION_OPTION=2&RADIUS=80.4672&COUNTRY=1525&DAYSBA
CK=14&NUM_PER_PAGE=30&N=1525+1811&NUM_PER_PAGE=
30&FREE_TEXT=%22business+intelligence%22
https://www.linkedin.com/vsearch/j?orig=FCTD&keywords=business
%20intelligence&distance=50&locationType=I&countryCode=us&trk
=two_box_geo_fill&rsid=73997291440174363090&openFacets=L,C
&f_L=us%3A52
What are Employers Looking For?
48
Critical Knowledge and Skills
Three competencies
Technical, Business, Analytical
Technical knowledge
Knowledge of database systems and data warehousing technologies
Ability to manage database system integration, implementation and testing
Ability to manage relational databases and create complex reports
Knowledge and ability to implement data and information policies, security requirements,
and state and federal regulations
50
Where to Take BI Courses
51
BI/Analyics Education at KSU
MSIT/BSIT
IT 4713/6713 Business Intelligence
http://jackzheng.net/teaching/it6713/
IT 7113 Data Visualization http://jackzheng.net/teaching/it7113/
Certificate on data management and analytics
http://ccse.kennesaw.edu/it/programs/cert-dm.php
Other departments
Ph.D. in Analytics and Data Science
https://datascience.kennesaw.edu
ACS 8310 Data Warehousing
IS 8935 Business Intelligence - Traditional and Big Data Analytics
Certificate in High Performance Cluster Computing
http://ccse.kennesaw.edu/cs/programs/cert-hpcc.php
Data visualization overview
https://www.slideshare.net/jgzheng/business-data-visualization
Disclaimer: based on personal online search as of 2012. Please contact the instructor if there are more recent updates.
52
Our IT Graduates Working in BI
IT 4713 BI Graduates
https://www.linkedin.com/in/ydagnoko
Other IT alumni
http://www.linkedin.com/pub/amir-shahrokhi/11/a76/1b6
http://www.linkedin.com/in/tanvirspsu
53
Good General BI Resources
Organizations and communities
The Data Warehousing Institute: http://tdwi.org
Data Warehousing and Business Intelligence Organization: http://dwbi.org
54