1 Data Warehouse Development Methodology

This chapter will explain the methodology of building a data warehouse. In software engineering, the discipline that studies the process people use to develop an information system is called the system development life cycle (SDLC) or system development methodology. The waterfall methodology is called the sequential methodology. The iterative methodology is also known as the incremental methodology. Waterfall Methodology: 1. 2. 3. 4. 5. 6. 7. 8. feasibility study Requirements architecture design development testing deployment and operation

There are other variations of the step names as: proposal, analysis, business requirements definition, technical architecture, functional design, technical design, unit testing, system testing, integration testing, user acceptance testing, rollout, implementation, maintenance, and support. Feasibility study: You gather the requirements at a high level (for example, determining why you need a data warehouse and whether data warehousing is the right solution), you have a quick look at the source systems to find out whether it is possible to get the data you need, you get sample data to assess the data quality, and write a proposal (some people prefer to call this the business case). Important things to mention in the proposal or business case are: i. Benefits ii. How long it will take iii. How much it will cost iv. And the business justifications v. Requirements (Summary) vi. The project organization vii. Project plan Feasibility: I find it best to set the feasibility step as its own project on separate budget. In this phase, the work is to analyze the requirements and data volume at a high level and determine suitable architecture and toolset so you can produce a rough estimate. For this work you will need a business analyst to analyze the requirements, a data warehouse architect to determine the architecture and a project manager to produce the estimate and high-level project plan. Other types of resources that may be required during the feasibility study phase are a hardware and infrastructure person (to verify the details of the proposed architecture), a business owner (such as a customer service manager), and the source system DBA (to get the information about data volume). Requirements: You talk to users to understand the details of the processes, the business, the data, and the issues. Arrange site visits to get firsthand experience. Discuss the meaning of the data, the user interface, and so on, and document them. You also have to list the nonfunctional requirements such as performance.

2
Architecture: You need to determine which data flow architecture you are going to use and what system architecture you are going to use in detail, including specifications for database servers, the type of network, the storage solution, and so on. Design: You need to design the three main parts of the data warehouse system: the data stores, the ETL system, and front-end applications. Development: You need to build the three parts that you design: data stores, ETL system and front-end applications. Testing: Basically, you need to test the data stores, the front-end applications and the ETL system. Deployment: Once the system is ready, you put all the components in the production boxes: the ETL system, the data stores, and the front-end applications. You do the initial load; that is, you load the data from the source system for the first time. You do a few tests and the front-end applications are producing the correct figures. You produce a user guide, the operational guide, and the troubleshooting guide for the operations team. You train the users and the operational team. You support the system, the users, and the operations team for a few weeks. Operation: The users continue to use the data warehouse and the application. The operations team continues to administer the data warehouse and to support the users. Infrastructure setup: One of the biggest tasks when you build an application is to prepare the production environment where you are going to run the application and to build the development and testing environments.

The Amadeus Entertainment Case Study:

Lets first assume that the solution for this project to build four data marts: Sales, Purchasing, Inventory, and CRM. We will be using the NDS + DDS data architecture. We have a PM, a DWA, business analyst, and two developers. We will deliver functionality by building the easiest data mart (purchasing). The BA and DWA will work together to design and create part of the NDS and DDS databases required by the purchasing data mart. They will also specify the front-end application for three reports and one OLAP cube. The two developers will build the ETL and the report and set up the CUBE. The data warehouse system will extract the purchase order table from the Jupiter, Jade and WebTower9 systems every day and load into the NDS database. The DDS ETL will get the data from the NDS database. At this time, the NDS will contain perhaps seven tables, and the DDS will contain perhaps five tables only the necessary entities, fact tables, and dimension table required to support the purchasing data mart to function properly.

3
At the heart of the iterative methodology you will do all the system components (ETL, stage, NDS, DDS, DQ, control, cubes, reports) including the training and the handover to the operations team but on a smaller scale.