AUTHOR: Sourav Das Gupta

Ensure Accuracy in Data Transformation with

Data Testing Framework (DTF)
Contents

Abstract .................................................................................... 2
Need for a Data Testing Framework .................................................. 3
DTF Overview ......................................................................... 3
What is DTF? .............................................................................. 4
Execution Steps ....................................................................... 4
Building Blocks of DTF ............................................................... 6
Software Requirements .............................................................. 7
Hardware Requirements ............................................................. 7
Benefits offered by DTF................................................................. 8
Differentiators ........................................................................ 8
Conclusion ............................................................................. 8
References ................................................................................ 9
Abbreviations and Acronyms ........................................................ 9
About the Author ....................................................................... 10
About L&T Infotech .................................................................... 10

L&T Infotech Proprietary Data Testing Framework Page 1 of 10

Abstract
Information stored in a data warehouse is critical to organizations for decision making and
predictive analysis. The huge volume of data loaded onto a data warehouse makes exhaustive
manual comparison of data impractical. The existing quality tools are either manual or have
other limitations, and do not cover all aspects of data warehouse testing. Therefore, a holistic
solution is required to test high-volume applications that are built on Data Warehouse (DW) or
Business Intelligence (BI) architecture.
L&T Infotechs Data Testing Framework (DTF) is a comprehensive, easy-to-use tool designed to
test high-volume, data-centric DW/BI applications.
This paper discusses the need for such a DTF along with its features, course of action, software
and hardware requirements. The benefits offered by DTF are also briefly discussed towards the
end of the paper.

L&T Infotech Proprietary Data Testing Framework Page 2 of 10

Need for a Data Testing Framework
Testing holds high stakes in helping businesses make insightful and intelligent decisions using
available information. Given the growing complexity in Data Warehousing and Business
Intelligence space in the IT Industry; L&T Infotech has developed a cost-effective solution to
address the following challenges faced by clients:
Unavailability of comprehensive testing tools
Varied skill sets required to understand various file formats
Voluminous data from heterogeneous sources
100 % data validation is not be feasible
Manual comparison of data is tedious and error-prone
Data Testing Framework is a testing framework that easily integrates with users needs for
different types of data validation processes. It enables users to compare and validate data across
various types of data sources and databases.

DTF Overview
DTF is L&T Infotechs Open Source data validation and comparison framework that allows a user
to perform data-centric testing. Its simple User Interface (UI) enables users to easily configure
the tool as per their testing needs. The framework also provides detailed results of the test
cases enabling faster analysis of test results.

L&T Infotech Proprietary Data Testing Framework Page 3 of 10

What is DTF?
The DTF has been developed by synthesizing years of experience in the Database Testing area.
DTF can be used for comparison of data from two different data feeds after data migration or
reconciliation. These source and target data-feeds can be database table, database query, flat
file, CSV, PSV or an Excel file.
DTF has a proven track record of comparing high volume of data and supports leading databases
in the market.
DTF can be configured to perform the following types of comparisons:
File to File comparator
File to Database table comparator
Database to Database comparator
Query output to File comparator
Database Table comparator
Database Table to Query output
Database table to Fixed Length File Comparator
Database table to XML comparator
Database table to Stored Procedure output comparator
DTF provides a user-friendly UI to the testers from non-technical background and allows them to
configure the tool to operate in different modes for different types of comparisons.

Execution Steps
Common test scenarios required for data conversion testing can be broadly classified into the
following categories:
Table/schema validation (includes the verification of indexes, stored procedures and trigger)
Count and data validation
Data character set conversion
File processing (In cases where the source is a file)
Batch job and business rule validation
Interface testing

L&T Infotech Proprietary Data Testing Framework Page 4 of 10

 Figure 1: DTF Process

The process for data testing using DTF is as follows:

1. Analyze Study the data model of the source and the target databases to understand the
conversion process. If the source is a flat file, analyze the files structure and its
mappingwith the target database.
2. Data Mapping The mapping between the source and the target databases & tables needs to
be configured in the DTF. If there are no schema changes, the mapping of the source and
target databases at database level is enough. There may be scenarios where either the data
of one source table is distributed to multiple target tables or the data of multiple source
tables is merged in one target table. In such cases, the mapping of source and the target
tables will be required to be configured in the DTF at column level.
3. Test Case Creation The test cases for various data comparison and validation scenarios can
be created in DTF using the data mappings done. DTF also provides the user an option to
create test suites and execute multiple test cases in a single framework execution.
4. Execute & Report DTF test case or test suite can be executed in DTF by providing different
run time DTF execution options. Following are some DTF execution options:
Trim Data before Comparison
Ignore case in Comparison
Database Schema Comparison
Full Database Comparison

L&T Infotech Proprietary Data Testing Framework Page 5 of 10

Once the execution is complete, a detailed report is generated which gives the following details:
Summary report
Mismatched records
Extra records in source
Extra records in Target
All reports are generated in a spreadsheet, which are detailed and convenient to analyze.

Building Blocks of DTF

Figure 2: DTF Building Blocks

DTF comprises the following three blocks:

1. DTF Util Manager - DTF Util Manager is responsible for reading/writing data into
files/databases and data conversion, if required, for internal DTF logic. It ensures that the
source and target data arein same format before data goes to the DTF Compare Engine. It
implements logic for all other activities other than actual data comparison and report
generation.

L&T Infotech Proprietary Data Testing Framework Page 6 of 10

2. DTF Compare Engine - DTF Compare Engine is responsible for actual comparison of source
and target data. If the data is huge, it divides the data into predefined sized chunks and does
the comparison. Formation of the data chunks and data comparison is done in parallel to
have faster comparison. This engine communicates with DTF Report Manager to give details
of comparison execution result.
3. DTF Report Manager - DTF Report Manager is responsible for generating DTF reports by
taking comparison execution results from DTF Compare Engine. It generates reports in excel
format. Reports are generated in two categories: summary reports and detailed reports. It
takes comparison execution time as a reference and creates folders with that name to store
reports for every execution.
In addition to the three primary blocks, DTF has the following building blocks, each of which
represents different data feeds:
Excel Files
Flat Files
Database Tables
Database Query
Excel Config file block represents configuration input excel files. Typically, a user lists the
parameters for comparison between the source and destination in these configuration file(s).
DTF Report block represents DTF summary as well as DTF detailed reports generated after
comparison execution.

Software Requirements
JRE 1.6
Microsoft Office
Windows Operating System

Hardware Requirements
1 GB RAM or greater
3 GHz CPU

L&T Infotech Proprietary Data Testing Framework Page 7 of 10

Benefits offered by DTF
DTF is a very cost-effective solution as it is developed using Open Source Tool
Detailed reports help in identifying problems Reduction in test execution effort
Reusability of the framework across different Data Warehousing projects
Less maintenance because of the modular structure of the framework
Ability to work with different types of data feeds
Easier result analysis through Excel sheets

Differentiators
Simple test script creation and execution
Tester productivity increased with improved quality of testing
Cost savings of 30%
Compressed testing cycle

Conclusion
DTF, the open source technology based framework that supports all databases currently
available in the market, creates detailed reports that help organizations identify defects and
take corrective actions based on the inputs. Enterprises are thus able to achieve cost and efforts
savings with enhanced test coverage through automation. Accurate, real-time information is
readily available to help in making informed decisions.

L&T Infotech Proprietary Data Testing Framework Page 8 of 10

References
No external references.

Abbreviations and Acronyms

DTF Data Testing Framework
DW Data Warehouse
BI Business Intelligence
UI User Interface
CSV Comma Separated Value
PSV Prefix Separated Value

L&T Infotech Proprietary Data Testing Framework Page 9 of 10

About the Author
Sourav Das Gupta, L&T Infotech
Sourav Das Gupta is an ISTQB certified tester with over 6 years experience
in the testing domain. He is currently associated with L&T Infotech as a
Test Lead. He has experience in the implementation of QA Processes, end-
to-end testing for Business Intelligence (BI) applicatons, Database Testing
and Web-based Application Testing.

About L&T Infotech

Larsen & Toubro Infotech (L&T Infotech), one of the fastest growing IT Services companies, is a
part of USD 11.7 billion L&T Group, Indias Best Managed Company with presence in the areas
of engineering, manufacturing and financial services. It is ranked by NASSCOM as 8th largest
Indian software & services exporter from India and is ranked 7th in DATAQUEST-IDC top 20 IT Best
Employers Survey 2010. It offers comprehensive, end-to-end technology solutions and services in
banking & financial services, energy & petrochemicals, insurance, manufacturing (automotive,
consumer packaged goods/retail, industrial products,) and product engineering services
(telecom). Its horizon is filled with the promise of new and cutting edge offerings in the
technology space including launch of an end-to-end cloud computing adoption toolkit and cloud
advisory consulting services; and a new service to provide enterprise mobility solutions to clients
and launch of a smart access platform. This is in addition to exciting offerings in media &
entertainment; and life sciences & healthcare.
Business solutions are also offered in SAP, Oracle, infrastructure management, testing,
consulting, domain services, business intelligence/data warehousing, legacy modernization,
applications outsourcing, architecture consulting, enterprise integration, SOA, systems
integration, PLM and software as a service.
In addition to delivering solutions to vertical industries served, L&T Infotech maintains its own
intellectual property; the flagship IP being its wealth management platform Unitrax.
L&T Infotechs unique brand differentiation is Business-to-IT Connect which enables the
Company to convert the business knowledge acquired, into a winning edge for clients, leading to
faster time to market.

For more information, visit us at www.Lntinfotech.com or email us at info@Lntinfotech.com.

Follow L&T Infotech on:

L&T Infotech Proprietary Data Testing Framework Page 10 of 10