Data Warehousing with VFP

A smart, cost effective, and complete hand-coding alternative

TOD J MCKENNA

Whitepaper

Page 2

2007, Tod J McKenna; Blackstone Providence. Creative Commons Attribution 3.0 Unported license: You are free
to Share (to copy, distribute, and transmit the work) and to Remix (to adapt the work) with the following condition:
Attribution. You must attribute the work in this paper to either Blackstone Providence, or Tod J McKenna.
For any reuse or distribution, you must make clear to others the license terms of this work.
Any of the above conditions can be waived if you get permission from the Tod J McKenna.
Nothing in this license impairs or restricts the author's moral rights.
While every attempt has been made to ensure that the information in this document is accurate and complete, some
typographical errors or technical inaccuracies may exist. Tod J McKenna or Blackstone Providence does not accept
responsibility for any kind of loss resulting from the use of information contained in this document. The information
contained in this document is subject to change without notice.

Data Warehousing with VFP

TOD J MCKENNA

Page 3

Table of Contents:
Table of Contents:......................................................................................................... 3
Abstract ......................................................................................................................... 4
Business Intelligence and the Data Warehouse ............................................................ 5
The Hand-coding Alternative ....................................................................................... 6
Why FoxPro? ................................................................................................................ 7
From Here ..................................................................................................................... 8

Data Warehousing with VFP

TOD J MCKENNA

Page 4

Abstract
The purpose of this whitepaper is to discuss why a data warehousing
solution can be implemented using VFP. Data Warehousing is a process
that ultimately consolidates, federates, or propagates an organizations data
in order to create a single version of the truth for delivering Business
Intelligence. Business Intelligence is a broad term used to describe an
organizations need to turn raw data into valuable information for supporting
decisions. The data warehouse can be used for extracting and exploring
interesting information patterns and for gaining valuable business insights
that can assist management in decision-making. Traditionally, data
warehousing has been the domain of expensive and feature-rich database
management systems and companies (Informatica, Microsoft, Business
Objects, IBM, et. al.). However, almost half of all projects are still handcoded. In this paper, hand-coding data warehouse technology is explored
through the lens of Visual FoxPro: A smart, cost effective, and complete
hand-coding alternative.

Data Warehousing with VFP

TOD J MCKENNA

Page 5

Business Intelligence and the Data Warehouse

Data Warehousing is a process that ultimately consolidates, federates, or
propagates an organizations data in order to create a single version of the
truth for delivering Business Intelligence. Business Intelligence is a broad
term used to describe an organizations need to turn raw data into valuable
information for supporting decisions. In todays dynamic and highly
competitive business environment, an organization must have the ability to
analyze, plan, and react to rapidly changing conditions. To achieve this, top
managers, analysts, and other decision-makers depend on accurate and
timely information.
Many organizations in the past decade have turned to data warehousing as
a strategic tool to aid in decision-making. The trend continues. This has
been compelled by the need to integrate data from disparate OLTP (Online
Transaction Processing) systems into one single homogenous whole: to
create a single version of the truth in which all important decisions can be
made. Organizations who achieve this gain a competitive advantage. Some
challenges that these organizations face when deciding to implement a data
warehouse include determining:
how to catalog and discover usable source data;
what database software should be used;
what tools will need to be purchased and what in-house tools
can be leveraged;
what human resources the project will need for development
and maintenance;
what skills and training the staff will need;
the budget;
how to fix data quality issues that may exist throughout the
organization;
how to conform business rules, metrics, and definitions so
that an apple becomes an apple and an orange stays and
orange.
None of the items listed above are trivial matters. Whats more, the decision
to incorporate a data warehouse into daily operations will fundamentally
change the organization.
Over the years, data warehousing has been limited to large companies due
to the enormous costs, time, and risks involved in the implementation.
Cursory research conducted by Pegasys Systems and Technologies
revealed that most vendors (Trillium, Informatica, IBM, and others) price
solutions from 36k to 150k. This is just to get started and does not include
education, resource allocation, licensing, and maintenance.
And despite best efforts by the companies who invest this kind of money,
more than 80% of all data warehousing projects overrun time and cost
estimates. This, according to the Standish Group report Migrate
Headaches. These costs, which can easily run into the millions of dollars,
can be divided into four key data warehousing components:
Data Profiling and Quality

Data Warehousing with VFP

TOD J MCKENNA

Page 6

Integration software
Database software
Dashboards, analysis and reporting tools

Because of these costs, small and medium sized businesses are generally
faced with four choices:
1. Purchase software from vendors, distributing the costs for
each component listed above across many years
2. Hand-code the entire solution using scripting languages
(PHP, VB, Java) and inexpensive database servers (MySQL,
PostgreSQL)
3. Use a hybrid of the two (hand code profilers and integration,
but purchase analysis tools for example)
4. Forego or defer developing a Business Intelligence solution

The Hand-coding Alternative

Many small and medium-sized organizations, who have recognized their data
as a valuable asset, decide to hand-code Profiling, Integration, Reporting,
and Analysis solutions using available in-house tools. In fact, almost half of all
data warehousing projects are hand-coded (according to a TDWI Forrester
Research survey). A web-design firm might leverage their technicians PHP
and MySQL skills, for example. A company with trained VB developers goes
out and purchases Microsoft SQL Server 2005 to build a hybrid solution.
More often than not, though, companies simply turn away from the BI
bandwagon, relying on their Microsoft Excel sheets, gut, and inadequate
reports to make key company decisions. How do they know if last weeks
promotion was the true reason sales increased? How do they know if they
can afford to send their salespeople to a training in the next state? How do
they analyze their shipping costs against time of the day, month of the year,
or the weather?
Hand-coding is an accepted and proven alternative. Organizations can utilize
their in-house resources, use licensed software that they have already paid
for, and for the most part keep their warehousing initiatives coupled tightly to
their OLTP sources (which is important for compatibility, performance, and
understandability). In addition, off-the-shelf solutions are usually too
excessive and feature-rich for the needs of the organization; hand coding
forces developers to code only what is needed.
Another variable worth mentioning is the effort involved in selecting an offthe-shelf product. To properly research, purchase, setup, and learn these
tools could take anywhere from 3, 6, or even 9 months! In 9 months, a handcoded solution could be well under way, not just starting.
This isnt to say that buying proven industry warehousing products doesnt
have merit or value. In fact, if you can afford to procure a solution from a
company like Informatica, then chances are you will be satisfied with both the
product and the service. These vendors do a great many things right and
their pioneering and research has added value and excitement to business

Data Warehousing with VFP

TOD J MCKENNA

Page 7

intelligence, but unfortunately for most, the total cost of ownership is simply
out of reach.
And heres a little secret about off-the-shelf warehousing products: You need
to code them too! For example, SSIS makes a great pipe (as a colleague
recently put it), but beyond that, you are forced to engineer the rest of the
plumbing: writing scripts, SQL statements, and expressions. You cant
escape writing code.

Why FoxPro?
Data Warehousing's best-kept secret might be Visual FoxPro 9. VFP9 is a
data-centric, object oriented language that is fast, reliable, and easy to learn
and use. VFP is mature, will not break the budget, and it contains all the
tools necessary natively to create and maintain a data warehousing
solution. On top of that, it integrates well with 3rd party databases (Oracle,
SQL Server, MySQL, etc.). It allows you to extend the IDE, create your own
builders and wizards, and customize the environment to suit your needs.
You could, for example, create toolbox categories with relevant classes and
components, wizards and builders to automate heavy lifting, custom
adapters to connect to and process source data; and schedulers to run tasks
when needed. You can multithread and consume web services. You can
chain reports. You can build custom queries. And on and on.
(It is suspected by some VFP developers that Microsofts decision to stop
developing FoxPro is due to its strong potential in this market. If
organizations knew that they didnt need Microsoft SQL Server components
such as SSIS or SSRS, but could accomplish much more for much less
using VFP, they would jump at the chance!)
Microsoft will continue standard support for VFP9 through January 2010 and
extended support through January 2015. Windows 7, the next generation of
Windows, will support 32-bit applications (see http://news.zdnet.com/21009593_22-6197943.html). If it is any measure, Fox2.6 applications still exist
and run today, long after support has ended.
So what are some advantages to using VFP9 as a hand-coding solution?
Heres a list of the items I feel give it an edge:
1. VFPs position as a proven RAD tool facilitates the prototype-iterate
cycles that accompany warehouse development and ever-changing
requirements. An entire package could be developed and deployed
in an afternoon (extracting data, cleansing it, conforming it, and
delivering it). A dimensional model can be adjusted easily by using
the database designer, table designer, and query designer.
2. Hundreds of commands exist to manipulate data. There are over 60
string functions alone, and with the ability to develop Visual FoxPro
library (.fll) files, there really is no limit to what you can transform.
3. VFP licensing is not prohibitive. You can purchase a license to the
full version, off the shelf, for around $600.00.

Data Warehousing with VFP

TOD J MCKENNA

Page 8

4. Visual FoxPro has long been praised for its tremendous speed.
Rushmore technology, together with great string processing
capabilities, makes VFP9 one of the fastest data processing
languages out there.
5. You can combine set (SQL) and procedural syntax in a single
method or program, giving you amazing flexibility when it comes to
performance, tactics, and readability.
6. EXECSCRIPT and macro substitution allows you to write business
logic scripts in isolation, and gives you tremendous flexibility when
developing metadata driven solutions.
7. The native database capability allows you to work seamlessly with
staged data, log files, metadata, and the warehouse itself. Even if
you use other software (Oracle, for example) for your database
engine, the connections are simple, fast, and reliable.
8. Upsize your working prototype to SQL Server when all prototypeiterate cycles are complete. You may decide early on that using SQL
Server is right for your organization. FoxPro ships with an Upsizing
Wizard that allows you to convert your native FoxPro databases to
SQL Server. It isnt too difficult to move over to Oracle or MySQL
either.

From Here
I invite you to visit my blog, Tod means Fox at http://blog.todmeansfox.com.
There, I discuss various data integration and warehousing issues from a VFP
perspective (among other things). There are articles about integration,
quality, profiling, data conforming, and much more.
Also, I have published 3 lengthy articles in Advisor Guide to Microsoft Visual
FoxPro (http://advisor.com/pub/MSVisualFoxProAdvisor) about warehousing
data using FoxPro. These articles detail an approach to dimensional
modeling and ETL, using only VFP.
You may contact me directly using the contact form at Blackstone Providence
(http://www.blackstoneprovidence.com); through my personal e-mail
tod@grengama.com; or via Blackberry: bb@todmckenna.com.

Understanding what a data warehouse is and how it functions is critical for

building Business Intelligence solutions. While Visual FoxPro is a great tool
for developing OLTP systems, it can also be used to create all the
components of a true data warehouse. For small and medium sized projects,
it is a robust, sensible, and efficient way of providing Business Intelligence.
Even in larger environments, it can be useful for initial development,
prototyping, and testing of different data warehouse designs and
implementations.

Data Warehousing with VFP

TOD J MCKENNA