Demand Forecasting TODAY

New challenges, opportunities in e-business environment require practitioners to step

beyond old boundaries and start marching into uncharted territory.

By Roger R. Gung, Ying Tat Leung, Grace Y. Lin and Roger Y. Tsai

Demand forecasting is a well-studied topic, and most techniques used in practice today
are relatively mature. There are many commercial software packages for forecasting
[12], ranging from stand-alone, specialized packages to modules in enterprise
management systems. One would be inclined to conclude that there is no shortage of
commercial software for demand forecasting. On the other hand, the now well-known
paradigms of just-in-time manufacturing [9], time-based competition [2], quick response
manufacturing [8], and more recently sense-and-respond [5] propose speed and
responsiveness as a major competitive advantage, and hence a goal for manufacturing
organizations. Under these frameworks, an organization can rely much less on a plan that
is based on a demand forecast. One may then argue that demand forecasting no longer
plays a critical role in the day-to-day operation of a business.

With a mature technology base, plenty of commercial software packages and a seemingly
less critical role in an organization, is demand forecasting then a "solved problem"? Our
experience has shown quite the opposite. We explore this issue by first discussing the
business environment today, and then the new opportunities and challenges to demand
forecasting presented by this new environment. As we shall see, the new opportunities
and challenges are related to the abundance of information today and the intelligent
choice and use of it. For each of several categories of information, we describe the nature
of the data, the challenges in using them in forecasting, and offer some of our
experiences whenever applicable. Finally, we emphasize that forecasting is only a means
to an end. The revenue and profit perspective is important in setting priorities.

In this article, we discuss demand forecasting in the context of predicting future

demand for products of a business organization. We focus on products of the physical
type (such as electronic components, computers or automobiles), as opposed to services
(such as long distance telephony or Web hosting). While demand forecasting for
services exemplifies some of the issues discussed here, in other ways the service sector
presents a different set of challenges suitable for another article.

The e-Business Environment

Despite the recent burst of the dot-com bubble, the new economy has indeed begun; it is
an economy of fierce competition on a global basis. The consumer or any potential buyer
has access to information any time of the day and more product choices than ever. "Word
of mouth" on the Internet worldwide will increasingly become such a strong force in the
market that any organization interested in surviving beyond a few weeks will have to
present a clear and justifiable value proposition to the consumer. And if a business cannot
deliver the value it promises, someone else will.

IBM coined the term "e-business" seven years ago. Today, 28 percent of global
companies have substantially adopted e-business technology to derive a number of
benefits — new revenue sources, better delivery of products, and significant cost savings.
IBM has recently announced the next phase of adopting today's computing resources and
global connectivity technologies to run an e-business: e-business on demand [1]. An on-
demand e-business is embodied with end-to-end integrated processes across the entire
company as well as with key partners, suppliers and customers such that the value chain
as a whole can quickly respond to changes in demand, supply, customer preferences,
market opportunity and competition.

Today, most consumers or buyers expect any respectable business to be an e-business, or

at least to have the appearance of an e-business — that is, they can contact the business
through electronic means, perform transactions with the business on the Web, and have
their historical transaction record electronically available when they call. When a buyer
attempts to purchase a product from a seller, he expects the product to be immediately
available, or at the very least be told when it will be available with a high certainty.
Otherwise he will most likely go to a competitor, which is usually one mouse click or
phone call away. Such order fulfillment capability delivered on a consistent manner over
time requires not only a lean and responsive manufacturing approach, but also serious
effort in materials, capacity and logistics planning. Demand forecasting is an initial step
for such planning and is therefore critical to the success of an e-business.

New Opportunities and Challenges in Demand Forecasting

From the viewpoint of a demand forecaster, the key opportunity presented by the new e-
business environment is the abundance and availability of information, driven by the
proliferation of information technology globally among businesses and consumers. To
exploit such an opportunity, we need to be aware of the value of the different types of
information and the subsequent exploration of it.

1. Information on demand throughput. The most well-established forecasting techniques

are based on historical demand. In today's business environment, changes in the
marketplace are swift and sudden, and may not follow the historical pattern; hence future
demand may not be predicted accurately by relying on past demand alone.

First, we note that historical demand information need not be information about the past
in the traditional sense, such as realized demand in the last month. Demand information
about the present is commonly available. For instance, predicting demand in a time
period when some customer orders are already placed can benefit from information on
the incoming orders. Take for example a manufacturer serving other businesses. If the
time period is a month, then a useful indicator might be the cumulative, month-to-date
shipment quantity, quantity sold, or quantity sold to end-users.
Shipment refers to the quantity of shipment from a manufacturing site to a distribution
center, owned either by the manufacturer or the customer. Quantity sold is the quantity of
product shipped for which revenue has been received. Quantity sold to end-users is the
quantity of products sold to the end consumers rather than the sales channels. Generally,
end-user sales information becomes available to the manufacturer at a later time than that
of the others, because it is usually provided by the sales channel — a middleman between
the manufacturer and the end consumer. For IBM servers, end-user sales data availability
generally lags behind the others by about a week. We found that for some geographic
areas and product families, end-user sales are very useful for forecasting, while for
others it can be erratic and unstable.

In some industries an initial "order" may not be a firm order; in this case the certainty of
the order estimated subjectively or using an objective criterion will be useful information.
A particular view of order information we have found useful for predicting demand is
order quantity grouped by lead-time.

Second, some information, however small an amount, about demand in the near future
may also be possible. In many industries, especially those who primarily serve other
businesses, product sales are an important and often lengthy business process. It begins
with the recognition of the need of a customer by a salesperson. It then proceeds from
initial contact, through several stages of interaction between customer and supplier
(including customer testing of some products such as large servers), to the creation and
the signing of a contract. In general, when demand is strong, the speed and quantity of
the flow of the sales process will have a corresponding indication. Such information is a
longer range "headlight" than orders on hand, as it captures the flow of opportunities
through the steps of the customer buying process.

In IBM, we refer to the identification of opportunities through their closure as sales as the
"opportunity pipeline." We have found it to be very valuable in predicting aggregate
demand or revenue, particularly in a time of change. In addition, we found it useful for
predicting the success of sales for an individual customer or group of customers with
similar attributes. This can be particularly useful for effective sales force resource
allocation and future product design.

To take advantage of the information provided by these indicators, the data must be
readily available. Clearly, such up-to-date order information would not be practical to
obtain on a regular basis without fairly extensive use of information technology.
Assuming that the raw data is available, data pre-processing and preparation is possibly
the most time-consuming process among all of the forecasting challenges discussed in
this article. For example, for IBM's own opportunity pipeline data, it requires a design of
data-recording rules and/or processes so that sales personnel know what to record for a
sales opportunity. The processes or rules need to be consistent, clear cut and easy to
execute, and they need to produce adequate data for future forecasting. Furthermore,
data collection during the execution of the sales process must be carried out consistently.
This takes the cooperation of a wide range of functional units and different levels of
management within the corporation. Also, forecasting efforts cannot start until there is
sufficient history available.

2. Information on selling price and product promotion. Changes in the selling price and
the presence of product promotions are known to have a significant effect on demand in
many industries. Today, in large part due to the proliferation of information and other
technologies, price changes are less costly. Price changes in electronic business-to-
business product catalogs or Web-based retail businesses incur little incremental cost.
Even in traditional retail stores, the day will soon come when a button on a computer is
pressed to issue a price change, and new prices will be reflected on a liquid crystal shelf
label in a physical store a few seconds later. Such opportunities imply that price changes
and promotion actions may be used very frequently, and so they can no longer be
analyzed separately from "normal" demand.

Product promotions are getting very sophisticated. Targeted marketing, and ultimately
one-on-one marketing, has created complications in the analysis of promotion effects.
The traditional way of applying a general "lift factor" to nominal demand when a certain
promotion is performed may not be adequate. At the very least, this "lift factor" needs to
take into account the promotion's target portion of the entire market, a quantity to be
estimated itself.

To enhance forecast accuracy with pricing and promotion information, a prerequisite is a

well-maintained and managed database of price and promotions corresponding to
historical demand information. This requires a disciplined process to capture the
information in a timely manner. Even more challenging is collecting and maintaining
historical data of competitors' promotions and price changes. Such information is
especially crucial in industries where products are of the commodity type. It is generally
easier to collect such information for retail businesses. For traditional brick-and-mortar
stores, organizations have long used third-party vendors to do "competitive shopping"
(i.e., to collect product prices of competitors' stores). To collect price information from
Web-based retailers, crawler software is available [7].

Although it is well publicized [6, 10] that good price and promotion management
improves revenue, it is interesting to note that it is generally not clear whether the
improvement comes from an accurate forecast of price or promotion effects, or if it is the
result of an optimization based on fairly rough forecasts.

3. Information on product life cycle. One of the serious challenges facing a demand
forecaster in the e-business environment is ever shortening product life cycles. In many
industries, a product can be expected to have a life of at most one year. As is customary,
it can inherit older history from its predecessor product, which can in turn inherit history
from its own predecessor and so on. This means that in order to get, say, two to three
years history, we need a well-organized product map over time. At this point in time, we
have found that many organizations do not have such product map data stored in a usable
manner. The upcoming industry of product life cycle management software will no doubt
provide a better infrastructure to maintain such information. However, even with a
product map, one would not go too far back since the entire business environment was
different. For many products, we have practically at most two to three years of
appropriate history.

For ease of product management, most, if not all, organizations use a product hierarchy.
The hierarchy is organized by a set of attributes of the products, such as marketing
attributes (e.g. geographical area, customer type) or technical attributes of the products
(e.g. speed or other capacity). Because individual products are phased out and new
products come in constantly over time, and the fact that the hierarchy might be
reorganized fairly frequently to reflect the fast changing business environment (e.g.
gaining or losing major customers or markets), the product hierarchy is dynamic.

With a sizable number of products in its portfolio, organizations now have a different
product hierarchy every month. This presents at least two challenges. First, we need to
find a way to obtain the history of every node in this changing hierarchy. For nodes
representing individual products, we can use inheritance as described. For nodes at levels
higher than the individual products, we need to "reconstruct" its history every time the
hierarchy is used, based on the current children of the node. Note, the history of each
child can be a constructed history of several generations. This results in a complex data
preprocessing step. Further, it is arguable that such construction may not necessarily be
the best or even correct way of handling a dynamic product hierarchy. Second, the
historical forecast made in the past no longer corresponds to the newly constructed
history or to the most current product hierarchy. It is not clear how we can obtain running
statistics of historical forecast errors. Subsequently, forecast monitoring becomes
difficult.

In planning product transition, we are interested in forecasting the demand of a single

product, not the sum of itself and its predecessor. In this case it is natural to try using the
current stage of a product's life cycle to help its demand prediction. Existing state of the
art in applying life cycle information to forecasting has not been fully proven and
demonstrated in a real business environment, partially due to a lack of organized data
mentioned before and a lack of serious effort in applying it in full scale. Fortunately,
when forecasting demand of aggregated product groups in which the rise and fall in
demand within the life cycle of an individual product compensates for each other, it is
less critical to consider life cycle information.

4. Information on the marketplace. As econometricians have long known, demand

history is only one of many streams of information from which a forecast can be made.
The e-business environment presents at least two key opportunities on forecast
information. First, high level indicators of economic activities such as total production
output of an industry are more up-to-date than previously possible. Data are collected
continuously and automatically in electronic transactions, and should also be less error
prone. This will be increasingly so as more and more business-to-business as well as
business-to-consumer transactions are performed electronically. Note that this comment
applies to retail store transactions as well, where the transaction is performed
electronically at the point-of-sale and will be recorded into some central database.
Second, more detailed economic data are available, such as those by product types within
an industry. Our experience shows that detailed data are more useful as a predictor of the
demand of a single product of an individual organization.

5. Information on consumers. We have already mentioned the use of end user sales or
consumer demand as a source of demand information. Demand history consisting of the
quantity sold, and perhaps selling price, is no longer the only piece(s) of information
coming directly from past consumers. Customer database collected by an organization
over time, previously limited only to expensive products such as mainframe computers or
automobiles, is now likely to cover regular products such as end-user software packages
or even children's products. Curiously, for these two product types, end users have very
different incentives to register with the manufacturer: for future product updates in the
form of software downloads, or product recalls. As more products incorporate elements
of software that will go through a typical life cycle of updates, customers are more likely
to register with the product manufacturer. Thus the existing customer base is no longer
characterized only by a total sales number, but rather a database of information at the
manufacturer's own request.

Customer loyalty programs, heralded by frequent-flyer programs offered by airlines,

identify the customer, who can be anonymous, whenever a product is sold. This enables
an organization to have a view of a single customer over time. One can now go to a
grocery store's Web site, enter one's loyalty card number, and retrieve a record of all one
has purchased from that store for the last 12 months. It would seem that such a trend of
having centralized information about customers over time will only continue. For
example, with the proliferation of consumer credit cards, the purchase history of a
household across many product categories is available in the transaction database of the
credit card issuer.

Perhaps even more importantly, the e-business environment provides an opportunity to

readily collect information about potential or future buyers. Along the different channels
of communication between an organization and its potential customers, most companies
now routinely log every visit to the product Web pages, every call made to an inquiry
response center and every e-mail that was received. Many organizations also use every
customer touch point as an opportunity to perform a brief customer survey to collect
information about their customers and comments on their products.

The challenge here, of course, is making use of such information to predict future
demand more reliably.

A Means to an End: The Profit and Revenue Perspective

Although forecast accuracy is one of the most important targets driving this forecasting
enhancement effort, one should not lose sight of the fact that one of the most important
purposes of doing forecasting is to improve the efficiency of the business including the
bottom-line profit. Forecasting is only a means to an end. Understanding what roles
forecasting plays in the process of optimizing profit would set the right perspective for
the forecaster and enable him to place the right resource in the right area, and to set the
appropriate goal. Our own work in this area of relating forecasting to profit optimization
[11] has generated some interesting observations. For example, for products with a high-
profit margin, the decisions on product supply and inventory to optimize profit is not very
sensitive to the forecast accuracy, when compared to products with a low-profit margin.
We also found that the "process window" for the product supply to vary, while
maintaining near-maximum profit, increases as the profit margin increases. This process
window gives a window of opportunity for the supply decision-maker to take other
factors into consideration, such as customer satisfaction, financial objectives and other
business constraints. Furthermore, the wider the window, the more room it leaves for
forecast error. That is, the tolerance or target for forecast accuracy depends on the profit
margin and the corresponding width of the process window. It is obvious from this
discussion that for a high-margin product, the forecast can be fairly inaccurate while the
product can remain highly profitable. But for a product with a low-profit margin, unless
the forecast accuracy is very high, it is hard to remain profitable.

Another observation points to the importance of forecasting the bias and variation of the
demand rather than just forecasting an average demand. In fact, not only the mean and
standard deviation of the demand needs to be estimated, but the uncertainty of the mean
and standard deviation estimate plays a role in the supply optimization. The uncertainty
of the demand estimate helps pinpoint which portion of the process window is reliable.
This analysis also quantifies the effect of forecast uncertainty on the profit, giving some
guidance to where and how resources for forecasting should be placed to maximize
profit. Without a perspective of profit versus supply given the demand uncertainty, it
would be difficult for the forecaster to know what is important to measure, what metric is
needed, and what performance target to shoot for.

Conclusion

We have offered our view of the new challenges and opportunities for demand
forecasting in the e-business environment today. Among other works, we have
developed some enhanced forecasting tools, such as a proprietary forecasting tool
(BIA), Forecast Improvement Tool (FIT), forecasting using customer surveys [4] and
diffusion-based forecasting model to predict product demand over its entire life [3].
These tools take advantage of some of the information mentioned above. We have
applied these tools to some of our diverse businesses and have experienced certain
success. We have also developed a methodology called ProfitMax [11] for supply
decision-making in several major lines of hardware products and have found
improvement on our bottom line. Our experience with these applications form the basis
for our views and observations here.

We recommend that forecasting practitioners step beyond the old boundaries and start
marching into this new territory. Look for all those indicators mentioned above in your
company and initiate effort to collect, store and maintain them. Start thinking of ways to
relate this information to future demand, but do not limit yourselves to traditional time-
series methods or linear regression models. Find ways to strengthen the forecast using
more indicators, but do not limit yourself to a linearly-weighted combination of the
multiple forecasts. There are ample opportunities to develop your own methodologies and
apply them to your business to gain the value of new information now available.
Forecasting may never be the same.

References

1. "e-Business on Demand," http://www.ibm.com/ondemand

2. J.D. Blackburn, "Time-Based-Competition: The Next Battleground in American

Manufacturing," Irwin Professional Publication, 1990.

3. R.R. Gung, Y. Jang, G. Lin, R. Tsai, "A Bass Diffusion-based Product Lifecycle and
Demand Forecasting Model," IBM Research Report, November 2002.

4. A. Heching, Y.T. Leung, J. Caruso, "Using Surveys to Understand the Present and
Predict the Future," in Proceedings of the 14th Annual Conference of the Northeast SAS
Users' Group, September 2001.

5. G. Lin et al., "The Sense and Response Enterprise," OR/MS Today, Vol. 29, No. 2,
April 2002.

6. G. McWilliams, "Dell Fine-Tunes Its PC Pricing to Gain an Edge in Slow Market,"

The Wall Street Journal, June 8, 2001.

7. R.C. Miller and K. Bharat, "SPHINX: A Framework for Creating Personal, Site-
Specific Web Crawlers," in Proceedings of WWW7, Brisbane, Australia, April 1998.

8. R. Suri, "Quick Response Manufacturing: A Competitive Strategy for the 21st

Century," Proceedings of the Quick Response Manufacturing Conference, June 2000,
Dearborn, Mich.: Society of Manufacturing Engineers.

9. C.A. Swanson, W.M. Lankford, "JIT Manufacturing," Business Process Management

Journal, Vol. 4, No. 4, 1998, pp. 333-341.

10. B. Tedechi, "Scientifically Priced Retail Goods," The New York Times, Sept. 2,
2002.

11. R. Tsai, M. Ettl, Y. Lee, J. Konopka, S. Liu, F. Cheng, "ProfitMax, Risk Management
and Supply Decision Support: Theory and Methodology," IBM Research Report,
November 2002.
12. J. Yurkiewicz, "Software Survey: Forecasting 2000," OR/MS Today, Vol. 27, No. 1,
February 2000.