MEMORANDUM
To: Company X
From: Mr. Ajay Yadagiri
MANAGEMENT SCIENCE ASSOCIATES, INC.
6565 Penn Avenue
Pittsburgh, PA 15206
Cc: Bernie Ryan – MSA
Subject: Evaluation of Autobox
Date: September 27, 2000
Background
Philip
Morris has a need for a sophisticated forecasting system that would have the
ability to use causal / promotional variables to forecast their weekly
shipments in order to facilitate efficient demand planning. Currently, Philip
Morris uses a system which is based on relatively less sophisticated techniques
such as smoothing which can be easily influenced by load / deload conditions
and is also incapable of handling infrequent ordering patterns in the data. PM
commissioned MSA to evaluate alternate forecasting software, specifically
Autobox, to determine its ability to meet PM’s forecasting needs.
OBJECTIVE
The
objective of this study was to evaluate Autobox , and to determine if it can
meet PM’s demand forecasting needs.
SOFTWARE
Autobox is a product of Automatic Forecasting Systems Inc. (AFS). It utilizes an automatic model building system developed by AFS. Autobox is in rarified air, in terms of the flexibility it offers the user. The user can choose a specific model or the user can ask the built-in expert system to pick the best suited Box-Jenkins model for a specific data series. The expert system follows a heuristic process to determine the best model. Autobox has several sophisticated algorithms which provide several options to the user, for eg., intervention detection for level shifts, pulses, seasonal pulses, local time trends, variance change detection and identification of lead or lag relationships.
Response time to problems ( mostly gliches ) was nothing short of remarkable reflecting a strong customer support philosophy by AFS. AFS was building or significantly enhancing it’s batch capabilities while we were performing our review. Startup issues , as you might expect were detected and promptly resolved to our 100% satisfaction.
database
Data: PM Weekly Shipment Data
Geos: Wholesaler data
Time period: 150 weeks ending 1998
13 weeks - validation period
Brands: Brands – 88 series
METHODOLOGY
The evaluation was based on two dimensions:
1) Robustness of the forecasting algorithms and logic
2) Software interface - reliability and ease of use
A subset of the 88 time series data showing different ordering patterns was selected. The selection was based on the size of the wholesaler and that of the brand packings. These series’ were representative of the data that is typically used in PM’s demand forecast system.
These series were run through the model using different user level options:
- Novice
- Advanced, No Tournament
- Advanced, Pick Best
- Combination forecasts
All 88 series were chosen to test the automated ‘batch’ capability of the software. MSA analysts conducted an evaluation based on the criteria of reliability, accuracy, clarity and ease of use.
RESULTS
Robustness of algorithms
Overall, the forecasts provided by Autobox were impressive in terms of the low prediction error from actual. The accuracy of the forecasts varied based on the options selected. However, across all the series, the error measurement was relatively small. Autobox was able to handle series with irregular demand patterns (although it did require some data manipulation) and in identifying lead/lag situations. A detailed report of forecast results is included in the appendix.
Software interface -
reliability and ease of use
In the course of the evaluation process, MSA encountered a few problems with the software. These ‘bugs’ need to rectified before Autobox can be used in a production environment. A detailed problem log is available for review.
Key benefits
- Easy to use interface
- Automated modeling
- High quality charts
- Detailed reports on forecast process
- Well written manuals / user guides
Key drawbacks
-Inability to print graphs from within Autobox – charts need to be copied to other applications to be printed
-No automatic procedure to handle series with irregular demand patterns
-No descriptive statistics, such as minimum, mean, variance, quantiles, maximum, are given for the original series, residuals, MAPE, etc…
-Data input processes not automated
RECOMMENDATION
Autobox makes the use of tedious time series techniques such as Box-Jenkins easy, by incorporating expert systems using logical heuristic processes. A lot of other vendors offer ‘the pick best option’ which is usually based on relatively unsophisticated average/smoothing based forecasts which are rigidly applied to any type of data. Autobox is unique in this respect, that it has the ability to run sophisticated causal / time series models by customizing them to each individual series. For Philip Morris, this is of great significance because of the idiosyncracies of the different types of data that is used in their forecasts, ie, small packings for small wholesalers show highly irregular demand patterns which can’t be handled by simpler algorithms. With a click of a button, the expert system automatically goes through the process of model selection, estimation, diagnostics, and reporting. Conceptually it is very easy to use and understand. As mentioned earlier, there do exist some software glitches that might hamper its functionality in a production environment. However, MSA feels that these glitches / bugs can be fixed and would strongly recommend Philip Morris to consider Autobox as a forecasting tool for their demand planning needs. It’s strong features in detecting and incorporating relocation of demand is enough to place it far in advance of its competitors. AUTOBOX can detect and remedy lag effects reflecting effect of promotions as well as lead effects reflecting reduced demand in the anticipation of promotions.