M3 Competition; Description of Data and AUTOBOX forecasts

Introduction

The Institute of International Forecasters (IIF) is conducting a third competition which is designed to overcome the glaring deficiencies of the two prior competitions. Prof. Makridakis , the M in the M3 title has collected Monthly (M) , Quarterly (Q) , Annual (A) and Other (O) series ... 3003 in all. AFS includes here a brief description and summary of these series and the call for participants to prepare forecasts. AUTOBOX was used to analyze , automatically without any human intervention, all

How AUTOBOX was used in this competition

AUTOBOX was set up in a BATCH mode and conditions were set under which modelling was to be performed. Each time series was analyzed under four sets of conditions and the model selected was the one that minimized the error sums of squares. This is equivalent to maximizing R SQUARED or the AIC. The four conditions were: 1. Perform ARIMA modelling first then do INTERVENTION DETECTION, allowing local time trends to be identified. 1. Perform ARIMA modelling first then do INTERVENTION DETECTION , NOT allowing local time trends to be identified. 3. Perform ARIMA modelling second after INTERVENTION DETECTION, allowing local time trends to be identified. 4. Perform ARIMA modelling second after INTERVENTION DETECTION, NOT allowing local time trends to be identified. The best of these four approaches was then declared the "winner" and its forecasts saved for submission to IIF

How long did AUTOBOX take to perform the 3003 analysis ?

Since this analysis was done while multi-tasking, it was impossible to exactly estimate the time required. We ran it on a Pentium 133 with 16mg ram. Print options were set to a minimum as no details were required for this production run. The time to develop a model varied as a result of the length of the series and the strength of the model structure and of course the presence of anomalies or outliers, be they Pulse, Level Shift, Seasonal Pulse or Time Trends. We would estimate that if AUTOBOX had been run without any other tasks active the run time would have been around 4 to 6 hours.

When will they get it right ?

The evaluation of a forecasting method , even with 3003 different time series still fails to provide generality due to the design of the comptetion. The single largest confusion in measuring and conducting forecasting competitions is the confusion between forecast errors from a single origin and forecast errors for different lead times. Single origin forecasts generate a correlated set of forecasts due to the inherent bootstrapping procedures. That is to say the forecast error for one period out is correlated with the forecast error for two periods out, etc. To correctly measure forecast errors one has to compute k period projections from n origins. In this way, one gets n independent measures of one period out erors, two period out errors, etc. This requires an iterative process where the modeller is given a set and asked for a k period forecasts and is then given 1 new value and is asked to return another set of k period forecasts. In this way, the effect of the origin or launch is designed out by virtue of the n replications. The developers of the M3 competition could have done this by scaling and coding these series thus masking the data and defeating any attempt to "cheat". A more important point is the continued silliness inherent in auto-projective models, i.e univariate models. The history of a series never causes or is responsible for the future. It is simply a surrogate for the omitted "cause" series. Box and Jenkins not only codified "rear-window driving" models (ARIMA) but developed a rigorous approach to causal modelling known as Transfer Functions. Transfer Functions are simply distributed lag models which are optimally tuned to the data. By extracting the impacts or elasticities associated with causals or exogenous series one can project using the drivers rather than the rear-view mirror.

Until both of these issues are spoken to the question of which approach or model is optimal will remain unanswered.

M3-International Journal of Forecasting Competition

The M3-IJF Competition will compare the forecasts produced by a variety of extraoplative forecasting methods. Participants from multiple disciplines will use techniques in which they have expertise to forecast 3003 time series. Resulting forecasts will be compared with actuals using multiple error measures.

The Series

The 3003 time series are distributed as follows


Types of Time Series Data
Time Interval	Micro	Industry	Macro	Finance	Demographic	Other	Total
Yearly	146	102	83	58	245	11	645
Quarterly	204	83	336	76	57	0	756
Monthly	474	334	312	145	111	52	1428
Other	4	0	0	29	0	141	174
Total	828	519	731	308	413	204	3003

For each time series the following will be available

title
starting date
type of series (macro, micro, demographic, financial, others)
time interval (yearly, quarterly, monthly, others)
historical values

For those who want more details and to download the original distribution go to M3-IJF Competition Data.

Click here to return to the home page

P.O. Box 563
Hatboro, PA 19040
Tel: (215) 675-0652
Fax: (215) 672-2534

sales@autobox.com

CLICK HERE:Home Page For AUTOBOX