QUESTION: 

allstat@mailbase.ac.uk From: Franz Walter Klein X-List: allstat@mailbase.ac.uk X-Unsub: To leave, send text 'leave allstat' to mailbase@mailbase.ac.uk Reply-To: Franz Walter Klein list Hello list members. I need to eliminate outliers from some 15.000 time series. So I am looking for a reliable method to do this automatically. There seems to be a compound smoothing transformation that is called "T4253H" Method in SPSS X 4.0 but there is no description on how it works. I found nothing about it in the docs nor on the internet. # Can you tell me how that method works? # Can you recommend any other method for (automatically) eliminating outliers. Any help much appreciated may it be big or small. Cheers Franz

ANSWER: 

Unfortunately, I can not tell you how their proprietary smoothing works, but I can refer you to publicly available procedures from the statistical literature, particularly Bell, Chiang, Tiao and Tsay among others. The problem or opportunity here requires the classic decomposition of observations to signal (prediction) and noise. In order to assess or declare " an unusual value " one must develop " the expected or usual value". Time series techniques extended for outlier detection , i.e. intervention variables like pulses, seasonal pulses, level shifts and local time trends can be useful in "data cleansing" or pre -filtering of observations. The pre-filtering or data cleansing is often done in tasks unrelated to forecasting. Note that AUTOBOX has been extended to adjust the original series for identified outliers providing a "cleansed series". This is not the same as the fitted values from a time series intervention model. Please see a small writeup of work being done by the Swiss Government on some 15,000 time series each month. Yes, by chance the number of series is the approximately the same. You can get info on this from our web site http://www.autobox.com/educatio.html and pursue the link to Case Study #1: MODELLING UNIT -VALUES SERIES: A TIME SERIES APPROACH TO IMPROVE PRICE INDICES IN FOREIGN TRADE.  Another in a series of educational case studies presented by AFS. This case study, written by Winfried Stier of the University of St. Gallen in conjunction with the Swiss federal Custom Agency, examines how time series techniques can be used to adjust government data series. Prof. Stier can be reached at the University of St. Gallen in Switzerland 41 (071) 224 23 17 or via email at klaus.edel@few.unisg.ch or you can go direct to http://www.autobox.com/ozd.html David Reilly AUTOMATIC FORECASTING SYSTEMS (AFS) _____________________________________________________________________ First, the processing of outliers and the smoothing of times series are two quite different things. Second, there is a vast amount of literature on both -- try signal processing (e.g. IEEE Trans. on Signal Processing). --VS I strongly disagree with this as it would appear that the above author thinks that time series are analyzed without the explicit identification and incorporation of outliers. Early researchers found that the methods of times series analysis needed to be extended to make them more robust in the presence of deterministic structure. One can not nor should not study time series data without being sensitive to outliers. Outliers can be one-time onlies ( poor word ! ) or seasonal pulses or a sequential set of outliers with nearly the same magnitude and direction (level shift) or local time trends. A pulse is a difference of a step while a step is a difference of a time trend. We include all in our tour de force. The sample ACF, PACF , Power Spectrum are all effected by outliers thus the need to deal with both kinds of structure, i.e. stochastic and deterministic, is mandated.