The M3 Forecasting Competition Calculations were off for Monthly Data
Guess What We Uncovered ? The 2001 M3 Competition's Monthly calculations for SMAPE were off for most of the entries. How did we find this? We are very detailed.
14 off the 24 to be exact. The accuracy rate was underestimated. Some entries were completely right. ARARMA was almost off by 2%. Theta-SM was off by almost 1%. Theta-SM's 1 to 18 SMAPE goes from 14.66 to 15.40. Holt and also Winter were both off by 1/2%.
The underlying data wasn't released for many years so this made this check impossible when this was released. Does it change the rankings? Of course. The 1 period out forecast and the averaged of 1 to 18 are the two that I look at. The averaged rankings had the most disruption. Theta went from 13.85 to 13.94. It's not much of a change.
The three other seasonalities accuracies were correctly computed.
if you saw our release of Autobox for R, you would know that Autobox would place 2nd for the 1 period out forecast. You can use our spreadsheet and the forecasts from each of the competitors and prove it yourself.
See Autobox's performance in the NN3 competition here. SAS sponsored the competition, but didn't submit any forecasts.