Link to the original text:http://tecdat.cn/?p=22849
When it is necessary to choose the most suitable prediction model or method for data, forecasters usually divide the available samples into two parts: inner samples (also known as “training set”) and reserved samples (or outer samples, or “test set”). Then, the model is estimated in the sample, and some error indicators are used to evaluate its prediction performance.
If such a program is done only once, it is called a “fixed origin” assessment. However, time series may contain outliers, and a poor model may perform better than a more suitable model. To enhance the evaluation of the model, we use a method called “rolling origin”.
Rolling origin is a prediction method. According to this method, the prediction origin is continuously updated, and the prediction is generated by each origin (tashman 2000). This method allows the prediction errors of several time series to be obtained, so as to better understand the performance of the model.
How to achieve it?
The figure below describes the basic idea of rolling the origin. The white cell corresponds to the data in the sample, while the light gray cell corresponds to the prediction of the first three steps. There are 25 observations in the time series, and the prediction starts from 8 origins and starts from 15 origins. The model is re estimated in each iteration, and the prediction results are produced. After that, a new observation is added at the end of the series, and the process continues. When there is no more data to add, the process stops. This can be considered as a rolling origin with a fixed reserved sample size. The result of this program is eight one to three-step predictions. On this basis, we can calculate the error measurement method and choose the best model.
Another option to generate predictions from eight origins is to start at origin 17 instead of 15 (see figure below). In this case, the program continues until the origin 22, that is, when the last three-step ahead prediction is generated, and then continues with a decreasing prediction range. Therefore, two-step prediction is generated from origin 23, and only one-step prediction is generated from origin 24. So we get eight one-step predictions, seven two-step predictions and six three-step predictions. This can be considered as a rolling origin with an unfixed reserved sample size. It can be used in the case of small samples when we have no redundant observations.
Finally, in both cases, our sample size is increasing. However, for some research purposes, we may need a constant internal sample. The figure below shows such a situation. In this case, in each iteration, we add an observation at the end of the series and delete an observation (dark gray cell) from the beginning of the series.
R implementation: a case study of ARIMA
R realizes the rolling origin estimation of any function, has a predefined call, and returns the expected value.
Let’s start with a simple example, generating a sequence from a normal distribution.
x <- rnorm(100,100,10)
We use ARIMA (0,1,1) in this example.
The call includes two important elements: data and H. Data specifies the position of the values in the sample in the function we want to use. H will tell our function that the range of prediction is specified in the selected function. In this example, we use ARIMA (x = data, order = C (0,1,1)) to generate a desired ARIMA (0,1,1) model, and then we use predict (…, N. ahead = h) to generate a prediction from the model.
You also need to specify what the function should return. It can be conditional mean (point forecast), forecast interval, model parameters. However, depending on what the function you use returns, the rolling prediction returns something different. If it is a vector, the rolling prediction will produce a matrix (with values for each origin in the column). If it is a matrix, then an array is returned. Finally, if it is a list, then a list of lists is returned.
Let’s first collect the conditional average from the predict() function.
We can use the rolling origin to generate predictions from the model. For example, we want three-step prediction and eight origins, the default values of all other parameters.
predro(x, h , orig )
This function returns a list of all the values we requested, plus the actual values of the reserved samples. We can calculate some basic error indicators based on these values, such as the proportional average absolute error.
apply(abs(holdo - pred),1,mean) / mean(actual)
In this example, we use the apply() function to distinguish different forecast periods and understand the performance of the model in each forecast period. In a similar way, we can evaluate the performance of some other models and compare the errors generated by the first model. These numbers don’t mean much by themselves, but if we compare the performance of this model with that of another model, we can infer whether one model is more suitable for the data than the other.
We can also plot predictions from the rolling origin.
In this case, predictions from different sources are close to each other. This is because the data is stable and the model is quite stable.
If we look at the returned matrices, we will notice that they contain missing values.
This is because by default, reserved samples are set to non constant. The inner samples are also set to non constant, which is why the model will re estimate the added samples in each iteration. We can modify that.
predro(x, h , ori )
Note that the values of return2 and Return1 cannot be directly compared because they are generated from different starting points. This can be seen in our drawing.
If you use the functions in the prediction package, you can modify the call and return values in the following ways.
"forecast(ets(data) ,level=95" c("mean","lower","upper")
Multiple time series Arima case
Rolling prediction is a convenient method when you have a model and a time series. But what if you need to apply different models to different time series? We will need a cycle. In this case, there is a simple way to use rolling prediction. Now let’s introduce a few time series.
For this example, we need an array of return values.
Here, we will have three time series, two models and three-step lead predictions from eight sources. Our model will be saved in a separate list. In this example, we will have ARIMA (0,1,1) and ARIMA (1,1,0).
We return the same prediction from the function, but we need to change the way we call it, because now we have to take these two different models into account.
Instead of specifying the model directly, we use the ith element in the list.
We also want to save the actual value from the reserved sample so that we can calculate the error.
This array has three time series and three-step ahead prediction dimensions from eight origins.
Finally, we can write a cycle and produce a prediction.
for(j in 1:3) for(i in 1:2)predro(data, h , or=8)
Compare their performance in different time series.
exp(mean(log(apply(Holdout - Fore / apply(abs(Holdout - Fore ))
Therefore, based on these results, we can conclude that ARIMA (0,1,1) is more accurate than ARIMA (1,1,0) on average in our three time series.
Linear regression and arimax cases
In our final example, we create a data frame and fit the linear regression.
Note that in this example, the regression implemented in the LM () function depends on the data framework and does not use the prediction range.
In addition, the function predict. Lm() returns a matrix with values, not a list. Finally, we call rolling prediction.
pred(y, h , ori )
In this case, we need to provide the dependent variable in the data parameter of the call, because the function needs to extract the value of holdout
predict(lm( xreg ,new =xreg " predro( $y, h , or ) plot( Return)
As a final example, we consider the arimax model for the following data.
And modify the call accordingly.
ourCall <- "predict(arima(x=data, order=c(0,1,1), xreg=xreg\[counti,\]), n.ahead=h, newxreg=xreg\[counto,\])"
Considering that we are dealing with ARIMA, we need to specify both data and H. In addition, xreg is different from the previous example because it should not contain dependent variables now.
If you use ETSX model, the call can be simplified as follows:
"es(x=dat, xreg, h=h"
Finally, all of the above examples can be completed in parallel, especially when the data is large and the sample size is large.
Davydenko, Andrey, and Robert Fildes. 2013. “Measuring Forecasting Accuracy: The Case of Judgmental Adjustments to Sku-Level Demand Forecasts.” _International Journal of Forecasting_ 29 (3). Elsevier B.V.: 510–22. https://doi.org/10.1016/j.ijforecast.2012.09.002.
Petropoulos, Fotios, and Nikolaos Kourentzes. 2015. “Forecast combinations for intermittent demand.” _Journal of the Operational Research Society_ 66 (6). Nature Publishing Group: 914–24. https://doi.org/10.1057/jors.2014.62.
Most popular insights