# Iterative Forecasting with Nested Ensembles

> Iteratively forecast with nested modeling

Why is nested forecasting important? For starters, the ability to iteratively forecast time series with many models that are trained on many individual groups has been a huge request from students in our Time Series Course. Why? Because two methods exist that get results:

1. Global Modeling: Best for scalability using a Global Models and a Panel Data structure. See Forecasting with Global Models.

2. Iterative Forecasting: Best for accuracy using a Nested Data Structure. Takes longer than global model (more resources due to for-loop iteration), but can yield great results.

We’ve incorporated a new approach called “nested forecasting” to help perform Iterative Forecasting.

## What is Nested Forecasting?

The core idea of nested forecasting is to convert a dataset containing many time series groups into a nested data set, then fit many models to each of the nested datasets. The result is an iterative forecasting process that generates Nested Modeltime Tables with all of the forecast attributes needed to make decisions.

## What is Nested Ensembling?

Nested ensembling applies the concept of ensembling, which is generally averaging many individual models (called submodels) to produce a more stable model that sometimes improves over the best individual model.

We can apply the ensembling techniques to iterative or nested forecasting. In this tutorial, we will show you how to perform:

- Average Ensembles using `ensemble_nested_average()`. These are the simplest models.

- Weighted Ensembles using `ensemble_nested_weighted()`. These allow the user to provide “loadings” to distribute the weighting to the top models, which can sometimes improve over the simple average ensembles.

Let’s go!

# Nested Ensemble Tutorial

We’ll showcase nested ensembling for iterative forecasting in this short tutorial.

## Libraries

Load the following libraries.

    library(tidymodels)
    library(modeltime)
    library(modeltime.ensemble)
    library(dplyr)
    library(timetk)
    library(gt)

nested-ensembles.R

## Data

Read in the Walmart Sales Weekly data (comes with `timetk`).

    data_tbl <- walmart_sales_weekly %>%
        select(id, date = Date, value = Weekly_Sales) %>%
        filter(id %in% c("1_1", "1_3"))

    data_tbl

    #> # A tibble: 286 × 3
    #>    id    date        value
    #>    <fct> <date>      <dbl>
    #>  1 1_1   2010-02-05 24924.
    #>  2 1_1   2010-02-12 46039.
    #>  3 1_1   2010-02-19 41596.
    #>  4 1_1   2010-02-26 19404.
    #>  5 1_1   2010-03-05 21828.
    #>  6 1_1   2010-03-12 21043.
    #>  7 1_1   2010-03-19 22137.
    #>  8 1_1   2010-03-26 26229.
    #>  9 1_1   2010-04-02 57258.
    #> 10 1_1   2010-04-09 42961.
    #> # ℹ 276 more rows

nested-ensembles.R

We can get a quick visual of the two time series we will forecast.

    data_tbl %>%
      group_by(id) %>%
      plot_time_series(date, value, .facet_ncol = 1, .interactive = FALSE)

nested-ensembles.R

## Prepare the Data in Nested Format

The most critical stage in “Nested Forecasting” is data preparation, making sure that the input to the nested forecasting workflow is in the appropriate structure. We’ve included several functions to help that involve a bit of forethought that can be broken into 3 steps:

1. Extending each of the times series: How far into the future do you need to predict for each time series? See `extend_timeseries()`.

2. Nesting by the grouping variable: This is where you create the nested structure. You’ll identify the ID column that separates each time series, and the number of timestamps to include in the “.future_data” and optionally “.actual_data”. Typically, you’ll select the same `.length_future` as your extension from the previous step. See `nest_timeseries()`.

3. Train/Test Set Splitting: Finally, you’ll take your `.actual_data` and convert into train/test splits that can be used for accuracy and confidence interval estimation. See `split_nested_timeseries()`.

Here are the 3-steps in action:

    nested_data_tbl <- data_tbl %>%
        # Step 1: Extend
        extend_timeseries(
            .id_var        = id,
            .date_var      = date,
            .length_future = 52
        ) %>%
        # Step 2: Nest
        nest_timeseries(
            .id_var        = id,
            .length_future = 52,
            .length_actual = 52*2
        ) %>%
        # Step 3: Split Train/Test
        split_nested_timeseries(
            .length_test = 52
        )

    nested_data_tbl

    #> # A tibble: 2 × 4
    #>   id    .actual_data       .future_data      .splits
    #>   <fct> <list>             <list>            <list>
    #> 1 1_1   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]>
    #> 2 1_3   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]>

nested-ensembles.R

## Nested Modeltime Workflow

Next, we move into the Nested Modeltime Workflow now that nested data has been created. The Nested Modeltime Workflow includes 3 steps:

1. Modeling Fitting: This is the training stage where we fit to training data. The test forecast is generated from this step.

   1. Create tidymodels workflows.

   2. `modeltime_nested_fit()`: Used to fit the submodels to the training data.

   3. `ensemble_nested_average()` or `ensemble_nested_weighted()`: Used to make ensembles from the submodels.

2. Model Evaluation and Selection: This is where we review model performance and select the best model by minimizing or maximizing an error metric. See `modeltime_nested_select_best()`.

3. Model Refitting: This is the final fitting stage where we fit to actual data. The future forecast is generated from this step. See `modeltime_nested_refit()`.

### Step 1A: Create Tidymodels Workflows

First, we create `tidymodels` workflows for the various models that you intend to create.

#### Prophet

A common modeling method is prophet, that can be created using `prophet_reg()`. We’ll create a workflow. Note that we use the `extract_nested_train_split(nested_data_tbl)` to help us build preprocessing features.

    rec_prophet <- recipe(value ~ date, extract_nested_train_split(nested_data_tbl))

    wflw_prophet <- workflow() %>%
        add_model(
            prophet_reg("regression", seasonality_yearly = TRUE) %>%
                set_engine("prophet")
        ) %>%
        add_recipe(rec_prophet)

nested-ensembles.R

#### XGBoost

Next, we can use a machine learning method that can get good results: XGBoost. We will add a few extra features in the recipe feature engineering step to generate features that tend to get better modeling results. Note that we use the `extract_nested_train_split(nested_data_tbl)` to help us build preprocessing features.

    rec_xgb <- recipe(value ~ ., extract_nested_train_split(nested_data_tbl)) %>%
        step_timeseries_signature(date) %>%
        step_rm(date) %>%
        step_zv(all_predictors()) %>%
        step_dummy(all_nominal_predictors(), one_hot = TRUE)

    wflw_xgb <- workflow() %>%
        add_model(boost_tree("regression") %>% set_engine("xgboost")) %>%
        add_recipe(rec_xgb)

nested-ensembles.R

### Step 1B: Nested Modeltime Tables

With a couple of modeling workflows in hand, we are now ready to test them on each of the time series. We start by using the `modeltime_nested_fit()` function, which iteratively fits each model to each of the nested time series train/test “.splits” column.

    nested_modeltime_tbl <- modeltime_nested_fit(
      # Nested data
      nested_data = nested_data_tbl,

      # Add workflows
      wflw_prophet,
      wflw_xgb
    )

    #> Fitting models on training data... ■■■■■■■■■■■■■■■■                  50% | ETA:…

    #> Fitting models on training data... ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■  100% | ETA:…

    nested_modeltime_tbl

    #> # Nested Modeltime Table
    #>

    #> Trained on: .splits | Forecast Errors: [0] | Conf Method: conformal_default |
    #> Conf Interval: 0.95

    #> # A tibble: 2 × 5
    #>   id    .actual_data       .future_data      .splits         .modeltime_tables
    #>   <fct> <list>             <list>            <list>          <list>
    #> 1 1_1   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [2 × 5]>
    #> 2 1_3   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [2 × 5]>

nested-ensembles.R

This adds a new column with `.modeltime_tables` for each of the data sets and has created several logged attributes that are part of the “Nested Modeltime Table”. We also can see that the models were trained on “.splits” and none of the models had any errors.

#### Accuracy Check

This is kind of advanced, but because our accuracy functions (`table_modeltime_accuracy(.interactive = FALSE)`) produce static `gt` table, we can make a function to highlight rows by group.

    tab_style_by_group <- function(object, ..., style) {

      subset_log <- object[["_boxhead"]][["type"]]=="row_group"
      grp_col    <- object[["_boxhead"]][["var"]][subset_log] %>% rlang::sym()

      object %>%
        tab_style(
          style = style,
          locations = cells_body(
            rows = .[["_data"]] %>%
              tibble::rowid_to_column("rowid") %>%
              group_by(!!grp_col) %>%
              filter(...) %>%
              ungroup() %>%
              pull(rowid)
          )
        )
    }

nested-ensembles.R

And now we can see which models are the winners, performing the best by group with the lowest RMSE (root mean squared error).

    nested_modeltime_tbl %>%
      extract_nested_test_accuracy() %>%
      group_by(id) %>%
      table_modeltime_accuracy(.interactive = FALSE) %>%
      tab_style_by_group(
        rmse == min(rmse),
        style = cell_fill(color = "lightblue")
      )

Accuracy Table
.model_id | .model_desc | .type | mae | mape | mase | smape | rmse | rsq
--- | --- | --- | --- | --- | --- | --- | --- | ---
1_1
1 | PROPHET | Test | 10071.47 | 45.88 | 1.99 | 59.97 | 11776.87 | 0.38
2 | XGBOOST | Test | 6236.79 | 25.31 | 1.23 | 24.57 | 9017.22 | 0.19
1_3
1 | PROPHET | Test | 3539.80 | 29.87 | 1.37 | 25.46 | 4707.77 | 0.80
2 | XGBOOST | Test | 3085.78 | 18.81 | 1.20 | 20.40 | 5085.81 | 0.79

nested-ensembles.R

### Step 1C: Make Ensembles

Now that we’ve fitted submodels, our goal is to improve on the submodels by leveraging ensembles.

#### Average Ensemble

We’ll give a go at an average ensemble using a simple mean with the `ensemble_nested_average()` function. We select `type = "mean"` for simple average (another option is median ensemble, which is better when you have models with large spikes).

    nested_ensemble_1_tbl <- nested_modeltime_tbl %>%
        ensemble_nested_average(
            type           = "mean",
            keep_submodels = TRUE
        )

    nested_ensemble_1_tbl

    #> # Nested Modeltime Table
    #>   # A tibble: 2 × 5
    #>   id    .actual_data       .future_data      .splits         .modeltime_tables
    #>   <fct> <list>             <list>            <list>          <list>
    #> 1 1_1   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [3 × 5]>
    #> 2 1_3   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [3 × 5]>

nested-ensembles.R

We can check the accuracy again. This time the Ensemble (MEAN) outperforms both the prophet and xgboost submodels.

    nested_ensemble_1_tbl %>%
      extract_nested_test_accuracy() %>%
      group_by(id) %>%
      table_modeltime_accuracy(.interactive = FALSE) %>%
      tab_style_by_group(
        rmse == min(rmse),
        style = cell_fill(color = "lightblue")
      )

Accuracy Table
.model_id | .model_desc | .type | mae | mape | mase | smape | rmse | rsq
--- | --- | --- | --- | --- | --- | --- | --- | ---
1_1
1 | PROPHET | Test | 10071.47 | 45.88 | 1.99 | 59.97 | 11776.87 | 0.38
2 | XGBOOST | Test | 6236.79 | 25.31 | 1.23 | 24.57 | 9017.22 | 0.19
3 | ENSEMBLE (MEAN): 2 MODELS | Test | 5419.21 | 20.22 | 1.07 | 22.23 | 8654.52 | 0.42
1_3
1 | PROPHET | Test | 3539.80 | 29.87 | 1.37 | 25.46 | 4707.77 | 0.80
2 | XGBOOST | Test | 3085.78 | 18.81 | 1.20 | 20.40 | 5085.81 | 0.79
3 | ENSEMBLE (MEAN): 2 MODELS | Test | 2661.82 | 18.97 | 1.03 | 17.75 | 4038.38 | 0.82

nested-ensembles.R

#### Weighted Ensemble

Next, we can give a go at a weighted ensemble with the `ensemble_nested_weighted()` function. A few key points about the arguments:

- `loadings`: This parameter allows us to weight models differently. Providing `c(2,1)` places a 2-to-1 weighting on the two submodels.

- `metric`: This parameter is determined by the accuracy table. The default is to use the “rmse” column. The loadings are then applied to the best (lowest) “rmse” first. The best model will have 2/3 (66% weight) loading and the second best will have 1/3 (33% weight).

- `model_ids`: This is a filtering mechanism to help us isolate which model ID’s that we want to include as submodels. We want to exclude Model ID 3, because this is our Ensemble Average (MEAN) model.

- `control`: This uses `control_nested_fit()` to control aspects of the fitting process like running in Parallel vs Sequential and outputting verbose to provide additional information during the fitting process.

    nested_ensemble_2_tbl <- nested_ensemble_1_tbl %>%
        ensemble_nested_weighted(
            loadings        = c(2,1),
            metric          = "rmse",
            model_ids       = c(1,2),
            control         = control_nested_fit(allow_par = FALSE, verbose = TRUE)
        )

    #> ℹ [1/2] Starting Modeltime Table: ID 1_1...

    #> ✔ Model 4 Passed ENSEMBLE WEIGHTED.

    #> ✔ [1/2] Finished Modeltime Table: ID 1_1

    #> ℹ [2/2] Starting Modeltime Table: ID 1_3...

    #> ✔ Model 4 Passed ENSEMBLE WEIGHTED.

    #> ✔ [2/2] Finished Modeltime Table: ID 1_3

    #> Finished in: 1.628696 secs.

    nested_ensemble_2_tbl

    #> # Nested Modeltime Table
    #>

    #> Trained on: .splits | Forecast Errors: [0] | Conf Method: conformal_default |
    #> Conf Interval: 0.95

    #> # A tibble: 2 × 5
    #>   id    .actual_data       .future_data      .splits         .modeltime_tables
    #>   <fct> <list>             <list>            <list>          <list>
    #> 1 1_1   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [4 × 5]>
    #> 2 1_3   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [4 × 5]>

nested-ensembles.R

Next, let’s check the accuracy on the new ensemble. The Weighted Ensemble has improved the 1_1 time series, but not the 1_3 time series.

    nested_ensemble_2_tbl %>%
      extract_nested_test_accuracy() %>%
      group_by(id) %>%
      table_modeltime_accuracy(.interactive = FALSE) %>%
      tab_style_by_group(
        rmse == min(rmse),
        style = cell_fill(color = "lightblue")
      )

Accuracy Table
.model_id | .model_desc | .type | mae | mape | mase | smape | rmse | rsq
--- | --- | --- | --- | --- | --- | --- | --- | ---
1_1
1 | PROPHET | Test | 10071.47 | 45.88 | 1.99 | 59.97 | 11776.87 | 0.38
2 | XGBOOST | Test | 6236.79 | 25.31 | 1.23 | 24.57 | 9017.22 | 0.19
3 | ENSEMBLE (MEAN): 2 MODELS | Test | 5419.21 | 20.22 | 1.07 | 22.23 | 8654.52 | 0.42
4 | ENSEMBLE (WEIGHTED): 2 MODELS | Test | 4414.19 | 14.75 | 0.87 | 15.99 | 8320.87 | 0.41
1_3
1 | PROPHET | Test | 3539.80 | 29.87 | 1.37 | 25.46 | 4707.77 | 0.80
2 | XGBOOST | Test | 3085.78 | 18.81 | 1.20 | 20.40 | 5085.81 | 0.79
3 | ENSEMBLE (MEAN): 2 MODELS | Test | 2661.82 | 18.97 | 1.03 | 17.75 | 4038.38 | 0.82
4 | ENSEMBLE (WEIGHTED): 2 MODELS | Test | 2771.82 | 21.25 | 1.08 | 19.12 | 4067.88 | 0.82

nested-ensembles.R

### Step 2: Select Best

Using the accuracy data, we can pick a metric and select the best model based on that metric. The available metrics are in the `default_forecast_accuracy_metric_set()`. Make sure to select `minimize` based on the metric. The `filter_test_forecasts` parameter tells the function to filter the logged test forecasts to just the best.

    best_nested_modeltime_tbl <- nested_ensemble_2_tbl %>%
        modeltime_nested_select_best(
          metric                = "rmse",
          minimize              = TRUE,
          filter_test_forecasts = TRUE
        )

nested-ensembles.R

#### Extract Nested Best Model Report

The best model selections can be accessed with `extract_nested_best_model_report()`.

    best_nested_modeltime_tbl %>%
      extract_nested_best_model_report() %>%
      table_modeltime_accuracy(.interactive = FALSE)

Accuracy Table
id | .model_id | .model_desc | .type | mae | mape | mase | smape | rmse | rsq
--- | --- | --- | --- | --- | --- | --- | --- | --- | ---
1_1 | 4 | ENSEMBLE (WEIGHTED): 2 MODELS | Test | 4414.19 | 14.75 | 0.87 | 15.99 | 8320.87 | 0.41
1_3 | 3 | ENSEMBLE (MEAN): 2 MODELS | Test | 2661.82 | 18.97 | 1.03 | 17.75 | 4038.38 | 0.82

nested-ensembles.R

#### Extract Nested Best Test Forecasts

Once we’ve selected the best models, we can easily visualize the best forecasts by time series. Note that the nested test forecast logs have been modified to isolate the best models.

    best_nested_modeltime_tbl %>%
      extract_nested_test_forecast() %>%
      group_by(id) %>%
      plot_modeltime_forecast(
        .facet_ncol  = 1,
        .interactive = FALSE
      )

nested-ensembles.R

### Step 3: Refitting and Future Forecast

With the best models in hand, we can make our future forecasts by refitting the models to the full dataset.

- If the best models have been selected, the only the best models will be refit.

- If best models have not been selected, then all models will be refit.

We’ve selected our best models, and will move forward with refitting and future forecast logging using the `modeltime_nested_refit()` function.

    nested_modeltime_refit_tbl <- best_nested_modeltime_tbl %>%
        modeltime_nested_refit(
            control = control_nested_refit(verbose = TRUE)
        )

    #> ℹ [1/2] Starting Modeltime Table: ID 1_1...

    #> ✔ Model 4 Passed ENSEMBLE (WEIGHTED): 2 MODELS.

    #> ✔ [1/2] Finished Modeltime Table: ID 1_1

    #> ℹ [2/2] Starting Modeltime Table: ID 1_3...

    #> ✔ Model 3 Passed ENSEMBLE (MEAN): 2 MODELS.

    #> ✔ [2/2] Finished Modeltime Table: ID 1_3

    #> Finished in: 1.089689 secs.

nested-ensembles.R

We can see that the nested modeltime table appears the same, but has now been trained on `.actual_data`.

    nested_modeltime_refit_tbl

    #> # Nested Modeltime Table
    #>

    #> Trained on: .actual_data | Forecast Errors: [0] | Conf Method:
    #> conformal_default | Conf Interval: 0.95

    #> # A tibble: 2 × 5
    #>   id    .actual_data       .future_data      .splits         .modeltime_tables
    #>   <fct> <list>             <list>            <list>          <list>
    #> 1 1_1   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [1 × 5]>
    #> 2 1_3   <tibble [104 × 2]> <tibble [52 × 2]> <split [52|52]> <mdl_tm_t [1 × 5]>

nested-ensembles.R

#### Extract Nested Future Forecast

After the refitting process completes, we can now access the future forecast, which is logged.

    nested_modeltime_refit_tbl %>%
      extract_nested_future_forecast() %>%
      group_by(id) %>%
      plot_modeltime_forecast(
        .interactive = FALSE,
        .facet_ncol  = 2
      )

nested-ensembles.R

# Summary

Nested ensembling is a powerful technique that can improve forecasting accuracy. But, this is just a small portion of what can be done to take your forecasting to the next level… If you want to become a forecasting expert for your organization, then take the read on!

## Take the High-Performance Forecasting Course

> Become the forecasting expert for your organization

High-Performance Time Series Course

### Time Series is Changing

Time series is changing. Businesses now need 10,000+ time series forecasts every day. This is what I call a High-Performance Time Series Forecasting System (HPTSF) - Accurate, Robust, and Scalable Forecasting.

High-Performance Forecasting Systems will save companies by improving accuracy and scalability. Imagine what will happen to your career if you can provide your organization a “High-Performance Time Series Forecasting System” (HPTSF System).

### How to Learn High-Performance Time Series Forecasting

I teach how to build a HPTFS System in my High-Performance Time Series Forecasting Course. You will learn:

- Time Series Machine Learning (cutting-edge) with `Modeltime` - 30+ Models (Prophet, ARIMA, XGBoost, Random Forest, & many more)
- Deep Learning with `GluonTS` (Competition Winners)
- Time Series Preprocessing, Noise Reduction, & Anomaly Detection
- Feature engineering using lagged variables & external regressors
- Hyperparameter Tuning
- Time series cross-validation
- Ensembling Multiple Machine Learning & Univariate Modeling Techniques (Competition Winner)
- Scalable Forecasting - Forecast 1000+ time series in parallel
- and more.

Become the Time Series Expert for your organization.

Take the High-Performance Time Series Forecasting Course