Detrending Series

The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Detrending Series

The trend extraction methods covered in the other vignettes return a smooth trend. Often, though, the trend is exactly the part we want to remove: in macroeconomics the object of interest is usually the deviation from trend — the cycle, or the output gap when the series measures aggregate activity. detrend_series() removes the trend from a series and adds the detrended series as a column to the original data frame.

library(trendseries)
library(dplyr)
library(tidyr)

The theme below is used throughout the vignette for consistent styling.

library(ggplot2)

theme_series <- theme_minimal(paper = "#fefefe") +
  theme(
    legend.position = "bottom",
    panel.grid.minor = element_blank(),
    strip.background = element_rect(fill = "#2c3e50"),
    strip.text = element_text(color = "#fefefe"),
    axis.ticks.x = element_line(color = "gray40", linewidth = 0.5),
    axis.line.x = element_line(color = "gray40", linewidth = 0.5),
    axis.title.x = element_blank(),
    palette.colour.discrete = c(
      "#2c3e50",
      "#e74c3c",
      "#f39c12",
      "#1abc9c",
      "#9b59b6"
    )
  )

Trend extraction vs detrending

detrend_series() is the mirror image of augment_trends().

augment_trends() returns the trend (trend_* columns) and discards the fluctuations around it.
detrend_series() returns the fluctuations (detrend_* columns): the trend is fitted with the same methods and then subtracted from the series, so the exact identity value = trend + detrend holds.

Any of the 20 trend methods supported by augment_trends() can be used for detrending. The default is the Hodrick-Prescott filter ("hp"), the most common detrending choice for economic data, with the smoothing parameter set automatically from the frequency of the series. And unlike decompose_series(), which needs a seasonal component to isolate, detrending is well defined at any frequency.

A first detrended series

Let’s start with the ibcbr dataset, a monthly index of Brazilian economic activity (IBC-Br) compiled by the Central Bank.

ggplot(ibcbr, aes(date, index)) +
  geom_line(lwd = 0.7) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  labs(
    title = "Brazilian economic activity (IBC-Br)",
    y = "Index (2003 = 100)"
  ) +
  theme_series

Passing the data to detrend_series() adds a single new column, detrend_hp, holding the deviation from the HP trend. The frequency is detected automatically from the date column.

ibcbr_cycle <- ibcbr |>
  detrend_series(value_col = "index")

ibcbr_cycle
#> # A tibble: 276 × 3
#>    date       index detrend_hp
#>    <date>     <dbl>      <dbl>
#>  1 2003-01-01  67.1     -1.93 
#>  2 2003-02-01  68.8     -0.482
#>  3 2003-03-01  72.2      2.53 
#>  4 2003-04-01  71.3      1.37 
#>  5 2003-05-01  70.0     -0.250
#>  6 2003-06-01  68.8     -1.75 
#>  7 2003-07-01  71.9      1.07 
#>  8 2003-08-01  70.8     -0.332
#>  9 2003-09-01  71.8      0.344
#> 10 2003-10-01  73.3      1.57 
#> # ℹ 266 more rows

ggplot(ibcbr_cycle, aes(date, detrend_hp)) +
  geom_hline(yintercept = 0, color = "gray40", lty = 2) +
  geom_line(lwd = 0.7) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  labs(
    title = "Detrended economic activity",
    subtitle = "Deviation from the HP trend",
    y = "Index points"
  ) +
  theme_series

The big picture is right — the 2008–09 recession, the 2015–16 crisis, and the COVID collapse all show up as deep negative deviations. But the line is also covered in a regular saw-tooth pattern. That is not the business cycle: it is seasonality, and it points to an important caveat.

Detrending does not deseasonalize

The IBC-Br index above is not seasonally adjusted, and detrending only removes the slow-moving part of the series. The seasonal swings are too fast for the trend to absorb, so they end up in the detrended series, where they can drown out — or be mistaken for — cyclical movements.

The fix is to remove the seasonal component first and detrend the seasonally adjusted series. The two wrappers compose naturally: deseason_series() adds a seasadj_stl column, which detrend_series() can then take as its input.

ibcbr_sa_cycle <- ibcbr |>
  deseason_series(value_col = "index") |>
  detrend_series(value_col = "seasadj_stl")

ibcbr_sa_cycle
#> # A tibble: 276 × 4
#>    date       index seasadj_stl detrend_hp
#>    <date>     <dbl>       <dbl>      <dbl>
#>  1 2003-01-01  67.1        71.4     2.00  
#>  2 2003-02-01  68.8        72.5     2.77  
#>  3 2003-03-01  72.2        68.7    -1.30  
#>  4 2003-04-01  71.3        71.1     0.918 
#>  5 2003-05-01  70.0        70.5    -0.0433
#>  6 2003-06-01  68.8        69.6    -1.16  
#>  7 2003-07-01  71.9        69.0    -2.06  
#>  8 2003-08-01  70.8        68.1    -3.26  
#>  9 2003-09-01  71.8        71.6    -0.0420
#> 10 2003-10-01  73.3        72.0     0.0601
#> # ℹ 266 more rows

We can compare the approaches side by side.

cycles <- bind_rows(
  list(
    "Raw series" = ibcbr_cycle,
    "Seasonally adjusted" = ibcbr_sa_cycle
  ),
  .id = "input"
)

cycles <- cycles |>
  rename(cycle = detrend_hp)

ggplot(cycles, aes(date, cycle)) +
  geom_hline(yintercept = 0, color = "gray40", lty = 2) +
  geom_line(aes(color = input), lwd = 0.7, show.legend = FALSE) +
  facet_wrap(vars(input), ncol = 1) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  labs(
    title = "Detrending before and after seasonal adjustment",
    subtitle = "HP deviations from trend",
    y = "Index points"
  ) +
  theme_series

The seasonally adjusted cycle tells the same story more clearly. For seasonal data this two-step workflow — deseason, then detrend — should be the default.

Percentage deviations from trend

The ibcbr series in measured in index points, which can make it hard to comapre across series or different time periods. A common solution in macroeconomics is to report deviations as a percentage of the trend — this is how output gaps are usually stated.

Setting transform = "log" fits the trend on the log scale and returns the log deviation from trend, log(value) - log(trend). Multiplied by 100, this is approximately the percentage deviation. On the original scale the identity becomes multiplicative: value = trend * exp(detrend).

ibcbr_gap <- ibcbr |>
  deseason_series(value_col = "index") |>
  detrend_series(value_col = "seasadj_stl", transform = "log")

ggplot(ibcbr_gap, aes(date, detrend_hp)) +
  geom_hline(yintercept = 0, color = "gray40", lty = 2) +
  geom_line(lwd = 0.7) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  scale_y_continuous(labels = scales::percent) +
  labs(
    title = "An HP-filter activity gap",
    subtitle = "Log deviation from trend of the seasonally adjusted IBC-Br (x 100)",
    y = "% of trend"
  ) +
  theme_series

The COVID trough now reads directly as “activity was about 13% below trend”.

Comparing detrending methods

As with similar functions in trendseries, the methods argument accepts multiple methods in a single function call. Since the actual “cycle” is not observed, different filters take different stances on what counts as “trend”.

Passing several methods adds one detrend_* column per method, so the implied cycles can be compared side by side. Here we contrast the HP filter with the Hamilton filter, a regression-based alternative proposed as an alternative for the HP filter.

ibcbr_methods <- ibcbr |>
  deseason_series(value_col = "index") |>
  detrend_series(
    value_col = "seasadj_stl",
    methods = c("hp", "hamilton"),
    transform = "log"
  )

methods_long <- ibcbr_methods |>
  pivot_longer(
    cols = starts_with("detrend_"),
    names_to = "method",
    names_prefix = "detrend_",
    values_to = "cycle"
  )

ggplot(methods_long, aes(date, 100 * cycle)) +
  geom_hline(yintercept = 0, color = "gray40", lty = 2) +
  geom_line(aes(color = method), lwd = 0.7) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  labs(
    title = "HP vs Hamilton detrending",
    subtitle = "Log deviation from trend (x 100)",
    y = "% of trend",
    color = NULL
  ) +
  theme_series

The two cycles agree on the major swings but differ in amplitude and timing — a useful reminder that detrended series are estimates, not data. Note also that the Hamilton filter projects the series two years ahead from a year of lags (h = 24, p = 12 for monthly data), so its first three years of detrended values are missing; methods with boundary effects (such as "bk") behave similarly at both ends.

The unified parameters of augment_trends() — window, smoothing, band, align, and params — all pass through unchanged. For instance, the Baxter-King filter isolates fluctuations between 1.5 and 8 years directly:

ibcbr |>
  deseason_series(value_col = "index") |>
  detrend_series(
    value_col = "seasadj_stl",
    methods = "bk",
    band = c(18, 96) # periods in months
  )

The Econometric Filters vignette discusses what each of these filters does and how to choose between them.

Keeping the fitted trend

By default only the detrended column is added. Setting components = TRUE also keeps the fitted trend_* columns, which is handy for plotting the trend against the series or for verifying the identity.

ibcbr_parts <- ibcbr |>
  detrend_series(value_col = "index", components = TRUE)

all.equal(ibcbr_parts$trend_hp + ibcbr_parts$detrend_hp, ibcbr_parts$index)
#> [1] TRUE

ggplot(ibcbr_parts, aes(date)) +
  geom_line(aes(y = index, color = "Observed"), lwd = 0.7, alpha = 0.5) +
  geom_line(aes(y = trend_hp, color = "Trend (HP)"), lwd = 1) +
  scale_x_date(date_breaks = "2 years", date_labels = "%Y") +
  labs(
    title = "Economic activity and its HP trend",
    y = "Index (2003 = 100)",
    color = NULL
  ) +
  theme_series

With transform = "log" the trend is reported back in the units of the series, so the same plot works unchanged; the identity is then value = trend * exp(detrend).

Grouped detrending

Like the other functions in the package, detrend_series() accepts a group_cols argument to detrend several series at once. The full workflow — seasonal adjustment followed by detrending, in percent of trend — carries over group by group. Here we use the electricity dataset, which records monthly electricity consumption for three sectors.

elec_cycles <- electricity |>
  dplyr::filter(date >= as.Date("2003-01-01")) |>
  deseason_series(group_cols = "name_series") |>
  detrend_series(
    value_col = "seasadj_stl",
    group_cols = "name_series",
    transform = "log"
  )

glimpse(elec_cycles)
#> Rows: 828
#> Columns: 5
#> $ date        <date> 2003-01-01, 2003-02-01, 2003-03-01, 2003-04-01, 2003-05-0…
#> $ name_series <chr> "electric_commercial", "electric_commercial", "electric_co…
#> $ value       <dbl> 4182, 4153, 4221, 3948, 3938, 3711, 3667, 3720, 3686, 4001…
#> $ seasadj_stl <dbl> 3862.461, 3819.806, 3778.933, 3678.709, 4063.355, 4147.311…
#> $ detrend_hp  <dbl> 0.004224489, -0.011316838, -0.026511318, -0.057827607, 0.0…

ggplot(elec_cycles, aes(date, detrend_hp)) +
  geom_hline(yintercept = 0, color = "gray40", lty = 2) +
  geom_line(color = "#2c3e50", lwd = 0.7) +
  facet_wrap(vars(name_series), ncol = 1) +
  scale_x_date(date_breaks = "5 years", date_labels = "%Y") +
  scale_y_continuous(labels = scales::percent) +
  labs(
    title = "Electricity consumption cycles by sector",
    subtitle = "Log deviation from HP trend of the seasonally adjusted series (x 100)",
    y = "% of trend"
  ) +
  theme_series

Commercial consumption swings the hardest — the collapse of in-person services during COVID stands out — while industrial demand stays closest to its trend.

Summary

detrend_series() removes the trend from a series, adding a detrend_{method} column with the deviation from trend (the cycle). The exact identity value = trend + detrend holds.
Any of the 20 trend methods of augment_trends() can be used; the default is the HP filter with frequency-appropriate smoothing.
Detrending does not remove seasonality: for seasonal data, run deseason_series() first and detrend the seasadj_* column.
Use transform = "log" for deviations in percent of trend (the output-gap convention); the identity becomes value = trend * exp(detrend).
components = TRUE keeps the fitted trend columns alongside the detrended series.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.