The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Nested pipelines

A pipeline can be a step of another pipeline. This allows to reuse a standard analysis pipeline inside a larger workflow. The key is as.function(), which turns a pipeline into a function that can be used in pipeline steps.

Inner pipeline

We start with the same pipeline as in the previous “Split, map, and reduce” vignette, fitting a linear model and returning its coefficients. This will serve as our inner pipeline.

library(pipeflow)

inner <- pip_new("coefficients") |>
    pip_add("input", \(input_df = NULL) input_df) |>
    pip_add("fit",
        \(data = ~input, xVar = "x", yVar = "y") {
            lm(paste(yVar, "~", xVar), data = data)
        }
    ) |>
    pip_add("coefs",
        \(fit = ~fit, digits = 3) {
            round(coefficients(fit), digits)
        }
    )

inner
# <pipeflow> coefficients (3 steps)
# ---------------------------------
#     step         params depends state
# 1: input       input_df           new
# 2:   fit data,xVar,yVar   input   new
# 3: coefs     fit,digits     fit   new
# ---------------------------------
# <ready> last run: never
# steps: 3 new

A pipeline as a function

as.function() returns a function whose arguments are the unbound parameters of the pipeline, with the current values as defaults, and whose result is the output of the step(s) defined by out.

f_inner <- as.function(inner, out = "coefs")

formals(f_inner) |> str()
# Dotted pair list of 4
#  $ input_df: NULL
#  $ xVar    : chr "x"
#  $ yVar    : chr "y"
#  $ digits  : num 3

As you see, the function version of the inner pipeline has the same default params as the pipeline itself.

Under the hood, the function runs a copy of the inner pipeline. Let’s see this now:

params <- list(input_df = iris, xVar = "Sepal.Length", yVar = "Sepal.Width")
pip_set_params(inner, params) |> pip_run()
# info [2026-11-15 18:27:57.305 UTC]: Starting run of pipeflow 'coefficients'
# info [2026-11-15 18:27:57.305 UTC]: Step 1/3 input
# info [2026-11-15 18:27:57.306 UTC]: Step 2/3 fit
# info [2026-11-15 18:27:57.308 UTC]: Step 3/3 coefs
# info [2026-11-15 18:27:57.308 UTC]: Finished in 0.004 s (3 done, 0 skipped, 0 failed)

inner[["coefs", "out"]]
#  (Intercept) Sepal.Length 
#        3.419       -0.062

do.call(f_inner, params)
#  (Intercept) Sepal.Length 
#        3.419       -0.062

Note that since f_inner gets its own copy of the inner pipeline, later changes to the inner pipeline will not change the function.

Outer pipeline

As any other function, the function can be added as a step of another pipeline. We start with the simplest case, where the outer pipeline provides the data and the inner pipeline analyses it as a whole.

outer_all <- pip_new("analysis") |>
    pip_add("df", \(df = NULL) df) |>
    pip_add(
        "inner_run",
        as.function(inner, out = "coefs"),  # coefs is returned from inner run
        params = list(input_df = ~df)       # df is passed to the inner pipeline
    )

outer_all
# <pipeflow> analysis (2 steps)
# -----------------------------
#         step                    params depends state
# 1:        df                        df           new
# 2: inner_run input_df,xVar,yVar,digits      df   new
# -----------------------------
# <ready> last run: never
# steps: 2 new

As for any other step function, params of pip_add() wires the function into the outer pipeline. list(input_df = ~df) reads: the argument input_df of the function is bound to the output of the step df from the outer pipeline, that is, the step inner_run now depends on df.

The remaining arguments of the function, xVar, yVar and digits, are not bound. They stay unbound parameters, and as such are parameters of the outer pipeline, without having to forward them:

pip_get_params(outer_all) |> str()
# List of 4
#  $ df    : NULL
#  $ xVar  : chr "Sepal.Length"
#  $ yVar  : chr "Sepal.Width"
#  $ digits: num 3

The df listed here belongs to the outer step df.

Let’s set the analysis parameters and run the outer pipeline:

outer_all |>
    pip_set_params(
        list(df = iris, xVar = "Sepal.Length", yVar = "Sepal.Width")
    ) |>
    pip_run()
# info [2026-11-15 18:27:57.369 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:57.369 UTC]: Step 1/2 df
# info [2026-11-15 18:27:57.370 UTC]: Step 2/2 inner_run
# info [2026-11-15 18:27:57.377 UTC]: Finished in 0.009 s (2 done, 0 skipped, 0 failed)

outer_all[["inner_run", "out"]]
#  (Intercept) Sepal.Length 
#        3.419       -0.062

The inner pipeline was run as a single step and returned the same coefficients as above where we called the function directly.

Returning something computed

Instead of step names, out can also be a function. It receives the inner pipeline after the run and returns whatever you need from it, e.g. a value computed from several steps:

f_summary <- as.function(
    inner,
    out = \(p) list(
        r2 = summary(p[["fit", "out"]])$r.squared,
        coefs = p[["coefs", "out"]]
    )
)

f_summary(input_df = iris, xVar = "Sepal.Length", yVar = "Sepal.Width")
# $r2
# [1] 0.01382265
# 
# $coefs
#  (Intercept) Sepal.Length 
#        3.419       -0.062

To simply return the outputs of some steps, use out = c("fit", "coefs"). Functions like pip_collect() are useful in out when you want more, e.g. to collect the outputs by tags (see ?as.function.pipeflow).

Outer pipeline with split data

The real strength shows when the inner pipeline is used multiple times in a “split-data” scenario. Here, the outer pipeline splits the data into subsets (see also the previous vignette), and input_df is bound to the output of the split step. Then the function is called once per subset.

outer <- pip_new("full analysis") |>
    pip_add("df", \(df = NULL) df) |>
    pip_add(
        "split_df",
        \(df = ~df, byVar = "by") split(df, f = df[[byVar]]),
        exec = "split"
    ) |>
    pip_add(
        "inner_run",
        as.function(inner, out = "coefs"),
        params = list(input_df = ~split_df)    # bind to split data
    ) |>
    pip_add(
        "combine",
        \(coefs = ~inner_run) as.data.frame(do.call(rbind, coefs)),
        exec = "reduce"
    )

outer
# <pipeflow> full analysis (4 steps)
# ----------------------------------
#         step                    params   depends state   exec
# 1:        df                        df             new   auto
# 2:  split_df                  df,byVar        df   new  split
# 3: inner_run input_df,xVar,yVar,digits  split_df   new   auto
# 4:   combine                     coefs inner_run   new reduce
# ----------------------------------
# <ready> last run: never
# steps: 4 new

The split output is partitioned, so the function is called once per subset, and its output is partitioned as well. The combine step is a reduce step and receives all partitions. As before, the unbound parameters of the inner pipeline are parameters of the outer pipeline, now next to byVar of the split step:

pip_get_params(outer) |> str()
# List of 5
#  $ df    : NULL
#  $ byVar : chr "by"
#  $ xVar  : chr "Sepal.Length"
#  $ yVar  : chr "Sepal.Width"
#  $ digits: num 3

Let’s set the analysis parameters and run the full pipeline:

outer |>
    pip_set_params(
        list(
            df = iris,
            xVar = "Sepal.Length",
            yVar = "Sepal.Width",
            byVar = "Species"
        )
    ) |>
    pip_run()
# info [2026-11-15 18:27:57.473 UTC]: Starting run of pipeflow 'full analysis'
# info [2026-11-15 18:27:57.473 UTC]: Step 1/4 df
# info [2026-11-15 18:27:57.474 UTC]: Step 2/4 split_df
# info [2026-11-15 18:27:57.475 UTC]: Step 3/4 inner_run
# info [2026-11-15 18:27:57.487 UTC]: Step 4/4 combine
# info [2026-11-15 18:27:57.488 UTC]: Finished in 0.015 s (4 done, 0 skipped, 0 failed)

The output of the inner_run step is a partitioned list of coefficient vectors, one for each species,

outer[["inner_run", "out"]]
# $setosa
#  (Intercept) Sepal.Length 
#       -0.569        0.799 
# 
# $versicolor
#  (Intercept) Sepal.Length 
#        0.872        0.320 
# 
# $virginica
#  (Intercept) Sepal.Length 
#        1.446        0.232 
# 
# attr(,"class")
# [1] "list"                 "pipeflow_partitioned"

and the combine step returns the expected combined table.

outer[["combine", "out"]]
#            (Intercept) Sepal.Length
# setosa          -0.569        0.799
# versicolor       0.872        0.320
# virginica        1.446        0.232

Now suppose we want to change one of the model settings, say use Petal.Length instead of Sepal.Length as the predictor.

pip_set_params(outer, params = list(xVar = "Petal.Length"))

outer
# <pipeflow> full analysis (4 steps)
# ----------------------------------
#         step                    params   depends    state                 out   exec
# 1:        df                        df               done <data.frame[150x5]>   auto
# 2:  split_df                  df,byVar        df     done           <list[3]>  split
# 3: inner_run input_df,xVar,yVar,digits  split_df outdated           <list[3]>   auto
# 4:   combine                     coefs inner_run outdated   <data.frame[3x2]> reduce
# ----------------------------------
# <ready> last run: 2026-11-15 19:27:57 (0.015 s)
# steps: 2 done, 2 outdated

Since xVar is a parameter of the inner_run step, this step and its downstream steps have been marked as “outdated”.

Caching of inner pipelines

A pipeline function by default caches the inner pipelines it has run: one for each step and subset. In a step that is mapped over subsets, each subset gets an inner pipeline of its own. A later call sets all parameters and inputs in the pipeline of its subset, so that exactly the steps that are outdated by the change are run again.

To see this at work, we use a minimal toy example. The inner pipeline squares numbers and scales the result. The counters tell us how often each step has actually run.

runs <- new.env()
runs$square <- runs$scale <- 0L
count <- function(step) runs[[step]] <- runs[[step]] + 1L
n_runs <- function() unlist(mget(c("square", "scale"), envir = runs))

toy_inner <- pip_new("toy inner") |>
    pip_add("input", \(x = NULL) x) |>
    pip_add("square", \(x = ~input) {
        count("square")
        x^2
    }) |>
    pip_add("scale", \(sq = ~square, factor = 1) {
        count("scale")
        sq * factor
    })

The outer pipeline splits a list of two number vectors, a and b, and runs the inner pipeline on each of them.

toy_outer <- pip_new("toy outer") |>
    pip_add(
        "numbers",
        \(values = list(a = 1:3, b = 4:6)) values,
        exec = "split"
    ) |>
    pip_add(
        "inner_run",
        as.function(toy_inner, out = "scale"),
        params = list(x = ~numbers)
    )

pip_run(toy_outer, lgr = NULL)
n_runs()
# square  scale 
#      2      2

Each step ran twice, once in the inner pipeline of a and once in the one of b. Now we change factor, which only the last step uses. Only this step is outdated in the inner pipelines, so only this step runs again:

pip_set_params(toy_outer, list(factor = 10))
pip_run(toy_outer, lgr = NULL)
n_runs()
# square  scale 
#      2      4

Next we change the input data and give a other numbers while b stays the same, so that we expect that the inner pipeline is run once for a.

pip_set_params(toy_outer, list(values = list(a = 1:4, b = 4:6)))
pip_run(toy_outer, lgr = NULL)
n_runs()
# square  scale 
#      3      5

toy_outer[["inner_run", "out"]]
# $a
# [1]  10  40  90 160
# 
# $b
# [1] 160 250 360
# 
# attr(,"class")
# [1] "list"                 "pipeflow_partitioned"

To look at the kept inner pipelines, e.g. while debugging, use pip_inner(). For a mapped step, it returns one pipeline per partition key:

inner_pips <- pip_inner(toy_outer, "inner_run")
names(inner_pips)
# [1] "a" "b"

inner_pips[["a"]]
# <pipeflow> toy inner (3 steps)
# ------------------------------
#      step    params depends state             out
# 1:  input         x          done         1,2,3,4
# 2: square         x   input  done      1, 4, 9,16
# 3:  scale sq,factor  square  done  10, 40, 90,160
# ------------------------------
# <ready> last run: 2026-11-15 19:27:57 (0.003 s)
# steps: 3 done

A few things to know:

See the “Pipeline functions with partitioned inputs” section of ?as.function.pipeflow for how the inner pipelines are kept and reused.

Running subsets in parallel

The map argument of pip_run() runs the subsets of a mapped step in parallel (see the previous vignette). A pipeline function that keeps its inner pipelines can’t do this, as the pipelines it updates in another process would be lost, so its subsets are still run one after another. A function made with cache = FALSE keeps nothing and runs a new copy of the inner pipeline for each call, so its subsets can run in parallel. Here we replace the function of the inner_run step in a copy of outer. With keep_params = TRUE, pip_replace() keeps the model settings set before. The binding to split_df is not kept, so we give it again:

par_outer <- pip_clone(outer) |>
    pip_replace(
        "inner_run",
        as.function(inner, out = "coefs", cache = FALSE),
        params = list(input_df = ~split_df),
        keep_params = TRUE
    )

mc2 <- \(X, FUN) parallel::mclapply(X, FUN, mc.cores = 2)
pip_run(par_outer, map = mc2)
# info [2026-11-15 18:27:57.658 UTC]: Starting run of pipeflow 'full analysis'
# info [2026-11-15 18:27:57.658 UTC]: Step 1/4 df - skipping done step
# info [2026-11-15 18:27:57.659 UTC]: Step 2/4 split_df - skipping done step
# info [2026-11-15 18:27:57.659 UTC]: Step 3/4 inner_run
# info [2026-11-15 18:27:57.683 UTC]: Step 4/4 combine
# info [2026-11-15 18:27:57.685 UTC]: Finished in 0.027 s (2 done, 2 skipped, 0 failed)

par_outer[["combine", "out"]]
#            (Intercept) Petal.Length
# setosa           2.861        0.388
# versicolor       1.175        0.374
# virginica        1.673        0.234

The price is that every run of the step runs all inner steps again, so this pays off when each subset takes long to compute and is mostly computed anew anyway.

Full control: running the inner pipeline yourself

as.function() shares the parameters of the inner pipeline between all subsets. If you need more control, e.g. different settings per group or mapping over something else than the output of a split step, you can always “manually” clone and run the inner pipeline in a step function yourself.

# Helper to run inner pipeline
run_inner_pip <- function(pip, name, df) {
    pip$name <- sprintf("coefs for *%s*", name)

    # Set data subset for inner pipeline and run it
    pip_set_params(pip, list(input_df = df)) |> pip_run(lgr = NULL)

    pip[["coefs", "out"]]
}

manual <- pip_new("full analysis") |>
    pip_add("df", \(df = NULL) df) |>
    pip_add(
        "split_df",
        \(df = ~df, byVar = "by") split(df, f = df[[byVar]])
    ) |>
    pip_add(
        "inner_run",
        \(dfList = ~split_df, ...) {
            p <- pip_clone(inner)

            # Forward the parameters of the step to the inner pipeline
            pip_set_params(p, list(...))

            Map(
                f = run_inner_pip,
                name = names(dfList),
                df = dfList,
                MoreArgs = list(pip = p)
            )
        },
        params = pip_get_params(inner) # <--- default parameters of inner
    ) |>
    pip_add(
        "combine",
        \(coefs = ~inner_run) as.data.frame(do.call(rbind, coefs))
    )

manual
# <pipeflow> full analysis (4 steps)
# ----------------------------------
#         step                           params   depends state
# 1:        df                               df             new
# 2:  split_df                         df,byVar        df   new
# 3: inner_run dfList,input_df,xVar,yVar,digits  split_df   new
# 4:   combine                            coefs inner_run   new
# ----------------------------------
# <ready> last run: never
# steps: 4 new

In this version, the inner_run step does not declare the parameters of the inner pipeline as arguments. Instead, its parameters are seeded with the inner pipeline’s parameters via params = pip_get_params(inner), and the step forwards them to the inner pipeline at run time:

Let us now run the manual pipeline to see it in action.

manual |>
    pip_set_params(
        list(
            df = iris,
            xVar = "Sepal.Length",
            yVar = "Sepal.Width",
            byVar = "Species"
        )
    ) |>
    pip_run()
# info [2026-11-15 18:27:57.729 UTC]: Starting run of pipeflow 'full analysis'
# info [2026-11-15 18:27:57.729 UTC]: Step 1/4 df
# info [2026-11-15 18:27:57.730 UTC]: Step 2/4 split_df
# info [2026-11-15 18:27:57.732 UTC]: Step 3/4 inner_run
# info [2026-11-15 18:27:57.750 UTC]: Step 4/4 combine
# info [2026-11-15 18:27:57.751 UTC]: Finished in 0.022 s (4 done, 0 skipped, 0 failed)

manual[["combine", "out"]]
#            (Intercept) Sepal.Length
# setosa          -0.569        0.799
# versicolor       0.872        0.320
# virginica        1.446        0.232

This version runs each subset from scratch every time the step is outdated. With as.function(), the same result is obtained with less code and with caching.

Built-in exec modes vs. nested pipelines

Both this and the previous vignette solve the same “split, apply, and combine” problem. The built-in execution modes (exec = "split"/"reduce") are the recommended default whenever the work fits into the steps of a single pipeline, while nested pipelines are the more general tool. As a rule of thumb:

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.