The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Handling failures

In practice, steps fail: data is missing, a model does not converge, or an input violates an assumption. This vignette shows how {pipeflow} reports such failures, how a run can be continued for unaffected steps, and how to pick up the pieces once the cause is fixed.

A pipeline that fails

We use a small pipeline whose check step fails if the data exceeds a limit.

library(pipeflow)

pip <- pip_new("analysis") |>
    pip_add("data", \(n = 5) seq_len(n)) |>
    pip_add("check", \(x = ~data, limit = 3) {
        if (max(x) > limit) stop("data exceeds limit")
        x
    }) |>
    pip_add("model", \(x = ~check) mean(x)) |>
    pip_add("total", \(x = ~data) sum(x))

With n = 5 and limit = 3, the check step will fail, and the model step downstream of it cannot run. The total step, however, only needs the data step.

Stop at the first error

By default, pip_run() stops at the first step that fails.

pip_run(pip)
# info [2026-11-15 18:27:58.537 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.538 UTC]: Step 1/4 data
# info [2026-11-15 18:27:58.539 UTC]: Step 2/4 check
# error [2026-11-15 18:27:58.539 UTC]: data exceeds limit
# Error:
# ! step 'check': data exceeds limit

pip
# <pipeflow> analysis (4 steps)
# -----------------------------
#     step  params depends  state                 out
# 1:  data       n           done           1,2,3,4,5
# 2: check x,limit    data failed <error/simpleError>
# 3: model       x   check    new              [NULL]
# 4: total       x    data    new              [NULL]
# -----------------------------
# <failed> last run: 2026-11-15 19:27:58 (0.002 s)
# steps: 1 done, 1 failed, 2 new

pip_run_state(pip)
# [1] "failed"

The pipeline stays in a defined state. The failed step is marked as failed, the steps that were not reached are still new, and the run state indicates that the last run ended with an error.

Let’s inspect the error object returned by tryCatch().

err <- tryCatch(pip_run(pip, lgr = NULL), error = \(e) e)

class(err)
# [1] "pipeflow_step_error" "error"               "condition"
err$step
# [1] "check"

class(err$parent)
# [1] "simpleError" "error"       "condition"
err$parent$message
# [1] "data exceeds limit"

The error is of class pipeflow_step_error. Its field step names the failed step and the field parent holds the original condition, which keeps its own class, so that errors can be handled by class:

Continue after errors

In the previous run, the total step was not run, although it does not depend on the failed step, because the run stopped early. With on_error = "continue" in pip_run(), a failing step is still marked as failed but the run can continue for all other steps that do not depend on it:

pip_reset(pip)

suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.595 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.596 UTC]: Step 1/4 data
# info [2026-11-15 18:27:58.596 UTC]: Step 2/4 check
# error [2026-11-15 18:27:58.597 UTC]: data exceeds limit
# info [2026-11-15 18:27:58.598 UTC]: Step 3/4 model - skipping step, input 'x' failed
# info [2026-11-15 18:27:58.598 UTC]: Step 4/4 total
# info [2026-11-15 18:27:58.599 UTC]: Finished in 0.004 s (2 done, 1 skipped, 1 failed)
# warn [2026-11-15 18:27:58.600 UTC]: Run of pipeline 'analysis' continued after 1 failed step:
#   'check': data exceeds limit
# Steps not run because an input failed: 'model'

pip
# <pipeflow> analysis (4 steps)
# -----------------------------
#     step  params depends  state                 out
# 1:  data       n           done           1,2,3,4,5
# 2: check x,limit    data failed <error/simpleError>
# 3: model       x   check    new              [NULL]
# 4: total       x    data   done                  15
# -----------------------------
# <continued> last run: 2026-11-15 19:27:58 (0.004 s)
# steps: 2 done, 1 failed, 1 new

pip_run_state(pip)
# [1] "continued"

Now total was calculated, while model was skipped, because its input depends on the failed step. The out column of a failed step holds the condition, which is printed in short form as <kind/class>.

To work with it programmatically, we can extract it as usual:

out <- pip[["check", "out"]]

class(out)
# [1] "simpleError" "error"       "condition"
out$message
# [1] "data exceeds limit"

Let’s inspect the warning object returned by tryCatch().

w <- tryCatch(
    pip_run(pip, lgr = NULL, on_error = "continue"),
    warning = \(w) w
)

names(w$failed)
# [1] "check"
w$skipped
# [1] "model"

The warning is signalled after all states and outputs are stored, so catching it with tryCatch() does not lose any results.

Mitigate failed inputs with try()

Sometimes a step should run even if one of its inputs failed, for example a report that mentions what is missing or if a fallback is available. Such an argument is marked by wrapping the step reference in try():

pip_add(
    pip,
    "report",
    \(total = ~total, m = ~try(model)) {
        if (inherits(m, "pipeflow_failure")) {
            paste("total:", total, "- no model:", m$message)
        } else {
            paste("total:", total, "- mean:", m)
        }
    }
)

suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.657 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.658 UTC]: Step 1/5 data - skipping done step
# info [2026-11-15 18:27:58.658 UTC]: Step 2/5 check
# error [2026-11-15 18:27:58.659 UTC]: data exceeds limit
# info [2026-11-15 18:27:58.660 UTC]: Step 3/5 model - skipping step, input 'x' failed
# info [2026-11-15 18:27:58.660 UTC]: Step 4/5 total - skipping done step
# info [2026-11-15 18:27:58.660 UTC]: Step 5/5 report
# info [2026-11-15 18:27:58.661 UTC]: Finished in 0.004 s (1 done, 3 skipped, 1 failed)
# warn [2026-11-15 18:27:58.661 UTC]: Run of pipeline 'analysis' continued after 1 failed step:
#   'check': data exceeds limit
# Steps not run because an input failed: 'model'

pip[["report", "out"]]
# [1] "total: 15 - no model: step 'model' was not run: step 'check' failed: data exceeds limit"

Instead of the output of model, the m argument received a failure object, a condition of class pipeflow_failure, which stands in for the missing output. The message tells why the input is not available. It has the fields

A step is only run despite a failed input if all its failed inputs are marked with try(). Everything else is skipped, as model above. Note that try() has an effect only in a continuing run. With on_error = "stop", the run ends at the failed step anyway.

Recover from failures

Both skipped and failed steps are retried in the next run, and so are steps that got a failure object. Steps that were completed are not run again. Let’s fix the cause by raising the limit and re-run:

pip_set_params(pip, list(limit = 10))
pip_run(pip)
# info [2026-11-15 18:27:58.678 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.678 UTC]: Step 1/5 data - skipping done step
# info [2026-11-15 18:27:58.679 UTC]: Step 2/5 check
# info [2026-11-15 18:27:58.679 UTC]: Step 3/5 model
# info [2026-11-15 18:27:58.680 UTC]: Step 4/5 total - skipping done step
# info [2026-11-15 18:27:58.680 UTC]: Step 5/5 report
# info [2026-11-15 18:27:58.684 UTC]: Finished in 0.006 s (3 done, 2 skipped, 0 failed)

pip
# <pipeflow> analysis (5 steps)
# -----------------------------
#      step  params     depends state                 out
# 1:   data       n              done           1,2,3,4,5
# 2:  check x,limit        data  done           1,2,3,4,5
# 3:  model       x       check  done                   3
# 4:  total       x        data  done                  15
# 5: report total,m total,model  done total: 15 - mean: 3
# -----------------------------
# <ready> last run: 2026-11-15 19:27:58 (0.006 s)
# steps: 5 done

The run log shows that data and total were skipped, since they were completed already. The failed step check and the skipped step model were executed now, and so was report, because it had received a failure object instead of the real output. The report now contains the model result and the run state is ready again.

pip[["report", "out"]]
# [1] "total: 15 - mean: 3"

pip_run_state(pip)
# [1] "ready"

Failures in split pipelines

Failures also work with the execution modes split and reduce, which are described in the vignette on split, map, and reduce. Let’s split a small data set by group and let the spread step fail for groups with a single row.

d <- data.frame(g = c("a", "a", "b", "b", "c"), y = c(1, 2, 3, 5, 8))

pip <- pip_new("groups") |>
    pip_add("data", \(data = NULL) data) |>
    pip_add("by_group", \(data = ~data) split(data, data$g), exec = "split") |>
    pip_add("spread", \(d = ~by_group) {
        if (nrow(d) < 2) stop("need at least 2 rows")
        diff(range(d$y))
    }) |>
    pip_add("label", \(s = ~spread) sprintf("spread: %g", s)) |>
    pip_add(
        "combine",
        \(s = ~try(spread)) {
            failed <- vapply(
                s,
                FUN = inherits,
                what = "pipeflow_failure",
                FUN.VALUE = logical(1)
            )
            unlist(s[!failed])
        },
        exec = "reduce"
    )

pip_set_params(pip, list(data = d))

If one partition (group c) fails, spread fails for that key only. It gets state failed, but its output keeps the results of the groups a and b. For group c it holds a failure object, which tells the key and keeps the original condition as parent of a pipeflow_key_error:

suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.739 UTC]: Starting run of pipeflow 'groups'
# info [2026-11-15 18:27:58.739 UTC]: Step 1/5 data
# info [2026-11-15 18:27:58.740 UTC]: Step 2/5 by_group
# info [2026-11-15 18:27:58.741 UTC]: Step 3/5 spread
# error [2026-11-15 18:27:58.744 UTC]: key 'c': need at least 2 rows
# info [2026-11-15 18:27:58.745 UTC]: Step 4/5 label
# info [2026-11-15 18:27:58.747 UTC]: Step 5/5 combine
# info [2026-11-15 18:27:58.747 UTC]: Finished in 0.009 s (3 done, 0 skipped, 2 failed)
# warn [2026-11-15 18:27:58.748 UTC]: Run of pipeline 'groups' continued after 2 failed steps:
#   'spread': key 'c': need at least 2 rows (1 of 3 keys)
#   'label': key 'c': input step 'spread' failed (1 of 3 keys)

pip
# <pipeflow> groups (5 steps)
# ---------------------------
#        step params  depends  state                       out   exec
# 1:     data   data            done         <data.frame[5x2]>   auto
# 2: by_group   data     data   done                 <list[3]>  split
# 3:   spread      d by_group failed <partitioned: 1/3 failed>   auto
# 4:    label      s   spread failed <partitioned: 1/3 failed>   auto
# 5:  combine      s   spread   done                       1,2 reduce
# ---------------------------
# <continued> last run: 2026-11-15 19:27:58 (0.009 s)
# steps: 3 done, 2 failed

out <- pip[["spread", "out"]]
out$a
# [1] 1
fc <- out$c
class(fc)
# [1] "pipeflow_failure" "error"            "condition"
fc$key
# [1] "c"
fc$message
# [1] "step 'spread' failed: key 'c': need at least 2 rows"

The label step maps over spread like spread maps over the groups, so the failure is handled per key: label runs for groups a and b, and is not run for group c, whose output is again a failure object:

pip[["label", "out"]]$a
# [1] "spread: 1"
pip[["label", "out"]]$c$message
# [1] "step 'label' was not run: step 'spread' failed: key 'c': need at least 2 rows"

The combine step reduces the outputs of all groups in one call. Since it takes spread via try(), it ran and received all of them, with the failure in group c:

pip[["combine", "out"]]
# a b 
# 1 2

Without try(), combine would not be run, as for any failed input. With s = ~try(spread) in label, label would be called for group c too and receive the failure object. If the pipeline is run with on_error = "stop", the first failed group stops the run with a pipeflow_step_error, with the pipeflow_key_error as its parent.

If nothing changed in the pipeline since, the next run calls spread only for the failed group c and keeps the results of the other groups, which helps if the cause was outside of the pipeline, e.g. a lost connection. Here the cause is in the data, so we fix it by removing the group. That changes the input of all groups, so spread is run for all of them, and the pipeline completes:

pip_set_params(pip, list(data = d[d$g != "c", ]))
pip_run(pip)
# info [2026-11-15 18:27:58.801 UTC]: Starting run of pipeflow 'groups'
# info [2026-11-15 18:27:58.802 UTC]: Step 1/5 data
# info [2026-11-15 18:27:58.802 UTC]: Step 2/5 by_group
# info [2026-11-15 18:27:58.804 UTC]: Step 3/5 spread
# info [2026-11-15 18:27:58.805 UTC]: Step 4/5 label
# info [2026-11-15 18:27:58.806 UTC]: Step 5/5 combine
# info [2026-11-15 18:27:58.809 UTC]: Finished in 0.008 s (5 done, 0 skipped, 0 failed)

pip[["combine", "out"]]
# a b 
# 1 2

Failures in nested pipelines

A pipeline can be turned into a function with as.function() and added as a step to another pipeline, as described in the vignette on nested pipelines. Such a step runs a pipeline of its own, which means it also has its own error handling.

Let’s put the spread step of the previous section into an inner pipeline that reports on it, and map that pipeline over the groups:

inner <- pip_new("inner") |>
    pip_add("data", \(data = NULL) data) |>
    pip_add("spread", \(d = ~data) {
        if (nrow(d) < 2) stop("need at least 2 rows")
        diff(range(d$y))
    }) |>
    pip_add("report", \(s = ~try(spread)) {
        if (inherits(s, "pipeflow_failure")) "no spread" else s
    })

nested <- pip_new("nested") |>
    pip_add("data", \(data = NULL) data) |>
    pip_add("by_group", \(data = ~data) split(data, data$g), exec = "split") |>
    pip_add(
        "per_group",
        as.function(inner, out = "report"),
        params = list(data = ~by_group)
    )

pip_set_params(nested, list(data = d))

By default, the inner pipeline is run with on_error = "stop", so it ends at the first failed inner step and the outer step fails with it. The try() reference of the report step has no effect here, because try() only matters in a continuing run:

pip_run(nested)
# info [2026-11-15 18:27:58.851 UTC]: Starting run of pipeflow 'nested'
# info [2026-11-15 18:27:58.851 UTC]: Step 1/3 data
# info [2026-11-15 18:27:58.852 UTC]: Step 2/3 by_group
# info [2026-11-15 18:27:58.853 UTC]: Step 3/3 per_group
# error [2026-11-15 18:27:58.868 UTC]: key 'c': step 'spread': need at least 2 rows
# Error:
# ! step 'per_group': key 'c': step 'spread': need at least 2 rows

The message names the outer step, the group and the inner step. Each of the conditions keeps the one of the level below as its parent, so these can also be read from the error:

e <- tryCatch(pip_run(nested, lgr = NULL), error = identity)
e$step               # the outer step
# [1] "per_group"
e$parent$key         # the group
# [1] "c"
e$parent$parent$step # the inner step
# [1] "spread"

original <- e$parent$parent$parent
conditionMessage(original)
# [1] "need at least 2 rows"

With as.function(on_error = "continue"), the inner pipeline is run as pip_run(on_error = "continue") runs it: inner steps that do not depend on the failed step still run, and ~try(spread) gives the report step the failure object. Group c is handled inside the inner pipeline, and the outer run completes:

pip_replace(
    nested,
    "per_group",
    as.function(inner, out = "report", on_error = "continue"),
    params = list(data = ~by_group)
)

pip_run(nested)
# info [2026-11-15 18:27:58.921 UTC]: Starting run of pipeflow 'nested'
# info [2026-11-15 18:27:58.921 UTC]: Step 1/3 data - skipping done step
# info [2026-11-15 18:27:58.921 UTC]: Step 2/3 by_group - skipping done step
# info [2026-11-15 18:27:58.921 UTC]: Step 3/3 per_group
# warn [2026-11-15 18:27:58.934 UTC]: Run of pipeline 'inner' continued after 1 failed step:
#   'spread': need at least 2 rows
# Warning: Run of pipeline 'inner' continued after 1 failed step:
#   'spread': need at least 2 rows
# info [2026-11-15 18:27:58.935 UTC]: Finished in 0.014 s (1 done, 2 skipped, 0 failed)

nested[["per_group", "out"]]
# $a
# [1] 1
# 
# $b
# [1] 2
# 
# $c
# [1] "no spread"
# 
# attr(,"class")
# [1] "list"                 "pipeflow_partitioned"

The warning of the continued inner run is passed on like any other warning of a step, so the outer run logs it and the result of every group is available.

The function still fails if one of the steps named in out failed or was not run because an input failed: the inner pipeline handles what it can, and everything else is left to the outer run, as before.

pip_replace(
    nested,
    "per_group",
    as.function(
        inner,
        out = "spread",
        on_error = "continue"           # <--- this time we continue
    ),
    params = list(data = ~by_group)
)

pip_run(nested, lgr = NULL)
# Warning: Run of pipeline 'inner' continued after 1 failed step:
#   'spread': need at least 2 rows
# Error:
# ! step 'per_group': key 'c': step 'spread': need at least 2 rows

Since out = NULL requests the output of all inner steps, name the steps you need in out to profit from a continuing inner run.

Summary

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.