The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
In practice, steps fail: data is missing, a model does not converge, or an input violates an assumption. This vignette shows how {pipeflow} reports such failures, how a run can be continued for unaffected steps, and how to pick up the pieces once the cause is fixed.
We use a small pipeline whose check step fails if the
data exceeds a limit.
library(pipeflow)
pip <- pip_new("analysis") |>
pip_add("data", \(n = 5) seq_len(n)) |>
pip_add("check", \(x = ~data, limit = 3) {
if (max(x) > limit) stop("data exceeds limit")
x
}) |>
pip_add("model", \(x = ~check) mean(x)) |>
pip_add("total", \(x = ~data) sum(x))With n = 5 and limit = 3, the
check step will fail, and the model step
downstream of it cannot run. The total step, however, only
needs the data step.
By default, pip_run() stops at the first step that
fails.
pip_run(pip)
# info [2026-11-15 18:27:58.537 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.538 UTC]: Step 1/4 data
# info [2026-11-15 18:27:58.539 UTC]: Step 2/4 check
# error [2026-11-15 18:27:58.539 UTC]: data exceeds limit
# Error:
# ! step 'check': data exceeds limit
pip
# <pipeflow> analysis (4 steps)
# -----------------------------
# step params depends state out
# 1: data n done 1,2,3,4,5
# 2: check x,limit data failed <error/simpleError>
# 3: model x check new [NULL]
# 4: total x data new [NULL]
# -----------------------------
# <failed> last run: 2026-11-15 19:27:58 (0.002 s)
# steps: 1 done, 1 failed, 2 new
pip_run_state(pip)
# [1] "failed"The pipeline stays in a defined state. The failed step is marked as
failed, the steps that were not reached are still
new, and the run state indicates that the last run ended
with an error.
Let’s inspect the error object returned by
tryCatch().
err <- tryCatch(pip_run(pip, lgr = NULL), error = \(e) e)
class(err)
# [1] "pipeflow_step_error" "error" "condition"
err$step
# [1] "check"
class(err$parent)
# [1] "simpleError" "error" "condition"
err$parent$message
# [1] "data exceeds limit"The error is of class pipeflow_step_error. Its field
step names the failed step and the field
parent holds the original condition, which keeps its own
class, so that errors can be handled by class:
In the previous run, the total step was not run,
although it does not depend on the failed step, because the run stopped
early. With on_error = "continue" in
pip_run(), a failing step is still marked as failed but the
run can continue for all other steps that do not depend on it:
pip_reset(pip)
suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.595 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.596 UTC]: Step 1/4 data
# info [2026-11-15 18:27:58.596 UTC]: Step 2/4 check
# error [2026-11-15 18:27:58.597 UTC]: data exceeds limit
# info [2026-11-15 18:27:58.598 UTC]: Step 3/4 model - skipping step, input 'x' failed
# info [2026-11-15 18:27:58.598 UTC]: Step 4/4 total
# info [2026-11-15 18:27:58.599 UTC]: Finished in 0.004 s (2 done, 1 skipped, 1 failed)
# warn [2026-11-15 18:27:58.600 UTC]: Run of pipeline 'analysis' continued after 1 failed step:
# 'check': data exceeds limit
# Steps not run because an input failed: 'model'
pip
# <pipeflow> analysis (4 steps)
# -----------------------------
# step params depends state out
# 1: data n done 1,2,3,4,5
# 2: check x,limit data failed <error/simpleError>
# 3: model x check new [NULL]
# 4: total x data done 15
# -----------------------------
# <continued> last run: 2026-11-15 19:27:58 (0.004 s)
# steps: 2 done, 1 failed, 1 new
pip_run_state(pip)
# [1] "continued"Now total was calculated, while model was
skipped, because its input depends on the failed step. The
out column of a failed step holds the condition, which is
printed in short form as <kind/class>.
To work with it programmatically, we can extract it as usual:
out <- pip[["check", "out"]]
class(out)
# [1] "simpleError" "error" "condition"
out$message
# [1] "data exceeds limit"Let’s inspect the warning object returned by
tryCatch().
w <- tryCatch(
pip_run(pip, lgr = NULL, on_error = "continue"),
warning = \(w) w
)
names(w$failed)
# [1] "check"
w$skipped
# [1] "model"The warning is signalled after all states and outputs are stored, so
catching it with tryCatch() does not lose any results.
try()Sometimes a step should run even if one of its inputs failed, for
example a report that mentions what is missing or if a fallback is
available. Such an argument is marked by wrapping the step reference in
try():
pip_add(
pip,
"report",
\(total = ~total, m = ~try(model)) {
if (inherits(m, "pipeflow_failure")) {
paste("total:", total, "- no model:", m$message)
} else {
paste("total:", total, "- mean:", m)
}
}
)
suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.657 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.658 UTC]: Step 1/5 data - skipping done step
# info [2026-11-15 18:27:58.658 UTC]: Step 2/5 check
# error [2026-11-15 18:27:58.659 UTC]: data exceeds limit
# info [2026-11-15 18:27:58.660 UTC]: Step 3/5 model - skipping step, input 'x' failed
# info [2026-11-15 18:27:58.660 UTC]: Step 4/5 total - skipping done step
# info [2026-11-15 18:27:58.660 UTC]: Step 5/5 report
# info [2026-11-15 18:27:58.661 UTC]: Finished in 0.004 s (1 done, 3 skipped, 1 failed)
# warn [2026-11-15 18:27:58.661 UTC]: Run of pipeline 'analysis' continued after 1 failed step:
# 'check': data exceeds limit
# Steps not run because an input failed: 'model'
pip[["report", "out"]]
# [1] "total: 15 - no model: step 'model' was not run: step 'check' failed: data exceeds limit"Instead of the output of model, the m
argument received a failure object, a condition of class
pipeflow_failure, which stands in for the missing output.
The message tells why the input is not available. It has the fields
step: the step that provided the input
(model),failed_step: the step that actually failed
(check), andparent: the original error condition.A step is only run despite a failed input if all its failed
inputs are marked with try(). Everything else is skipped,
as model above. Note that try() has an effect
only in a continuing run. With on_error = "stop", the run
ends at the failed step anyway.
Both skipped and failed steps are retried in the next run, and so are steps that got a failure object. Steps that were completed are not run again. Let’s fix the cause by raising the limit and re-run:
pip_set_params(pip, list(limit = 10))
pip_run(pip)
# info [2026-11-15 18:27:58.678 UTC]: Starting run of pipeflow 'analysis'
# info [2026-11-15 18:27:58.678 UTC]: Step 1/5 data - skipping done step
# info [2026-11-15 18:27:58.679 UTC]: Step 2/5 check
# info [2026-11-15 18:27:58.679 UTC]: Step 3/5 model
# info [2026-11-15 18:27:58.680 UTC]: Step 4/5 total - skipping done step
# info [2026-11-15 18:27:58.680 UTC]: Step 5/5 report
# info [2026-11-15 18:27:58.684 UTC]: Finished in 0.006 s (3 done, 2 skipped, 0 failed)
pip
# <pipeflow> analysis (5 steps)
# -----------------------------
# step params depends state out
# 1: data n done 1,2,3,4,5
# 2: check x,limit data done 1,2,3,4,5
# 3: model x check done 3
# 4: total x data done 15
# 5: report total,m total,model done total: 15 - mean: 3
# -----------------------------
# <ready> last run: 2026-11-15 19:27:58 (0.006 s)
# steps: 5 doneThe run log shows that data and total were
skipped, since they were completed already. The failed step
check and the skipped step model were executed
now, and so was report, because it had received a failure
object instead of the real output. The report now contains the model
result and the run state is ready again.
Failures also work with the execution modes split and
reduce, which are described in the vignette on split, map, and reduce. Let’s
split a small data set by group and let the spread step
fail for groups with a single row.
d <- data.frame(g = c("a", "a", "b", "b", "c"), y = c(1, 2, 3, 5, 8))
pip <- pip_new("groups") |>
pip_add("data", \(data = NULL) data) |>
pip_add("by_group", \(data = ~data) split(data, data$g), exec = "split") |>
pip_add("spread", \(d = ~by_group) {
if (nrow(d) < 2) stop("need at least 2 rows")
diff(range(d$y))
}) |>
pip_add("label", \(s = ~spread) sprintf("spread: %g", s)) |>
pip_add(
"combine",
\(s = ~try(spread)) {
failed <- vapply(
s,
FUN = inherits,
what = "pipeflow_failure",
FUN.VALUE = logical(1)
)
unlist(s[!failed])
},
exec = "reduce"
)
pip_set_params(pip, list(data = d))If one partition (group c) fails, spread
fails for that key only. It gets state failed, but its
output keeps the results of the groups a and
b. For group c it holds a failure object,
which tells the key and keeps the original condition as
parent of a pipeflow_key_error:
suppressWarnings(pip_run(pip, on_error = "continue"))
# info [2026-11-15 18:27:58.739 UTC]: Starting run of pipeflow 'groups'
# info [2026-11-15 18:27:58.739 UTC]: Step 1/5 data
# info [2026-11-15 18:27:58.740 UTC]: Step 2/5 by_group
# info [2026-11-15 18:27:58.741 UTC]: Step 3/5 spread
# error [2026-11-15 18:27:58.744 UTC]: key 'c': need at least 2 rows
# info [2026-11-15 18:27:58.745 UTC]: Step 4/5 label
# info [2026-11-15 18:27:58.747 UTC]: Step 5/5 combine
# info [2026-11-15 18:27:58.747 UTC]: Finished in 0.009 s (3 done, 0 skipped, 2 failed)
# warn [2026-11-15 18:27:58.748 UTC]: Run of pipeline 'groups' continued after 2 failed steps:
# 'spread': key 'c': need at least 2 rows (1 of 3 keys)
# 'label': key 'c': input step 'spread' failed (1 of 3 keys)
pip
# <pipeflow> groups (5 steps)
# ---------------------------
# step params depends state out exec
# 1: data data done <data.frame[5x2]> auto
# 2: by_group data data done <list[3]> split
# 3: spread d by_group failed <partitioned: 1/3 failed> auto
# 4: label s spread failed <partitioned: 1/3 failed> auto
# 5: combine s spread done 1,2 reduce
# ---------------------------
# <continued> last run: 2026-11-15 19:27:58 (0.009 s)
# steps: 3 done, 2 failed
out <- pip[["spread", "out"]]
out$a
# [1] 1
fc <- out$c
class(fc)
# [1] "pipeflow_failure" "error" "condition"
fc$key
# [1] "c"
fc$message
# [1] "step 'spread' failed: key 'c': need at least 2 rows"The label step maps over spread like
spread maps over the groups, so the failure is handled per
key: label runs for groups a and
b, and is not run for group c, whose output is
again a failure object:
pip[["label", "out"]]$a
# [1] "spread: 1"
pip[["label", "out"]]$c$message
# [1] "step 'label' was not run: step 'spread' failed: key 'c': need at least 2 rows"The combine step reduces the outputs of all groups in
one call. Since it takes spread via try(), it
ran and received all of them, with the failure in group
c:
Without try(), combine would not be run, as
for any failed input. With s = ~try(spread) in
label, label would be called for group
c too and receive the failure object. If the pipeline is
run with on_error = "stop", the first failed group stops
the run with a pipeflow_step_error, with the
pipeflow_key_error as its parent.
If nothing changed in the pipeline since, the next run calls
spread only for the failed group c and keeps
the results of the other groups, which helps if the cause was outside of
the pipeline, e.g. a lost connection. Here the cause is in the data, so
we fix it by removing the group. That changes the input of all groups,
so spread is run for all of them, and the pipeline
completes:
pip_set_params(pip, list(data = d[d$g != "c", ]))
pip_run(pip)
# info [2026-11-15 18:27:58.801 UTC]: Starting run of pipeflow 'groups'
# info [2026-11-15 18:27:58.802 UTC]: Step 1/5 data
# info [2026-11-15 18:27:58.802 UTC]: Step 2/5 by_group
# info [2026-11-15 18:27:58.804 UTC]: Step 3/5 spread
# info [2026-11-15 18:27:58.805 UTC]: Step 4/5 label
# info [2026-11-15 18:27:58.806 UTC]: Step 5/5 combine
# info [2026-11-15 18:27:58.809 UTC]: Finished in 0.008 s (5 done, 0 skipped, 0 failed)
pip[["combine", "out"]]
# a b
# 1 2A pipeline can be turned into a function with
as.function() and added as a step to another pipeline, as
described in the vignette on nested
pipelines. Such a step runs a pipeline of its own, which means it
also has its own error handling.
Let’s put the spread step of the previous section into
an inner pipeline that reports on it, and map that pipeline over the
groups:
inner <- pip_new("inner") |>
pip_add("data", \(data = NULL) data) |>
pip_add("spread", \(d = ~data) {
if (nrow(d) < 2) stop("need at least 2 rows")
diff(range(d$y))
}) |>
pip_add("report", \(s = ~try(spread)) {
if (inherits(s, "pipeflow_failure")) "no spread" else s
})
nested <- pip_new("nested") |>
pip_add("data", \(data = NULL) data) |>
pip_add("by_group", \(data = ~data) split(data, data$g), exec = "split") |>
pip_add(
"per_group",
as.function(inner, out = "report"),
params = list(data = ~by_group)
)
pip_set_params(nested, list(data = d))By default, the inner pipeline is run with
on_error = "stop", so it ends at the first failed inner
step and the outer step fails with it. The try() reference
of the report step has no effect here, because
try() only matters in a continuing run:
pip_run(nested)
# info [2026-11-15 18:27:58.851 UTC]: Starting run of pipeflow 'nested'
# info [2026-11-15 18:27:58.851 UTC]: Step 1/3 data
# info [2026-11-15 18:27:58.852 UTC]: Step 2/3 by_group
# info [2026-11-15 18:27:58.853 UTC]: Step 3/3 per_group
# error [2026-11-15 18:27:58.868 UTC]: key 'c': step 'spread': need at least 2 rows
# Error:
# ! step 'per_group': key 'c': step 'spread': need at least 2 rowsThe message names the outer step, the group and the inner step. Each
of the conditions keeps the one of the level below as its
parent, so these can also be read from the error:
e <- tryCatch(pip_run(nested, lgr = NULL), error = identity)
e$step # the outer step
# [1] "per_group"
e$parent$key # the group
# [1] "c"
e$parent$parent$step # the inner step
# [1] "spread"
original <- e$parent$parent$parent
conditionMessage(original)
# [1] "need at least 2 rows"With as.function(on_error = "continue"), the inner
pipeline is run as pip_run(on_error = "continue") runs it:
inner steps that do not depend on the failed step still run, and
~try(spread) gives the report step the failure
object. Group c is handled inside the inner pipeline, and
the outer run completes:
pip_replace(
nested,
"per_group",
as.function(inner, out = "report", on_error = "continue"),
params = list(data = ~by_group)
)
pip_run(nested)
# info [2026-11-15 18:27:58.921 UTC]: Starting run of pipeflow 'nested'
# info [2026-11-15 18:27:58.921 UTC]: Step 1/3 data - skipping done step
# info [2026-11-15 18:27:58.921 UTC]: Step 2/3 by_group - skipping done step
# info [2026-11-15 18:27:58.921 UTC]: Step 3/3 per_group
# warn [2026-11-15 18:27:58.934 UTC]: Run of pipeline 'inner' continued after 1 failed step:
# 'spread': need at least 2 rows
# Warning: Run of pipeline 'inner' continued after 1 failed step:
# 'spread': need at least 2 rows
# info [2026-11-15 18:27:58.935 UTC]: Finished in 0.014 s (1 done, 2 skipped, 0 failed)
nested[["per_group", "out"]]
# $a
# [1] 1
#
# $b
# [1] 2
#
# $c
# [1] "no spread"
#
# attr(,"class")
# [1] "list" "pipeflow_partitioned"The warning of the continued inner run is passed on like any other warning of a step, so the outer run logs it and the result of every group is available.
The function still fails if one of the steps named in
out failed or was not run because an input failed: the
inner pipeline handles what it can, and everything else is left to the
outer run, as before.
pip_replace(
nested,
"per_group",
as.function(
inner,
out = "spread",
on_error = "continue" # <--- this time we continue
),
params = list(data = ~by_group)
)
pip_run(nested, lgr = NULL)
# Warning: Run of pipeline 'inner' continued after 1 failed step:
# 'spread': need at least 2 rows
# Error:
# ! step 'per_group': key 'c': step 'spread': need at least 2 rowsSince out = NULL requests the output of all
inner steps, name the steps you need in out to profit from
a continuing inner run.
pipeflow_step_error that names the step and holds the
original condition in parent.pip_run(on_error = "continue") marks the step as
failed, stores its condition as output, skips the steps
that depend on it, and runs everything else. The run state becomes
continued, and a pipeflow_run_failed warning
lists both failed and skipped steps.try(),
e.g. x = ~try(step), makes a step run despite a failed
input. The argument then receives a pipeflow_failure
condition.key. Steps mapped downstream are not run for
that key (unless they use try()), and a reduce
step gets the output with the failed keys if it uses try(),
and is skipped otherwise.as.function() runs a pipeline of its
own. as.function(on_error = "continue") lets that inner
pipeline continue after a failed inner step, which is also what makes
try() references inside it work. The step still fails if
one of the steps named in out failed or was not run.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.