The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
pip_add() and pip_replace() now signal an
error for params whose names are no arguments of the step
function, unless the function has a ... argument that takes
them.after of pip_add() moved to
the end, after exec, so that pip_add() and
pip_replace() share the order
x, step, fun, tags, params, exec. A call that gave
after by position must name it now.params of pip_add() and
pip_replace() override the defaults of fun
again, as before 0.4.0 (in 0.5.0 the defaults won and
pip_add() warned). They must be a list with unique names,
and the params of a step list the arguments of fun first,
followed by the other params.pip_set_params() on a view sets the values in every
step of the pipeline that has the param, also outside the view, so that
a param keeps one value. scope = "view" gives the old
behavior. An option named scope in ... no
longer reaches the methods of pip_param_value<-()..self is released when a step returns, so that step
outputs no longer keep the pipeline alive. A function returned by a step
that refers to .self fails with
object '.self' not found when it is called after the
run..self), or that a later step changes during the
run, now ends the run as "outdated" instead of
"done" and runs again in the next run, as do the steps that
ran with its output.on_error = "continue", a step mapped over
partitioned inputs fails per partition key: its output keeps the results
of the keys that succeeded, and each failed key holds a
pipeflow_failure with the field key. The step
gets state "failed". Mapped steps downstream are not run
for the failed keys, unless they take the input via ~try().
The pipeflow_run_failed warning has a new field
keys.pip_collect_out() and
pip_get_graph() were removed. Use
pip_collect() and pip_graph().%like%, %ilike%, %flike%,
%plike%, %chin%, %between%,
%inrange% and %notin%. They still work in the
filters of [, e.g. p[tags %like% "model"],
without attaching data.table. To use them elsewhere, call
library(data.table).as.function() turns a pipeline
into a function whose arguments are the unbound params of the pipeline,
so that a pipeline can be a step of another pipeline, e.g.
pip_add(outer, "step", as.function(inner), params = list(df = ~split)).
The exec modes map it over partitions, and the function keeps one inner
pipeline per outer step and partition key, so that a changed param only
reruns the affected inner steps. out can be a function that
is called with a light copy of the run inner pipeline,
on_error = "continue" continues the inner run,
lgr sets the logger of the inner run, and
cache = FALSE switches the kept pipelines off. New
pip_inner(x, step) returns the inner pipelines kept by such
a step, e.g. to debug an inner run. See
?as.function.pipeflow and the “Nested pipelines”
vignette.pip_add() takes a pipeline or view as step
and copies its steps, with their functions, params, tags and exec modes,
but without their upstream steps: references are resolved by step name
in the target pipeline, e.g.
pip_add(p, q[c("fit", "plot")]). If a step can’t be added,
the pipeline is left unchanged.pip_clone(out = ) copies a pipeline with the outputs of
the named steps only, e.g. to send the results of a run back from a
parallel worker.pip_output_value() is a generic for annotating the
output of a step when the step returns, e.g. to stamp a result with the
step it came from and the time of the run.
`pip_param_value<-`() is the write side of
pip_param_value(): pip_set_params() uses it to
keep the annotations of a stored object when it is given a plain value,
and passes ... on to its methods. It determines the values
for all affected steps before it changes anything, and reports all
failures at once with an error of class
pipeflow_set_params_error. See the vignette “Annotated step
outputs”.pip_set_params() can bind params to steps: a one-sided
formula as value,
e.g. pip_set_params(p, list(model = ~fit)), binds the param
in all selected, unlocked steps that have it unbound, as a formula does
in the defaults of a step function and in params of
pip_add(). (A formula was set as a value before; to do so
now, give it as a string and turn it into a formula in the step.)params of pip_add() and
pip_replace() can give values to parameters of
fun that have no default, so that existing functions can be
steps without a wrapper,
e.g. pip_add(p, "avg", mean, params = list(x = ~data, na.rm = TRUE)).pip_replace() gets the argument
keep_params. With TRUE, the new step keeps the
current values of the unbound params that are arguments of the new
function, unless params give them or the new function binds
them.pip_run() has a new argument map, a
function like lapply() that runs the keys of a step mapped
over partitions (exec = "auto"), e.g. in parallel with
parallel::mclapply or
future.apply::future_lapply.progress callback of
pip_run(): if it returns FALSE, the run is
halted before the next step.pip_clone(), which runs all keys again (known issue,
#172).pip_run() warns (class
pipeflow_outdated_before) if a step changes itself or a
done step before it via .self, as these steps then run
again in the next run.time_start (the start of
the step’s last run, also set if the run fails) and
duration (how long the last run took, as a
difftime). Both are NA for a step that has not
run since it was added, replaced or reset, and pipelines saved with an
earlier version get the columns when read.pip_run() names the pipeline and ends with a
summary of the whole run, including restarts, e.g.
Finished in 0.251 s (3 done, 1 skipped, 0 failed). The
footer of print() shows the duration of the last run and
the number of steps per state.pipeflow.lgr sets the default logger of
pip_run(), e.g.
options(pipeflow.lgr = function(level, msg) NULL) switches
logging off.pip_get_params(x, simplify = FALSE) returns the values
of all steps, grouped by param name and named by step, e.g. to find
params whose values differ between steps.pip_add(), pip_replace() and
rbind() warn (class pipeflow_param_conflict)
if an unbound param of a new step has another value than the same param
in other steps. Turn it off with
options(pipeflow.warn_param_conflicts = FALSE).[ warns if the selector is a column name that is also a
variable of the caller: with step <- "load",
p[step] selects by the step column.pip_graph(): the nodes have a title column
(shown by visNetwork on hover), the edges a dashes column
(a ~try() reference is dashed), and
format = "mermaid" returns the graph as a mermaid
flowchart, which renders e.g. on GitHub and in Quarto.pip_run() tracks the steps of a run by node id, so a
step that changes its own pipeline via .self (or a
progress callback) no longer makes the run run the wrong
step, fail with “subscript out of bounds” (#157) or run the dependents
of a renamed failed step with the failure as input (#160). A step that
was removed during the run is skipped, and a step that renames itself
keeps its output.pip_run() skips a step whose input is still
"new", e.g. a step added via .self during the
run, instead of running it with a NULL input.on_error = "stop"
stores its error as its output, as with "continue", instead
of keeping the output of an earlier attempt.fit(x = x), so errors and
pipeflow_failures no longer carry the inputs of the step in
their call (megabytes of data), and an output that keeps the frame of
the step call no longer keeps the pipeline.NULL is
kept, and a step mapped over it gets the argument.pip_lock() on a step that is not done marks its
downstream steps as outdated (#147).pip_reset() marks the downstream steps of the reset
steps as "outdated", and on a view it keeps the time of the
last run and the run state of the pipeline.exec or out of a step with
p[[step, "exec"]] <- or
p[[step, "out"]] <- (also via p[i, j] <-
and views) marks the affected steps as "outdated", so the
next run uses the new value. See ?Extract.pipeflow.p[i, j] <- value or
p[i, ] <- q[i, ], changes all selected steps or
none.p[i] <- q[j] keeps the params
given via params and resolves relative references such as
~-1 by name, also onto themselves.rbind() no longer drops arguments with a
NULL value from the step functions of the second pipeline,
and no longer fails if it references a step with a non-syntactic
name."source data", can
be referenced with backticks, x = ~`source data`.pip_add() and pip_rename() reject step
names that start with -, which a reference ~-1
would take as a relative reference.pip_rename() also renames the step in the failures of
failed keys if the step name has a name attribute, e.g. if it is taken
from p[["step"]].p[i, view = FALSE] no longer warns that an edge already
exists if a step refers to the same step with several arguments, and
v[, view = FALSE] returns a standalone pipeline instead of
the view.pip_set_params() on a pipeline or view without steps no
longer says that all selected steps are locked.pip_add(),
pip_remove(), pip_rename() and
pip_replace() for step name, function and params are the
same, e.g. pip_remove(p, "") says that the step must be a
non-empty string.pip_add(),
pip_clone() or rbind() points at
x[, view = FALSE].pip_set_params() about unknown params no
longer shows the call.%||% helper is used for NULL defaults.tar_meta()-like
table, reproducible random numbers, saved results, runs in a fresh R
process and outdated steps) and “Comparison and benchmarks” (formerly
“pipeflow vs targets”), and the vignettes “Annotated step outputs” and
“Nested pipelines”.?as.function.pipeflow shows the error messages of inner
steps, ?pip_add explains how outputs that keep an
environment can keep the whole pipeline, and ?pip_view says
that running a view also runs the upstream steps of its steps.v[["view"]] returns the names of the covered steps
instead of row indices..self is only made available in step functions that
refer to it, i.e. if the name .self appears in the function
body.pip_replace() by default now keeps the tags and the
execution mode of the replaced step.pip_run(on_error = "continue") continues the run of
unaffected steps after a step fails with an error.~try()
references (x = ~try(step)), e.g. by creating fallback
results for failed steps.pip_param_value() is a generic for unwrapping parameter
values, enabling users to set pipeline parameters that consist of
non-trivial types or classes.pip_set_params() gains
unknown = c("warn", "ignore", "error") for parameters that
no step defines, and force = TRUE to outdate steps even if
the values are identical.pip_run_state() returns the run state of a pipeline or
view.pipeflow_lgr() is exported, so
custom loggers can build on it.saveRDS() and readRDS(),
and can be sent to other R processes (see ?pip_new).pip_set_params() no longer outdates steps if the values
are identical to the stored ones.pip_run() name the step
(step 'fit': ...) and have class
pipeflow_step_error with the fields step and
parent. A failing partition key gives a
pipeflow_key_error with key and
parent."<name> view").pip_run() also re-runs a "done" step if
one of its inputs was executed in the same run.halt() ends in run state
"halted" (before: "ready").pip_add() and pip_replace() warn if
params contain names that are also arguments of
fun.print() now shows conditions, matrices, arrays, tables,
functions and environments in the out column in short form,
e.g. <error/simpleError>."new", and a run only
outdates the steps downstream of steps it executed.x[[step, j]] on a view read the wrong row.pip_clone(), p[] or
p[i, view = FALSE] shared .self with the
original.pip_set_params() no longer warns about parameters
defined only in locked steps.pip_add() and pip_replace() reject
primitive functions such as sum with a clear error, and
pip_add() calls .self a reserved
parameter name.unknown = "ignore", when .self is available,
and saving pipelines.p$add(...), p$run(...),
p$set_params(...), …p[[step, "out"]]),p[...] and
p[i, j] <- value,p[...] (data.table-style
expressions),p[[step, "step"]] <- NULL,p[[step, "time"]] <- value,p[["view"]] <- ....pip_collect() (formerly pip_collect_out())
supports grouping via a by argument,
as.table = TRUE to return a compact table, and a
simplify argument to control flattening of single-step
groups.pip_data() provides direct access to the underlying
step table.pip_graph() (formerly pip_get_graph())
returns visNetwork-compatible graph data. The old function names remain
available as deprecated aliases.depends, unbound and
nodeId are read-only and can’t be assigned directly.pip_view() gains a join = "union" argument
to combine filters as a logical OR..self is now automatically available inside every step
function (and is a protected parameter name), enabling pipelines to
modify themselves at runtime, including .self$restart() and
.self$halt().nodeId is now shown in the full print view
(cols = "all").lgr and jsonlite were
removed.pip_rename() no longer corrupts the parameters of
single-step pipelines.dim/length,
lock/unlock,
tag/untag,
set_params/get_params).pip_run() documents the runtime control flow
(restart(), halt()), and
pip_add() gained an example for the after
argument.src/dag.cpp): All
graph operations (node/edge add/remove, topological ordering,
reachability queries) run at C++ speed via Rcpp, eliminating R object
overhead for dependency resolution.pip_* API: Functional API
replacing the R6 Pipeline class — pip_new(),
pip_add(), pip_run(),
pip_replace(), pip_clone(),
pip_bind(), pip_view(),
pip_tag()/pip_untag(), etc.auto/split/reduce/plain):
Native support for map-reduce style workflows where steps split output
into named partitions and downstream steps auto-map over them.pip_add(), pip_replace(), and
pip_remove() fail fast on broken references.vignettes/articles/v07-vs-targets.Rmd) with an 18-row
feature comparison table and benchmarks.pipe_* functions and Pipeline R6
class are deprecated (preserved as aliases in R/aliases.R,
R/pipelineR6.R).pip_collect_out() no longer accepts
grouped or by parameters. It returns a flat
named list of step outputs. Use pip_view() with tags and
manual list composition for grouped output.group column has been removed from the pipeline
data.table. Steps previously using group = "..." in
pip_add() should use tags = "...".pip_* API.vignettes/v04-collect-output.Rmd to demonstrate
tag-based grouping via pip_view() composition.pip_* API,
aliases, execution modes, and recursive runs.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.