The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
Streamlining Workflows
r Sys.Date()
2021-03-01
moab
and sge
schedulers (see
#74 for details)2017-04-30
moab
and sge
schedulers (see
#74 for details)2016-04-18
tl;dr (summary of changes)
run
function now accepts paths.# 'cherries' version
cd <path to pipelines>
flowr x=mypipeline
# 'dates' version
flowr x=<path to pipelines>/mypipeline
~/.flowr.conf
, one may specify their own structure -
enabling flexibility.# 'cherries' version: a fixed directory structure was recommended:
~/flowr
├── conf
│ ├── flowr.conf
├── pipelines
│ ├── sleep_pipe.R
├── runs
# 'dates' version: one may change the default paths to run, config files etc using
~/.flowr.conf
# this file controls the location of these folders:
flow_base_path ~/flowr # flowr home
flow_conf_path {{flow_base_path}}/conf # path to configuration files, not required if using ~/.flowr.conf
flow_run_path ~/flowr/runs # default home of all executed flows, you may change this
flow_pipe_paths ~/flowr/pipelines,<add new paths...> # multiple paths can be specified using ","
2015-12-03
tl;dr (summary of changes)
flowr.conf
(modules_cmds
and local_cores
), after
upgrading the package. Everything else is compatible with previous
versions.additions/changes to flowr.conf
file
New: option local_cores, which determines (max) number of cores to use when running local jobs.
New: flow_pipe_paths
now supports
multiple paths, separated by comma. The fetch_pipes()
would
split the vector at commas.
IMP: New version needs additional components in
the flowr.conf
file
New: Now you can add a module_cmds
variable to the config file, and this will be prefixed in all script of
the pipeline. An example could be:
# version >= 0.9.8.9004
# max number of cores to use when running on a local server
local_cores 4
# default module of a pipeline
# version >= 0.9.8.9015
module_cmds ''
# examples: one may define all modules used in a pipeline here,
# further one may specify any other command which should be run before
# script executes. #module_cmds 'module load samtools;export PATH=$PATH:/apps/bin'
addition/changes to status()
get_wds
/status
, so that if current wd contains
a flow_details file, status is shown for this folder and not
sub-folder(s).## now this works well !
flowr status x=.
addition/changes to run()
and
rerun()
functions
New: run function now accepts a custom
configuration [conf
], parameter. See
help(flowr::run)
for more details. This enables one, to
specify custom parameters used for that pipeline.
New: Now rerun()
supports multiple
folders. Basically, one may specify a parent folder which has multiple
flowr runs and ask it to re-run ALL of them again, from
a specific intermediate step.
New: Flowr creates a new folder if there are multiple samples in the flowmat; basically containerizes the run, keeping the logs clean and debugging life easier.
New: run()
now supports, re-running
as well. i.e. One would generate a new set of commands etc. but execute
in the previous folder; possibly from a inter-mediate step (experimental
feature).
other changes
to_flowdef()
can now guess
submission and dependency types (experimental feature).to_flowdef
now adds a parameter
nodes
, to enable specifying number of nodes required
per-job.opts_flow$get
replaces
get_opts
, for reliability etc. Also this closely follows
how knitr options are set.2015-10-02
status
column. Specifically, this uses information from other columns and
summarizes whether a specific step is pending
,
processing
, completed
or
errored
.================================================================================
| | total| started| completed| exit_status|status |
|:---------------|-----:|-------:|---------:|-----------:|:----------|
|001.alnCmd1 | 109| 109| 109| 0|completed |
|007.markCmd | 3| 3| 0| 0|processing |
[0.9.7.11]
Introducing verbose levels:
One can set the level of verboseness using
opts_flow$set(verbose=2)
. Where the level may be 0, 1, 2….
Level 1 is good for most purposes, where as, level 0 is almost silent,
producing messages only when necessary. While level 2 is good when
developing a new pipeline, additional details useful for debugging are
provided by level 3.
Detailed checking of flowdef
checking if required columns are present...
checking if resources columns are present...
checking if dependency column has valid names...
checking if submission column has valid names...
checking for missing rows in def...
checking for extra rows in def...
checking submission and dependency types...
jobname prev.sub_type --> dep_type --> sub_type: relationship
1: aln1_a none --> none --> scatter
2: aln2_a scatter --> none --> scatter
3: sampe_a scatter --> serial --> scatter rel: complex one:one
4: fixrg_a scatter --> serial --> scatter rel: complex one:one
5: merge_a scatter --> gather --> serial rel: many:one
6: markdup_a serial --> serial --> serial rel: simple one:one
7: target_a serial --> serial --> serial rel: simple one:one
8: realign_a serial --> burst --> scatter rel: one:many
9: baserecalib_a scatter --> serial --> scatter rel: complex one:one
10: printreads_a scatter --> serial --> scatter rel: complex one:one
2015-08-22 (0.9.7.10)
fetch_pipe()
mypipeline()
, mypipeline
is the name of the
pipeline.load_conf()
fetch_pipe
, and load it
using as.flowdef()
to_flow()
kill()
: now a S3 functions, and operates on both a flow
object and flow_wd foldercheck()
: Now works on flowdef and flowmatfetch()
along with fetch_pipes()
and
fetch_conf()
simplify finding filesparams
pkg
read_sheet()
, write_sheet()
get_opts()
, set_opts()
.load_conf()
load_conf()
2015-04-07
2015-03-16
TODO
2014-12-05
Versioning:
Loosely, following fruits A-Z.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.