The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
data.table
(thanks
@D3SL, #166).serialization()
deprecated in mirai 1.2.0 is now
removed.daemons(dispatcher = "thread")
implements threaded
dispatcher (experimental), a faster and more efficient alternative to
running dispatcher in a separate process.mirai_map()
adds [.progress_cli]
as an
alternative progress indicator, using the cli package to show % complete
and ETA.daemons()
gains argument ‘force’ to control whether
further calls reset previous settings for the same compute profile.daemon()
gains argument ‘asyncdial’ to allow control of
connection behaviour independently of what happens when the daemon
exits.daemons()
:
...
arguments are now dropped instead
of throwing an error. This allows ...
containing unused
arguments to be more easily passed from other functions.mirai_map()
:
x[.stop, .progress]
.mirai::mirai_map(1:4, Sys.sleep)[.progress]
.dispatcher()
drops argument ‘asyncdial’ as it is rarely
useful to set this here.everywhere()
now errors if the specified compute
profile is not yet set up, rather than fail silently.launch_local()
and launch_remote()
now
strictly require daemons to be set, and will error otherwise.serial_config()
now validates the arguments provided
and returns them as a list. This means any saved configurations from
previous package versions must be re-generated.daemons()
to correctly handle a vector of URLs
passed to ‘url’ again.mirai_map()[.flat]
assigning a
variable ‘typ’ to the calling environment.mirai()
,
mirai_map()
and the promises method.nanonext
>= 1.3.0.everywhere()
adds argument ‘.serial’ to accept
serialization configurations created by serial_config()
.
These allow normally non-exportable reference objects such as Arrow
Tables or torch tensors to be used seamlessly across parallel processes
without additional marshalling steps. Configurations apply on a per
compute profile basis.serialization()
is now deprecated in favour of the
above usage of everywhere()
, and will be removed in a
future version.mirai_map()
enhanced to perform multiple map over 2D
lists/vectors, allowing advanced patterns such as mapping over the rows
of a dataframe or matrix.[]
method gains the option
[.flat]
to collect and flatten results, avoiding
coercion.daemons(dispatcher = FALSE)
and
make_cluster()
by using asynchronous launches (thanks @mtmorgan #123).everywhere()
not reaching all
daemons when using dispatcher.stats
and utils
base packages.nanonext
>= 1.2.0.serialization()
function signature and return value
slightly modified for clarity. Successful registration / cancellation
messages are no longer printed to the console.dispatcher()
argument ‘retry’ now defaults to FALSE for
consistency with non-dispatcher behaviour.remote_config()
gains argument ‘quote’ to control
whether or not to quote the daemon launch command, and now works with
Slurm (thanks @michaelmayer2 #119).nanonext
>= 1.1.1.mirai_map()
for asynchronous parallel/distributed
map using mirai
, with promises
integration.
Allows recovery from partial failure or else early stopping, together
with optional progress reporting.
x[]
collects the results of a mirai_map x
,
waiting for all asynchronous operations to complete.x[.progress]
collects the results whilst showing a text
progress bar.x[.stop]
collects the results applying early-stopping,
which stops at the first error, and aborts remaining in-progress
operations.x[]
as a more efficient
equivalent of the interruptible call_mirai_(x)$data
.collect_mirai()
as a more efficient equivalent of
the non-interruptible call_mirai(x)$data
.unresolved()
, call_mirai()
,
collect_mirai()
and stop_mirai()
now accept a
list of ‘mirai’ such as that returned by mirai_map()
.daemons()
with new settings when the compute
profile is already set now implicitly resets daemons before applying the
new settings instead of silently doing nothing.daemons()
as automatic
re-tries are no longer performed for non-dispatcher daemons.dispatcher()
governs whether to
auto-retry in the dispatcher case.nanonext
>= 1.1.0.mirai()
and
everywhere()
:
environment()
to be supplied
to ‘.args’ or as the only element of ‘…’.mirai(x, x = 1)
.ssh_config()
improvements:
stop_mirai()
now resolves to an ‘errorValue’ 20
(operation canceled) in the case the asynchronous task was still ongoing
(thanks @jcheng5
#110).[
method for ‘miraiCluster’ no longer produces a
‘miraiCluster’ object (thanks @HenrikBengtsson #83).parallel
package is now only
loaded when first used.nanonext
>= 1.0.0.mirai()
and everywhere()
behaviour changed
such that ‘…’ args are now assigned to the global environment of the
daemon process.with()
method for mirai daemons, allowing for
example: with(daemons(4), {expr})
, where the daemons last
for the duration of ‘expr’.register_cluster()
for registering ‘miraiCluster’
as a parallel Cluster type (requires R >= 4.4).is.promising()
method for ‘mirai’ for the promises
package.$stack.trace
, and includes the trailing line
break for consistency with ‘as.character.error()’.serialization
adds arguments ‘class’ and ‘vec’ for
custom serialisation of all reference object types.ssh_config()
argument ‘timeout’ widened to
10 (seconds).daemons()
specifying ‘output = FALSE’ registering
as TRUE instead.everywhere()
specifying ‘.args’ as an
unnamed list or ‘.expr’ as a language object.local_url()
helper to construct a random
inter-process communications URL for local daemons (thanks @noamross #90).daemon()
argument ‘autoexit’ now accepts a signal value
such as tools::SIGINT
in order to raise it upon exit.daemon()
now records the state of initial global
environment objects (e.g. those created in .Rprofile) for cleanup
purposes (thanks @noamross #91).as.promise()
method for
‘mirai’.serialization()
for registering custom
serialization and unserialization functions when using daemons.call_mirai_()
, a user-interruptible version
of call_mirai()
suitable for interactive use.mirai()
interface:
everywhere()
now returns invisible NULL in the case the
specified compute profile is not set up, rather than error.mirai()
specifying a timeout when
daemons()
has not been set - the timeout begins immediately
rather than after the ephemeral daemon has connected - please factor in
a small amount of time for the daemon to launch.make_cluster()
now prints daemon launch commands where
‘url’ is specified without ‘remote’ whether or not interactive.make_cluster()
specifying only ‘url’ now succeeds with
implied ‘n’ of one.mirai()
specifying a language object by name for
‘.expr’ in R versions 4.0 and earlier.%...>%
with a ‘mirai’.everywhere()
evaluates an expression ‘everywhere’ on
all connected daemons for a compute profile. Resulting changes to the
global environment, loaded pacakges or options are persisted regardless
of the ‘cleanup’ setting (request by @krlmlr #80).host_url()
implemented as a helper function to
automatically construct the host URL using the computer’s hostname.daemon()
adds argument ‘autoexit’, which replaces
‘asyncdial’, to govern persistence settings for a daemon. A daemon can
now survive a host session and re-connect to another one (request by
@krlmlr #81).daemons(NULL)
implemented as a variant of
daemons(0)
which also sends exit signals to connected
persistent daemons.dispatcher()
argument ‘lock’ removed as this is now
applied in all cases to prevent more than one daemon dialling into a
dispatcher URL at any one time.daemon()
argument ‘cleanup’ simplified to a logical
argument, with more granular control offered by the existing integer
bitmask (thanks @krlmlr #79).ssh_config()
specifying tunnelling
to the ‘remote’ argument of daemons()
.%>>%
.make_cluster()
creates a ‘miraiCluster’, compatible
with all existing functions taking a ‘cluster’ object, for example in
the ‘parallel’ and ‘doParallel’ / ‘foreach’ packages.status()
can now take a ‘miraiCluster’ as the argument
to query its connection status.launch_remote()
improvements:
daemons()
gains the following features:
launch_remote()
.remote_config()
added to generate configurations for
directly launching remote daemons, and can be supplied directly to a
‘remote’ argument.ssh_config()
added as a convenience method to generate
launch configurations using SSH, including SSH tunnelling.mirai()
adds logical argument ‘.signal’ for whether to
signal the condition variable within the compute profile upon resolution
of the ‘mirai’.daemon()
argument ‘exitlinger’ retired as daemons now
synchronise with the host/dispatcher and exit as soon as possible
(although a default ‘exitlinger’ period still applies to ephemeral
daemons).daemons()
or
dispatcher()
now raise an error early rather than
attempting to launch daemons that fail.status()
if there is no longer a connection to
dispatcher.daemons()
,
dispatcher()
and daemon()
to clearly delineate
core vs peripheral options.%>>%
in
favour of a recommendation to use package mirai.promises
for performing side effects upon ‘mirai’ resolution.server()
for
daemon()
is retired.daemons()
gains the new argument ‘seed’ to set a random
seed for generating these streams.daemon()
and dispatcher()
gain the
argument ‘rs’ which takes a L’Ecuyer-CMRG random seed.nextstream()
and
nextget()
, opening interfaces for packages which extend
mirai
.--vanilla
flags for
efficiency, avoiding lengthy startup configurations (thanks @alexpiper).status()
occasionally failing to
communicate with dispatcher.saisei()
:
status()
:
saisei()
or
launch_local()
.saisei()
no longer resets the counter but instead
turns it negative until a new daemon connects.launch_remote()
.daemons()
and dispatcher()
gain the
argument ‘pass’ to support password-protected private keys when
supplying TLS credentials (thanks @wlandau #76).daemons()
,
launch_local()
or launch_remote()
now works
correctly as documented.tls+tcp://
or wss://
URL in
daemons()
. Single-use keys and certificates are
automatically generated.daemons()
and daemon()
, such as
those generated via a Ceritficate Signing Request (CSR) to a Certificate
Authority (CA).launch_remote()
launches daemons on remote machines
and/or returns the shell command for launching daemons as a character
vector.
launch_remote("ws://192.168.0.1:5555", command = "ssh", args = c("-p 22 192.168.0.2", .)
.daemon()
replaces server()
, which is
deprecated (although currently retained as an alias).launch_local()
replaces launch_server()
and now accepts a vector argument for ‘url’ as well as numeric values to
select the relevant dispatcher or host URL, returning invisible NULL
instead of an integer value.status()
now retrieves connections and daemons status,
replacing the call to daemons()
with no arguments (which is
deprecated). The return value of $daemons
is now always the
host URL when not using dispatcher.output=TRUE
for daemons()
or
launch_local()
. daemon()
accepts a new
‘output’ argument.saisei()
argument validation now happens prior to
sending a request to dispatcher rather than on dispatcher.saisei(i = 1L)
is called.daemons()
status requests have been rendered more
robust.saisei()
always returns NULL
if
‘tcp://’ URLs are being used as they do not support tokens.server()
and dispatcher()
argument
‘asyncdial’ is now FALSE by default, causing these functions to exit if
a connection is not immediately available. This means that for
distributed computing purposes, daemons()
should be called
before server()
is launched on remote resources, or else
server(asyncdial = TRUE)
allows servers to wait for a
connection.launch_server()
now parses the passed URL for
correctness before attempting to launch a server, producing an error if
not valid.%>>%
gains the
following enhancements:
.()
implemented to wrap a piped expression, ensuring
return of either an ‘unresolvedExpr’ or ‘resolvedExpr’.unresolved()
in the
same way as a ‘mirai’.mirai()
gains the following enhancements (thanks @HenrikBengtsson):
dispatcher()
gains the following arguments:
token
for appending a unique token to each URL the
dispatcher listens at.lock
for locking sockets to prevent more than one
server connecting at a unique URL.saisei()
implemented to regenerate the token used by a
given dispatcher socket.launch_server()
replaces launch()
for
launching local instances, with a simpler interface directly mapping to
server()
.mirai()
with timeouts and no connection to a server.dispatcher()
re-implemented using an innovative
non-polling design. Efficient process consumes zero processor usage when
idle and features significantly higher throughput and lower latency.
launch()
exported as a utility for easily re-launching
daemons that have timed out, for instance....
variables in the
daemons()
call..compute
argument in daemons()
and
mirai()
for sending tasks with heterogeneous compute
requirements.
daemons()
can create new profiles to connect to
different resources e.g. servers with GPU, accelerators etc.mirai()
tasks can be sent using a specific compute
profile.daemons()
interface has a new url
argument
along with dispatcher
for using a background dispatcher
process to ensure optimal FIFO task scheduling (now the default).
:0
will
automatically assign a free ephemeral port, with the actual port number
subsequently reported by daemons()
.daemons("view")
functionality.dispatcher()
is implemented as a new function for the
dispatcher.server()
gains the following arguments:
asyncdial
to specify how the server dials into the
client.maxtasks
for specifying a maximum number of tasks
before exiting.idletime
for specifying an idle time, since completion
of the last task before exiting.walltime
for specifying a soft walltime before
exiting.timerstart
for specifying a minimum number of task
completions before starting timers.daemons()
and
server()
now error and return immediately instead of
potentially causing a hang.eval_mirai()
is removed as an alias for
mirai()
.--vanilla
argument to enable site / user profile and
environment files to be read.call_mirai()
again (regression
in 0.7.0), now returning a ‘miraiInterrupt’.is_mirai_interrupt()
to test if
an object is a ‘miraiInterrupt’.daemons()
now takes ‘n’ and ‘.url’ arguments. ‘.url’ is
an optional client URL allowing mirai tasks to be distributed across the
network. Compatibility with existing interface is retained.server()
is exported for creating
daemon / ephemeral processes on network resources.daemons()
and
%>>%
deferred evaluation pipe.mirai()
gains a ‘.args’ argument for passing a list of
objects already in the calling environment, allowing for example
mirai(func(x, y, z), .args = list(x, y, z))
rather than
having to specify
mirai(func(x, y, z), x = x, y = y, z = z)
.is_mirai_error()
should
be used in place of is_nul_byte()
, which is no longer
re-exported.is_error_value()
can be used to test for all errors,
including timeouts where the ‘.timeout’ argument has been used.%>>%
deferred evaluation
pipe.mirai()
to ensure a mirai
always resolves even if the child process crashes etc.is_mirai()
to test if an object
is a mirai.$data
to
optimally align with the underlying implementation.mirai()
added as an alias for
eval_mirai()
; supports evaluating arbitrary length
expressions wrapped in {}
.call_mirai()
. Access the $value
directly and
an ‘unresolved’ logical NA will be returned if the async operation is
yet to complete.stop_mirai()
added as a function to stop evaluation of
an ongoing async operation.is_nul_byte()
and
unresolved()
re-exported from {nanonext} to test for
evaluation errors and resolution of a ‘mirai’ respectively.daemons()
interface to set and manage persistent
background processes for receiving ‘mirai’ requests.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.