Introduction to jobqueue

Create a Queue

library(jobqueue)
q <- Queue$new()

When you create a Queue, several Worker processes are created in the background. You can then evaluate R code on those background processes while your main R process is free to do other work.

Important

The background processes are persisent R sessions. Avoid modifying their .GlobalEnv, otherwise your R code will produce different results based on which Worker evaluates it.

Create a Job

Main article: vignette('eval')

job <- q$run({ paste('Hello', 'World!') })

This Job will begin evaluating immediately, assuming no other Jobs are ahead of it in the Queue.

Access the Result

Main article: vignette('results')

job$result
#> [1] "Hello World!"

Running <Job>$result will block until the Job finishes and the result is ready.

Add a Callback Hook

Main article: vignette('hooks')

job$on('done', function (job) message(job$result))
#> [1] "Hello World!"

Adding a callback hook to trigger when the Job is done allows the result to be handled without blocking the main R process.

Interrupt Evaluation

Main article: vignette('stops')

job <- q$run({ Sys.sleep(10); 'Zzzzz' })
job$stop()
job$result
#> <interrupt: job stopped by user>

If the Job’s result is no longer needed and you want to free up compute resources, calling <Job>$stop() will terminate the background process. Terminated background process are automatically replaced by new ones.

Passing Data

# Variables to permanently store on the Worker.
q <- Queue$new(globals = list(MY_DATA = mtcars))

# Variables to temporary add to the Worker.
vars <- list(n = 2, fields = c('mpg', 'cyl', 'disp'))

# The expression to evaluate on the Worker.
expr <- quote(head(MY_DATA, n)[,fields])

job <- q$run(expr = expr, vars = vars)

job$result
#>               mpg cyl disp
#> Mazda RX4      21   6  160
#> Mazda RX4 Wag  21   6  160