The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Getting Started with staRburst

Introduction

staRburst makes it trivial to scale your parallel R code from your laptop to 100+ AWS workers. This vignette walks through setup and common usage patterns.

Which API should I use?

staRburst offers a few entry points. They share the same engine — pick the one that fits how your code is written:

Need	Use
Map a function over inputs (simplest)	`starburst_map()`
Existing `future` / `furrr` code	`plan(starburst)` then `future_map()`
Long-running job that survives disconnects	`starburst_session()`
Explicit, reusable worker cluster	`starburst_cluster()`
Configure account/backend defaults	`starburst_config()`

If you’re new, start with starburst_map() — the next section walks through a first job with it. If you already have future/furrr code, skip to Migrating existing future / furrr code; the two styles are equivalent and shown side by side there.

Which backend? EC2 (default) vs Fargate

Every API above runs on one of two compute backends, chosen with launch_type:

EC2 — the default and recommended choice. Faster (no cold start), cheaper with Spot (the default), and more flexible (any instance type, warm pools). starburst_setup() provisions its capacity provider for you, so it works out of the box.
Fargate — an optional serverless alternative (launch_type = "FARGATE"). Fully managed and needs no starburst_setup_ec2 / capacity provider, but has cold-start latency and is bounded by your account’s Fargate vCPU quota. Reach for it if you specifically want task-based serverless execution.

You don’t have to choose up front — leave the default (EC2) and add launch_type = "FARGATE" later if you want to compare.

How it works (the mental model)

        Local R session
              |
              |  (1) serialize your function + inputs + detected globals (qs2)
              |      AWS credentials are read HERE, from your local environment
              v
   S3 bucket  +  staRburst control plane (ECS/ECR)
              |
              |  (2) launch workers: EC2 (default, Spot) or Fargate
              |      each worker's R packages come from your renv.lock, baked
              |      into a Docker image cached in ECR
              v
   Remote R workers  (one task per input element)
              |
              |  (3) each worker pulls a task from S3, runs it, writes the
              |      result + logs back to S3
              v
   Local R session  <-- results collected in order (starburst_map / cluster)
        or
   Detached-session store in S3  <-- collected later via starburst_session_attach()

What to know from this picture:

Credentials are used only on your machine (to call AWS) and by the workers’ IAM role — you never put keys in your code or in S3.
Uploaded per task: your function, its inputs, and auto-detected globals — not your whole workspace. Large data should go to S3 yourself and be read on the worker.
Dependencies come from your renv.lock, installed into the worker image once and cached in ECR (this is the first-run build cost).
After a failure or disconnect, workers keep running against S3 until a timeout; nothing is silently lost. session$cleanup() (or normal completion of starburst_map()) tears down workers and removes the job’s S3 objects.
Detached sessions differ only in step 3’s return path: results persist in S3 so you can close R and collect later, instead of blocking the local session.

Installation

# Install from GitHub
remotes::install_github("scttfrdmn/starburst")

One-Time Setup

Before using staRburst, you need to configure AWS resources. This only needs to be done once.

library(starburst)

# Interactive setup wizard
starburst_setup()

This will: - Validate your AWS credentials - Create an S3 bucket for data transfer - Create an ECR repository for Docker images - Set up an ECS cluster and VPC resources - Provision the default EC2 capacity (launch template + Auto Scaling Group + capacity provider for c7g.xlarge) so the default EC2 backend works immediately — created at zero instances, so no compute cost until you run a job (skip with setup_ec2 = FALSE if you only use Fargate) - Check Fargate quotas and offer to request increases - Build the initial worker image

Provisioning the AWS resources takes about 2 minutes. On the first run, staRburst also builds the worker Docker image, which adds 5–10 minutes (it is cached and reused afterwards, and you can skip it with starburst_setup(build_image = FALSE) and build lazily on first job launch).

Your first job

The simplest way to use staRburst is starburst_map() — hand it your inputs and a function, and it runs the function over each input on AWS workers:

library(starburst)

# Define your work
expensive_simulation <- function(i) {
  # Some computation that takes a few minutes
  results <- replicate(1000, {
    x <- rnorm(10000)
    mean(x^2)
  })
  mean(results)
}

# Run across 50 cloud workers (EC2 by default)
results <- starburst_map(1:100, expensive_simulation, workers = 50)
#> [Starting] Starting starburst cluster with 50 workers
#> [Status] Processing 100 items with 50 workers
#> [Starting] Submitting 100 tasks...
#> [Wait] Progress: 100/100 (128.0s)
#> [OK] Completed in 128.0 seconds
#> [Cost] Estimated cost: $0.85

That is the whole workflow: starburst_setup() once, then starburst_map() per job. Everything below builds on this.

Migrating existing `future` / `furrr` code

If you already use future or furrr, you don’t need starburst_map() — set the plan to starburst and your existing future_map() / future_lapply() calls run on AWS unchanged. The two styles are equivalent; use whichever matches your code:

library(furrr)
library(starburst)

# Local baseline
plan(sequential)
results_local <- future_map(1:100, expensive_simulation)

# Same code, now on 50 cloud workers — just change the plan
plan(starburst, workers = 50)
results_cloud <- future_map(1:100, expensive_simulation)

# Results are identical to the local run
identical(results_local, results_cloud)
#> [1] TRUE

The two calls below do the same thing — pick the one that fits your codebase:

# Direct API
results <- starburst_map(1:100, expensive_simulation, workers = 50)

# future / furrr
plan(starburst, workers = 50)
results <- future_map(1:100, expensive_simulation)

Real-world examples

Rather than repeat toy snippets here, the article gallery carries complete, measured end-to-end examples — each with real runtimes, costs, and the batching choices that make (or break) a workload:

Monte Carlo simulation — many small simulations batched into a handful of tasks.
Bootstrap resampling — resampling a shared dataset across workers.
Grid search, risk modeling, geospatial, and more.

Before you size a job, read the two guides that report real numbers on when the cloud wins, when it loses, and how to pick task/worker counts: Workload Shapes and Performance. The short version: each task should be real work (seconds or more), and thousands of tiny tasks should be batched into dozens–hundreds — see Batch Small Tasks below.

Working with Data

Base R’s read.csv()/readRDS() cannot read s3:// URLs — you need an S3 client on the worker. A small helper keeps the examples below concrete and copy-pasteable (the worker image already includes paws.storage):

# Download an s3://bucket/key object to a temp file and read it on the worker.
read_s3 <- function(s3_uri, reader = readRDS) {
  m <- regmatches(s3_uri, regexec("^s3://([^/]+)/(.+)$", s3_uri))[[1]]
  bucket <- m[2]; key <- m[3]
  tmp <- tempfile()
  obj <- paws.storage::s3()$get_object(Bucket = bucket, Key = key)
  writeBin(obj$Body, tmp)
  reader(tmp)
}

Data Already in S3

If your data is already in S3, workers can read it directly:

plan(starburst, workers = 50)

results <- future_map(file_list, function(file) {
  # Each worker pulls its file from S3 (read_s3 defined above)
  data <- read_s3(sprintf("s3://my-bucket/%s", file), reader = read.csv)
  process(data)
})

Uploading Local Data

For smaller datasets, you can pass data as arguments:

# Load data locally
data <- read.csv("local_file.csv")

# staRburst automatically uploads to S3 and distributes
plan(starburst, workers = 50)

# `replicates` is your vector of inputs (e.g. bootstrap replicate ids)
results <- future_map(replicates, function(i) {
  # Each worker gets a copy of 'data'
  bootstrap_analysis(data, i)
})

Large Data Optimization

For very large objects, upload once to S3 yourself and have each worker read it from there, rather than serializing the object into every task:

# Upload once from your machine
paws.storage::s3()$put_object(
  Bucket = "my-bucket", Key = "large_data.rds",
  Body = "huge_file.rds"
)
s3_path <- "s3://my-bucket/large_data.rds"

# Workers read from S3 inside the task (read_s3 helper defined above)
plan(starburst, workers = 100)

# `tasks` is your vector of work items
results <- future_map(tasks, function(i) {
  data <- read_s3(s3_path)   # readRDS by default
  process(data, i)
})

Cost Management

Estimate Costs

# Check cost before running
plan(starburst, workers = 100, cpu = 4, memory = "8GB")
#> Estimated cost: ~$3.50/hour

Set Cost Limits

# Set an hourly cost ceiling (USD/hour) — jobs above this rate won't start
starburst_config(
  max_hourly_cost = 10,       # Don't start jobs estimated over $10/hour
  cost_alert_threshold = 5     # Warn at $5/hour
)

# Now jobs exceeding limit will error before starting
plan(starburst, workers = 1000)  # Would cost ~$35/hour
#> Error: Estimated cost ($35/hr) exceeds limit ($10/hr)

Cost estimates

After a run, staRburst reports an estimated cost — the measured worker runtime multiplied by the current AWS price for your instance type (On-Demand or Spot), looked up live from the AWS Pricing API (and cached; it falls back to built-in rates offline). It is a close estimate, not a figure from AWS billing / Cost Explorer.

plan(starburst, workers = 50)

results <- future_map(data, process)

#> Cluster runtime: ~23 minutes
#> Estimated cost: ~$1.34

Quota Management (Fargate backend)

This section applies to the Fargate backend. On the default EC2 backend, worker count is bounded by your normal EC2 On-Demand / Spot instance limits, not the Fargate vCPU quota below. If you stay on EC2 you can usually skip this.

Check Your Quota

starburst_quota_status()
#> Fargate vCPU Quota: 100 / 100 used
#> Allows: ~25 workers with 4 vCPUs each
#>
#> Recommended: Request increase to 500 vCPUs

Request Quota Increase

starburst_request_quota_increase(vcpus = 500)
#> Requesting Fargate vCPU quota increase:
#>   Current: 100 vCPUs
#>   Requested: 500 vCPUs
#>
#> [OK] Quota increase requested (Case ID: 12345678)
#> [OK] AWS typically approves within 1-24 hours

Wave-Based Execution

If you request more workers than your quota allows, staRburst automatically uses wave-based execution:

# Quota allows 25 workers, but you request 100
plan(starburst, workers = 100, cpu = 4)

#> [!] Requested: 100 workers (400 vCPUs)
#> [!] Current quota: 100 vCPUs (allows 25 workers max)
#>
#> [Plan] Execution plan:
#>   - Running in 4 waves of 25 workers each
#>
#> [TIP] Request quota increase to 500 vCPUs? [y/n]: y
#>
#> [OK] Quota increase requested
#> [Starting] Starting wave 1 (25 workers)...

results <- future_map(inputs, expensive_function)

#> [Wave] Wave 1: 100% complete (250 tasks)
#> [Wave] Wave 2: 100% complete (500 tasks)
#> [Wave] Wave 3: 100% complete (750 tasks)
#> [Wave] Wave 4: 100% complete (1000 tasks)

Troubleshooting

View Worker Logs

# View logs from most recent cluster
starburst_logs()

# View logs from specific task
starburst_logs(task_id = "abc-123")

# View last 100 log lines
starburst_logs(last_n = 100)

Check Cluster Status

starburst_status()
#> Active Clusters:
#>   • starburst-xyz123: 50 workers running
#>   • starburst-abc456: 25 workers running

Common Issues

Environment mismatch: Packages not found on workers

# Rebuild environment
starburst_rebuild_environment()

Task failures: Some tasks failing

# Check logs
starburst_logs(task_id = "failed-task-id")

# Often due to memory limits - increase worker memory
plan(starburst, workers = 50, memory = "16GB")  # Default is 8GB

Slow data transfer: Large objects taking too long

# Use Arrow for data frames
library(arrow)
write_parquet(my_data, "s3://bucket/data.parquet")

# Workers read Arrow
results <- future_map(1:100, function(i) {
  data <- read_parquet("s3://bucket/data.parquet")
  process(data, i)
})

Best Practices

1. Use for Right-Sized Workloads

✅ Good: Each task takes >5 minutes

# 100 tasks, each takes 10 minutes
# Local: 1000 minutes, Cloud: ~10 minutes

❌ Bad: Each task takes <1 minute

# 10000 tasks, each takes 30 seconds
# Startup overhead (45s) dominates

2. Batch Small Tasks

Instead of:

# 10,000 tiny tasks
results <- future_map(1:10000, small_function)

Do:

# 100 batches of 100 tasks each
batches <- split(1:10000, ceiling(seq_along(1:10000) / 100))

results <- future_map(batches, function(batch) {
  lapply(batch, small_function)
})

# Flatten results
results <- unlist(results, recursive = FALSE)

3. Use S3 for Large Data

Don’t:

big_data <- read.csv("10GB_file.csv")  # Upload for every task
results <- future_map(1:1000, function(i) process(big_data, i))

Do:

# Upload once to S3
tmp <- tempfile(fileext = ".csv"); write.csv(big_data, tmp, row.names = FALSE)
paws.storage::s3()$put_object(Bucket = "bucket", Key = "big_data.csv", Body = tmp)

# Workers read from S3 (read_s3 helper from "Working with Data" above)
results <- future_map(tasks, function(i) {
  data <- read_s3("s3://bucket/big_data.csv", reader = read.csv)
  process(data, i)
})

4. Set Reasonable Limits

starburst_config(
  max_hourly_cost = 50,            # Cap the hourly rate (USD/hour)
  cost_alert_threshold = 25        # Get warned early
)

5. Clean Up

# staRburst auto-cleans, but you can force it
plan(sequential)  # Switch back to local
# Old cluster resources are cleaned up automatically

Advanced: Custom Configuration

CPU and Memory

# High CPU, low memory (CPU-bound work)
plan(starburst, workers = 50, cpu = 8, memory = "16GB")

# Low CPU, high memory (memory-bound work)
plan(starburst, workers = 25, cpu = 4, memory = "32GB")

Timeout

# Increase timeout for long-running tasks (default 1 hour)
plan(starburst, workers = 10, timeout = 7200)  # 2 hours

Region

# Use specific region (default from config)
plan(starburst, workers = 50, region = "us-west-2")

Next Steps

Learn about Detached Sessions for long-running jobs
Explore Example Vignettes for real-world patterns
Review Security Best Practices guide
Read Troubleshooting Guide when stuck

Getting Help

GitHub Issues: https://github.com/scttfrdmn/starburst/issues
Email: help@starburst.ing
Documentation: https://starburst.ing

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.