Performance and caching

The read lifecycle

Every snapshot or change read follows the same broad path:

1. R requests the relevant file actions from the Delta Sharing server.
2. Missing Parquet and deletion-vector files are downloaded into the table’s session cache.
3. R writes a new local Delta log that points to the cached files.
4. Delta Kernel scans the local files and emits Arrow batches.
5. An eager materializer, when used, collects those batches into its result.

The stages have different reuse behavior:

Reusing cached files avoids downloading them again, but each materializer call still performs the Sharing request, local log construction, Delta Kernel scan, and result collection.

What the cache contains

Each table has a deterministic cache directory under R’s session temporary directory. Its identity includes the profile endpoint and the table’s share, schema, and name. New handles for the same table therefore share cached files, even when they use different download concurrency:

client <- sharing_client("~/config.share")

orders <- client$table("sales.default.orders", concurrency = 4)
faster_downloads <- client$table("sales.default.orders", concurrency = 8)

identical(orders$cache_path, faster_downloads$cache_path)
#> TRUE

Snapshot and change data feed reads for the same table use this shared cache. When the server selects the same underlying file for both reads, its local copy is reused. Files that do not overlap are downloaded normally.

The cache contains local copies of selected Parquet and deletion-vector files. Files are identified using immutable IDs supplied by the sharing server, so they can be reused even after a signed download URL changes. When the server provides a file size, the package checks it before reusing the cached file.

Downloads are completed in temporary files before becoming available to the cache, so interrupted transfers are not reused. Cache directories and files are created with user-only permissions.

Cache lifetime

The cache lasts for the R session, not for the lifetime of a table handle. It lives under tempdir(), R’s per-session temporary directory. It is normally removed when the session ends, but it is not durable storage: abnormal termination may leave files behind, and system temporary-directory policies may remove them during a long-running session. See ?tempdir for how R chooses the directory.

The cache is not persistent across sessions and does not contain materialized tibbles or Arrow tables.

The read-only cache_path property is available for inspection:

orders$cache_path
fs::dir_info(orders$cache_path)

After all lazy readers and streams for the table have been closed, the directory can be removed to reclaim disk space:

fs::dir_delete(orders$cache_path)

The next read recreates the directory and downloads any required files. A long session that touches many large tables can retain substantial data, so cache_path is also useful for monitoring local disk usage.

Download concurrency

Set concurrency when creating a table handle:

orders <- client$table(
  "sales.default.orders",
  concurrency = 8
)

The default is four. concurrency controls the maximum number of missing files downloaded at once. It does not affect Sharing API pagination, Delta Kernel scanning, Arrow batch size, or reads whose files are already cached.

Higher concurrency is most useful when a query selects many files and network latency leaves the connection idle. It may make little difference when there are few files, a single file saturates the connection, or the provider throttles requests. Benchmark representative uncached reads before increasing it.

Interactive reads show progress for missing downloads. When every file has a known size, the display also shows the total bytes to download. Cache hits are not counted as downloads.

Shape the read before tuning it

Query options affect different stages:

• columns reduces local scan and materialization work, but not network transfer or cache disk usage. The sharing server selects files before Delta Kernel applies the projection, and the complete selected Parquet files are downloaded.
• limit is sent to the server as a hint and is enforced exactly by Delta Kernel. A provider may use the hint to return fewer files, but this is not guaranteed.
• predicate is a best-effort server hint. It may improve file pruning, but it is not an exact row filter; apply an exact filter in the downstream consumer.

Choose a materializer

Materializers differ in how they consume Arrow batches:

Arrow is a required dependency. Eager R materializers use Arrow’s converter and return BIGINT columns as bit64::integer64. Close an Arrow reader when it is no longer needed, and release a low-level stream if its consumer does not take ownership.

reader <- orders$snapshot(
  columns = c("status", "amount")
)$to_arrow_reader()

con <- DBI::dbConnect(duckdb::duckdb())
duckdb::duckdb_register_arrow(con, "orders", reader)

summary <- withr::with_options(
  list(arrow.use_threads = FALSE),
  DBI::dbGetQuery(con, "
    SELECT
      status,
      count(*) AS orders,
      sum(amount) AS total_amount
    FROM orders
    GROUP BY status
    ORDER BY status
  ")
)

duckdb::duckdb_unregister_arrow(con, "orders")
DBI::dbDisconnect(con)

Arrow batch size

batch_size controls the maximum number of rows emitted in each Arrow batch. The default is 65,536 rows. It does not control the number or size of downloaded files.

Smaller batches can reduce peak memory while a batch is being processed, at the cost of more conversion and boundary overhead. Larger batches can reduce that overhead but use more memory. An eager materializer still collects every batch, so reducing batch_size does not make to_tibble(), to_data_frame(), or to_arrow() safe for a result larger than memory.

Change the default only after measuring a representative workload.

A practical tuning order

1. Use columns to reduce local scan and materialization work. Use limit and predicates as server-side pruning hints, while respecting their exact and best-effort semantics.
2. Choose the execution path for the result size: eager R materialization when it fits comfortably in memory, or DuckDB for larger-than-memory SQL analytics.
3. Reuse an existing result when the same data is needed again. Let the session cache handle overlapping reads automatically.
4. If downloading uncached files is the bottleneck, compare a small number of concurrency values on a representative read.
5. Tune batch_size last and only when measurements show batch overhead or memory pressure.

Measure end-to-end elapsed time, peak memory, and cache disk usage. Network throughput, latency, provider behavior, file count, compression, schema width, query shape, and local hardware all affect performance, so there is no universally optimal configuration.