Citing a Set of Tables

This article continues Starting on S3 in the same R session. It reuses that article’s secrets, settings and connections, and the four tables it onboarded into s3://study001/imported/.

The aim

You want to track liver safety for study001 as data arrives. For that you keep a small, versioned collection: the input tables you used, and the output tables you derived from them. Any result can then be traced back to the exact data behind it.

What a datom set is

A named, versioned list of pointers to exact table versions. It holds no data.
It never drifts. Each member stays pinned until you move it.
Any change, to members or to labels, makes a new version. Old versions stay readable.
Reading a set needs access to its own project only. Reading a member needs access to that member’s project.

Set up the project

The set lives in its own project, in a folder beside the imported tables:

s3://study001/
    imported/datom/        onboarded tables        (Starting on S3)
    liver-safety/datom/    the set and its outputs  (this article)

The settings this article adds:

project_liver_safety <- "study001-liver-safety"
prefix_liver_safety  <- "liver-safety/"
repo_liver_safety    <- "study001-liver-safety"   # GitHub repo name
set_liver_safety     <- "study001-liver-safety"   # the set this project owns
workdir_liver_safety <- fs::path(tempdir(), "study001-liver-safety")

A writer store and a reader store, built the same way as before:

store_write_liver_safety <- datom_store(
  data = datom_store_s3(
    bucket     = bucket,
    prefix     = prefix_liver_safety,
    region     = region,
    access_key = Sys.getenv("AWS_ACCESS_KEY_ID"),
    secret_key = Sys.getenv("AWS_SECRET_ACCESS_KEY")
  ),
  github_pat = Sys.getenv("GITHUB_PAT")
)

store_read_liver_safety <- datom_store(
  data = datom_store_s3(
    bucket     = bucket,
    prefix     = prefix_liver_safety,
    region     = region,
    access_key = Sys.getenv("AWS_ACCESS_KEY_ID"),
    secret_key = Sys.getenv("AWS_SECRET_ACCESS_KEY")
  )
)

Create the repository:

datom_init_repo(
  path         = workdir_liver_safety,
  project_name = project_liver_safety,
  store        = store_write_liver_safety,
  create_repo  = TRUE,
  repo_name    = repo_liver_safety,
  mode         = "product",
  set          = set_liver_safety
)
#> v Created GitHub repo ".../study001-liver-safety".
#> v Initialized datom repository "study001-liver-safety" at '.../study001-liver-safety'

mode = "product" makes this a repository that builds from other projects’ tables instead of taking files in. It owns exactly one set, the one named here, and its datom_sync() maps that set against other projects rather than importing files. mode accepts "product" or nothing, so mode = "yolo" is an error.

Connect, once to write and once to read:

conn_write_liver_safety <- datom_get_conn(
  path  = workdir_liver_safety,
  store = store_write_liver_safety
)

conn_read_liver_safety <- datom_get_conn(
  store        = store_read_liver_safety,
  project_name = project_liver_safety
)

From here on, writes go through a writer connection and reads through a reader

Version 1: register every input

Register every table the imported project holds, at its current version. Some may go unused; that is fine. A set records exact versions, so there is nothing to pick.

This is the same two steps as syncing files: map what is there, review it, then apply it. First the map. On a product repo, datom_sync_manifest() compares the repo’s set with the projects you pass as sources:

m <- datom_sync_manifest(
  conn    = conn_write_liver_safety,
  sources = list(conn_read_imported)
)
#> i Mapped 4 artifacts from 1 source: 4 new, 0 changed, 0 unchanged.

m[, c("project", "name", "kind", "status")]
#>             project name  kind status
#> 1 study001-imported   ae table    new
#> 2 study001-imported   dm table    new
#> 3 study001-imported   ex table    new
#> 4 study001-imported   lb table    new

m is a data frame with one row per table in the source. There is no set yet, so every row is new. It also carries version_from and version_to, the full version each member would move from and to, left out here to fit the page. The call writes nothing. To leave a table out, drop its row, for example with subset(m, name != "ae").

Then apply it:

x <- datom_sync(
  conn     = conn_write_liver_safety,
  manifest = m,
  sources  = list(conn_read_imported)
)
#> v Applied 4 rows: 4 added, 0 repointed.
#> project study001-imported:
#>   ae  added at 075773e9
#>   dm  added at 773e6862
#>   ex  added at 8dbcc9a7
#>   lb  added at 435bccb0
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v1 <- datom_write_set(
  conn    = conn_write_liver_safety,
  members = x,
  tags    = list(description = "Liver safety, study001")
)
#> v Wrote set "study001-liver-safety" (4 members): "d2d0456b"

datom_sync() adds one member per new row, labelled type = "input" unless you pass other tags. It takes sources again because it reads each new member from its own project first, which confirms the version exists. A reader connection is enough for that.

Syncing a set writes nothing: it hands back the edited set, x, and says so. datom_write_set() stores it. That is the one difference from syncing files, where each table is saved as it syncs. A set is saved in one step, so you can look at it, or add to it, before it becomes a version.

The write returns the new version invisibly:

v1$metadata_sha
#> [1] "d2d0456b719244fe18d01f74a59b4e77637108405b716037c8e8b960dc15d786"

That string is the citation. Put it in a report and it resolves to these four tables at these versions.

Version 2: derive the output

The derivation is one function. Save it as R/derive_liver_flags.R inside workdir_liver_safety:

# R/derive_liver_flags.R
#
# One row per subject: age, sex, dose, and peak ALT and AST as a multiple of
# the upper limit of normal. Reads its inputs through the set `x`, so it uses
# exactly the versions the set pins.
derive_liver_flags <- function(x, conn_input) {
  dm <- datom_fetch_member(conn = conn_input, x = x, member = "dm")
  ex <- datom_fetch_member(conn = conn_input, x = x, member = "ex")
  lb <- datom_fetch_member(conn = conn_input, x = x, member = "lb")

  peak_xuln <- function(test) {
    rows <- lb[lb$LBTESTCD == test, ]
    tapply(X = rows$LBORRES / rows$LBORNRHI, INDEX = rows$USUBJID, FUN = max)
  }

  liver_flags <- merge(
    x  = dm[, c("USUBJID", "AGE", "SEX")],
    y  = ex[, c("USUBJID", "EXDOSE")],
    by = "USUBJID"
  )
  liver_flags$ALT_PEAK_XULN <- as.numeric(peak_xuln("ALT")[liver_flags$USUBJID])
  liver_flags$AST_PEAK_XULN <- as.numeric(peak_xuln("AST")[liver_flags$USUBJID])
  liver_flags$ELEVATED <- liver_flags$ALT_PEAK_XULN > 1 |
    liver_flags$AST_PEAK_XULN > 1

  liver_flags
}

It reads dm, ex and lb through the set, not by reaching into the imported project for whatever is current. ae stays registered and unused: a derivation can use a subset of the inputs. The function returns a data frame and writes nothing, so it is easy to run and check on its own.

Load it, run it against version 1, write the output, and add it to the set:

source(fs::path(workdir_liver_safety, "R", "derive_liver_flags.R"))

x <- datom_get_set(conn = conn_read_liver_safety, name = set_liver_safety)

liver_flags <- derive_liver_flags(x = x, conn_input = conn_read_imported)

liver_flags_written <- datom_write(
  conn    = conn_write_liver_safety,
  data    = liver_flags,
  name    = "liver_flags",
  parents = datom_parent(
    conn  = conn_read_imported,
    table = c("dm", "ex", "lb"),
    x     = x
  )
)
#> v Wrote "liver_flags" (full): "dafdb954"

x <- datom_add_member(
  x       = x,
  member  = "liver_flags",
  version = liver_flags_written$metadata_sha,
  tags    = list(type = "output"),
  conn    = conn_read_liver_safety
)
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v2 <- datom_write_set(
  conn          = conn_write_liver_safety,
  members       = x,
  include_paths = "R"
)
#> v Wrote set "study001-liver-safety" (5 members): "66d721f3"

datom_parent() with x records each input as a parent of liver_flags, at the version the set pins. The output’s own record then says exactly which data it was derived from.

datom_add_member() adds the output by name. The name is looked up through conn, a connection to the output’s own project.

include_paths = "R" commits the derivation script in the same commit as the set, so checking out that commit gives you the pointers and the code that produced the output. Only the derivation code is kept. The setup and onboarding code in these articles is not, because datom already records what it did.

Every output is derived from members of the set, and records them as its parents. So the set carries its full provenance. datom_write_set() checks this: if an output’s parents name an input the set pins at a different version, the write stops before anything is written.

Use it

The labels you gave the members become the structure you read them through:

x <- datom_get_set(conn = conn_read_liver_safety, name = set_liver_safety)
print(x)
#> 
#> -- datom set: "study001-liver-safety"
#> * Project: "study001-liver-safety"
#> * Version: "66d721f34ab2d8419b952fc267560218b06df6ea065669206277c1a1aa0a21ad"
#> * Members: 5
#> * Tags: description=Liver safety, study001
#>   * ae (table) type=input
#>   * dm (table) type=input
#>   * ex (table) type=input
#>   * lb (table) type=input
#>   * liver_flags (table) type=output
#> i Fetch a member with `datom_fetch_member(conn, x, "ae")`.

dp <- datom_structure_members(x = x, by = "type")

head(dp$output$liver_flags(conn = conn_read_liver_safety))
#> # A tibble: 6 x 7
#>   USUBJID         AGE SEX   EXDOSE ALT_PEAK_XULN AST_PEAK_XULN ELEVATED
#>   <chr>         <int> <chr>  <int>         <dbl>         <dbl> <lgl>   
#> 1 STUDY-001-001    71 F        200         0.416         0.648 FALSE   
#> 2 STUDY-001-002    27 M        200         0.711         0.585 FALSE   
#> 3 STUDY-001-003    35 M          0         0.911         0.455 FALSE   
#> 4 STUDY-001-004    68 M        200         0.752         0.728 FALSE   
#> 5 STUDY-001-005    43 M        200         0.846         0.535 FALSE   
#> 6 STUDY-001-006    60 F          0         0.404         0.722 FALSE

nrow(dp$input$lb(conn = conn_read_imported))
#> [1] 205

type = "input" and type = "output" are what produce dp$input and dp$output. Each entry is a function: call it with a connection to the member’s project and it reads that member at its pinned version.

The same members as a data frame, one row per label. It also carries each member’s full version and kind, left out here to fit the page:

datom_list_members(x = x)[, c("name", "project", "key", "value")]
#>          name               project  key  value
#> 1          ae     study001-imported type  input
#> 2          dm     study001-imported type  input
#> 3          ex     study001-imported type  input
#> 4          lb     study001-imported type  input
#> 5 liver_flags study001-liver-safety type output

The set’s own history, and version 1 read back by its version:

set_history <- datom_history(conn = conn_read_liver_safety,
                             name = set_liver_safety, short_hash = TRUE)
set_history[, c("version", "commit_message")]
#>    version                              commit_message
#> 1 66d721f3  Update study001-liver-safety: add 1 member
#> 2 d2d0456b Update study001-liver-safety: add 4 members

datom_get_set(
  conn    = conn_read_liver_safety,
  name    = set_liver_safety,
  version = v1$metadata_sha
)
#> 
#> -- datom set: "study001-liver-safety"
#> * Project: "study001-liver-safety"
#> * Version: "d2d0456b719244fe18d01f74a59b4e77637108405b716037c8e8b960dc15d786"
#> * Members: 4
#> * Tags: description=Liver safety, study001
#>   * ae (table) type=input
#>   * dm (table) type=input
#>   * ex (table) type=input
#>   * lb (table) type=input
#> i Fetch a member with `datom_fetch_member(conn, x, "ae")`.

Refresh: new data arrives

The month-4 extract arrives. It updates the four tables and brings a fifth, vital signs (vs). The imported project takes it in, as before:

for (domain in c("dm", "ex", "lb", "ae", "vs")) {
  write.csv(
    x         = datom_example_data(domain = domain, cutoff_date = "2026-04-28"),
    file      = fs::path(inputs_imported, paste0(domain, ".csv")),
    row.names = FALSE
  )
}

manifest <- datom_sync_manifest(conn = conn_write_imported)
#> i Scanned 5 files: 1 new, 4 changed, 0 unchanged.

synced <- datom_sync(conn = conn_write_imported, manifest = manifest)
#> i Syncing 5 tables...
#> v Wrote "ae" (full): "97e2a95a"
#> v "ae" synced (changed).
#> v Wrote "dm" (full): "a87789e0"
#> v "dm" synced (changed).
#> v Wrote "ex" (full): "b59939d6"
#> v "ex" synced (changed).
#> v Wrote "lb" (full): "dc682195"
#> v "lb" synced (changed).
#> v Wrote "vs" (full): "28c748b0"
#> v "vs" synced (new).
#> i Sync complete: 5 succeeded, 0 failed, 0 skipped.

The set still points at the month-3 versions. A citation does not move on its own. Move it in this order: the inputs, then the output derived from them, then one write.

The inputs move the way they were first registered, with the same two calls:

m <- datom_sync_manifest(
  conn    = conn_write_liver_safety,
  sources = list(conn_read_imported)
)
#> i Mapped 5 artifacts from 1 source: 1 new, 4 changed, 0 unchanged.

m[, c("project", "name", "kind", "status")]
#>             project name  kind  status
#> 1 study001-imported   ae table changed
#> 2 study001-imported   dm table changed
#> 3 study001-imported   ex table changed
#> 4 study001-imported   lb table changed
#> 5 study001-imported   vs table     new

x <- datom_sync(
  conn     = conn_write_liver_safety,
  manifest = m,
  sources  = list(conn_read_imported)
)
#> v Applied 5 rows: 1 added, 4 repointed.
#> project study001-imported:
#>   ae  075773e9 -> 97e2a95a
#>   dm  773e6862 -> a87789e0
#>   ex  8dbcc9a7 -> b59939d6
#>   lb  435bccb0 -> dc682195
#>   vs  added at 28c748b0
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

The four tables the set holds are changed, and vs is new, so it joins the set as an input. liver_flags has no row: it belongs to this project, not to a source, and moves only once it is derived again. A changed row keeps the member’s labels.

Then the output, derived from the inputs as x now pins them:

liver_flags <- derive_liver_flags(x = x, conn_input = conn_read_imported)

liver_flags_written <- datom_write(
  conn    = conn_write_liver_safety,
  data    = liver_flags,
  name    = "liver_flags",
  parents = datom_parent(
    conn  = conn_read_imported,
    table = c("dm", "ex", "lb"),
    x     = x
  )
)
#> v Wrote "liver_flags" (full): "0f89dd1b"

x <- datom_update_members(
  x    = x,
  conn = conn_read_liver_safety,
  tags = list(type = "output")
)
#> v Repointed 1 member, of 1 selected.
#> project study001-liver-safety:
#>   liver_flags  dafdb954 -> 0f89dd1b
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v3 <- datom_write_set(
  conn          = conn_write_liver_safety,
  members       = x,
  include_paths = "R"
)
#> v Wrote set "study001-liver-safety" (6 members): "35e39f62"

datom_update_members() moves the selected members to their project’s current version and reports what moved. Nothing is written until datom_write_set().

The derivation runs on x as you hold it, with the inputs already moved, so the new liver_flags is built from month-4 data. The order matters, and the write checks it. Had you written the set without deriving again, the old liver_flags would still name the month-3 inputs as its parents while the set pins month-4 ones, and datom_write_set() would stop and name them. Writing once, after both the inputs and the output have moved, is what keeps version 3 from pairing new inputs with an output built from old ones.

Who can read what

A set and one reader connection per project open everything in it:

You hold	You can read
`conn_read_liver_safety`	the set, every version of it, and `liver_flags`
`conn_read_imported`	the input tables

Neither needs a GitHub token or a clone. Someone who can read the liver-safety project but not the imported one can still read and cite the set, and see which inputs it names; they cannot open the inputs.

Where you are

One project holds the study’s onboarded tables. A second holds a set that names exact versions of them, plus the output derived from them, the code that derived it, and three versions of history. One string cites any of those versions.

Pooling several studies into one set is a separate article.

Teardown

Delete both projects, storage first and then the repository each time:

datom_storage_delete_prefix(conn = conn_write_liver_safety)
datom_repo_delete(conn = conn_write_liver_safety, confirm = project_liver_safety)

datom_storage_delete_prefix(conn = conn_write_imported)
datom_repo_delete(conn = conn_write_imported, confirm = project_imported)

datom_storage_delete_prefix() deletes everything under that project’s datom/ folder, and it does not ask first. The rest of the bucket, and the bucket itself, are left alone. datom_repo_delete() deletes the GitHub repository and the local clone, and does not touch storage.