---
title: "Citing a Set of Tables"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Citing a Set of Tables}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment  = "#>",
  eval     = FALSE
)
```

```{=html}
<link rel="stylesheet" href="../inst/vignette-setup/datom.css">
```

This article continues [Starting on S3](start-on-s3.html) **in the same R
session**. It reuses that article's secrets, settings and connections, and the
four tables it onboarded into `s3://study001/imported/`.

## The aim

You want to track liver safety for study001 as data arrives. For that you keep
a small, versioned collection: the input tables you used, and the output tables
you derived from them. Any result can then be traced back to the exact data
behind it.

<div class="datom-callout datom-callout-tip" markdown="1">
**What a datom set is**

- A named, versioned list of pointers to exact table versions. It holds no data.
- It never drifts. Each member stays pinned until you move it.
- Any change, to members or to labels, makes a new version. Old versions stay
  readable.
- Reading a set needs access to its own project only. Reading a member needs
  access to that member's project.
</div>

## Set up the project

The set lives in its own project, in a folder beside the imported tables:

```
s3://study001/
    imported/datom/        onboarded tables        (Starting on S3)
    liver-safety/datom/    the set and its outputs  (this article)
```

The settings this article adds:

```{r settings-liver-safety}
project_liver_safety <- "study001-liver-safety"
prefix_liver_safety  <- "liver-safety/"
repo_liver_safety    <- "study001-liver-safety"   # GitHub repo name
set_liver_safety     <- "study001-liver-safety"   # the set this project owns
workdir_liver_safety <- fs::path(tempdir(), "study001-liver-safety")
```

A writer store and a reader store, built the same way as before:

```{r stores-liver-safety}
store_write_liver_safety <- datom_store(
  data = datom_store_s3(
    bucket     = bucket,
    prefix     = prefix_liver_safety,
    region     = region,
    access_key = Sys.getenv("AWS_ACCESS_KEY_ID"),
    secret_key = Sys.getenv("AWS_SECRET_ACCESS_KEY")
  ),
  github_pat = Sys.getenv("GITHUB_PAT")
)

store_read_liver_safety <- datom_store(
  data = datom_store_s3(
    bucket     = bucket,
    prefix     = prefix_liver_safety,
    region     = region,
    access_key = Sys.getenv("AWS_ACCESS_KEY_ID"),
    secret_key = Sys.getenv("AWS_SECRET_ACCESS_KEY")
  )
)
```

Create the repository:

```{r init-liver-safety}
datom_init_repo(
  path         = workdir_liver_safety,
  project_name = project_liver_safety,
  store        = store_write_liver_safety,
  create_repo  = TRUE,
  repo_name    = repo_liver_safety,
  mode         = "product",
  set          = set_liver_safety
)
#> v Created GitHub repo ".../study001-liver-safety".
#> v Initialized datom repository "study001-liver-safety" at '.../study001-liver-safety'
```

`mode = "product"` makes this a repository that builds from other projects'
tables instead of taking files in. It owns exactly one set, the one named here,
and its `datom_sync()` maps that set against other projects rather than
importing files. `mode` accepts `"product"` or nothing, so `mode = "yolo"` is an
error.

Connect, once to write and once to read:

```{r conns-liver-safety}
conn_write_liver_safety <- datom_get_conn(
  path  = workdir_liver_safety,
  store = store_write_liver_safety
)

conn_read_liver_safety <- datom_get_conn(
  store        = store_read_liver_safety,
  project_name = project_liver_safety
)
```

From here on, writes go through a writer connection and reads through a reader

## Version 1: register every input

Register every table the imported project holds, at its current version. Some
may go unused; that is fine. A set records exact versions, so there is nothing
to pick.

This is the same two steps as syncing files: **map** what is there, review it,
then **apply** it. First the map. On a product repo, `datom_sync_manifest()`
compares the repo's set with the projects you pass as `sources`:

```{r v1-preview}
m <- datom_sync_manifest(
  conn    = conn_write_liver_safety,
  sources = list(conn_read_imported)
)
#> i Mapped 4 artifacts from 1 source: 4 new, 0 changed, 0 unchanged.

m[, c("project", "name", "kind", "status")]
#>             project name  kind status
#> 1 study001-imported   ae table    new
#> 2 study001-imported   dm table    new
#> 3 study001-imported   ex table    new
#> 4 study001-imported   lb table    new
```

`m` is a data frame with one row per table in the source. There is no set yet,
so every row is `new`. It also carries `version_from` and `version_to`, the
full version each member would move from and to, left out here to fit the page.
The call writes nothing. To leave a table out, drop its row, for example with
`subset(m, name != "ae")`.

Then apply it:

```{r v1}
x <- datom_sync(
  conn     = conn_write_liver_safety,
  manifest = m,
  sources  = list(conn_read_imported)
)
#> v Applied 4 rows: 4 added, 0 repointed.
#> project study001-imported:
#>   ae  added at 075773e9
#>   dm  added at 773e6862
#>   ex  added at 8dbcc9a7
#>   lb  added at 435bccb0
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v1 <- datom_write_set(
  conn    = conn_write_liver_safety,
  members = x,
  tags    = list(description = "Liver safety, study001")
)
#> v Wrote set "study001-liver-safety" (4 members): "d2d0456b"
```

`datom_sync()` adds one member per `new` row, labelled `type = "input"` unless
you pass other `tags`. It takes `sources` again because it reads each new
member from its own project first, which confirms the version exists. A reader
connection is enough for that.

Syncing a set writes nothing: it hands back the edited set, `x`, and says so.
`datom_write_set()` stores it. That is the one difference from syncing files,
where each table is saved as it syncs. A set is saved in one step, so you can
look at it, or add to it, before it becomes a version.

The write returns the new version invisibly:

```{r v1-version}
v1$metadata_sha
#> [1] "d2d0456b719244fe18d01f74a59b4e77637108405b716037c8e8b960dc15d786"
```

That string is the citation. Put it in a report and it resolves to these four
tables at these versions.

## Version 2: derive the output

The derivation is one function. Save it as `R/derive_liver_flags.R` inside
`workdir_liver_safety`:

```{r derive-script}
# R/derive_liver_flags.R
#
# One row per subject: age, sex, dose, and peak ALT and AST as a multiple of
# the upper limit of normal. Reads its inputs through the set `x`, so it uses
# exactly the versions the set pins.
derive_liver_flags <- function(x, conn_input) {
  dm <- datom_fetch_member(conn = conn_input, x = x, member = "dm")
  ex <- datom_fetch_member(conn = conn_input, x = x, member = "ex")
  lb <- datom_fetch_member(conn = conn_input, x = x, member = "lb")

  peak_xuln <- function(test) {
    rows <- lb[lb$LBTESTCD == test, ]
    tapply(X = rows$LBORRES / rows$LBORNRHI, INDEX = rows$USUBJID, FUN = max)
  }

  liver_flags <- merge(
    x  = dm[, c("USUBJID", "AGE", "SEX")],
    y  = ex[, c("USUBJID", "EXDOSE")],
    by = "USUBJID"
  )
  liver_flags$ALT_PEAK_XULN <- as.numeric(peak_xuln("ALT")[liver_flags$USUBJID])
  liver_flags$AST_PEAK_XULN <- as.numeric(peak_xuln("AST")[liver_flags$USUBJID])
  liver_flags$ELEVATED <- liver_flags$ALT_PEAK_XULN > 1 |
    liver_flags$AST_PEAK_XULN > 1

  liver_flags
}
```

It reads `dm`, `ex` and `lb` **through the set**, not by reaching into the
imported project for whatever is current. `ae` stays registered and unused: a
derivation can use a subset of the inputs. The function returns a data frame
and writes nothing, so it is easy to run and check on its own.

Load it, run it against version 1, write the output, and add it to the set:

```{r v2}
source(fs::path(workdir_liver_safety, "R", "derive_liver_flags.R"))

x <- datom_get_set(conn = conn_read_liver_safety, name = set_liver_safety)

liver_flags <- derive_liver_flags(x = x, conn_input = conn_read_imported)

liver_flags_written <- datom_write(
  conn    = conn_write_liver_safety,
  data    = liver_flags,
  name    = "liver_flags",
  parents = datom_parent(
    conn  = conn_read_imported,
    table = c("dm", "ex", "lb"),
    x     = x
  )
)
#> v Wrote "liver_flags" (full): "dafdb954"

x <- datom_add_member(
  x       = x,
  member  = "liver_flags",
  version = liver_flags_written$metadata_sha,
  tags    = list(type = "output"),
  conn    = conn_read_liver_safety
)
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v2 <- datom_write_set(
  conn          = conn_write_liver_safety,
  members       = x,
  include_paths = "R"
)
#> v Wrote set "study001-liver-safety" (5 members): "66d721f3"
```

`datom_parent()` with `x` records each input as a parent of `liver_flags`, at
the version the set pins. The output's own record then says exactly which data
it was derived from.

`datom_add_member()` adds the output by name. The name is looked up through
`conn`, a connection to the output's own project.

`include_paths = "R"` commits the derivation script in the same commit as the
set, so checking out that commit gives you the pointers and the code that
produced the output. Only the derivation code is kept. The setup and onboarding
code in these articles is not, because datom already records what it did.

Every output is derived from members of the set, and records them as its
parents. So the set carries its full provenance. `datom_write_set()` checks
this: if an output's parents name an input the set pins at a different
version, the write stops before anything is written.

## Use it

The labels you gave the members become the structure you read them through:

```{r use-structure}
x <- datom_get_set(conn = conn_read_liver_safety, name = set_liver_safety)
print(x)
#> 
#> -- datom set: "study001-liver-safety"
#> * Project: "study001-liver-safety"
#> * Version: "66d721f34ab2d8419b952fc267560218b06df6ea065669206277c1a1aa0a21ad"
#> * Members: 5
#> * Tags: description=Liver safety, study001
#>   * ae (table) type=input
#>   * dm (table) type=input
#>   * ex (table) type=input
#>   * lb (table) type=input
#>   * liver_flags (table) type=output
#> i Fetch a member with `datom_fetch_member(conn, x, "ae")`.

dp <- datom_structure_members(x = x, by = "type")

head(dp$output$liver_flags(conn = conn_read_liver_safety))
#> # A tibble: 6 x 7
#>   USUBJID         AGE SEX   EXDOSE ALT_PEAK_XULN AST_PEAK_XULN ELEVATED
#>   <chr>         <int> <chr>  <int>         <dbl>         <dbl> <lgl>   
#> 1 STUDY-001-001    71 F        200         0.416         0.648 FALSE   
#> 2 STUDY-001-002    27 M        200         0.711         0.585 FALSE   
#> 3 STUDY-001-003    35 M          0         0.911         0.455 FALSE   
#> 4 STUDY-001-004    68 M        200         0.752         0.728 FALSE   
#> 5 STUDY-001-005    43 M        200         0.846         0.535 FALSE   
#> 6 STUDY-001-006    60 F          0         0.404         0.722 FALSE

nrow(dp$input$lb(conn = conn_read_imported))
#> [1] 205
```

`type = "input"` and `type = "output"` are what produce `dp$input` and
`dp$output`. Each entry is a function: call it with a connection to the
member's project and it reads that member at its pinned version.

The same members as a data frame, one row per label. It also carries each
member's full `version` and `kind`, left out here to fit the page:

```{r use-list}
datom_list_members(x = x)[, c("name", "project", "key", "value")]
#>          name               project  key  value
#> 1          ae     study001-imported type  input
#> 2          dm     study001-imported type  input
#> 3          ex     study001-imported type  input
#> 4          lb     study001-imported type  input
#> 5 liver_flags study001-liver-safety type output
```

The set's own history, and version 1 read back by its version:

```{r use-history}
set_history <- datom_history(conn = conn_read_liver_safety,
                             name = set_liver_safety, short_hash = TRUE)
set_history[, c("version", "commit_message")]
#>    version                              commit_message
#> 1 66d721f3  Update study001-liver-safety: add 1 member
#> 2 d2d0456b Update study001-liver-safety: add 4 members

datom_get_set(
  conn    = conn_read_liver_safety,
  name    = set_liver_safety,
  version = v1$metadata_sha
)
#> 
#> -- datom set: "study001-liver-safety"
#> * Project: "study001-liver-safety"
#> * Version: "d2d0456b719244fe18d01f74a59b4e77637108405b716037c8e8b960dc15d786"
#> * Members: 4
#> * Tags: description=Liver safety, study001
#>   * ae (table) type=input
#>   * dm (table) type=input
#>   * ex (table) type=input
#>   * lb (table) type=input
#> i Fetch a member with `datom_fetch_member(conn, x, "ae")`.
```

## Refresh: new data arrives

The month-4 extract arrives. It updates the four tables and brings a fifth,
vital signs (`vs`). The imported project takes it in, as before:

```{r refresh-sync}
for (domain in c("dm", "ex", "lb", "ae", "vs")) {
  write.csv(
    x         = datom_example_data(domain = domain, cutoff_date = "2026-04-28"),
    file      = fs::path(inputs_imported, paste0(domain, ".csv")),
    row.names = FALSE
  )
}

manifest <- datom_sync_manifest(conn = conn_write_imported)
#> i Scanned 5 files: 1 new, 4 changed, 0 unchanged.

synced <- datom_sync(conn = conn_write_imported, manifest = manifest)
#> i Syncing 5 tables...
#> v Wrote "ae" (full): "97e2a95a"
#> v "ae" synced (changed).
#> v Wrote "dm" (full): "a87789e0"
#> v "dm" synced (changed).
#> v Wrote "ex" (full): "b59939d6"
#> v "ex" synced (changed).
#> v Wrote "lb" (full): "dc682195"
#> v "lb" synced (changed).
#> v Wrote "vs" (full): "28c748b0"
#> v "vs" synced (new).
#> i Sync complete: 5 succeeded, 0 failed, 0 skipped.
```

The set still points at the month-3 versions. A citation does not move on its
own. Move it in this order: the inputs, then the output derived from them, then
one write.

The inputs move the way they were first registered, with the same two calls:

```{r refresh-inputs}
m <- datom_sync_manifest(
  conn    = conn_write_liver_safety,
  sources = list(conn_read_imported)
)
#> i Mapped 5 artifacts from 1 source: 1 new, 4 changed, 0 unchanged.

m[, c("project", "name", "kind", "status")]
#>             project name  kind  status
#> 1 study001-imported   ae table changed
#> 2 study001-imported   dm table changed
#> 3 study001-imported   ex table changed
#> 4 study001-imported   lb table changed
#> 5 study001-imported   vs table     new

x <- datom_sync(
  conn     = conn_write_liver_safety,
  manifest = m,
  sources  = list(conn_read_imported)
)
#> v Applied 5 rows: 1 added, 4 repointed.
#> project study001-imported:
#>   ae  075773e9 -> 97e2a95a
#>   dm  773e6862 -> a87789e0
#>   ex  8dbcc9a7 -> b59939d6
#>   lb  435bccb0 -> dc682195
#>   vs  added at 28c748b0
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.
```

The four tables the set holds are `changed`, and `vs` is `new`, so it joins the
set as an input. `liver_flags` has no row: it belongs to this project, not to a
source, and moves only once it is derived again. A `changed` row keeps the
member's labels.

Then the output, derived from the inputs as `x` now pins them:

```{r refresh-output}
liver_flags <- derive_liver_flags(x = x, conn_input = conn_read_imported)

liver_flags_written <- datom_write(
  conn    = conn_write_liver_safety,
  data    = liver_flags,
  name    = "liver_flags",
  parents = datom_parent(
    conn  = conn_read_imported,
    table = c("dm", "ex", "lb"),
    x     = x
  )
)
#> v Wrote "liver_flags" (full): "0f89dd1b"

x <- datom_update_members(
  x    = x,
  conn = conn_read_liver_safety,
  tags = list(type = "output")
)
#> v Repointed 1 member, of 1 selected.
#> project study001-liver-safety:
#>   liver_flags  dafdb954 -> 0f89dd1b
#> i Nothing has been written. Write the set with `datom_write_set(conn, x)`.

v3 <- datom_write_set(
  conn          = conn_write_liver_safety,
  members       = x,
  include_paths = "R"
)
#> v Wrote set "study001-liver-safety" (6 members): "35e39f62"
```

`datom_update_members()` moves the selected members to their project's current
version and reports what moved. Nothing is written until `datom_write_set()`.

The derivation runs on `x` as you hold it, with the inputs already moved, so the
new `liver_flags` is built from month-4 data. The order matters, and the write
checks it. Had you written the set without deriving again, the old
`liver_flags` would still name the month-3 inputs as its parents while the set
pins month-4 ones, and `datom_write_set()` would stop and name them. Writing
once, after both the inputs and the output have moved, is what keeps version 3
from pairing new inputs with an output built from old ones.

## Who can read what

A set and one reader connection per project open everything in it:

| You hold | You can read |
|---|---|
| `conn_read_liver_safety` | the set, every version of it, and `liver_flags` |
| `conn_read_imported` | the input tables |

Neither needs a GitHub token or a clone. Someone who can read the liver-safety
project but not the imported one can still read and cite the set, and see which
inputs it names; they cannot open the inputs.

## Where you are

One project holds the study's onboarded tables. A second holds a set that names
exact versions of them, plus the output derived from them, the code that derived
it, and three versions of history. One string cites any of those versions.

Pooling several studies into one set is a separate article.

## Teardown

Delete both projects, storage first and then the repository each time:

```{r teardown}
datom_storage_delete_prefix(conn = conn_write_liver_safety)
datom_repo_delete(conn = conn_write_liver_safety, confirm = project_liver_safety)

datom_storage_delete_prefix(conn = conn_write_imported)
datom_repo_delete(conn = conn_write_imported, confirm = project_imported)
```

`datom_storage_delete_prefix()` deletes everything under that project's
`datom/` folder, and it does not ask first. The rest of the bucket, and the
bucket itself, are left alone. `datom_repo_delete()` deletes the GitHub
repository and the local clone, and does not touch storage.