Package {datom}

datom: A Unified Framework for Versioned, Traceable Tabular Data

Description

Provides versioned storage for tabular data without a database or a server. Each table is written as an immutable, content-addressed version – identical content is detected and stored only once – while its version history and metadata are kept as code in a 'git' repository and the data itself in a local filesystem or cloud object storage ('S3'). Any past version can be read back exactly by its identifier, and each table records the sources it was derived from, so a project carries full data lineage. A lightweight reader role retrieves current or historical data from storage alone, without 'git' or write access, giving downstream analyses and pipelines a single versioned source of truth. It targets analytical and scientific data management, such as preparing clinical study datasets, and is designed as a foundation for higher-level governance tooling.

Key terms

• Project: one body of data you manage together, such as one clinical study. It has a name, a GitHub repository that records every change, and a storage location that holds the data itself; the data never goes into git. Created once with datom_init_repo().

• Store: tells datom where a project's data lives (a local folder or an S3 bucket) and, if you will write, your GitHub token. Built with datom_store().

• Developer and reader: the two roles. A developer has a GitHub token and a local copy of the project, and can write. A reader needs only access to the storage – no token, no git – and can only read. datom picks the role from whether the store carries a token.

• Connection (conn): a pointer to one project, returned by datom_get_conn(). It records which project, where its data is kept, and your role, and it is the first argument to almost every other function.

• Table: a data frame saved in datom under a name, such as dm. Saved with datom_write() or datom_sync(), read with datom_read().

• Version: a long identifier for one exact saved state of a table or a set. Old versions are never overwritten, and a save that changes nothing makes no new version. datom_history() lists them; pass one as ⁠version =⁠ to read it back.

• Sync manifest: the preview datom_sync_manifest() returns – one row per file in ⁠input_files/⁠, each marked new, changed, unchanged, or in a format datom cannot read – which you then pass to datom_sync().

• Set: a named, versioned list of exact versions of tables (or other sets), so a whole collection can be cited with one version string. It holds no data.

• Member: one entry in a set – one table or set, pinned at one version, with optional labels such as type = "input".

• Parent, source and lineage: a parent is a table another table was made from, named with datom_parent() before writing. A source is an original imported table at the start of the chain. Lineage is the record of both, read with datom_get_lineage().

Author(s)

Maintainer: Afshin Mashadi-Hossein amashadihossein@gmail.com [copyright holder]

See Also

Useful links:

• https://github.com/amashadihossein/datom

• https://amashadihossein.github.io/datom/

• Report bugs at https://github.com/amashadihossein/datom/issues

Abbreviate SHA Hash

Description

Truncates a SHA-256 hash to a short prefix for display. Accepts character vectors; NA values pass through unchanged.

Usage

.datom_abbreviate_sha(sha, n = 8L)

Arguments

Value

Character vector of abbreviated hashes.

Add This Edit's Rows to Whatever Log the Object Already Carries

Description

One log for every edit verb, appended to rather than replaced, and that is what makes a chain of edits produce one honest commit message. With a verb owning its own attribute, ⁠update |> remove |> write⁠ commits a message naming the repoints and silent about the removal – and a destructive edit is the one a ⁠git log⁠ reader most wants named. So an entry says which action it records, and a third editing verb costs an action value rather than a new attribute.

Usage

.datom_append_edits(x, rows)

Arguments

Details

The log is an attribute rather than a field, following the link's carried member record: the write reads tags and members off a set and nothing else, so an attribute cannot reach the payload by construction.

Repointing a member and then removing it leaves both entries. That is an honest history of the edits and slightly odd in a commit message; collapsing them would mean one verb reasoning about the other's rows.

Three actions are written: repoint by datom_update_members(), remove by datom_remove_members(), and add by datom_add_member().

Value

x, with the log extended.

Build the Relative Key for an Artifact's Current-State Metadata

Description

Build the Relative Key for an Artifact's Current-State Metadata

Usage

.datom_artifact_meta_key(name, which = c("metadata", "version_history"))

Arguments

Value

Character relative key, e.g. "dm/.metadata/metadata.json".

Build the Relative Key for an Artifact's Payload

Description

The stored data object for an artifact: parquet for a table, JSON for a set. This is the single place that decision is made.

Usage

.datom_artifact_payload_key(name, sha, kind = c("table", "set"))

Arguments

Value

Character relative key, e.g. "dm/9f2a....parquet".

Build the Relative Key for a Versioned Metadata Snapshot

Description

Note this is a different directory from the payload key: the snapshot lives under ⁠.metadata/⁠ and is addressed by metadata_sha (the version), whereas the payload sits beside it addressed by data_sha (the content). Both end in .json for a set, which is exactly why they are easy to confuse.

Usage

.datom_artifact_snapshot_key(name, metadata_sha)

Arguments

Value

Character relative key, e.g. "dm/.metadata/c3d4....json".

Select the Artifacts of One Kind

Description

The one place the artifact list is filtered by kind. Four counters need it – two in the manifest's stored summary block, plus the numbers datom_summary() and datom_status() count for themselves – and a predicate written out at each of them is a predicate that can differ at one of them.

Usage

.datom_artifacts_of_kind(artifacts, kind)

Arguments

Details

An entry that is not a named list is skipped rather than dereferenced. The upgrade step deliberately passes such an entry through untouched, because it has no shape to convert; without the check here, that preserved entry reaches entry$kind and aborts with "$ operator is invalid for atomic vectors". datom_status() is the one that must not do that: it exists to describe a connection when the manifest cannot be trusted, and this count sits outside the error handling that gives it that tolerance. A hand-edited manifest is exactly the document most likely to reach it.

Skipping is not the same as tolerating a missing kind, which stays deliberately uncounted: a typed entry with no type means the conversion was skipped, and a visibly wrong count is the intended signal for that.

Value

The entries of that kind, names preserved.

Assign One Leaf Into a Nested List, Creating Branches on the Way

Description

Recursive rather than iterative because the depth is the length of the axis vector, and tree[[c("a", "b")]] <- v fails when the intermediate list does not exist yet.

Usage

.datom_assign_leaf(tree, path, value)

Arguments

Value

tree, with the leaf assigned.

Human Label for a Connection's Storage Backend

Description

The word to put in a user-facing message for the backend a connection uses: "S3", "local", or the raw backend name for anything not in the table.

Usage

.datom_backend_label(conn)

Arguments

Details

One table, not five. This mapping was written out at four call sites and was about to be written at a fifth, which is the shape of the defect the artifact-kind predicate produced: the same rule copied per site, until one copy lost a term. There is nothing to disagree about here yet, which is exactly when it is cheap to make disagreement impossible.

Value

A single string.

Build a datom_conn from Store Components

Description

Backend-aware helper that creates the appropriate client (S3 client or NULL) and assembles a datom_conn. Used by datom_init_repo(), .datom_get_conn_developer(), and .datom_get_conn_reader().

Usage

.datom_build_init_conn(
  project_name,
  data_store,
  path,
  role,
  endpoint = NULL,
  gov_store = NULL,
  gov_local_path = NULL,
  data_repo_url = NULL,
  github_pat = NULL,
  github_api_url = NULL
)

Arguments

Value

A datom_conn object.

Build Metadata Object

Description

Constructs the metadata list for a table write, including auto-computed fields (data_sha, dimensions, colnames, timestamp, datom_version) and any user-supplied custom metadata.

Usage

.datom_build_metadata(
  data,
  data_sha,
  custom = NULL,
  table_type = "derived",
  size_bytes = NULL,
  parents = NULL,
  source_lineage = NULL,
  original_file_sha = NULL,
  original_format = NULL,
  project = NULL
)

Arguments

Value

Named list suitable for writing as metadata.json. Always carries kind = "table" (which artifact kind the document describes), schema_version (the format the document is written in) and hash_algo = "datom-cv1", and declares parquet_sha (left NULL here and populated by datom_write() after change detection, since the stored- object hash is not knowable until then; it is excluded from metadata_sha so this deferred assignment is safe).

Build Full S3 URI

Description

Convenience function that combines bucket and key into an S3 URI.

Usage

.datom_build_s3_uri(bucket, key)

Arguments

Value

Character string S3 URI.

Build the Metadata Document for a Set Write

Description

A set's metadata.json is a collapsed version of a table's: schema_version, kind, data_sha, hash_algo, document_sha, project, created_at, datom_version, and no more. Everything a table carries that describes a rectangle (nrow, ncol, colnames), the provenance axis (table_type, parents, source_lineage), the stored-parquet facts (parquet_sha, size_bytes) and the user-metadata channel (custom) are all omitted, not nulled – a set's members and its user metadata both live in the payload as tags, and no counter reads a set's byte size.

Usage

.datom_build_set_metadata(payload, document_sha = NULL, project = NULL)

Arguments

Details

Kept beside .datom_build_metadata() on purpose: the two documents are close enough that a field copied from the wrong one is easy to miss, and two of the values here are exactly that kind of trap.

• data_sha comes from .datom_canonical_set_hash(), the datom-sv1 identity engine, not from the table hash. Computed here rather than passed in, so a caller cannot hand a set a table-regime hash.

• hash_algo is the literal "datom-sv1". The encoder embeds that string inside the digest but nothing stamps the field, so the builder must. A copied "datom-cv1" would leave a set claiming one regime while hashing under the other, and no hash comparison would notice.

Value

Named list of exactly the fields a set's metadata.json carries, named in the description above. Deliberately not stated as a count: the count was written down in six places and went stale in all of them the first time a field was added.

Build S3 Object Key

Description

Constructs S3 keys from path components, inserting the ⁠datom/⁠ segment per the storage structure convention.

Usage

.datom_build_storage_key(prefix = NULL, ...)

Arguments

Details

Mapping from arguments to key, for reference:

("proj", "customers", "abc123.parquet")
  -> "proj/datom/customers/abc123.parquet"

("proj", "customers", ".metadata", "metadata.json")
  -> "proj/datom/customers/.metadata/metadata.json"

("proj", ".metadata", "dispatch.json")
  -> "proj/datom/.metadata/dispatch.json"

(NULL, "customers", "abc123.parquet")
  -> "datom/customers/abc123.parquet"

Value

Character string S3 key.

Compute the datom-cv1 Canonical Content Hash

Description

The I/O-free identity engine for datom-cv1. Computes data_sha from the in-memory logical values only – no parquet write, no CSV, no temp files, no as.data.frame() or coercion, and never invokes arrow. Columns are read via data[[i]] / names(data) and dimensions via nrow() / ncol(), so two frames with equal values hash identically regardless of container class (tibble vs data.frame vs grouped_df), row names, or arrow version.

Usage

.datom_canonical_hash(data)

Arguments

Details

Before encoding, every column is scanned through .datom_hash_recourse(); if any are unsupported the function aborts once, listing every offender with its class and canonical recourse. This fires during data_sha computation (step 1 of datom_write()), before any git or storage mutation, so a refusal leaves no partial state.

The final hash is sha256( "datom-cv1" || f64le(nrow) || f64le(ncol) || concat(col_digest_hex...) ).

The per-column digests are an intermediate only and are never returned or persisted. A per-column digest lets anyone holding metadata confirm a guess about one column's values, and metadata is meant to describe a table's shape without revealing its values.

Value

A list with data_sha (character).

Compute the datom-sv1 Canonical Set-Content Hash

Description

The identity engine for a set artifact: data_sha for the payload's semantic content. data_sha = h(0x06 || utf8("datom-sv1") || set(payload)).

Usage

.datom_canonical_set_hash(payload)

Arguments

Details

The hash domain is the parsed-JSON data model, not the in-memory R object, and the write path – which necessarily starts from an in-memory object – agrees with it by construction rather than through a serialize-and-reparse pass. Each way a JSON round trip could mutate a value is unrepresentable instead of handled: there are no numbers (so integer-versus-double cannot arise), NA aborts and absence is omission (so neither the string "NA" nor null can appear), and a single string hashes equal to a one-element array (so scalar-versus-length-1 is not a question). One structural condition remains: the payload must be parsed with simplifyVector = FALSE, so members[] stays a list of records instead of collapsing into a data frame. Both storage backends already do that.

No I/O, no serializer, and no dependency that carries versioned data – which is what makes the hash stable across R versions, jsonlite versions, platforms, and architectures.

Value

A 64-character SHA-256 hex string.

Carry Unrecognised Top-Level Fields Onto a Rebuilt Document

Description

Copies onto rebuilt every top-level field of prior whose name is not in known, so a field this build cannot place survives being rewritten.

Usage

.datom_carry_unknown_fields(rebuilt, prior, known)

Arguments

Details

Only unrecognised fields are carried, and that narrowness is the design. A field datom knows about keeps exactly the behaviour it has today, including disappearing when this write does not set it. original_format is the case that makes the difference concrete: a table first imported from a CSV and later written straight from a data frame has no format to declare, and the row is meant to stop claiming one. Carrying every absent field forward instead of only the unplaceable ones would leave that claim standing against a version it does not describe – a wrong statement, which is worse than a missing one.

Where rebuilt already has a field, rebuilt wins. That cannot happen for a genuinely unrecognised field, since this build only writes names it knows; stating the precedence costs one term and removes the question.

Top-level only, at each level separately. A field nested inside a value datom does understand – inside custom, or inside the manifest's summary block – is not this function's business: custom is carried whole as one recognised field, and summary is a derived aggregate that is meant to be recomputed.

A field whose value is JSON null gets no special handling. Absence in a datom document is spelled by omitting the key, never by nulling it, so such a field is already off-convention; it is carried, but a null re-serialises as an empty object rather than as null.

Value

rebuilt, with the unrecognised fields of prior appended.

Refuse to Write One Kind of Artifact Over Another

Description

One name means one artifact, whatever its kind, because both kinds store everything under ⁠{name}/⁠ – a set named dm beside a table named dm would write the same dm/.metadata/metadata.json and each would clobber the other.

Usage

.datom_check_artifact_kind(
  current,
  name,
  expected,
  operation = c("write", "read")
)

Arguments

Details

Checked against the metadata document in storage, not against the manifest. The manifest is a projection and can lag behind a write that got partway through, so it can say a name is free when it is not. The document is also the copy .datom_has_changes() has just read, so the comparison costs no extra round trip – which is why the current document is passed in rather than fetched here.

An absent kind reads as "table": every document written before the field existed describes a table, because sets did not exist. The pairing with a format check is not needed here the way it is in datom_member() – a document from a future datom has already been refused at the write entry, and on the read path .datom_read_metadata() has just checked the same document.

Both directions of the same invariant, in one function. A read that meets the other kind needs different words and a different suggested verb from a write that does, which operation selects – following .datom_check_schema_version(), which took exactly that shape for exactly this reason. A separate read-side twin would let the two directions drift, each passing its own tests, while the rule they enforce is single: one name is one artifact. For the same reason both aborts carry one condition class, so no test can key on one direction alone.

The check is made by each verb after its own .datom_read_metadata() call rather than inside that function, because the two verbs want different answers from it.

Value

Invisibly NULL. Aborts on a kind mismatch.

Validate Data Store Reachability

Description

Checks that the data store at the ref-resolved location is reachable. For S3: HeadBucket. For local: dir_exists. Provides actionable error messages when data is unreachable after migration.

Usage

.datom_check_data_reachable(conn, migrated = FALSE)

Arguments

Value

Invisible TRUE on success. Warns on network error (offline use ok).

Refuse a Document Carrying a Field This Build Cannot Place

Description

Compares one document's top-level key names against the names this build can classify, and aborts naming any it cannot. The evidence is in the file: no version comparison, no configuration, no network.

Usage

.datom_check_document_vocabulary(doc, known, source)

Arguments

Details

Chosen over a declared version floor as the primary mechanism for one reason – it cannot be forgotten. A floor protects a repo only if somebody remembers to raise it; this fires on the evidence whether or not anyone did anything.

Top-level keys only, and that is a scope rather than a shortcut. It means do not descend into a value – custom holds arbitrary user keys by design and is classified as one recognised field, and the manifest's summary block is a derived aggregate that is rebuilt on every write. It does not mean skip the manifest's artifact entries: an entry is its own document for this purpose and gets checked against its own vocabulary.

The check cannot fire on the upgrade path, and no code guards against that: a newer build's vocabulary is a superset of every older build's, so it can never meet a name it does not know. A directional special case would be dead code protecting an unreachable state.

The accepted cost is that any release adding a field to a datom-owned document forces a fleet-wide writer upgrade, cosmetic additions included. Writes are infrequent, done by few people, and they change content – a false refusal costs one person an install, a miss costs corrupted data.

Value

Invisibly NULL. Aborts on an unclassifiable key.

Check git2r Availability

Description

Aborts with a helpful message if git2r is not installed.

Usage

.datom_check_git2r()

Value

Invisible TRUE if available.

Check Local Branch is Current with Remote

Description

Fetches from the remote and compares local HEAD SHA against the upstream HEAD SHA. If the local branch is behind, aborts with a clear message telling the developer to pull first.

Usage

.datom_check_git_current(path, pat = NULL)

Arguments

Details

Does NOT auto-pull - lets the developer decide how to resolve.

If the fetch fails (offline, unreachable remote), this warns and returns TRUE without comparing anything: the cached remote-tracking refs may be arbitrarily stale, so acting on them would abort an offline developer for being behind a remote they cannot reach. The backstop is .datom_git_push(), which pulls and aborts if the push is rejected, so a write cannot land on storage from a stale base.

Value

Invisible TRUE if the local branch is up to date.

Validate Git Remote Reachability

Description

Checks that the data git remote URL is reachable and that credentials work. Called at conn-construction time in .datom_get_conn_developer() alongside .datom_check_data_reachable().

Usage

.datom_check_git_reachable(conn)

Arguments

Details

Failure behaviour:

• No data_repo_url: returns invisibly (structural pass, no network needed).

• HTTPS, auth failure (HTTP 401/403): hard abort pointing to github_pat.

• HTTPS, URL not found (HTTP 404): hard abort.

• HTTPS, network error (timeout/DNS): warn-only (offline-tolerant).

• SSH, any error: warn-only (cannot reliably distinguish "no agent" from "offline").

• SSH, success: invisible TRUE.

Value

Invisible TRUE on success. Warns on network error (offline use ok).

Refuse a Repo With No Remote, Before git2r Does It Unhelpfully

Description

.datom_git_push() reads git2r::remotes(repo)[[1L]], which subscripts an empty list on a repo with no remote and fails with R's own out-of-bounds error – a message naming nothing the caller can act on. A data repo is required to have a remote, so this is an edge rather than a scenario, but datom_repo_push() is the first thing somebody points at a half-configured repo.

Usage

.datom_check_git_remote(path, verb)

Arguments

Details

Deliberately not inside .datom_git_push(): putting it there would change what four existing callers do on a repo they have never met in that state. Both new verbs call it, so there is no second copy.

Value

The remote name.

Connection Requirements Shared by the Two Git-Mutation Verbs

Description

Both verbs need the same three things and nothing else: a real connection, a developer role, and a local clone to operate on.

Usage

.datom_check_git_verb_conn(conn, verb)

Arguments

Value

Invisible TRUE.

Check the Caller's Extra Paths Before a Set Write Does Anything

Description

include_paths is the only way a commit datom makes on its own initiative may carry a path datom does not own, and it is allowed only because the caller enumerated it (R14.3). What it buys is that checking out a set version's commit yields the data pointers and the code and environment that produced them. So every refusal here is a refusal to produce a commit that would claim more than it holds.

Usage

.datom_check_include_paths(conn, name, include_paths)

Arguments

Details

Four refusals, in this order, each with its own condition class:

1. Not a path inside the clone. An absolute path, or one climbing out through .., refused lexically before the filesystem is touched. fs::path() joins an absolute second argument onto the clone path rather than replacing it, so ⁠/etc/passwd⁠ would otherwise be reported as a missing path inside the repo – a correct refusal whose message names the wrong thing.

2. A datom-owned path. ⁠.datom/⁠, the set being written, and any artifact directory already in the clone. The write stages those itself, so listing one is either a misunderstanding or an attempt to hand-place a datom document into a commit through a caller's argument.

3. A path that does not exist. An error, never a skip: a joint commit is deterministic or it is refused.

4. A path git is ignoring. git2r::add() on a gitignored path raises nothing and stages nothing, and .datom_git_commit() cannot notice, because it objects only when the staging area ends up empty and datom's own files are always in it. The commit would therefore succeed while omitting exactly the file the caller named, and the set version would claim a joint commit it does not have. Refused rather than dropped in silence (decided 2026-09-18).

This runs before the first hash and the first local write, the same placement as the two gates above, so a refused joint commit leaves nothing behind. One consequence, stated so nobody later softens it: change detection needs the hashes, so a bad path is an error even when the set is unchanged. The refusal wins over the no-op.

Value

Absolute paths in the clone, or NULL. Absolute, because .datom_commit_and_mirror() relativises what it is given against conn$path, and fs::path_rel() on an already-relative path resolves it against the working directory instead – which aborts with "files do not exist" pointing somewhere the caller never named.

Check Whether a Storage Namespace is Free

Description

Checks for the existence of .metadata/manifest.json in the target namespace. If found, the namespace is occupied by an existing datom project and this aborts with datom_namespace_occupied, naming the occupying project when it can be read.

Usage

.datom_check_namespace_free(conn, overridable = TRUE)

Arguments

Details

Checks for the object first (cheap) and only reads the manifest when the namespace is occupied, to extract the project name for the error message.

A store this connection cannot reach means unknown, and unknown fails closed. It used to warn and continue, which was not a deferral of the check but a silent removal of it: datom_init_repo() went on to push the git repo and then aborted at the manifest upload, and the recovery it pointed at performs no occupancy check of any kind. So the tolerance never produced a working offline init – storage is required to finish one – and its only reachable effect was getting past this check, with the outcome being a manifest written over another project's. Refusing here instead names the real problem at the moment it is known, rather than surfacing later as an unrelated upload failure.

There is no .force advice in that refusal, deliberately: .force skips this check but not the manifest upload, so it cannot rescue an init without storage either. Offering it would be advice that does not work.

The tolerated-failure detection lives here, around the one call that touches storage, rather than in a handler wrapping this whole function – which is what the caller used to do. That shape had two defects worth not reintroducing: it recognised the occupied refusal by matching its message text, so rewording the message would have quietly downgraded a refusal to a warning; and it swallowed anything it could not recognise, so any abort added to this function later would have been downgraded too, with nothing failing to say so.

The condition classes are what callers dispatch on – never the message.

Value

Invisible TRUE when the namespace is free. Aborts with class datom_namespace_occupied when it is occupied, or datom_namespace_unverified when the store could not be reached.

Refuse a Project Name That Cannot Be Cited

Description

The project field a metadata builder records exists to be quoted back by whoever cites the artifact, so a missing value or an empty string there is worse than no field at all: it reads as a project called nothing. Checked in the builders rather than at the call sites, because both builders take the value from the same place and a third caller will eventually appear.

Usage

.datom_check_project_field(project)

Arguments

Details

NULL passes. It means "not recorded", which is what every document written before the field existed looks like, and what a direct builder call in a test that is not about this field looks like.

Value

Invisibly TRUE.

Check project.yaml's Declared Format

Description

The same reader-side check every other datom-owned document gets, pinned to the config file's own ceiling (.datom_project_schema) rather than the repo-wide one. Absent means v1, which is every repo written so far, so no existing repo changes behaviour.

Usage

.datom_check_project_schema(cfg, source, operation = c("read", "write"))

Arguments

Details

Why the file needs a declared format at all. project.yaml carries fields a writer must obey, not merely fields it may read: min_writer_version already, and mode / set for a product repo. A build that does not recognise such a field walks past it and acts as though the repo had never asked for anything – so the file needs a way to say "this repo needs a newer datom", and a number is that way.

A number here, a vocabulary check there, and the two are not interchangeable. The vocabulary check that guards the manifest and per-artifact metadata draws its power from those documents being machine-written: an unrecognised key there is evidence a newer datom wrote it. project.yaml is hand-edited – storage migrations, prefixes, descriptions, private notes – so an unrecognised key is as likely a typo, and refusing on one would block every write in the repo until somebody found it. Never point the vocabulary check at this file; an unrecognised key here stays tolerated, and there is a test that says so.

This wrapper exists so the pairing of file and ceiling cannot be forgotten. A bare ⁠supported =⁠ argument at each call site is the same shape as the artifact-kind predicate that was written out at four sites and lost a tolerance at one of them. Callers pass the parsed config; the ceiling is not theirs to choose.

Value

Invisible resolved version as an integer. Aborts otherwise.

Check ref.json Matches Connection (Write-Time Guard)

Description

Re-resolves ref.json from the governance store and compares against the current connection's data location. Errors if they disagree, preventing writes to the wrong location after a migration.

Usage

.datom_check_ref_current(conn)

Arguments

Value

Invisible TRUE if current, or skips silently if no governance fields.

Check a Document's Declared Schema Version

Description

Reader-side compatibility check for one metadata or manifest document. Called wherever such a document enters datom from storage or from the local clone, so that a repo written by a newer datom fails with an actionable message instead of degrading silently – an older reader would otherwise find none of the fields it expects and report an empty repo.

Usage

.datom_check_schema_version(
  meta,
  source,
  operation = c("read", "write"),
  supported = .datom_supported_schema
)

Arguments

Details

The check is deliberately asymmetric:

• Newer than this build – abort, pointing at the upgrade. Continuing would mean interpreting a format this build does not know.

• Absent – treated as v1 and tolerated, so every repo written before schema_version existed keeps working unchanged.

• Equal or older – proceed.

A present-but-unusable value (a string, a fraction, NA, a vector) aborts as a corrupt document rather than being coerced. Coercion here would compare garbage against the supported version and could silently read as "supported"; and in R a comparison against NA propagates into ⁠if()⁠ as an opaque "missing value where TRUE/FALSE needed" error rather than anything a user can act on.

Both aborts carry a condition class so every call site is provably the same failure: datom_schema_unsupported for a too-new document, datom_schema_invalid for an unusable value.

Value

Invisible resolved schema version as an integer. Aborts otherwise.

Refuse a Set Whose Outputs Were Built From Versions It Does Not Pin

Description

A table records the parent versions it was derived from. When a set lists that table and one of its parents, the two claims can disagree: the set says "input lb is version A" while the output says "I was built from lb version B". A citation of that set would then describe a product that was never built. This check stops the write and names both versions.

Usage

.datom_check_set_parents(conn, name, members)

Arguments

Details

What is checked. Every member in the set's own project – all tables, since a product repo holds one set and a set listing itself is refused first. For each parent its snapshot records, the members naming that parent's project and table are found. None -> not checked (the parent is not part of the product). Some, and one of them at the parent's version -> fine, which is what keeps a live table beside a frozen baseline legal. Some, and none at that version -> a mismatch. All mismatches are collected and reported together.

Members of other projects are not read: the write holds a connection to its own project only.

An unreadable snapshot stops the write rather than being skipped. A version that does not exist means the set would point at data nobody can fetch, and storage that cannot be reached would fail the write anyway – only later, after the local files are written. Metadata only: one small JSON per member, no parquet, so a readable snapshot proves the version was recorded, not that its data is still in storage (that is datom_validate()'s job).

Called after the payload has been validated and before the first hash, so a malformed member gets the validator's message and a refusal leaves nothing behind.

Value

Invisibly TRUE.

Refuse a Payload Only a Whole-Payload View Can Judge

Description

Three refusals that .datom_validate_members() deliberately cannot make, because each needs the whole payload rather than one member:

Usage

.datom_check_set_payload(payload, name, project)

Arguments

Details

• Zero members. An empty citable product has no content to identify, and the recourse is simply to write the set once its first output exists.

• The same id listed twice with different tags. Deduplication does not catch this – the digest covers tags, so both entries survive – and the payload then holds one member twice with conflicting labels, which a consumer projecting tags into a folder view sees as one artifact in two places. Refused rather than tidied because both ways to tidy it guess: merging the tags is right if the caller meant both categories and nonsense if two code paths disagreed, and picking one entry is arbitrary.

• Self-reference. A set listing itself, at any version, is refused.

The same project and name at two different versions is legal and must stay legal: a product carrying a current table beside a locked baseline is atypical and entirely sensible. So the duplicate check keys on the full id, never on project + name – which is the tightening that looks natural and would break that use silently.

Self-reference is a nonsense check, not cycle detection. Cycles are structurally impossible: a member pins a version that already exists, so a set cannot reference anything containing it. Nothing here may grow into a visited set or a depth limit.

Value

Invisibly TRUE.

Refuse a Set That Belongs to Another Project

Description

A datom_set records the project it belongs to: the one it was read from, or the connection it was assembled on. The write stamps conn$project_name into the stored document, so a set from another project written here would be silently re-homed. The name gate does not catch that on its own, because two product repos may declare the same set name (two studies, each with a set called adam).

Usage

.datom_check_set_project(conn, name, set_project)

Arguments

Value

Invisibly NULL; aborts with class datom_set_project_mismatch.

Refuse a Set Write the Repo Has Not Declared

Description

Checks that read the clone's .datom/project.yaml directly, all of them before anything is hashed or written:

Usage

.datom_check_set_write_gates(conn, name = NULL)

Arguments

Details

1. The config's declared format must be one this build can read (.datom_check_project_schema()). It runs first because the two checks below read fields out of this file: a future format that renamed or moved ⁠set:⁠ would make check 3 report "this repo declares mode: product but names no set" and send the user to hand-edit a file that is already correct. An actionable-looking message that is wrong is worse than no answer. The connection-time gate does not make this one redundant – the file can be hand-edited or pulled between opening a connection and writing through it, which is the same reason the forward-compatibility door is re-run after a route's own pull.

2. The repo must declare mode: product. A set written into a repo that does not is a set with no declared owner, which defeats the one below it too.

3. The set's name must be the one the repo declares under ⁠set:⁠. This is what makes "one repo = one set = one product" true rather than aspirational, and it is the precondition the self-reference refusal depends on – that refusal needs the set's own identity, and this is where it is established.

Read from project.yaml, not from the connection. Only min_writer_version rides on a datom_conn; mode and set are read here, at the one place that needs them, so nothing has to be threaded through connection construction for two fields with one consumer. If a later caller wants them on the conn it is a move with one call site to update, rather than a decision to reopen.

⁠datom_init_repo(mode = "product", set = <name>)⁠ is what declares both fields, so the supported route into this check is a repo created that way. It shipped inert one release earlier, on purpose: a build that can notice the declaration has to exist before anything writes it, or the declaration reaches installs that walk straight past it. Hand-editing the file still works and some fixtures do it, which is also what a repo created before that argument existed needs.

Value

The resolved set name.

Everything a Write Must Clear Before It Starts

Description

The one entry sequence for every write route. Ordered, and the order matters:

Usage

.datom_check_write_entry(conn, artifact = NULL)

Arguments

Details

1. The floor – refuse if this repo has declared this build too old.

2. The manifest, read through the one shared reader, which checks the declared format and then converts an older document in memory. Refusing a format from the future has to happen before the conversion, because there is no conversion step for a version this build has never heard of.

3. The shape the conversion reached. If the artifact list is still absent after the chain has run, this document belongs to a lineage this build cannot produce – refuse rather than overwrite it in an older shape.

4. The vocabulary, on the manifest's top level, on each of its artifact entries, and on each per-artifact metadata document this write will touch.

Why step 3 is worded as "still absent after the chain" and not "absent". A current build meeting a pre-rename repo finds no artifact list either, and a rule that refused on that would deadlock the very upgrade it exists to protect: no repo could ever move forward. The discriminator is not the key, it is whether the chain can reach the shape. Note the deliberate asymmetry with the reader, which will one day rebuild on this same condition – same evidence, opposite response, because reads limp and writes stop.

Which per-artifact documents get checked depends on the route, which is why artifact exists. A table write or a metadata-only sync touches one artifact; the mirror-everything route touches all of them, and it is the route with no artifact name in its arguments at all. Both cases enumerate through .datom_clone_artifact_names(), the same helper the mirror route itself uses, so the door cannot end up inspecting a different set than the one that gets written.

Callable more than once, and one route calls it twice. Nothing here mutates anything, so re-running it is free. The metadata-only route pulls from the remote as its first act – after this check has already read the clone – so a collaborator's newer document can arrive in that pull; that route runs the sequence again afterwards. See .datom_sync_metadata().

Value

Invisibly NULL. Aborts on any refusal.

Refuse a Write This Repo Has Declared Too Old

Description

A repo may state the lowest version of datom it accepts writes from. The field is optional and lives in project.yaml; absent means no limit, so no repo written so far changes behaviour.

Usage

.datom_check_writer_floor(conn)

Arguments

Details

It exists for the two cases the vocabulary check structurally cannot see, because neither introduces a new field name: a change in what an existing field means, and a block for a reason that is not about format at all ("0.1.4 wrote bad hashes, do not let it write here"). A version number is the right currency for both – the schema number cannot carry them, since it does not move for a change that is reader-safe, and a package version directly answers the question a refusal raises.

The reading half ships even though nothing sets the field yet, and that ordering is the whole point: a build that does not look for the field can never be bound by it. This is exactly why no released datom can be stopped from writing – the looking has to be inside the build being stopped. Setting the field is a separate, later mechanism, and it owns the guard that whoever raises a floor must already satisfy it.

A value that will not parse as a version aborts rather than being ignored. Treating a malformed floor as no floor would turn a typo in a policy field into a silently disabled policy.

Value

Invisibly NULL. Aborts when the running build is older than the declared floor.

Human-Readable Class Label for a Column

Description

Renders the label shown for a column in the all-offenders abort bullets and in the datom_check_hashable() report: the collapsed class(x) string for an explicitly-classed column, or typeof(x) for an unclassed one (so a list column reads list, a complex column complex, and a units column units).

Usage

.datom_class_label(x)

Arguments

Value

A single character string.

The Artifacts Present in the Local Clone

Description

Enumerates the artifact directories in the git checkout by the one signal that identifies them: a directory holding a metadata.json. Deliberately independent of the manifest, so it still answers correctly when the manifest is the document under suspicion.

Usage

.datom_clone_artifact_names(conn)

Arguments

Details

Two callers, and they must agree. The data-side metadata sync mirrors exactly these artifacts to storage, and the write-entry check inspects exactly the documents that route is about to write – so discovering them twice, in two spellings, is how the door ends up checking a different set than the one that gets written.

The directory filter is the pre-existing one: dotfiles out, plus the fixed list of non-artifact directories a joint repo carries (⁠R/⁠, ⁠tests/⁠, ⁠renv/⁠, and so on). It is a convenience rather than the discriminator – metadata.json is what actually decides – which is why a foreign directory not on the list is tolerated rather than misread (R14.2).

Value

Character vector of artifact names, possibly empty.

Compute a Single Column's datom-cv1 Digest

Description

Encodes one column to its per-column SHA-256 hex digest for datom-cv1, as sha256( utf8(tag) || utf8(colname) || 0x00 || payload ). The tag is the kind returned by .datom_column_kind() and the payload is produced by the shared encoders. Labelled columns strip their class and attributes and re-dispatch on the bare underlying vector, so value labels never enter identity.

Usage

.datom_col_digest(name, x)

Arguments

Value

A 64-character SHA-256 hex string.

Classify a Column for Canonical Hashing

Description

The single supported-type classifier underneath the datom-cv1 hash. It returns the dispatch kind for a hashable column or NULL for an unsupported one. Both the all-offenders gate (.datom_hash_recourse()) and the per-column encoder in .datom_canonical_hash() consume this one function, so a column the gate accepts can never be one the encoder cannot encode.

Usage

.datom_column_kind(x)

Arguments

Details

Dispatch is evaluated in a fixed order (reordering can silently change hashes): bit64::integer64, factor, Date (incl. data.table::IDate), POSIXct, difftime/hms, data.table::ITime, then haven_labelled/labelled/labelled_spss (stripped to their underlying type and re-classified), then any other explicitly-classed column is refused, then unclassed atomics (logical/integer/double as "num", character as "chr"), and finally any other type is refused.

Detection uses inherits() / typeof() / is.object() class-string matching only – it adds no new package dependency (bit64, data.table, haven are recognised by their class strings, not by being loaded).

Value

One of the kind tags "i64", "chr", "date", "time", "drtn", "num" for a supported column, or NULL when unsupported.

Commit, Push, Then Mirror to Storage

Description

The tail of every artifact write, in the one order that is allowed: local files are already on disk, this commits and pushes them, and only then does it touch storage. Git push is the serialization point – a clone that is behind fails to push before it can upload anything, which is what makes the reuse decisions in .datom_resolve_parquet_sha() / .datom_resolve_document_sha() safe against a concurrent writer. Nothing may reorder these two halves.

Usage

.datom_commit_and_mirror(
  conn,
  name,
  meta,
  metadata_sha,
  git_paths,
  message,
  upload = NULL
)

Arguments

Details

Extracted when the set write arrived, and the extraction is the point rather than tidiness: this sequence was previously inline in datom_write(), so a second write verb had to either call it or grow a parallel copy – and a second copy of "git must succeed before storage is touched" is a second place for that rule to be broken by a change that only looks at one of them.

Value

The commit SHA.

Compute the datom-cv1 Content Hash of a Data Frame

Description

Thin wrapper over .datom_canonical_hash() returning only the scalar data_sha. Preserves the scalar-string contract for callers that need just the content hash (for example the datom_sync() self-lineage entry). Row and column order are significant; there is no sort option.

Usage

.datom_compute_data_sha(data)

Arguments

Value

Character SHA-256 data_sha.

Compute SHA-256 of Metadata (the datom Version)

Description

Hashes the fields named in .datom_metadata_identity_fields and ignores every other key in the document. See that constant for the field-by-field classification, for why selection is an allowlist rather than an exclusion list, and for the obligation that comes with adding a field to a builder.

Usage

.datom_compute_metadata_sha(metadata)

Arguments

Details

Hashes a JSON canonical form rather than the R object directly. This ensures that metadata read back from JSON (e.g., from S3) produces the same SHA as metadata built in-memory, despite R type differences (integer vs double, character vector vs list) introduced by JSON round-tripping.

Value

Character SHA-256 hash.

Compute SHA-256 of an Input File's Raw Bytes

Description

Answers "have this input artifact's bytes changed?". This is the original_file_sha of the three-SHA identity model – distinct from data_sha (canonical logical content) and parquet_sha (stored bytes).

Usage

.datom_compute_original_file_sha(path)

Arguments

Value

Character SHA-256 hash.

Scope-Selecting Connection Accessor

Description

Returns the connection shaped for either the data or governance store. The storage dispatch layer (⁠.datom_storage_*⁠) reads conn$root, conn$prefix, and conn$client; this accessor swaps those fields when callers need to operate on the governance store.

Usage

.datom_conn_for(conn, scope = c("data", "gov"))

Arguments

Details

Single source of truth for "which store am I talking to right now?" – replaces ad-hoc conn$gov_client peeking and the prior .datom_gov_conn() helper.

Value

A datom_conn object scoped to the requested store.

Copy a Single Storage Object Between Two Connections

Description

Dispatches on the (from_backend, to_backend) pair. For local->local uses fs::file_copy; all other combos transfer raw bytes.

Usage

.datom_copy_one(from_conn, to_conn, rel_key)

Arguments

Value

Named list with key (character) and bytes (numeric).

Create a GitHub Repository

Description

Creates a new GitHub repository via the REST API. Handles both org and personal repos.

Usage

.datom_create_github_repo(
  repo_name,
  pat,
  org = NULL,
  private = TRUE,
  api_url = "https://api.github.com"
)

Arguments

Details

Safety guard:

• Repo doesn't exist → create, return URL

• Repo exists + empty → reuse, return URL

• Repo exists + has content → abort

Value

The clone URL of the created/reused repository.

Build governance.json Content

Description

Constructs the governance pointer list that is written to both the local git copy and the data-store mirror.

Usage

.datom_create_governance_json(gov_repo_url, gov_store, attached_at = NULL)

Arguments

Value

Named list suitable for serialisation to JSON.

Create Initial ref.json Content

Description

Builds the initial ref.json structure from the data store component. No previous entries on first creation.

Usage

.datom_create_ref(data_store)

Arguments

Value

A list suitable for JSON serialization.

Each Artifact's Current Version in One Project, in One Read

Description

The manifest carries current_version per artifact, so learning what moved costs one read per project rather than one per member. Only the members that actually move then pay a snapshot read, through datom_member(), which is what keeps the new pointer trustworthy.

Usage

.datom_current_artifact_versions(conn)

Arguments

Details

The manifest is read directly rather than through datom_list() for two reasons: datom_list() abbreviates that column to 8 characters by default, which is not a version a member can record, and its abort would name S3 on a local backend.

An unreadable manifest refuses. It is the one answer that cannot be reported: a member whose project could not be read is a member whose state is unknown, which is the same situation as a missing connection.

Value

A named character vector of artifact name to current version, with NA for an entry that records none. Empty when the project has no artifacts.

Every Artifact in One Project, With Its Kind and Current Version, in One Read

Description

The same single manifest read as .datom_current_artifact_versions(), and the same refusal when it cannot be read – that function is built on this one. The set sync preview needs two things the name-to-version vector drops: each entry's kind, which each preview row reports and which drops any kind this build does not know, and the project name the manifest records, which is how a mislabelled source connection is caught before it shows every artifact as new.

Usage

.datom_current_artifacts(conn)

Arguments

Value

A list of project_name (the name the manifest records, or NULL when it records none) and artifacts, a data frame of name, kind and current_version, with NA for a field an entry does not record usably.

Which Project an Artifact Belongs To, From the Repo Rather Than a Label

Description

The cascade both pointer constructors use – datom_member() and datom_parent() – to answer "which project is this artifact in" without trusting the connection it was reached through.

Usage

.datom_declared_project(conn, snap, what = "member")

Arguments

Details

Why a label cannot be trusted. On a connection built from a clone, datom reads project_name out of .datom/project.yaml, so it is the repo's own declaration. On a reader connection it is a string the caller passed to datom_get_conn(): the namespace comes from the store's root and prefix, and nothing compares the label against the repo. Both constructors write the name they settle on into a stored document – a member's id$project is hashed into the set's data_sha and cited afterwards, and a parent's source is part of the declaring table's version – so a label nobody checked would be durable wrong data that no hash and no validator can notice.

Three steps, cheapest and most trustworthy first:

1. The artifact's own snapshot, which the caller has already read. Free, and it is the writing repo's declaration.

2. The manifest of the namespace the artifact lives in. One extra read, and only for an artifact written before datom recorded the field – which is every artifact in every existing repo, so this is the common path in this release rather than a rare one. Goes through the gated reader, so a manifest whose format this build cannot read is handled the one way datom handles that anywhere; when the manifest is unusable, that reader can escalate to reconstructing the index from a namespace listing, which is accepted because a repo in that state needs attention regardless.

3. The connection's label, said out loud to be unverified.

One gap, named rather than guarded. When the manifest has to be reconstructed and the document it replaced recorded no project name, the reconstruction fills that field from the connection (.datom_rebuild_manifest()), so step 2 can hand back the label while looking like the repo's declaration. What is lost there is the warning, not the value: the string is exactly the one step 3 would have returned. Closing it properly means the shared manifest reader reporting whether the document it returned was reconstructed, which is a change to that reader rather than to this cascade.

Value

A single non-empty string.

Delete a GitHub Repository

Description

Deletes a GitHub repository via the REST API. Requires a PAT with the delete_repo scope.

Usage

.datom_delete_github_repo(repo_full, pat, api_url = "https://api.github.com")

Arguments

Value

Invisible TRUE on success; aborts on failure.

Is This Member Already in the Set, and Is It the Same Member?

Description

Answers the question the write answers twice, one step earlier, so a repeat lands on the line that introduced it.

Usage

.datom_draft_member_clash(members, record)

Arguments

Details

Both of the write's rules are here, and they are deliberately different rules. .datom_order_set_members() drops an exact repeat – same id and same tags – silently, because the digest it dedupes on covers tags. The same id with different tags survives that and is then refused by .datom_check_set_payload(), because merging the labels and picking one entry both guess. So an exact repeat is a duplicate to skip, and a same-version disagreement is an error.

The comparison uses the write's own two mechanisms rather than restating them: the project / name / version key the payload check keys on, and the datom-sv1 member digest the dedup keys on. identical() on the two records is the spelling to avoid, and it fails in the direction that refuses working input: the encoder sorts a tag map's keys and encodes each value as a sorted, deduplicated set, so domain = c("a", "b") and c("b", "a") are one member to the write and to the digest, while identical() reads them as a disagreement and aborts.

That is also why nothing needs tidying first. Every spelling the write's tidy step collapses is a spelling the digest is already blind to, so a record can be compared – and stored in the set – exactly as the caller supplied it.

Value

A list with status – "new", "duplicate" or "conflict" – and, for the last two, at: the position of the member already in the set.

Drop Tag Keys Whose Value Is Empty

Description

The one tidy rule this file owns: a key that points at nothing is removed, because a tag with no values is spelled by omitting the key. Note this is about a tag value, not about a member having no tags at all – a member with no tags is the ordinary case and is simply accepted. Nothing is lost here – an empty value states no fact – and leaving it in would be worse than cosmetic: a present key with an empty value hashes differently from an absent key, so the same fact would mint two different data_sha values.

Usage

.datom_drop_empty_tags(tags)

Arguments

Details

Covers both empty spellings R produces. character(0) is the documented one; NULL is what an absent value looks like when a tag map is composed programmatically (list(domain = f()) where f() returned nothing), and it means exactly the same thing. Note the encoder still refuses a NULL value, and must: there it arrives from a parsed file rather than from a caller, so there is no caller intent to tidy toward.

Full canonicalization – sorting keys, sorting and deduplicating values, unboxing single values, ordering members – is not done here. It belongs to the set write, so that canonical form has exactly one implementation.

Value

tags with empty-valued keys removed; NULL unchanged. A non-list is returned untouched, so the validator reports the type rather than this function failing on it.

The Supplied Connections, Keyed by the Project Each One Claims

Description

One connection or a list of them, because a set legitimately spans projects and access in datom is per project. The key is conn$project_name, which nothing verifies – see this file's header for what confirms the choice afterwards.

Usage

.datom_edit_conns(conn, arg = "conn")

Arguments

Details

Two connections claiming one project are refused rather than ordered, since choosing between them would be a guess and the wrong one reads another project's namespace.

Value

A named list of connections.

One Display Line Per Edited Member, Grouped by Project

Description

Grouped by project because that is the axis connections are supplied along, so a surprise in the grouping is a surprise about which connection served what. A removal has no connection behind it, but it keeps the same grouping so one message can hold both kinds of entry.

Usage

.datom_edit_lines(edits, abbreviate = TRUE)

Arguments

Value

A character vector of lines.

The Columns of the Edit Log, in One Place

Description

The Columns of the Edit Log, in One Place

Usage

.datom_edit_log_fields()

Value

A character vector of column names.

The Member List of a Set, or an Abort Naming What Was Passed

Description

A datom_set however it was made – read back, or assembled with datom_assemble_set() – which is exactly what datom_write_set() takes, so the edit verbs and the write accept the same objects.

Usage

.datom_edit_members(x, arg = "x")

Arguments

Value

The member list, possibly empty.

The Zero-Row Shape of a Member Listing

Description

In one place, and carrying every column a populated result carries, so rbind() of an empty listing and a populated one works. datom_list() had exactly this defect twice: a zero-row frame built from zero rows loses its columns, and the failure only shows up when somebody binds two results.

Usage

.datom_empty_member_frame()

Value

A zero-row data frame.

A Set With No Version Yet

Description

What a set is before its first write: a name (or NULL, left for the write to resolve), the project it belongs to, and no members. Spelled list(version = NULL, ...) so the empty fields keep their names, which is the shape datom_get_set() returns for a read set. datom_assemble_set() returns one, and the sync preview starts from one when the repo's set has never been written.

Usage

.datom_empty_set(name, project)

Arguments

Value

A datom_set.

Encode a Character Payload for Canonical Hashing

Description

The character encoder for the chr column kind (character and factor columns). Emits a one-byte-per-row NA mask (0x01 where is.na(), 0x00 otherwise) followed by each value re-encoded to UTF-8 via enc2utf8() and NUL-terminated. The leading mask makes NA and the empty string "" distinguishable (both have an empty value section, but NA sets its mask byte). No Unicode normalization is applied, so NFC and NFD forms of the same text encode differently (a documented, benign limitation).

Usage

.datom_encode_character(x)

Arguments

Value

A raw vector: length(x) mask bytes followed by the NUL-terminated UTF-8 value bytes.

Encode a Numeric Payload for Canonical Hashing

Description

The single shared numeric encoder used by the num, date, time, and drtn column kinds of datom-cv1. Produces a fixed, platform-independent byte sequence: IEEE-754 doubles written little-endian regardless of host endianness, with three canonicalizations so that logically-equal values encode identically:

Usage

.datom_encode_numeric(x)

Arguments

Details

• Every NaN payload (e.g. 0/0, a signalling NaN, a negative NaN) is folded to the pinned canonical quiet NaN bit pattern 0x7ff8000000000000 (see .datom_nan_canonical).

• -0.0 is converted to +0.0.

• NA_real_ is preserved as its own distinct bit pattern – it is a specific NaN payload in R (high word 0x7ff00000, low word 1954), fixed by R itself and therefore portable, and is deliberately not folded into the canonical NaN, so NA_real_ and NaN encode differently.

No rounding is applied: doubles are encoded bit-exact.

Why the canonical NaN is written as bytes, not assigned as a value. Assigning R's NaN (d[nan_idx] <- NaN) folds NaN payloads but inherits the host's NaN sign bit: R's NaN is ⁠0x7ff8...⁠ on macOS/arm64 and ⁠0xfff8...⁠ on Linux/x86_64, because it comes from a C-level 0.0/0.0. That made data_sha platform-dependent for any table containing a NaN – caught by the CI golden matrix (the macOS job passed, the Linux job did not). Splicing the pinned bytes in directly removes the host from the equation, which is the whole premise of a canonical hash.

Value

A raw vector of 8 * length(x) bytes.

Expand a Member List to One Row Per Member Per Tag Value

Description

The shared expansion both shaping verbs are built on – see point 1 of this file's header for why there is exactly one of these.

Usage

.datom_expand_member_tags(members)

Arguments

Details

Carries a .member column holding the member's position, which is what lets datom_structure_members() get back from a row to the record it came from. datom_list_members() drops it, because a position is not a fact about a member.

Value

A data frame of .member, name, project, version, kind, key, value.

Find the One Member a Name Refers To

Description

An ambiguous name aborts and teaches. Two members can legitimately share a name – the same artifact at two versions, for instance a current table beside a locked baseline – so a name is not a key, and answering with the first match would be plausible and wrong. The abort lists the candidates with their versions and tags and names the two ways to narrow: tags, which is the navigation axis people reach for, and version, for exact pinning.

Usage

.datom_find_member(members, name, tags = NULL, version = NULL)

Arguments

Value

One member record.

Forget the Version an Edited Set Was Read As

Description

datom_get_set() fills version and data_sha from the payload it read. Once a member moves, those two describe a payload that no longer exists – and a set exists to be cited, so a stale version is a wrong statement rather than a missing one. Left in place when nothing moved: there the object still describes exactly the stored version, and dropping a true fact would cost the common "refresh found nothing" case its citability for no reason.

Usage

.datom_forget_set_identity(x)

Arguments

Details

Spelled x["f"] <- list(NULL), never x$f <- NULL, which would REMOVE the element and change names(x). A read set may legitimately report a NULL version, so the field exists and is empty rather than being absent.

A set never written has both fields empty already, so this changes nothing there.

Value

x, with version and data_sha emptied when it had them.

Format a Tag Map for One Line of Output

Description

key=value pairs, several labels joined by |, - when there are no tags. Tags are open-keyed, so a fixed column layout is impossible – do not try.

Usage

.datom_format_tag_line(tags)

Arguments

Value

A single string.

Build Connection from Local Repo + Store (Developer Path)

Description

Reads .datom/project.yaml for project identity and cross-checks against the store config. Uses the store for credentials.

Usage

.datom_get_conn_developer(path, store, endpoint = NULL)

Arguments

Value

A datom_conn object.

Build Connection from Store (Reader Path)

Description

Constructs a connection from a store object and project_name. Uses the data component of the store for S3 configuration.

Usage

.datom_get_conn_reader(store, project_name, endpoint = NULL)

Arguments

Value

A datom_conn object.

Commits on This Branch the Remote Does Not Have

Description

The ahead half of the count .datom_check_git_current() already computes for its behind half: git2r::ahead_behind() element ⁠[[1]]⁠ is ahead, ⁠[[2]]⁠ is behind. No new git machinery.

Usage

.datom_git_ahead(path)

Arguments

Details

Does not fetch. The comparison is against the cached remote-tracking ref, so a stale ref can only cause an unnecessary push – and a push pulls first and is idempotent, so the cost of being wrong in that direction is a round trip. Fetching here would instead make a clean-tree call fail when offline.

Value

Integer count of unpushed commits, or NA_integer_ when it cannot be determined – no upstream tracking ref yet, or the comparison failed. NA means cannot prove there is nothing to publish, so callers push: that is exactly the state of a branch that has never been pushed.

Get Author Info from Git Config

Description

Reads user.name and user.email from the repository's git config.

Usage

.datom_git_author(path)

Arguments

Value

Named list with name and email.

Get Current Branch

Description

Returns the name of the currently checked-out branch. Aborts on detached HEAD (datom requires a branch).

Usage

.datom_git_branch(path)

Arguments

Value

Branch name as a string.

Commit Changes

Description

Stages the specified files and creates a commit.

Usage

.datom_git_commit(path, files, message, staged_deletions = FALSE)

Arguments

Value

Commit SHA as a string.

Work Out Which Commit First Produced Each of an Artifact's Versions

Description

Walks the commits that touched {name}/metadata.json, oldest-first, hashing the document as each commit left it. A commit whose document hashes to version V is a commit that produced V, and the first one reached is the one recorded – which is what makes a code-only commit nobody's producer: it leaves that document untouched, so it is not in the walk at all.

Usage

.datom_git_commit_shas_by_version(repo_path, name)

Arguments

Details

One version maps to one-or-more commits by design, because a version is content-derived and code-invariant. Taking the oldest is not arbitrary tie-breaking; it answers "where did this version come from".

A repo git cannot answer for – a shallow clone, a rewritten history, a document that will not parse – yields no entry for the versions it lost. Callers omit the field in that case rather than recording a blank.

Every give-up here is silent on purpose, and that is not a house style: a version this cannot attribute is one the caller had no stored value for either, since a stored value is what stops it being asked about. So there is nothing to lose and nothing to report. The asymmetry with reading the stored copy, where a failure does lose something, is spelled out at the top of this file.

Value

Named character vector, commit sha named by version. Empty when nothing could be derived.

Build Git Credentials for HTTPS Remotes

Description

Returns a git2r::cred_user_pass object when the remote URL is HTTPS and a PAT has been supplied. Returns NULL for SSH remotes or when pat is absent.

Usage

.datom_git_credentials(remote_url, pat = NULL)

Arguments

Details

The PAT must be supplied explicitly – datom does not read environment variables internally. Callers obtain the PAT from conn$github_pat, which is populated at conn-construction time from store$github_pat.

Value

A git2r::cred_user_pass object or NULL.

Ensure a Repo Has a Local Git Identity

Description

Sets user.name and user.email on the local config of repo so that git2r::default_signature(repo) succeeds even when the host has no global git identity (e.g. CI runners). Values are taken from global config when present; otherwise fallback constants are used.

Usage

.datom_git_ensure_local_identity(
  repo,
  fallback_name = "datom",
  fallback_email = "datom@noreply"
)

Arguments

Details

Idempotent: re-setting the same values is a no-op from git's perspective.

Value

Invisible repo.

Paths git Is Ignoring in a Clone

Description

git2r::status(ignored = TRUE) is the only route – git2r exposes no check-ignore verb – and it reports an ignored directory with a trailing slash and does not recurse into it, so a caller's path has to be matched against these as prefixes rather than compared for equality. The trailing slash is stripped here so one comparison covers a file entry and a directory entry.

Usage

.datom_git_ignored(path)

Arguments

Details

Only ever called from .datom_check_include_paths(), and only when the caller supplied paths, so no existing write gains a git read.

Value

Character vector of ignored paths, possibly empty, without trailing slashes.

Pull from Remote (Fetch + Merge)

Description

Fetches from the remote and merges upstream changes into the current branch. Aborts on merge conflicts - user must resolve manually. This is the primary defense against diverged histories.

Usage

.datom_git_pull(path, pat = NULL)

Arguments

Value

Invisible TRUE on success.

Push to Remote

Description

Pulls (fetch + merge) first to detect conflicts, then pushes. Aborts on merge conflicts – user must resolve manually per spec.

Usage

.datom_git_push(path, pat = NULL, pull_first = TRUE)

Arguments

Value

Invisible TRUE on success.

Get GitHub Username from PAT

Description

Calls GET /user to get the authenticated user's login.

Usage

.datom_github_username(pat, api_url = "https://api.github.com")

Arguments

Value

Username string.

Check Whether a Gov Clone Exists

Description

Returns TRUE if gov_local_path is a directory that looks like a git repository (contains a .git folder). Does not validate the remote URL.

Usage

.datom_gov_clone_exists(gov_local_path)

Arguments

Value

Logical scalar.

Initialise Gov Clone (Clone If Missing, Reuse If Present)

Description

Ensures a valid gov clone exists at gov_local_path:

Usage

.datom_gov_clone_init(gov_repo_url, gov_local_path, pat = NULL)

Arguments

Details

• If the path does not exist: clones gov_repo_url into gov_local_path.

• If the path exists and is a git repo with matching remote URL: reuses it silently (idempotent).

• If the path exists with a different remote URL: hard abort (collision).

• If the path exists but is not a git repo: hard abort.

Value

Invisible gov_local_path (character).

Open an Existing Gov Clone

Description

Returns a git2r repository handle for the gov clone at gov_local_path. Aborts if the path is not a valid git repository.

Usage

.datom_gov_clone_open(gov_local_path)

Arguments

Value

A git2r::repository object.

List Registered Project Names

Description

Returns the set of project names registered in the governance repo. When a local gov clone is available, lists directories under ⁠{gov_local_path}/projects/⁠ (offline-friendly, reflects last gov-clone refresh). Otherwise lists keys under ⁠projects/⁠ via the gov storage client and extracts unique top-level segments.

Usage

.datom_gov_list_projects(gov_conn, gov_local_path = NULL)

Arguments

Details

Skips entries that don't contain a ref.json (corrupt registry rows).

Value

Character vector of project names (sorted, may be empty).

Build Project-Scoped Path Within Gov Clone

Description

Returns ⁠{gov_local_path}/projects/{project_name}/⁠. This is where dispatch.json, ref.json, and migration_history.json live for a given project in the shared governance repo.

Usage

.datom_gov_project_path(gov_local_path, project_name)

Arguments

Value

An fs_path character scalar.

Validate Gov Clone Remote URL

Description

Reads the first configured remote from the gov clone and compares it against expected_url. Aborts if they differ. This prevents silently reusing a clone that points at a different governance repo.

Usage

.datom_gov_validate_remote(gov_local_path, expected_url)

Arguments

Details

URL comparison is normalised: trailing .git is stripped from both sides before comparison so ⁠https://github.com/org/acme-gov⁠ and ⁠https://github.com/org/acme-gov.git⁠ are treated as equivalent.

Value

Invisible TRUE.

Detect Changes Against Current Metadata

Description

Compares the proposed metadata_sha against the current version in S3. Returns the type of change detected.

Usage

.datom_has_changes(conn, name, new_data_sha, new_metadata_sha)

Arguments

Value

Named list with two elements: change_type – "none" (no change), "metadata_only" (data same, metadata changed), or "full" (data changed) – and current, the already-read current metadata (or NULL for a brand-new table). Returning current lets datom_write() reuse it (the metadata_only parquet_sha carry-forward and the revert-to-older history scan) without a second storage read.

Canonical Recourse String for an Unhashable Column

Description

The single source of truth for the remediation advice attached to an unsupported column. Returns NULL when .datom_column_kind(x) classifies the column as hashable, otherwise the canonical recourse string for the first matching offender category. Both datom_check_hashable() and the .datom_canonical_hash() all-offenders abort call this one function, so the checker's advice and the abort's advice can never diverge.

Usage

.datom_hash_recourse(x)

Arguments

Details

The column name and class are added by the caller (a checker row or an abort bullet); the strings here are type-scoped only. Detection order matters: POSIXlt (a list under the hood) is matched before the generic list rows; the nested-data-frame list row before the generic list row; and the class-specific rows (units, sfc, yearmon/yearqtr/chron) before the "other classed" fallback.

Value

NULL when x is hashable, otherwise a canonical recourse string.

The Versions a History Names

Description

The Versions a History Names

Usage

.datom_history_versions(history)

Arguments

Value

Character vector, NA for an entry with no usable version.

Add commit_sha to a History on Its Way to Storage

Description

Returns history with a commit_sha on every entry whose producing commit is known, and unchanged entries where it is not. Called by each of the three functions that upload version_history.json.

Usage

.datom_history_with_commit_shas(
  conn,
  name,
  history,
  version = NULL,
  commit_sha = NULL
)

Arguments

Details

Two sources, in this order:

1. What storage already holds. Cheap, and it is the only source for a value git can no longer produce.

2. Derived from git, for the entries still missing after step 1 – and only then, so a repo whose history is complete pays no git walk.

version / commit_sha are the write path's shortcut: the caller has just made the commit that produced that version, so the walk is not needed for it. They are ignored when storage already records a commit for that version, since the recorded value is the first commit that introduced it and a later re-upload must not repoint it.

When step 1 failed rather than found nothing, and something was lost by it, this says so. The two states are not interchangeable: nothing to merge is the ordinary first write, whereas a stored copy that would not read means the values only storage had are now unknown, and the upload below replaces the file wholesale. The warning is raised only when a version actually ends up with no commit – if git could attribute every one of them, the same values were reconstructed and nothing is degraded.

Value

history with commit_sha filled in where it is known.

One id Field as Text, or NA

Description

A member read by datom_get_set() always has four single-string id fields – the read refuses a payload where one is not. This exists for the other input: a datom_set assembled by hand, which is supported and untrusted. NA rather than an abort so a listing still shows the member; the abort belongs to whoever tries to resolve it.

Usage

.datom_id_text(id, field)

Arguments

Value

A single string, or NA_character_.

Is a Value a Single Non-Empty, Non-Missing String?

Description

The field test used by the member validator. Deliberately stricter than .datom_validate_parents()'s equivalent, which accepts NA_character_: that value is character, has length 1, and nzchar(NA_character_) is TRUE, so the obvious three-part test lets it through. A missing value in a member's id would be spliced into a storage key or written into a citable payload, so it is refused here.

Usage

.datom_is_text_scalar(x)

Arguments

Value

TRUE or FALSE.

Turn a Vector of Description Lines into cli Bullets

Description

Each line is interpolated as a value rather than embedded as message text, because a tag value may legitimately contain a brace and cli reads {anything} in message text as markup. Embedding the lines directly turns an artifact called ⁠dm{1}⁠ into a cli parse error instead of a message.

Usage

.datom_line_bullets(lines)

Arguments

Value

A character vector of bullets, each named *.

Union and deduplicate source_lineage lists (internal wrapper)

Description

Thin wrapper retained for existing internal callers. Delegates to the exported datom_lineage_union().

Usage

.datom_lineage_union(lineage_lists)

Arguments

Value

Deduplicated list of source_lineage entries.

Say Which Project a Member Belongs To, When Fetching It Has Already Failed

Description

The highest-value message in the set design, and it is a hint on failure, never a gate. Access in datom is per project and not conjunctive, so resolving a member of another project through this connection genuinely does not work – but without this bullet it presents as a missing object, which names the wrong problem and sends the reader looking for corruption.

Usage

.datom_link_failure(cnd, name, kind, record, conn)

Arguments

Details

Why it cannot be a check that runs first. A connection's project_name is not a verified fact. On a reader connection – the primary consumer of a set – it is a label passed to datom_get_conn(): the namespace comes from the store's root and prefix and nothing compares the label against the repo. So a mismatch is the ordinary case, and refusing on it aborts fetches that resolve correctly. Verified end to end by a test that fetches through a deliberately wrong label.

Recording the writer's own project name in metadata made the member's side of that comparison trustworthy; the connection's side is unchanged. Comparing a verified value against an unverified one still refuses working reads, which is why this stayed a hint. What it did buy is the wording: the message names the project the member's own writer recorded, rather than a project someone typed.

When the two names agree, the original condition is re-signalled untouched – same object, same class – because callers dispatch on those classes and a failure that has nothing to do with projects must not be reworded.

Value

Never returns; always signals.

Delete a File from Local Storage

Description

Delete a File from Local Storage

Usage

.datom_local_delete(conn, key)

Arguments

Value

Invisible TRUE on success.

Delete All Files Under a Local Storage Prefix

Description

Removes the directory at root/{prefix}/datom/{prefix_key} and everything inside it. A missing prefix is a no-op.

Usage

.datom_local_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, 1L if the directory was removed, 0L if not found.

Download File from Local Storage

Description

Copies a file from the store directory to a local path. Creates parent directories if needed.

Usage

.datom_local_download(conn, key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if Local Storage Object Exists

Description

Check if Local Storage Object Exists

Usage

.datom_local_exists(conn, key)

Arguments

Value

TRUE or FALSE.

List Objects in Local Storage

Description

Lists files under a given prefix in the store.

Usage

.datom_local_list_objects(conn, prefix)

Arguments

Value

Character vector of relative keys (relative to conn$root).

Resolve a Storage Key to a Local Path

Description

Builds the full filesystem path from conn$root, conn$prefix, and the relative key segments.

Usage

.datom_local_path(conn, key)

Arguments

Value

An absolute filesystem path.

Read and Parse JSON from Local Storage

Description

Reads a JSON file from the store and parses it. Uses simplifyVector = FALSE to match S3 behavior.

Usage

.datom_local_read_json(conn, key)

Arguments

Value

Parsed R list.

Upload File to Local Storage

Description

Copies a local file to the store directory. Creates parent directories if needed.

Usage

.datom_local_upload(conn, local_path, key)

Arguments

Value

Invisible TRUE on success.

Write an R List to Local Storage as JSON

Description

Serializes data to JSON and writes to the store directory. Creates parent directories if needed.

Usage

.datom_local_write_json(conn, key, data)

Arguments

Value

Invisible TRUE on success.

Most-recent version_history document_sha for a data_sha

Description

The set half of .datom_lookup_history_object_sha(). Unlike its parquet sibling there is no legacy population to return NULL for: sets record document_sha from their first write, which is what lets a set read treat a missing one as an error rather than a skip.

Usage

.datom_lookup_history_document_sha(conn, name, data_sha)

Arguments

Value

Character document_sha, or NULL.

Most-recent version_history Stored-Object Hash for a data_sha

Description

Scans the developer's local version_history.json (newest-first) for the most recent entry whose data_sha matches and that carries a non-empty hash in field. Returns NULL when none is found. Reads the local git clone (offline-friendly); a stale clone is tolerated because the subsequent git push serializes concurrent writers (a behind clone fails to push before it can upload).

Usage

.datom_lookup_history_object_sha(conn, name, data_sha, field)

Arguments

Details

One scan serves both kinds, because the question is identical in each case – has this exact content already been stored, and under which byte hash? – and only the field name differs. Two copies would eventually disagree about what counts as a usable recorded value, and the reuse decision they feed is the one place where getting that wrong records a hash of bytes nobody stored.

Value

The recorded hash, or NULL.

Most-recent version_history parquet_sha for a data_sha

Description

The table half of .datom_lookup_history_object_sha(). Returns NULL for a pre-cv1 history, whose entries predate parquet_sha being recorded.

Usage

.datom_lookup_history_parquet_sha(conn, name, data_sha)

Arguments

Value

Character parquet_sha, or NULL.

Empty Manifest Skeleton

Description

The one shape of an empty manifest. Callers that need a manifest when none exists yet build it here rather than inline, so a later change to the manifest's shape has a single place to land.

Usage

.datom_manifest_skeleton(project_name = NULL)

Arguments

Details

artifacts is a named empty list on purpose: jsonlite serializes an empty bare list as a JSON array (⁠[]⁠) and an empty named list as an object ({}), and a manifest's artifact block must be an object. Inert today, since nothing writes a manifest that still has zero entries, and correct for the one case where it would.

The skeleton declares schema_version itself, so no repo ever exists in a state that declares no format at all – not even between being created and receiving its first artifact. This covers only the built-from-nothing path: a document read from disk in an older shape gets its version from .datom_manifest_upgrade() instead, because the skeleton is unreachable whenever a manifest file exists.

Value

A list with schema_version, project_name (when supplied), artifacts and summary.

Apply Every Upgrade Step from a Declared Version to Current

Description

The dispatcher. Runs each step from declared up to .datom_supported_schema in order, then records the version it reached.

Usage

.datom_manifest_upgrade(manifest, declared)

Arguments

Details

declared is a parameter rather than something read off the document, because the only correct source for it is .datom_check_schema_version(), which returns it after refusing a document this build cannot convert. Taking it as an argument is what makes "check first, then upgrade" structural: there is no way to call this without having obtained the number from the check.

Identity on a document already at the current version – zero steps run. A document declaring a version above current is returned untouched too, since no step exists for it; that state is unreachable through the check, which aborts first.

Value

The document in current shape, declaring the version it reached.

Upgrade a v1 Manifest to v2

Description

v1 is every manifest written before the artifact namespace existed: the artifact list sits under tables and no entry declares what kind of artifact it is. v2 renames that key to artifacts and types every entry with kind = "table", which is what all of them are – sets did not exist.

Usage

.datom_manifest_upgrade_v1_to_v2(manifest)

Arguments

Details

The rename is done in place (names() assignment rather than add-then-remove), so the key keeps its position in the document and any sibling key this build does not recognise survives untouched.

A v1 document with no tables key at all is left with no artifact key. That is deliberate: an absent key and an empty one are different states – a truncated document versus a repo with nothing in it – and flattening them here would destroy the distinction a later self-healing read depends on.

Frozen. See the file header.

Value

The same document in v2 shape. The version is stamped by .datom_manifest_upgrade(), not here, so a step is never mistaken for the thing that records the result.

Mask a Secret for Display

Description

By default shows the first 4 characters followed by ⁠****⁠. That prefix is fine for GitHub PATs (the ghp_/github_pat_ prefix is a public type tag), but for AWS secret access keys and session tokens the first characters are real entropy – pass reveal_prefix = FALSE to mask them fully.

Usage

.datom_mask_secret(secret, reveal_prefix = TRUE)

Arguments

Value

Masked string.

The Position of a Member Named by a Record or a Link

Description

Matched on the whole id, which is a member's only unique key: the same name can appear twice, and two projects may both hold a dm.

Usage

.datom_match_member_id(members, record)

Arguments

Value

An integer vector of positions, normally of length one.

Build the Resolvable Link for One Member Record

Description

The single route from a record to a callable link, used by datom_fetch_member() and by every leaf datom_structure_members() produces – and it is the same factory datom_get_set() uses for ⁠$fetch⁠. So kind dispatch, and the project hint that lives beside it, have one implementation reached by every route.

Usage

.datom_member_as_link(record, what = "member")

Arguments

Details

Any fetch already on the record is dropped first, so a record that came from a read produces a link over pure data rather than a link carrying a link.

Value

A datom_link.

Does a Member Carry All the Labels Asked For?

Description

Every key must be present and every value listed under it must be one the member carries. So tags = list(domain = "safety") matches a member tagged domain = c("safety", "efficacy") – narrowing by one label of a multi-valued tag is the ordinary case, since multi-valued tags are the point.

Usage

.datom_member_has_tags(member, tags)

Arguments

Details

The member's labels are read through .datom_tag_pairs(), not off the map, so there is genuinely one access path to a member's tag values and the duplicate-key hazard documented there cannot be reintroduced here. A member$tags[[k]] read is the same silent-first-match defect, and it fails in the direction that looks like missing data: the member is reported not found under a label the document says it carries.

The filter side is read by position for the same reason, even though datom_fetch_member() refuses a filter with duplicate keys before this runs.

Value

TRUE or FALSE.

A Member's id, or an Abort Saying the Pointer Cannot Be Resolved

Description

Checks only what resolution needs, and deliberately does not call .datom_validate_members(): that is the write-side contract, and it refuses an id field a newer datom added – which the read deliberately carries. Using it here would make an unknown field readable but unfetchable, which is a reads-limp violation arriving by a side door.

Usage

.datom_member_id(record, what = "member")

Arguments

Value

The id map.

One Line Describing a Member, for a Message That Has to Name Several

Description

Name, kind, the first 8 characters of its version, and its tags. The version costs nothing – a read member's id$version is the full recorded string – and it is what the reader needs to narrow an ambiguous name.

Usage

.datom_member_lines(members)

Arguments

Value

A character vector, one entry per member.

Resolve One Member Pointer Without a Connection in the Closure

Description

Builds the ⁠$fetch⁠ link every member of a read set carries: call it with a connection to the member's project and it resolves the pointer – a table member to data via datom_read(), a set member to references via datom_get_set().

Usage

.datom_member_link(name, kind, version, record)

Arguments

Details

fetch rather than read or get because it is genuinely both. This is the one polymorphic door in the design, and the member level is where the domain forces it: iterating members, the caller cannot know each kind in advance. At the top level they named one artifact they chose, which is why datom_read() and datom_get_set() stay separate verbs.

This function is namespace-level, and that is load-bearing. A factory defined inside datom_get_set() would put that call's frame – which holds conn, and therefore the PAT – on the closure's parent chain, and saveRDS() of the member would write the token into the file. Measured, same code both ways: nested, 2094 bytes with the token present; namespace-level, 1609 bytes without. Every argument is forced so that nothing is left as a promise pointing back at the caller's frame. The guard is a test on the serialized bytes, not on environment(link), because an environment check passes on the broken shape – there the connection sits one frame further up.

The link carries its own pointer as an attribute, so a consumer holding only a projection can still cite what they used. Links built without it cannot be repaired afterwards, which is why it ships with the factory rather than later.

It does not compare the member's project against the connection's, and it must not. That looks free – both names are in hand – and it would refuse working reads. For a reader connection, which is the primary consumer of a set, project_name is a label the caller passes to datom_get_conn(): the namespace comes from the store's bucket and prefix and nothing validates the label against the repo. So a mismatch is the ordinary case rather than the error case, and a gate here would abort a fetch that resolves correctly. Recording the writer's own project name in metadata does not change this. It makes the member's side of the comparison trustworthy; the connection's side is still a label nobody checked, so comparing them still refuses working reads. Pinned by a test that fetches through a deliberately mismatched label. A hint on an already-failed resolution is a different thing and is left to the task that owns that message.

Value

A function of one argument (conn), classed datom_link.

Read the Parents One Table Member Records

Description

The snapshot read behind .datom_check_set_parents(). The format check sits outside the read's handler, so a snapshot from a newer datom keeps its own refusal instead of being reworded as a read failure – the same pairing as .datom_parent_record().

Usage

.datom_member_parents(conn, id)

Arguments

Value

The snapshot's parents list, or NULL when it records none.

Resolve the Third Argument of datom_fetch_member() to a Member Record

Description

One accessor for the three shapes a caller holds, so a console call and a loop use the same verb: a name, a member record, or a link.

Usage

.datom_member_record(members, member, tags = NULL, version = NULL)

Arguments

Details

The shape dispatch itself is .datom_member_shape(), shared with datom_add_member(). Only the name half is here, and it genuinely differs between the two verbs: a name means "a member of this set" here and "an artifact in this project's storage" there, so a shared lookup would search the wrong thing on one of the two routes.

A record with no fetch on it is accepted, and that matters: it is the payload shape – what a caller who built a member with datom_member() holds, and what stripping a read set's links produces. The accessor keys on id and nothing else, which is what keeps this verb and datom_write_set() agreeing about what a member is.

tags and version narrow a name. Supplied beside a record or a link they are refused rather than ignored, because ignoring them would resolve a different version than the one asked for and report success.

Value

One member record.

Which of the Three Shapes a Member Argument Arrived In

Description

A caller naming one member holds one of three things, and every verb that takes a member accepts all three: a name, a member record (what datom_member() returns, and what stripping a read set's links produces), or a link (a member's fetch element, or a leaf of datom_structure_members()).

Usage

.datom_member_shape(member, arg = "member")

Arguments

Details

This is the shape dispatch alone, deliberately without the lookup. What a name means differs by verb – to datom_fetch_member() it is a member of the set already in hand, to datom_add_member() it is an artifact to look up in the project's storage – so handing the name back to the caller is what lets one dispatch serve both without either searching the wrong thing.

The refusal of tags / version beside a record or a link is left to each caller too: both refuse, and the reason differs enough to word differently (narrowing a search versus declaring a member twice). shape is the phrase to name it by, so the two messages at least agree on what the caller passed.

Value

A list of shape (a phrase naming what arrived) and record (the member record, or NULL when a name arrived).

Read a Metadata Document as One Commit Left It

Description

revparse_single(repo, "<sha>:<path>") is the whole mechanism: it resolves git's own commit:path syntax straight to the blob and raises when the path is absent at that commit. Indexing the tree object instead returns an empty list for a path that is not there, which reads as a successful lookup.

Usage

.datom_metadata_at_commit(repo, sha, rel)

Arguments

Value

The parsed document, or NULL when it cannot be read.

Every Metadata Field Name This Build Knows

Description

The two halves of the classification joined: the fields that make up a version's identity, and the fields datom deliberately keeps out of it. A name in neither half is a name this build cannot place.

Usage

.datom_metadata_known_fields()

Details

A function rather than a stored vector, for two reasons that both bite. ⁠R/⁠ is sourced alphabetically (DESCRIPTION declares no Collate), and this file sorts before R/utils-sha.R where both halves are defined – so a constant built from them here would be built from values that do not exist yet and the package would fail to install. Deriving it at call time also means it cannot fall out of step with either half.

Append-only. A name that has ever been written must keep classifying forever, including names datom no longer writes: a build that forgets one meets an older document, fails to place a field it should know, and starts preserving as unfamiliar something it could have handled – or, once the write-side refusal lands, refuses the document outright and blocks the upgrade direction, which must always work.

Value

Character vector of field names, unsorted.

Hash an Already-Selected Set of Metadata Fields

Description

The canonical-form half of metadata_sha, split from field selection so that each half is testable on its own: this function decides how a chosen set of fields becomes bytes and knows nothing about which fields are identity.

Usage

.datom_metadata_sha_from_fields(fields)

Arguments

Details

Sorts field names by C-locale byte order (method = "radix") before hashing so the result is deterministic regardless of field insertion order and regardless of the host's LC_COLLATE (default collation sorts differ between C and e.g. en_US.UTF-8, which would otherwise make the same metadata hash differently on different machines). Sorting here rather than relying on the declared order of .datom_metadata_identity_fields is deliberate: it means hash stability does not depend on how that constant happens to be written, so re-ordering it for readability cannot silently change every recorded version.

Value

Character SHA-256 hash.

Normalize a prefix value to NULL or a non-empty string

Description

A NULL prefix serializes to JSON as an empty object () and reads back as an empty list, not NULL. Empty strings can also creep in. This collapses all empty-ish forms (NULL, list(), "", NA) to NULL so that location equality checks survive a JSON round-trip.

Usage

.datom_normalize_prefix(prefix)

Arguments

Value

NULL or a single non-empty character string.

Say That a Manifest's Format Was Moved Forward

Description

Called from the two places that persist a converted manifest: the entry updater, which rewrites the git-tracked file, and the data-side metadata sync, which mirrors the converted document to storage. Reads convert too and stay silent, deliberately – a read changes nothing, and a line on every datom_list() call would be noise nobody can act on.

Usage

.datom_notify_manifest_upgraded(declared, where)

Arguments

Details

Why say anything: conversion is one-way for everybody else. Once this repo's manifest declares the newer format, a collaborator on an older datom no longer finds the artifact list where their build looks for it, and their datom_list() reports an empty repo without erroring. Their datom_read() keeps working, because the data path never touches the manifest. That is a real consequence of a command whose stated job was something else – datom_validate(fix = TRUE) in particular reads as a repair – and an unannounced one is the silent degradation the whole schema contract exists to remove.

No-op when the document was already current, which is every ordinary write.

Value

Invisibly NULL.

Deduplicate and Order a Member List for the File

Description

Drops exact duplicates – same id and same tags – by datom-sv1 member digest, then sorts by project, name, version.

Usage

.datom_order_set_members(members)

Arguments

Details

Two sort keys exist and each has its own reason. The identity hash orders member digests, which is what keeps the encoder from having to know what an id looks like. The file orders by name, which is what keeps an entry in place when its tags change so that ⁠git diff⁠ shows one changed field. version is in the key because two versions of one name are legal members, and would otherwise have no defined relative order.

No tiebreaker is required, and none may be added. The only way two members can share project || name || version is the same id with different tags, which survives dedup because the digest covers tags – and that payload is refused one step later. R's radix sort is stable, so the tie resolves to caller order in the meantime. A defensive tiebreaker would be dead code.

Runs after validation, unlike the rest of canonicalization, because the digest is computed by the identity encoder and the encoder refuses a value it cannot encode. Reaching it first would report a bad tag value in the encoder's words rather than the validator's.

Value

The members, deduplicated and ordered.

Resolve One Parent Record From Its Versioned Snapshot

Description

The body of datom_parent() for one table at one named version, shared by both of its routes so a parent declared by version and one declared from a set are read, checked and shaped by the same code.

Usage

.datom_parent_record(conn, table, version)

Arguments

Value

One parent record.

Declare Parents at the Versions a Set Pins

Description

The ⁠x =⁠ route of datom_parent(). Each name is resolved by .datom_find_member(), the resolver datom_fetch_member() uses for a name, so the two verbs cannot pick different members for the same name and labels. Tag validation is the same call with the same remedy, for the same reason.

Usage

.datom_parents_from_set(conn, table, x, tags)

Arguments

Value

An unnamed list of parent records, one per table.

Parse a ref.json structure into a location list

Description

Common parsing logic shared by storage-backed and clone-backed ref readers.

Usage

.datom_parse_ref(ref, source)

Arguments

Value

A named list with root, prefix, region.

Parse S3 URI into Components

Description

Extracts bucket and prefix from an ⁠s3://⁠ URI.

Usage

.datom_parse_s3_uri(uri)

Arguments

Details

Mapping from URI to components, for reference:

"s3://my-bucket/data/proj" -> list(bucket = "my-bucket", prefix = "data/proj")
"s3://my-bucket"           -> list(bucket = "my-bucket", prefix = NULL)

Value

Named list with bucket (character) and prefix (character or NULL).

The Metadata Document Already in the Clone, If Any

Description

Reads {name}/metadata.json from the local git checkout, for the one purpose of finding fields to carry forward. Returns NULL when there is no such file or it will not parse – both mean there is nothing to preserve, and neither is this function's business to report: a brand-new artifact legitimately has no prior document, and an unparseable one fails moments later on its own terms.

Usage

.datom_prior_metadata(conn, name)

Arguments

Details

The clone's copy, not storage's. Three reasons, any one sufficient: it is the file being overwritten, so preserving its own content is the claim being made; it is a local file read rather than a network round trip; and it is where a pull from a collaborator on a newer datom lands. Storage cannot legitimately hold a newer document than the clone, because git is written first and gates the storage mirror – if it does, that is drift, and datom_validate() owns drift.

Value

The parsed document, or NULL.

Push Metadata Files to S3

Description

Uploads metadata.json, version_history.json, and a versioned snapshot to S3. Called AFTER git commit+push succeeds to maintain local → git → S3 ordering.

Usage

.datom_push_metadata_s3(conn, name, metadata, metadata_sha, commit_sha = NULL)

Arguments

Details

The stored history carries one field the clone's copy cannot, and this is the reason commit_sha exists as an argument here: the clone's version_history.json is inside the commit that would name it, so only a storage-bound copy can say which commit produced a version. The upload sends the clone's file wholesale, so without the merge below the field would survive on the newest version only – the second write of an artifact would erase the first version's commit id.

commit_sha is derived, never authored. It reaches this function as an argument only because the caller one layer up already holds the commit it just made; no exported verb accepts it, and every other entry's value is worked out from git. See R/version-commit.R.

Value

Invisible character vector of S3 keys written.

Read governance.json from Local Git Clone

Description

Reads and validates {path}/.datom/governance.json. Returns NULL when the file is absent (project is not gov-attached). Aborts on malformed JSON or failed schema validation.

Usage

.datom_read_governance_json_local(path)

Arguments

Value

Parsed list or NULL.

Read a Manifest and Check Its Schema Version

Description

The single manifest read. Every reader that takes a manifest into datom goes through this, so the compatibility check happens once and cannot be softened by a caller's error handling.

Usage

.datom_read_manifest(
  conn,
  scope = c("storage", "clone"),
  operation = c("read", "write")
)

Arguments

Details

Two kinds of failure, handled deliberately differently:

• An IO failure is returned as data (ok = FALSE), because each caller has its own policy: datom_list() and datom_summary() abort, datom_status() reports the manifest unavailable and carries on, and the clone readers fall back to an empty manifest when the file does not exist yet.

• A schema refusal is thrown, so the "upgrade datom" message reaches the user intact. Placed inside a caller's tryCatch it would be reworded as "could not read manifest" at two sites and downgraded to a warning at a third. Throwing from in here means there is no handler for a caller to put it inside.

And one document that is not a failure at all. When the artifact list is missing from where this build looks for it – either because the format is newer than this build knows, or because the key is simply not there after the conversion has run – the index is reconstructed from storage and a warning says so. The manifest summarises documents that each hold the same facts, so it is the one datom-owned file with something to rebuild it from. A writer meeting either condition is refused instead (.datom_check_write_entry()): reads limp, writes stop.

Value

A list with:

• ok – TRUE when the manifest was read and parsed.

• absent – TRUE only when the document is known not to exist. That is decided for scope = "clone", where testing a local path is free. For scope = "storage" it is always FALSE, meaning "not known to be absent": separating a missing object from an unreachable store would cost an extra request on every read and no caller distinguishes them.

• manifest – the parsed document in current shape, or NULL when ok is FALSE. A document written in an older shape is converted in memory on the way through (.datom_manifest_upgrade()); one whose artifact list this build cannot reach is reconstructed from storage (.datom_rebuild_manifest()). Neither modifies the file on disk or in storage. So no caller ever sees a pre-current shape and none needs a fallback for one.

• error – the condition that stopped the read, or NULL. The whole condition rather than its text, so a caller can re-signal the original failure unchanged instead of manufacturing a look-alike.

• declared – the version the document declared before conversion, or NA_integer_ when nothing was read. Held so a caller that goes on to write the converted document can say the format moved, without re-deriving the comparison or reading the file twice.

Read Table Metadata from S3

Description

Fetches both metadata.json (current state) and version_history.json (version index) for a given table from S3.

Usage

.datom_read_metadata(conn, name)

Arguments

Value

Named list with current (metadata.json contents) and history (version_history.json contents as a list of entries).

Download and Read Parquet from S3

Description

Downloads ⁠{table}/{data_sha}.parquet⁠ from S3 to a temporary file and reads it via arrow::read_parquet(). When an expected parquet_sha is supplied (non-empty), the downloaded object's SHA-256 is verified against it BEFORE parsing, so corruption or tampering aborts rather than being silently read.

Usage

.datom_read_parquet(conn, name, data_sha, parquet_sha = NULL)

Arguments

Value

Data frame.

Normalize One Member Record Read Back from a Payload

Description

Normalizes representation in id and tags, then makes the one refusal the read owns: an id field that is not a single non-empty string after normalization aborts as a malformed document, naming the member.

Usage

.datom_read_set_member(m, at, name)

Arguments

Details

Why id is refused where a tag value is tolerated. id values are spliced into storage keys and compared against project names, and .datom_validate_members() enforces that contract on write only – so the read is the only place a payload's id is ever checked. Normalizing without refusing would silently accept a document datom_write_set() cannot produce, and a caller comparing a list against a string would conclude that a member of this project belongs to another one.

Fields outside the four are left alone rather than refused: a newer datom may have added one, and this build never reads it.

Value

The member record, normalized.

Turn a Parsed Member List into Resolvable Member Records

Description

Turn a Parsed Member List into Resolvable Member Records

Usage

.datom_read_set_members(members, name)

Arguments

Value

An unnamed list of member records, each carrying ⁠$fetch⁠.

Download, Verify and Parse a Set's Stored Payload

Description

Download, hash, then parse. The order is the point: a set read must not parse an unverified payload, which is the same gate position .datom_read_parquet() uses for parquet_sha.

Usage

.datom_read_set_payload(conn, name, data_sha, document_sha)

Arguments

Details

.datom_storage_read_json() cannot be used here, and it would work. It parses, so after calling it there is nothing left to hash but bytes re-serialized locally – a hash of bytes nobody stored, which is exactly the defect the write path guards against, inverted. It returns a structure identical to parsing the downloaded file, so nothing fails if you reach for it; the integrity check simply stops meaning anything.

A missing document_sha is an error, not a skip. parquet_sha's skip-on-absent branch exists purely as a grace for metadata written before that field did. Sets have recorded document_sha since their first write, so there is no legacy population to be lenient about, and reproducing the grace would build a silent-degradation path on purpose.

data_sha is deliberately not recomputed from the parsed payload. It is the address the payload was fetched from, so it catches nothing document_sha did not, and it would refuse a payload a newer datom wrote – the sv1 encoder aborts on a top-level payload key it does not know. Same reason the parsed payload is not re-validated. Reads limp.

Value

The parsed payload, with members kept as a list of records.

Normalize a Parsed JSON String Array to a Character Vector

Description

jsonlite::fromJSON(simplifyVector = FALSE) returns a JSON array of strings as a list of length-1 characters, and auto_unbox = TRUE on the write means a single label was written as a bare string. So one tag key comes back in three shapes – character(1), a list of 1, or a list of n – for what is one value in the document.

Usage

.datom_read_string_array(v)

Arguments

Details

Same strings, same order, same count: this is a representation change, not a content change, which is why order is preserved and duplicates are kept. Sorting or deduplicating here would be the write's canonicalization performed by a reader.

Anything that is not an all-text array is returned untouched. A reader has no caller intent to tidy toward and nothing downstream requires tag values to be text, so an odd value is reported by whoever tries to use it rather than refused here.

Value

A character vector when v was an all-text array, otherwise v.

Normalize a Parsed Tag Map's Values

Description

Applies .datom_read_string_array() to every value and does nothing else: no key sorting, no value sorting, no deduplication, no dropping of an empty-valued key. A map with no names is returned untouched rather than refused, for the same reason a single odd value is.

Usage

.datom_read_tag_map(tags)

Arguments

Value

The map with each value normalized, or NULL.

Reconstruct the Whole Artifact Index from Storage

Description

One storage listing plus two reads per artifact. The result is a complete manifest in this build's shape: the artifact rows, the summary counters recomputed from them, and the current format declared.

Usage

.datom_rebuild_manifest(conn, prior = NULL)

Arguments

Details

project_name and updated_at are carried from the document being replaced when it has them. Both are recorded facts about the repo rather than about the artifacts, so neither is recoverable from a listing – and inventing a fresh updated_at would state that the index was rewritten now, when nothing was written at all.

Aborts rather than returning a partial index. Half an artifact list is indistinguishable from a repo that only has half those artifacts, and the caller's job is to decide what an unreachable store means – see .datom_read_manifest(), which keeps a schema refusal separate from an IO failure on the way back out.

Value

A manifest in current shape.

Rebuild One Artifact's Manifest Row from Its Own Documents

Description

Every field on the row is copied from metadata.json, counted from version_history.json, or – for a set's member count – read from the payload. The row's shape has to match what .datom_update_manifest_entry() writes, field for field, or a rebuilt repo answers differently from a healthy one – so the two are pinned against each other by a test rather than by matching comments.

Usage

.datom_rebuild_manifest_entry(conn, name)

Arguments

Details

last_updated is the one field with no recorded source: the writer stamps the wall clock at the moment it rewrites the row, and that moment is not in any document. The version's own created_at is used instead, which is the closest true statement available – when this artifact's current state was written.

A set's row is built from different fields, and costs a third read. A set carries member_count where a table carries size_bytes, and that count lives in the payload rather than in either document read here – hence .datom_rebuild_member_count(). Putting a size_bytes on a set row instead would be worse than leaving the count out: the default is 0, which has length 1 and therefore survives purrr::compact(), so the row would state that the artifact is zero bytes.

Value

A named list: one manifest artifact row.

How Many Members a Set's Current Payload Holds

Description

The third storage read a set's row costs. Only a set needs it, and only a rebuild pays it: the healthy writer knows the count from the payload it just canonicalized.

Usage

.datom_rebuild_member_count(conn, name, data_sha)

Arguments

Details

Returns NULL for anything that is not a readable payload – an unusable data_sha, a missing object, a document that will not parse. That is the same trade the rest of this file makes: an absent count is a gap datom_validate() owns, while a stand-in count would be a statement about the set's contents that nothing supports.

Value

An integer count, or NULL.

The Set Name a Write Uses, From the Call and From the Set Itself

Description

A datom_set carries its name, and the caller may pass ⁠name =⁠ too. When both are given they must agree: preferring either would write a set under a name one of them did not say. When only one is given it is the one used, and .datom_check_set_write_gates() then checks it against the repo's declared set – so a set named for another repo stops there, with the gate's message. A set with no name (an assembled one, usually) takes the declared one.

Usage

.datom_reconcile_set_name(name, set_name)

Arguments

Value

The name to hand to the gate, or NULL.

The Version Storage Recorded for an Artifact's Current State

Description

Picks the version_history.json entry that describes metadata.json, and returns the version recorded on it.

Usage

.datom_recorded_current_version(meta, history)

Arguments

Details

Never recomputed, and that is the point of this function existing at all. Hashing metadata.json here would reach for the identity code in precisely the scenario a rebuild is for – a repo touched by a build whose field classification differs from this one's – and publish a current_version matching no version in the recorded history. An index pointing at a version that does not exist is worse than the empty list it replaced.

Which entry describes the current state is not simply the newest one. History is prepended newest-first, but a write that reverts to content already in the history appends nothing, so the current state can be an older entry. The selection therefore narrows by recorded fields only:

1. Entries whose data_sha equals the current document's. One match settles it – this is the revert case, and it is why the newest entry alone is wrong.

2. Several matches means metadata-only versions of the same content; the one whose timestamp equals the document's created_at is the current one, since a version's history entry copies that field verbatim.

3. Anything still ambiguous takes the newest candidate. That is a choice between two entries that both describe the current content, so the worst case is naming the wrong one of two versions of the same bytes.

No match at all returns NULL, and it deliberately does not fall back to the newest entry. No match means the history does not record the state metadata.json describes – a truncated or partly-synced history. The newest entry there is a version of different content, so naming it would be a wrong statement rather than a missing one, and a row already tolerates carrying no version. datom_validate() owns the inconsistency. This is the same trade the carry-forward rule makes in R/forward-compat.R: a stale claim that outlives what it described is worse than an absent one.

Value

The recorded version string, or NULL when the history records nothing usable – in which case the rebuilt row simply carries no version, rather than a manufactured one.

Refuse a File-Import Argument on a Product Repo

Description

Silently ignoring it would let a caller believe the argument did something.

Usage

.datom_refuse_file_arg_on_product(arg)

Arguments

Value

Does not return; aborts with class datom_sync_file_arg_on_product.

Refuse the File-Import Path on a Product Repo

Description

A mode: product repo builds its artifacts: derived tables written from data frames, and one set collecting them. It never onboards source files, so the two import verbs refuse instead of answering. Before this they answered unhelpfully – ⁠input_files/⁠ exists and is empty on such a repo, so the scan reported "no files found" and handed back a zero-row frame, which describes a repo with nothing to import rather than a repo that does not import.

Usage

.datom_refuse_import_on_product(verb, context)

Arguments

Details

Read from the file, not from the connection, and the rule behind that is worth carrying: a check that authorises a write must see the config as it is now, because a hand edit or a pull can replace it after the connection was built. Only datom_status(), which reports rather than decides, reads the mode off the connection. A future site applies the same test: does it authorise a write? Then it reads the file.

Three steps in one place, and the middle one is easy to leave out. Parsing this file makes this a new gated parse: every site that reads .datom/project.yaml checks its declared format first, or a build that cannot interpret the file acts on fields it has misread. Skipping that step here would reopen exactly that hole, on a path that writes.

Called from both sync verbs, not only the first. datom_sync() takes a manifest data frame, so a caller can hand it rows that a refusing datom_sync_manifest() would never have produced.

Above the input-file scan, never in its empty branch. A product repo with a file dropped into ⁠input_files/⁠ by accident would otherwise be imported, which is the thing this exists to prevent; the unhelpful no-op only happened when the directory was empty.

Both verbs have a set route on a product repo, reached by passing ⁠sources =⁠, so the message names that route rather than only the write verbs.

Value

Invisibly NULL. Aborts with class datom_import_on_product when the repo declares mode: product.

Refuse the Set's Own Project as a Source

Description

The set's own project holds its outputs, which are derived from the inputs and move only once they are re-derived. Checked on the labels, before any read.

Usage

.datom_refuse_own_project_source(own, labels)

Arguments

Value

Invisibly NULL; aborts with class datom_sync_own_project_source.

Refuse ⁠sources =⁠ (or Another Set-Path Argument) on an Ordinary Repo

Description

Refuse ⁠sources =⁠ (or Another Set-Path Argument) on an Ordinary Repo

Usage

.datom_refuse_sources_on_ordinary(arg)

Arguments

Value

Does not return; aborts with class datom_sync_sources_on_ordinary.

Render README.md from Template

Description

Reads the template from inst/templates/README.md and fills in project-specific values using {{{ }}} delimiters.

Usage

.datom_render_readme(
  project_name,
  backend = "s3",
  root,
  prefix,
  region = NULL,
  remote_url,
  gov = NULL
)

Arguments

Value

Character string — the rendered README content.

Repoint One Member at One Version

Description

Points 1, 2 and 3 of this file's header all live here: the labels are attached verbatim rather than passed through the constructor, the link is rebuilt through the shared factory, and the rebuilt record's recorded project is compared against the one it replaces.

Usage

.datom_repoint_member(record, conn, version)

Arguments

Value

The new member record, carrying the old labels and, when the old record had one, a link to the new version.

Say What Was Dropped, and That Nothing Was Written

Description

Say What Was Dropped, and That Nothing Was Written

Usage

.datom_report_member_removals(dropped, left, n = 20L)

Arguments

Value

Invisibly NULL.

Say What Moved, What Did Not, and That Nothing Was Written

Description

The report is the deliverable rather than decoration: nothing is written, so this is the dry run, and it is the only place the caller sees what an inferred "current" resolved to.

Usage

.datom_report_member_updates(changes, gone, skipped_lines, n_selected, n = 20L)

Arguments

Details

Lines are emitted with cli::cli_verbatim() because they embed artifact names and label values, and cli reads {anything} in message text as markup – an artifact called ⁠dm{1}⁠ would be a parse error rather than a line.

Value

Invisibly NULL.

Say What Apply Did, and That Nothing Was Written

Description

Say What Apply Did, and That Nothing Was Written

Usage

.datom_report_sync_apply(todo, n = 20L)

Arguments

Value

Invisibly NULL.

Say What the Set Sync Preview Found

Description

One summary line, then one warning per group of rows or members the caller has to know about, each with its remedy.

Usage

.datom_report_sync_preview(
  result,
  n_sources,
  ambiguous_lines,
  ambiguous_first,
  unpassed,
  gone,
  unversioned
)

Arguments

Value

Invisibly NULL.

Require Governance Attached on a Connection

Description

Guard helper used by gov-only commands (e.g. datom_projects) to fail with a single uniform message when called on a no-governance connection.

Usage

.datom_require_gov(conn, what)

Arguments

Value

Invisible TRUE when gov is attached. Aborts otherwise.

Resolve Data Location via Ref (Conn-Time Helper)

Description

Called during datom_get_conn() for both readers and developers when a governance store is present. Reads ref.json from governance, detects migration (store$data location != ref location), and returns the ref-resolved location.

Usage

.datom_resolve_data_location(
  store,
  role,
  project_name = NULL,
  path = NULL,
  gov_local_path = NULL,
  endpoint = NULL
)

Arguments

Details

Read path is role-aware:

• Developer with gov_local_path set: read projects/{name}/ref.json from the local gov clone (faster, works offline, reflects last gov-clone refresh).

• Otherwise (reader, or developer without a clone yet): read via the gov storage client.

Value

A named list with root, prefix, region from the ref, or NULL if no governance store is present (skip ref resolution).

Resolve the document_sha to Record and Whether to Upload

Description

The set analogue of .datom_resolve_parquet_sha(), kept beside it so the two cannot drift: the decision is the same decision, and both are the one place where "these are new bytes, so hash them" is the wrong answer.

Usage

.datom_resolve_document_sha(
  conn,
  name,
  data_sha,
  new_document_sha,
  change_type,
  current
)

Arguments

Details

Recomputing the hash from freshly emitted bytes while reusing the stored object records a hash of bytes nobody stored. Nothing fails at write time – it surfaces much later as a refused read of a valid version, when the integrity gate compares the stored payload against a hash taken from a different serialization of the same content. Sets reach that state far more easily than tables do: for a table it takes an arrow upgrade, while for a set an ordinary tag-value reorder is enough, because several payload spellings share one data_sha.

Cases, mirroring the parquet ones:

• metadata_only – the data_sha is unchanged, so the payload object already exists; carry the current metadata's document_sha forward and do not upload. Structurally unreachable for a set today (a set's hashed fields are data_sha, hash_algo and kind, so unchanged content means an unchanged version), and handled anyway rather than assumed away.

• full where a prior version already recorded a document_sha for this exact data_sha – reuse it and do not re-upload.

• full otherwise – upload these bytes and record their hash.

Value

List with document_sha (character or NULL) and upload (logical).

Resolve the Local Path for the Governance Clone

Description

Returns the explicit override if supplied. Otherwise, places the gov clone as a sibling of data_local_path named after the basename of gov_repo_url (stripping a trailing .git suffix). This ensures the gov clone directory name reflects the gov repo's own identity, not any specific data project.

Usage

.datom_resolve_gov_local_path(data_local_path, gov_repo_url, override = NULL)

Arguments

Value

Absolute path string for the gov clone.

Resolve Gov Clone Path with Store Defaults

Description

Convenience wrapper that derives a gov clone path from a datom_store: returns the store's explicit gov_local_path if set; otherwise derives a sibling-of-data default from gov_repo_url; otherwise returns NULL.

Usage

.datom_resolve_or_default_gov_path(store, data_local_path)

Arguments

Details

Centralises the three-arm pattern previously duplicated in datom_init_repo(), datom_clone(), and .datom_get_conn_developer().

Value

Character path string or NULL.

Resolve the parquet_sha to Record and Whether to Upload

Description

For a write that is not a no-op, decides which parquet_sha the new metadata should carry and whether the freshly-serialized parquet bytes need uploading. The caller performs the actual upload AFTER the git push (git push is the serialization point); this function only decides.

Usage

.datom_resolve_parquet_sha(
  conn,
  name,
  data_sha,
  new_parquet_sha,
  change_type,
  current
)

Arguments

Details

Cases:

• metadata_only – the data_sha is unchanged, so the parquet object already exists; carry forward the current metadata's parquet_sha (which may be NULL for a pre-cv1 table, leaving the integrity check skipped) and do not upload.

• full where a prior version already recorded a parquet_sha for this exact data_sha – the stored object exists and is pinned by that version; reuse its parquet_sha and do NOT re-upload (a fresh serialization can differ byte-for-byte and would break that version's integrity pin).

• full otherwise (brand-new content, or a legacy object with no recorded parquet_sha) – upload these bytes and record their hash.

This refines the design's literal step 7 (which gated on .datom_storage_exists()): a recorded parquet_sha is the precise thing we must not clobber, and its presence implies the object exists, so the history lookup subsumes the existence check with identical behavior and one fewer storage round-trip.

Value

List with parquet_sha (character or NULL) and upload (logical).

Resolve Data Location from Governance Store

Description

Reads projects/{project_name}/ref.json from the governance store and returns the current data location as a named list. Single read, no recursion, no chain-walking.

Usage

.datom_resolve_ref(gov_conn, project_name = NULL)

Arguments

Details

If the ref has previous entries, a deprecation-style warning is emitted to alert users that a migration occurred and old locations may sunset.

Value

A named list with root, prefix, region for the current data location.

Resolve Data Location from Local Gov Clone

Description

Reads projects/{project_name}/ref.json directly from a local gov clone on disk. Faster than storage reads, works offline, and reflects the last gov-clone refresh. Used for developer connections.

Usage

.datom_resolve_ref_from_clone(gov_local_path, project_name)

Arguments

Value

A named list with root, prefix, region for the current data location.

Resolve Version to data_sha, Stored-Object Hash and Recorded Version

Description

Given metadata from .datom_read_metadata(), resolves a version spec to the corresponding data_sha (the storage address), the recorded stored-object integrity hash, and the version string as recorded. If version is NULL, resolves from the current metadata.json; if a metadata_sha string (or a prefix of one), looks it up in version_history.json.

Usage

.datom_resolve_version(
  metadata_list,
  version = NULL,
  name = "table",
  field = "parquet_sha"
)

Arguments

Details

One function, two kinds, one field argument. A table's stored object is a parquet file pinned by parquet_sha; a set's is a JSON payload pinned by document_sha. The question is identical either way – which recorded hash pins the version I just resolved – so a second copy of this lookup would eventually disagree with this one about prefix matching or about what an absent hash means. The resolved hash comes back as object_sha regardless, because the caller already knows which field it asked for.

The object_sha may be NULL/"", and what that means is the caller's to decide, not this function's. For a table it is pre-cv1 metadata and tells .datom_read_parquet() to skip the integrity check – a grace for legacy metadata, not a gap in the current writer. For a set there is no legacy population, so the set read treats it as an error.

version is the version recorded for the resolved state, never recomputed: a pinned read echoes the matched history entry's own version string (so a caller who passed an 8-character prefix gets the full one back), and an unpinned read takes the current state's recorded version via .datom_recorded_current_version(). That helper returns NULL when the history records nothing matching the current document, which is a gap datom_validate() owns – a manufactured version would be a wrong statement rather than a missing one.

Value

Named list with data_sha (character), object_sha (character or NULL) and version (character or NULL) for the resolved version.

Create an S3 Client from Credentials

Description

Constructs a paws.storage::s3() client from credential values. Never stores raw credentials beyond the paws client object.

Usage

.datom_s3_client(
  access_key,
  secret_key,
  region = "us-east-1",
  endpoint = NULL,
  session_token = NULL
)

Arguments

Value

A paws.storage S3 client.

Delete All S3 Objects Under a Prefix

Description

Lists every key under {prefix}/datom/{prefix_key} and deletes in batches of up to 1000. A missing prefix is a no-op.

Usage

.datom_s3_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, the count of deleted objects.

Download File from S3

Description

Downloads an S3 object and writes it to a local path. Creates parent directories if needed.

Usage

.datom_s3_download(conn, s3_key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if S3 Object Exists

Description

Uses a HEAD request for efficiency. Returns TRUE if the object exists, FALSE on 404/NoSuchKey. Any other error (403, network) is re-thrown.

Usage

.datom_s3_exists(conn, s3_key)

Arguments

Value

TRUE or FALSE.

List S3 Objects Under a Prefix

Description

Lists every key under {prefix}/datom/{prefix_key} and returns relative keys (relative to the datom namespace, i.e. with the ⁠prefix/datom/⁠ part stripped). Paginates via ContinuationToken.

Usage

.datom_s3_list_objects(conn, prefix)

Arguments

Value

Character vector of relative keys (may be empty).

Read and Parse JSON from S3

Description

Downloads an S3 object, reads it as text, and parses as JSON. Uses simplifyVector = FALSE to keep lists as lists (matching how .datom_s3_write_json() writes them).

Usage

.datom_s3_read_json(conn, s3_key)

Arguments

Value

Parsed R list.

Upload File to S3

Description

Reads a local file as raw bytes and uploads via put_object().

Usage

.datom_s3_upload(conn, local_path, s3_key)

Arguments

Value

Invisible TRUE on success.

Write an R List to S3 as JSON

Description

Serializes data to JSON via jsonlite::toJSON() and uploads to S3.

Usage

.datom_s3_write_json(conn, s3_key, data)

Arguments

Value

Invisible TRUE on success.

Which Members Does This Call Refer To?

Description

The plural selector both edit verbs share. .datom_find_member() resolves exactly one member and aborts on an ambiguous name, which is right for a fetch and only half of what an edit needs: an edit legitimately acts on many.

Usage

.datom_select_members(
  members,
  member = NULL,
  tags = NULL,
  version = NULL,
  version_arg = "version"
)

Arguments

Details

Three routes, and the difference between them is how many members they can return:

An explicitly named member that is ambiguous aborts, and that is not in tension with the caller who sweeps: a sweep can honour "refresh everything" while skipping a name it cannot choose between, whereas a caller who named one member asked for something that cannot be done, so it is a user error and the narrowing arguments are what resolve it. The abort comes from .datom_find_member() rather than from a second copy of that message.

tags and version narrow a name or a sweep. Supplied beside a record or a link they are refused rather than ignored, because ignoring them would act on a different member than the one asked for and report success.

Value

An integer vector of positions in members, never empty.

Append One Member Record to a Set, as an Edit

Description

The three steps every addition takes – a ⁠$fetch⁠ link, the set's version forgotten, an add row in the edit log – in one place, so datom_add_member() and the set route of datom_sync() cannot drift apart in what they record. See points 5 and 6 of this file's header. It prints nothing: each caller says "nothing has been written" once, which for sync means once per call rather than once per added member.

Usage

.datom_set_add_record(x, record)

Arguments

Details

The link goes through the shared factory, never inline, so no frame holding a connection lands on its parent chain.

Value

x, one member longer.

The Commit Message a Set Write Uses

Description

A set write commits ⁠Update {name}⁠, which says nothing in ⁠git log⁠. When the object being written carries an edit log and the caller passed no message, the default names what changed instead – every action in the log, so a chained ⁠update |> remove⁠ produces one message describing both.

Usage

.datom_set_commit_messages(name, message, edits)

Arguments

Details

Two messages, because they go to two places. The subject is recorded as the version's commit_message, where one line is what datom_history() can show. The commit gets the subject plus the full list, with whole versions rather than prefixes: git is the durable record, so completeness belongs there rather than on screen.

An explicit message always wins, and a log of the wrong shape is ignored rather than trusted – it is an attribute, so a caller can put anything there.

Value

A list of history (a single line, or NULL to leave the existing default in place) and commit.

The Member List of a Set, or an Abort Naming What Was Passed

Description

Every verb in this file starts here, so "this is not a set" is reported once and identically rather than surfacing as a $ on a data frame returning NULL.

Usage

.datom_set_members(x, arg = "x")

Arguments

Details

A set with no members is not an error. The writer refuses an empty member list, but the reader does not – a hand-built payload, or one from a newer datom, reads back with none – so every verb below has to have an answer for zero members.

Value

The member list, possibly empty.

Which Project a Set Reports Itself As Belonging To

Description

Two steps, not the three .datom_declared_project() uses: the set's own metadata document, then the connection's name. See the call site in datom_get_set() for why the manifest step is deliberately absent here.

Usage

.datom_set_project(current, conn)

Arguments

Value

A single string, or whatever the connection carries.

Artifact Names Present in Storage

Description

Enumerates artifacts from a storage listing, by the one signal that identifies one: a {name}/.metadata/metadata.json object. Deliberately independent of the manifest, because the manifest is the document under suspicion whenever this is called.

Usage

.datom_storage_artifact_names(conn)

Arguments

Details

The clone-side equivalent is .datom_clone_artifact_names(). They are not interchangeable and neither can stand in for the other: a storage-only reader has no clone at all, and the clone can hold an artifact whose upload has not happened yet.

The listing returns FULL keys – including the ⁠{prefix}/datom/⁠ portion – while every other part of datom's business logic speaks in keys relative to the datom namespace root. Mixing the two shapes double-prefixes silently and does not error, so the root is stripped here, once, against the same builder the backends use.

Value

Character vector of artifact names, possibly empty. One storage listing, recursive.

Get Byte Size of a Single Storage Object

Description

Returns the byte size of the object at rel_key without reading its content. For S3 uses HEAD; for local uses fs::file_size(). Errors if the object is not found.

Usage

.datom_storage_byte_size(conn, rel_key)

Arguments

Value

Numeric byte count.

Compute SHA-256 Hash of a Storage Object's Content

Description

For S3, downloads the raw bytes and hashes in memory. For local, hashes the file directly. Used by datom_storage_verify() in content mode.

Usage

.datom_storage_content_hash(conn, rel_key)

Arguments

Value

Character SHA-256 hex string.

Delete governance.json Mirror from Data Storage

Description

Removes the governance.json mirror during project teardown. No-ops silently when the key is absent. Deletion is implemented via prefix-delete on the exact key path.

Usage

.datom_storage_delete_governance_json(conn)

Arguments

Value

Invisible NULL.

Delete All Objects Under a Storage Prefix

Description

Removes every file under prefix/datom/{prefix_key} from storage. For S3 this lists then batch-deletes. For local it removes the directory. A missing prefix is a no-op (returns 0L). Pass prefix_key = NULL to delete the entire datom namespace for this connection.

Usage

.datom_storage_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, the count of deleted objects.

Download File from Storage

Description

Download File from Storage

Usage

.datom_storage_download(conn, key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if Storage Object Exists

Description

Check if Storage Object Exists

Usage

.datom_storage_exists(conn, key)

Arguments

Value

TRUE or FALSE.

List Objects Under a Storage Prefix

Description

Returns the keys of every object under {prefix}/datom/{prefix_arg}. Keys are returned in their full storage-key form (i.e. including the ⁠{prefix}/datom/⁠ portion), matching what .datom_local_list_objects() and .datom_s3_list_objects() return.

Usage

.datom_storage_list_objects(conn, prefix)

Arguments

Value

Character vector of full storage keys (may be empty).

Read governance.json Mirror from Data Storage

Description

Returns the parsed list, or NULL when the key is absent. Aborts on any non-not-found storage error or on failed schema validation.

Usage

.datom_storage_read_governance_json(conn)

Arguments

Value

Parsed list or NULL.

Read and Parse JSON from Storage

Description

Read and Parse JSON from Storage

Usage

.datom_storage_read_json(conn, key)

Arguments

Value

Parsed R list.

Strip datom Namespace Prefix from a Full Storage Key

Description

Converts a full storage key (as returned by .datom_storage_list_objects()) to a relative key suitable for upload/download helpers (after ⁠{prefix}/datom/⁠).

Usage

.datom_storage_rel_key(full_key, conn)

Arguments

Value

Relative key string.

Upload File to Storage

Description

Upload File to Storage

Usage

.datom_storage_upload(conn, local_path, key)

Arguments

Value

Invisible TRUE on success.

Write governance.json Mirror to Data Storage

Description

Writes content to .metadata/governance.json in the data store. Uses .datom_storage_write_json() dispatch (backend-neutral).

Usage

.datom_storage_write_governance_json(conn, content)

Arguments

Value

Invisible NULL.

Write an R List to Storage as JSON

Description

Write an R List to Storage as JSON

Usage

.datom_storage_write_json(conn, key, data)

Arguments

Value

Invisible TRUE on success.

Get Backend Type from Store Component

Description

Get Backend Type from Store Component

Usage

.datom_store_backend(component)

Arguments

Value

"s3" or "local".

Build a Store-Constructor Snippet for a Component

Description

Renders a copy/paste datom_store_local(...) or datom_store_s3(...) call string for a store component, for embedding in a generated README. Secrets are shown as placeholders.

Usage

.datom_store_constructor_snippet(component)

Arguments

Value

Character scalar — an R constructor call as text.

Get Region from Store Component

Description

Returns the AWS region for S3, NULL for local.

Usage

.datom_store_region(component)

Arguments

Value

Region string or NULL.

Get Root from Store Component

Description

Returns the storage root: bucket name for S3, directory path for local.

Usage

.datom_store_root(component)

Arguments

Value

Root string.

The commit_sha Storage Already Holds, by Version

Description

Nothing there and could not look are separate answers, and only one of them is safe to pass over in silence. The first write of an artifact has no stored history, which is ordinary and silent. A stored copy that exists and will not read is the opposite: the values only storage had are now unknown, and the caller is about to replace that file wholesale – so an entry git cannot attribute loses a good value. Collapsing the two into "no known values" makes the loss invisible in exactly the case where it is unrecoverable, which is why this returns the distinction rather than just a map.

Usage

.datom_stored_commit_shas(conn, name)

Arguments

Details

The existence probe is what separates them. Its own failure counts as could not look, never as absence: an unreachable store cannot report that a file is missing.

A stored copy that reads but holds no usable pair is an absence, not a failure – the document was inspected and had nothing to contribute.

Value

A list with shas (named character vector, commit_sha named by version, empty when there are none) and unreadable (TRUE when storage holds a copy this call could not read).

Drop the ⁠$fetch⁠ Link a Read Puts on Every Member

Description

The one step that makes read-modify-write possible. A member record is payload-shaped – exactly id plus optional tags – and datom_get_set() adds a callable fetch to each one, which both .datom_validate_members() and the sv1 encoder refuse. Removing it here means neither of them needs a carve-out for a field that must never reach a payload.

Usage

.datom_strip_member_links(members)

Arguments

Details

Only a function is dropped. A hand-built fetch = "junk" is left in place so the validator reports it; stripping by name would turn a typo into a silent success.

Value

The member list with any callable fetch element removed.

Coerce a Parsed-JSON String Set to a Character Vector

Description

A tag value arrives in one of three spellings, all meaning the same thing: a length-1 character vector ("output"), a longer character vector (c("safety", "efficacy")), or – after a JSON round trip with simplifyVector = FALSE – a list of length-1 strings. All three normalise to a character vector here, which is what makes a single string and a one-element array hash identically.

Usage

.datom_sv1_as_strings(v, what)

Arguments

Details

character(0) and list() (the parsed form of ⁠[]⁠) both normalise to the empty set. Upstream, canonicalization drops a key whose value is empty – "no labels" is spelled by omitting the key – so the encoder should never meet one. It must not depend on that: an encoder whose correctness rests on an upstream rule breaks silently the day that rule moves.

Value

A character vector, possibly of length zero.

Hash Bytes for datom-sv1

Description

The single SHA-256 call of the datom-sv1 regime. Returns raw bytes rather than hex because every intermediate digest is concatenated into the next hash input; hex would double the width and put a text encoding in the identity path.

Usage

.datom_sv1_h(bytes)

Arguments

Value

A raw vector of 32 bytes.

Render a Digest as Lowercase Hex

Description

Used for the two places the specification names hex: the collation key for member digests, and the final data_sha string. Byte order and lowercase-hex C-locale order agree (00-09 before ⁠0a⁠-⁠0f⁠, digits before letters in ASCII), so sorting either representation gives the same result – hex is named in the spec because it is what a reader can compare by eye.

Usage

.datom_sv1_hex(x)

Arguments

Value

A character string of 2 * length(x) lowercase hex digits.

Encode a Map for datom-sv1

Description

⁠map(m) = h(0x03 || concat(str(k) || strset(m[k]) for k in sort(keys(m), radix)))⁠.

Usage

.datom_sv1_map(m, what = "map")

Arguments

Details

One encoder serves both slots of a member record – the id and the tags – so a fifth id field added later is just another key: no positional convention to maintain, and no absent-versus-empty question. id values are single strings, encoded as one-element string sets; enforcing "exactly these four keys, each single-valued" is validation's job, not the encoder's.

An absent map (NULL) and an empty map both encode as h(0x03). Writers never emit an empty map, but the encoder must not depend on that.

Value

A raw vector of 32 bytes.

Encode a Member Record for datom-sv1

Description

member(x) = h(0x04 || map(x.id) || map(x.tags)). Both slots are maps, so swapping content between them cannot collide, and a member with no tags encodes its tags slot as the empty map.

Usage

.datom_sv1_member(x, what = "member")

Arguments

Details

An unexpected field aborts. That is not grammar validation creeping in: a field the encoder ignored would be content that does not enter identity, so two payloads differing in it would share one data_sha and one storage address.

Value

A raw vector of 32 bytes.

Encode a Set Payload for datom-sv1

Description

set(p) = h(0x05 || map(p.tags) || concat(sort(unique(member(m)), radix))).

Usage

.datom_sv1_set(payload, what = "payload")

Arguments

Details

Member digests are deduped and sorted, exactly like tag values: arrangement is presentation, not content. The producer of a member list is normally a script, so an insertion-order refactor must not mint a new version of a citable artifact.

A zero-member payload aborts, mirroring .datom_canonical_hash()'s refusal of a zero-row or zero-column table.

Value

A raw vector of 32 bytes.

Encode a String for datom-sv1

Description

str(s) = h(0x01 || utf8(s)). No length prefix and no terminator: the string is the entire hash input, so nothing follows it to be confused with.

Usage

.datom_sv1_str(s, what = "value")

Arguments

Value

A raw vector of 32 bytes.

Encode a String Set for datom-sv1

Description

⁠strset(v) = h(0x02 || concat(str(e) for e in sort(unique(v), radix)))⁠. Order and multiplicity are not identity: a multi-valued tag models simultaneous membership in several categories, which has no order and no notion of a repeated element.

Usage

.datom_sv1_strset(v, what = "value")

Arguments

Details

The empty set is h(0x02) over an empty concatenation – a pinned golden.

Value

A raw vector of 32 bytes.

A Preview Frame Checked for Shape and Values, as Plain Text Columns

Description

Columns and values only, never where the frame came from: a subset or a hand-built frame is as good as the preview itself. Every check runs before any read.

Usage

.datom_sync_apply_frame(manifest)

Arguments

Details

Values are checked only on the rows apply acts on (new, changed); the others do nothing, so a hand-trimmed not_checked row is harmless. A short version_from would otherwise fail the exact comparison later and stop as stale, naming the wrong problem.

Value

A data frame of the six preview columns, each character.

Refuse a Row Whose Kind Disagrees With the Artifact

Description

Refuse a Row Whose Kind Disagrees With the Artifact

Usage

.datom_sync_check_kind(row, found)

Arguments

Value

Invisibly NULL; aborts with class datom_sync_kind_mismatch.

Which Context a Sync Call Is In: an Ordinary Repo or a Product Repo

Description

The two sync verbs do different jobs depending on the repo: an ordinary repo imports source files, a product repo maps its one set against source projects. This reads which, once per call, so every branch below it acts on one answer.

Usage

.datom_sync_context(conn)

Arguments

Details

Read from .datom/project.yaml, not from the connection, for the reason .datom_refuse_import_on_product() gives: the answer can authorise a write, and a hand edit or a pull can change the file after the connection was built. And the parse is gated – the file's declared format is checked before mode or set is read out of it.

Value

A list of product (TRUE for a mode: product repo) and set (the declared set name as written, possibly NULL). A repo with no config is reported as ordinary: the file path then fails with its own message about an uninitialised repo.

Sync Data-Side Metadata to Storage

Description

Mirrors the data repo's metadata to the data store so readers see current state: the manifest (.metadata/manifest.json) and each artifact's metadata ({name}/.metadata/metadata.json, version_history.json). Every artifact of either kind, not tables only – discovery is .datom_clone_artifact_names() and has been kind-agnostic since sets existed.

Usage

.datom_sync_data_metadata(conn, .confirm = TRUE)

Arguments

Details

It is not metadata-only, and the name understates it. For a set, a payload missing from storage is restored from the clone – see .datom_restore_set_payload() in this file for the three conditions on that. So this function can put content into storage, not just documents about content.

Data-only: governance files (dispatch.json, ref.json, migration_history.json) are not touched here. Governance sync is owned by the governance layer (gov_sync_dispatch()).

Two public routes reach this, and both get the restore: datom_write(conn) with no data and no name (the mirror-everything route), and datom_validate(fix = TRUE). Describing the restore as repair-only would leave a reader surprised to see it fire under a write verb.

Used after a failed upload, or by datom_validate(fix = TRUE), to bring storage back in line with the local data clone. Requires a developer connection with a local repo path.

Value

Invisibly, a list with repo_files (character vector of synced keys) and tables (list of per-table sync results).

Sync governance.json Storage Mirror from Git Copy

Description

Reads the git-canonical copy and overwrites the storage mirror. Call after a partial failure to repair a missing or stale storage mirror.

Usage

.datom_sync_governance_json(conn)

Arguments

Value

Invisible NULL.

Sync Single Table Metadata to S3

Description

Sync Single Table Metadata to S3

Usage

.datom_sync_metadata(conn, name)

Arguments

Value

Summary of sync operation.

Does a Name Match a Sync Glob?

Description

The same glob rule the file scan applies to file names.

Usage

.datom_sync_name_matches(names, pattern)

Arguments

Value

A logical vector.

Build the Member a new Row Adds, From Its Source

Description

Read through datom_member(), which confirms the version exists and records the project the artifact's own metadata declares. That project and the kind are then checked against the row: the connection's label is what routed the read, and nothing verifies a label.

Usage

.datom_sync_new_member(row, conn, tags)

Arguments

Value

A member record.

The Columns a Set Sync Preview Carries, in Order

Description

The Columns a Set Sync Preview Carries, in Order

Usage

.datom_sync_preview_cols()

Value

A character vector.

The Repo's Set As Stored, or an Empty One When It Has Never Been Written

Description

See point 1 of this file's header. The probe is on the set's current-state document, which is what datom_get_set() reads first. FALSE means the set has never been written; TRUE means read it, and any error from that read is the caller's to see. An error from the probe itself is never absence: an unreachable store cannot report that a file is missing.

Usage

.datom_sync_read_set(conn, name)

Arguments

Value

A datom_set.

Refuse Two Applied Rows for One Artifact

Description

A preview never produces them. Without this the second row would stop as stale, which names the wrong problem.

Usage

.datom_sync_refuse_duplicate_rows(todo)

Arguments

Value

Invisibly NULL; aborts with class datom_sync_manifest_duplicate_row.

Refuse Applied Rows Whose Project Has No Connection

Description

Only new and changed rows: they are the only ones apply acts on, and a full preview's not_checked rows belong by definition to projects not in sources, so checking every row would make an unedited preview impossible to apply.

Usage

.datom_sync_refuse_missing_source(todo, labels)

Arguments

Value

Invisibly NULL; aborts with class datom_sync_source_missing.

Refuse a Preview the Set Has Moved Away From

Description

A changed row must find exactly one member for its artifact, at version_from; a new row must find none. Anything else means the set was edited after the preview was built, and acting anyway would leave a removed member removed, move a member the preview never showed, or add a second member for one artifact.

Usage

.datom_sync_refuse_stale(todo, hits, members, x_given)

Arguments

Value

Invisibly NULL; aborts with class datom_sync_manifest_stale.

Apply a Set Sync Preview to a Product Repo's Set

Description

The product-repo route of datom_sync(). See point 5 of this file's header: every check that needs no read comes first, then the set is read (when not passed), then the stale and kind checks against it, and only then one snapshot read per applied row.

Usage

.datom_sync_set_apply(conn, set_name, manifest, sources, tags, x)

Arguments

Value

The edited datom_set.

The Product Repo's Declared Set Name, or an Abort Saying There Is None

Description

The Product Repo's Declared Set Name, or an Abort Saying There Is None

Usage

.datom_sync_set_name(set_name)

Arguments

Value

The set name.

Map a Product Repo's Set Against Its Sources

Description

The product-repo route of datom_sync_manifest(). Three kinds of read: the stored set (or none), one manifest per source, nothing per artifact.

Usage

.datom_sync_set_preview(conn, set_name, sources, pattern)

Arguments

Value

The preview data frame; see datom_sync_manifest().

One Source's Artifacts, After Checking Its Label Against Its Own Manifest

Description

See point 3 of this file's header.

Usage

.datom_sync_source_artifacts(conn)

Arguments

Value

The artifacts frame from .datom_current_artifacts().

One Member's Tags as Key/Value Pairs

Description

A member with no tags yields one pair of NA / NA, which is what keeps an untagged member visible in a listing rather than absent from it.

Usage

.datom_tag_pairs(tags)

Arguments

Details

A key whose value is empty also yields NA rather than no row. The writer drops such a key, so this reaches only a hand-built payload – and there the key IS in the document, so reporting the key with no value states what is there while dropping the row would not.

THE MAP IS READ BY POSITION, NEVER BY NAME, AND THAT IS THE WHOLE POINT OF THE FUNCTION. A tag map can carry the same key twice – jsonlite parses ⁠{"type": "output", "type": "baseline"}⁠ into two same-named elements, and a caller can write list(type = "a", type = "b") – and nothing on the read side refuses it, because a reader does not validate a tag map. tags[["type"]] returns the first match every time, so a by-name read reports one label twice and loses the other: the member lists a value it does not have, vanishes from a branch it belongs under, and cannot be found by the label the document says it carries. Verified in all three verbs before this was positional.

That is the file header's one-expander rule reappearing on the key axis. Having one expander closed the silent-first-value spelling on the value axis; reading that expander's own input by name reopened the identical failure one level up.

Duplicate keys are therefore treated exactly as one multi-valued key would be, which is also what they mean. Identical pairs are not collapsed across duplicate keys: the read reports what the document holds, and deduplicating here would be a reader canonicalizing.

Value

A data frame of key and value, at least one row.

A Tag Value as a Plain Character Vector

Description

This is only possible because the tag grammar is text-only. .datom_validate_tag_map() refuses numbers, booleans, null and nesting, so a tag value is a character vector and the value column of a member listing is a plain character column with no list-column anywhere. If the grammar ever widened, this function and the long format above it are what would have to change.

Usage

.datom_tag_values(v)

Arguments

Details

The two shapes handled are the two a value legitimately arrives in: a character vector, and the list-of-length-1-characters a JSON array parses as. A missing value is dropped rather than carried, because it states no label and no datom write can produce one – carried through, it would become a branch named NA.

Value

A character vector, possibly empty.

Tidy a Set Payload