Version SHAs: The Three-SHA Identity Model

Companion to: Getting Started. Read this when you’ve noticed that datom_history() shows a version column and a data_sha column and want to know why there is more than one hash – or when datom_write() has told you a column is not hashable.

datom identifies a table with three SHAs, each answering a different question:

SHA Question it answers Computed over
data_sha Is this the same table content? the table’s values, canonically encoded (datom-cv1)
metadata_sha Is this the same version? the table’s semantic metadata, as canonical JSON
parquet_sha Are the stored bytes intact? the exact parquet object that was uploaded

A fourth hash, original_file_sha, is recorded for tables onboarded from a file by datom_sync(). It is not part of table identity – it is file provenance, and it is what lets a re-scan say “this input has not changed” without reading the file.

The rest of this article explains each one, the type contract data_sha depends on, and the identity decisions baked into it.

data_sha: the canonical content hash

data_sha is a hash of the table’s values, computed with the datom-cv1 algorithm. Sketching the layout:

# per column, in table column order
col_digest <- sha256(utf8(type_tag) || utf8(column_name) || 0x00 || payload)

# then the table
data_sha <- sha256("datom-cv1" || f64le(nrow) || f64le(ncol) ||
                   concat(col_digest_hex ...))

Two properties matter more than the layout:

Why not just hash the parquet bytes

An earlier design hashed the serialized parquet file. That is simpler, and wrong for a versioning tool: parquet bytes move with the writer, not the data. An arrow upgrade, a different compression default, or a different dictionary-encoding heuristic all produce different bytes for identical content – and every one of those would have minted a spurious new version.

Hashing values instead of bytes decouples identity from the serializer. The serializer’s bytes are still hashed, but under a different name and for a different purpose: that is parquet_sha, below.

The datom table contract

Hashing values means datom must decide, per column type, what the bytes of that value are. That decision cannot be guessed, so datom-cv1 supports a defined set of types and refuses everything else with specific advice.

Supported: logical, integer, double, character, factor, Date (including data.table::IDate), POSIXct, difftime/hms, data.table::ITime, bit64::integer64, and labelled vectors (haven_labelled, labelled, labelled_spss) over any of those.

Where this bites

Most clinical and analytical tables are already inside the contract. The cases that are not tend to arrive the same handful of ways:

None of these are errors in your data. They are values whose canonical byte form is a modelling decision, and datom asks you to make it explicitly rather than picking one silently.

Check before you write

datom_check_hashable() is the pre-flight entry point. It is pure – no connection, no store, no network – so you can run it on any data frame:

library(datom)

clean <- data.frame(
  id    = 1:3,
  score = c(1.5, 2.5, 3.5),
  grp   = factor(c("x", "y", "x")),
  day   = as.Date(c("2026-01-01", "2026-01-02", "2026-01-03"))
)
datom_check_hashable(clean)
#> ✔ All 4 columns are hashable. This table is ready for `datom_write()`.
messy <- data.frame(id = 1:2)
messy$notes <- list(c("a", "b"), "c")
messy$z <- c(1 + 2i, 3 + 4i)

report <- datom_check_hashable(messy)
#> ✖ 2 of 3 columns are not hashable.
#> ✖ Column notes (<list>): List and blob columns are not hashable. Flatten to one
#>   value per row with tidyr::unnest(), or serialize each element to character
#>   (for example with jsonlite::toJSON() per element), before writing.
#> ✖ Column z (<complex>): Complex columns are not hashable. Split into separate
#>   real and imaginary numeric columns, or convert to character, before writing.
#> ℹ Fix these before `datom_write()`, which refuses the whole table until they are resolved.
report[report$status == "unsupported", c("column", "class")]
#>   column   class
#> 2  notes    list
#> 3      z complex

The report is returned invisibly as a data frame, so it composes: filter it, kable() it, or assert on it in a pipeline.

datom_write() runs the same check itself, before it touches git or storage, and reports every offending column in one error rather than one per attempt. A refusal leaves no partial state behind.

The recourse table

Each refused type has one canonical piece of advice. The table below is generated by running datom_check_hashable() over one fixture per case, so it is the same text datom_write() aborts with – the documentation cannot drift from the code.

Column class Recourse
POSIXlt/POSIXt POSIXlt columns are not hashable. Convert to POSIXct with as.POSIXct() before writing.
sfc sf geometry (sfc) columns are not hashable. Convert to WKT text with sf::st_as_text() before writing.
list Nested data-frame (list) columns are not hashable. Model the inner table as its own datom table joined by a key with datom_parent() lineage, or flatten it with tidyr::unnest(), before writing.
list List and blob columns are not hashable. Flatten to one value per row with tidyr::unnest(), or serialize each element to character (for example with jsonlite::toJSON() per element), before writing.
units units columns are not hashable. Drop the unit with units::drop_units() and record the unit in the column name or a companion column (audit-friendly), before writing.
yearmon zoo::yearmon / yearqtr and chron columns are not hashable. Convert to Date/POSIXct or ISO-8601 text before writing.
complex Complex columns are not hashable. Split into separate real and imaginary numeric columns, or convert to character, before writing.
raw Raw columns are not hashable. Encode the bytes as character (for example base64) before writing.
myclass Columns of this class are not hashable. Convert to a supported type (logical, integer, double, character, factor, Date, POSIXct, difftime/hms, or bit64::integer64) before writing.

The ingestion allowlist

datom_sync() onboards flat tabular files only: csv, tsv, txt, psv, parquet, sas7bdat, xpt, sav, zsav, por, dta, xls, xlsx. Anything else – .rds, .json, .xml – is refused up front. datom_sync_manifest() flags it as unsupported_format and still processes its allowlisted siblings; datom_sync() reports the same recourse in the error column for that row.

The reason is the table contract: a container format can deserialize to anything, including the list and exotic columns the hash refuses. Refusing at the door keeps the failure legible (“this format is not onboarded”) instead of deferring it to a type error deep in the write.

The escape hatch, and its tradeoff. You can always read an unsupported source yourself and pass the resulting data frame to datom_write():

dm <- readRDS("dm.rds")
datom_write(conn, data = dm, name = "dm")

The rule to know: that table is derived, not imported. It has no original_file_sha, so input-file change detection does not apply to it. A re-run will not report “unchanged” from the file’s bytes; it compares the data. If you need file-level change detection for that source, convert it to CSV or parquet once and let datom_sync() onboard it.

The identity decisions

Every entry below is a deliberate choice about what counts as “the same table”.

Decision Rationale
logical, integer, double unify (same encoding, same tag) 1L and 1 are the same number. Round-tripping a table through CSV, parquet, or a database routinely flips storage type without changing the data.
Factor levels and orderedness are not identity A factor’s values are hashed as character. Re-leveling for a plot, or dropping unused levels, must not mint a new data version.
Value labels are not identity haven_labelled columns strip to their underlying type. A relabelled SAS import carries the same numbers.
POSIXct tzone is not identity The value hashed is the instant (epoch seconds). tzone is a display attribute; the same instant printed in two zones is one instant.
All NaN payloads collapse to one canonical NaN; -0 encodes as +0 0/0 and NaN are indistinguishable arithmetically, and IEEE-754 leaves NaN payload bits and signed zero to the platform. Letting them differ would make identity platform-dependent.
NA_real_ stays distinct from NaN “Missing” and “not a number” are different statements about the data, and R keeps them apart.
Doubles are hashed bit-exact – no rounding, no tolerance A tolerance would make identity non-transitive (a ~ b and b ~ c but a != c). If a pipeline produces last-bit differences, that is a fact about the pipeline; round explicitly if you want it hidden.
Unicode NFC and NFD forms differ Normalizing would require an ICU dependency and a normalization-form decision baked into every future version. Byte-identical text is the narrow, stable promise.
Row order and column order are significant; nothing is sorted Sorting to make identity order-independent needs a collation order, which is locale-dependent – reintroducing the platform dependence the rest of this design removes. A reordered table is a different table; arrange() deliberately if order should not matter.

On the apparent asymmetry: NaN payloads are canonicalized while NFD text is not, and storage types unify while row order does not. The line is drawn at platform non-determinism, not at “things a human would call equivalent”. Where the same logical value can have different bytes for reasons outside the user’s control (NaN payload bits, signed zero, integer-vs-double storage after a round-trip), datom canonicalizes – otherwise identity would depend on the machine. Where a difference is a real, visible, user-controlled property of the data (row order, NFC vs NFD, secs vs mins), datom preserves it: collapsing those would need a policy decision that belongs to the user, and any policy datom picked would be wrong for someone.

metadata_sha: the version

metadata_sha is what datom_history() calls version and what you pass to datom_read(version = ...). It hashes the table’s semantic metadata:

identity  <- c("colnames", "custom", "data_sha", "hash_algo", "ncol", "nrow",
               "original_file_sha", "parents", "source_lineage", "table_type")
semantic  <- metadata[intersect(names(metadata), identity)]
sorted    <- semantic[sort(names(semantic), method = "radix")]
canonical <- jsonlite::toJSON(sorted, auto_unbox = TRUE)
digest::digest(canonical, algo = "sha256", serialize = FALSE)

Four details are load-bearing:

  1. The JSON canonical form is hashed, not the R object. R’s serialize() is type-sensitive (10L and 10 differ); JSON-parsed metadata has lost that distinction. A reader pulling metadata from storage must compute the same metadata_sha as the writer who built it in memory, so the shared representation is the JSON text. serialize = FALSE is what makes digest() hash those bytes.
  2. method = "radix" sorts field names by byte value. Plain sort() is locale-collated, so two machines with different LC_COLLATE would order fields differently and produce different versions for identical metadata.
  3. The fields are chosen by name, and anything else in the document is ignored. This is the part that keeps a pinned analysis working. The alternative – hash everything except a list of known exceptions – cannot survive a new field: a version of datom that has never heard of that field cannot know it was meant to ignore it, so it folds the field into the hash, disagrees with the version already recorded, and reports a change on content that did not move. Not once, but on every run. Selecting by name means an unfamiliar field is simply not identity, so a repo written by a newer datom still reads with the versions it already has.
  4. Everything datom deliberately leaves out is recorded as such, in .datom_metadata_excluded_fields: created_at and datom_version are wall-clock and package facts; parquet_sha and size_bytes move with the arrow version for identical content, so letting them in would undo the whole point of hashing values; document_sha is the same kind of fact for a stored JSON payload; column_hashes is no longer written (see “What metadata reveals about your data” below) and stays listed only so that files written by earlier releases still classify; and schema_version describes the file’s format rather than its content – in identity, a format change would mint a new version for every table in every repo while the data stood still.

Conversely, original_file_sha and hash_algo are semantic: a new source file is a new version of the table’s provenance, and a new hash algorithm is a new identity regime.

The two lists together are also datom’s record of every field name it recognises. Keeping them complete is a maintenance obligation on datom, not on you: the test suite derives the field list from the metadata builders and fails if one of them emits something neither list mentions.

The dedup guard is full-history

When a version’s metadata_sha already appears anywhere in version_history.json, no entry is appended – the current pointer in metadata.json is still updated. This matters when you re-sync an older file that content-matches: without a full-history scan, that write would append a second entry with a version value already present, and a later datom_read(version = ...) would report the SHA as ambiguous.

parquet_sha: stored-object integrity

parquet_sha is the SHA-256 of the exact parquet object in storage. It is not identity – it is a tamper and corruption check:

Change classification

Every datom_write() (including the one inside datom_sync()) lands in exactly one of three outcomes:

Outcome When Effect
none data_sha and metadata_sha both unchanged no commit, no upload, no history entry
metadata_only same data_sha, different metadata_sha (e.g. a re-export with new original_file_sha, or new parents) new version, parquet_sha carried forward, no upload
full data_sha changed new version, parquet stored at {name}/{data_sha}.parquet

The metadata_only case is the one that pays for the whole design: an EDC system re-exporting the same data with a new timestamp in the header produces different file bytes and identical content. datom records the new provenance and stores nothing twice.

What metadata reveals about your data

A table’s metadata describes its shape without its values: column names and order, row and column counts, lineage, provenance, and whatever you put in metadata =. That makes it useful on its own. You, a collaborator, or an AI coding assistant can write analysis code against a table’s metadata – a model of height on age, say – without ever seeing a value. And metadata lives in the data repo on GitHub, which is often shared more widely than the storage bucket.

For that to hold, metadata carries no per-column fingerprints. datom computes one digest per column as the input to data_sha, then discards them. A stored per-column digest would let anyone holding the metadata confirm a guess about one column: a flag column in a small table, a column that is constant, a few rows appended to a version the reader has already seen, or two same-named columns in different tables being identical. It would buy little in return, since each digest covers every row and adding rows – the common change – moves all of them.

What remains are whole-table hashes: data_sha, parquet_sha, and, for an imported table, original_file_sha. They cannot be dropped, because they are the storage address and the integrity checks. Like any hash, each can confirm a guess – but only a guess about the entire table at once, which is realistic only for a tiny or near-constant table.

Two things are outside datom’s control. Anything you put in metadata = is stored as given, so summary statistics there travel with the metadata. And datom 0.1.1 and 0.1.2 did write per-column digests, as column_hashes: a table’s current metadata.json loses the field the next time that table is written, while git history keeps the earlier copies.

If the people who can read your metadata can also read your data – the usual arrangement – none of this changes anything.

How far the cross-language claim goes

One caveat that is not ours to fix

data_sha is a portable identity of doubles. Getting from text to doubles is a separate step, and in base R it is neither correctly rounded nor platform-independent. R does its own decimal parsing (R_strtod in src/main/util.c): it accumulates digits into a long double and then scales by a power of ten. So the result depends on how wide long double is on the build – and as.numeric(), decimal literals, scan() and read.csv() all go through it.

This is known, accepted R behaviour rather than a bug: CRAN runs a dedicated check flavour with R configured --disable-long-double precisely because results differ, and package authors are told to tolerate it.

Check your own platform in one line:

.Machine$sizeof.longdouble   # 8 means long double == double
#> [1] 8

8 means no extra precision, and decimal parsing on that build deviates from the correctly-rounded value. It is not an Apple-versus-Intel split:

Build long double Decimal parsing
x86_64 (Linux, Windows, Intel mac) 80-bit extended agrees with correct rounding on everything tested
aarch64 Linux, s390x, riscv64 128-bit quad agrees
Apple silicon macOS, 32-bit ARM same as double deviates
any build with --disable-long-double (CRAN’s noLD flavour) forced to double deviates

Note that Linux and macOS on the same arm64 chip family fall on opposite sides – the variable is the long double width, not the architecture.

When it bites. Deviation is possible whenever mantissa * 10^exponent cannot be reached by a single correctly-rounded operation at the platform’s long double precision – that is, when the accumulated mantissa exceeds the significand, or 10^|exponent| is not itself exactly representable. 10^22 is the largest power of ten exactly representable as a double, and 10^23 the first that is not, which is why on Apple silicon a bare 1e-23 is the first one-digit value to drift. Longer mantissas drift sooner: 3.14159265358979e-9 is already 1 ULP off, and 1e300 / 1e-300 are 4. Short decimals – 0.1, 1, 1e15, 3.14159265358979 – are exact and agree everywhere, because only a single rounding occurs.

The consequence for datom. Two machines whose builds fall on opposite sides of that table can parse different doubles from the same CSV text, and will then compute different data_sha values – both correct, for genuinely different tables. No hash can paper over it; the divergence happens before datom sees the data.

Mitigations, in order of preference:

One warning: data.table::fread() is not a fix. It has its own parser that is also not correctly rounded, and its errors differ from base R’s – the two can disagree on the same string.

Worth keeping in perspective: this is a property of the platform’s decimal parser, not of datom, and it affects any analysis reading that CSV with or without datom. What datom adds is that the difference becomes visible instead of silent.

Derived columns: when arithmetic breaks cross-platform identity

The parsing caveat above is about text to doubles. A separate exposure exists for doubles to doubles: computing new columns from existing ones.

The guarantee direction

The key framing for both cases:

Scenario Possible? Consequence
same content → different identity Yes (wasteful, not harmful) duplicate storage, never corruption
different content → same identity Never the property that matters is preserved

datom is over-strict, never under-strict. When two platforms produce different data_sha values for the “same” pipeline, both are correct — they genuinely hold different doubles. No hash can paper over an arithmetic difference that already happened.

Which operations are stable

Not all floating-point work is equally portable. The dividing line is whether the operation’s result is fully specified by IEEE 754 or left to the implementation:

Category Examples Cross-platform identity
Data movement join, filter, reshape, select, bind Stable — no arithmetic, bits unchanged
IEEE basic ops +, -, *, /, sqrt Stable — IEEE 754 mandates a unique correctly-rounded result
Reductions sum(), mean(), cumsum() Unstable — accumulation order and intermediate precision vary
Transcendentals exp(), log(), sin(), cos() Unstable — IEEE 754 does not mandate correct rounding; libm implementations differ
BLAS / linear algebra %*%, crossprod(), solve() Unstable — tiling, FMA, and parallelism differ across BLAS builds

A pipeline that only moves data and applies basic arithmetic produces identical data_sha everywhere. One that calls sum() or log() on the same inputs can produce last-bit differences across platforms — and therefore different hashes.

Cross-language note

Even on a single machine, two languages can disagree. R’s sum() and numpy.sum() accumulate in different orders and with different intermediate widths, so they produce different doubles from the same input vector. An implementation of datom-cv1 in another language therefore reproduces the hash of given doubles — it does not reproduce the arithmetic that produced them. If you derive a column in R and then re-derive it in Python, expect different data_sha values even on the same hardware, unless the derivation uses only IEEE basic ops.

Practical advice

What you can rely on