The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

dtrackr - Grouping, Nesting and Long format data

Long format data

dtrackr assumes a tidy data paradigm where one row of data is relevant to one logical entity, whether it be cars, irises, diamonds, or anything else. This is not always the case, if for example the data you are processing comes from a join of data sets. Here we simulate a set of patients, test samples, and test results in a hypothetical trial:

age_cats = factor(sprintf("%02d-%02d",seq(0,80,5),seq(4,84,5)))

# A set of synthetic patients:
patients = tibble::tibble(
  patient_id = 1:100,
  age_category = sample(age_cats,100, replace=TRUE),
  ethnicity = sample(1:6, 100, replace = TRUE),
  gender = sample(c("Male","Female"), 100, replace=TRUE),
  group = sample(c("Cases","Controls"), 100, replace=TRUE)
) 

# each patient is going to have a random selection of tests
tests = tibble::tibble(
  test_id = 1:1000,
  patient_id = sample(1:100,1000, replace = TRUE),
  test_type = sample(c("FBC","LFT","Electrolytes"), 1000, replace=TRUE),
  test_date = as.Date("2025-01-01")+sample.int(50, 1000, replace=TRUE)
)

# and each test a random selection of results consisting of components and
# values:
tests = tests %>% mutate(
  result = purrr::map(test_type, ~ case_when(
    .x == "FBC" ~ list(tibble::tibble(
      component = c("HB","platelets","WCC"),
      value = c( runif(1,13.5,15), runif(1,100,1000), runif(1,0,30))
    )),
    .x == "LFT" ~ list(tibble::tibble(
      component = c("AST","GGT"),
      value = c( runif(1,0,100), runif(1,0,100))
    )),
    .x == "Electrolytes" ~ list(tibble::tibble(
      component = c("NA","K","Glucose"),
      value = c( runif(1,130,150), runif(1,3.3,5.2), runif(1,50,150))
    ))
  ))
)

data = patients %>% inner_join(
  tests %>% unnest(result) %>% unnest(result),
  by="patient_id"
)

data %>% glimpse()
## Rows: 2,654
## Columns: 10
## $ patient_id   <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, …
## $ age_category <fct> 10-14, 10-14, 10-14, 10-14, 10-14, 10-14, 10-14, 10-14, 1…
## $ ethnicity    <int> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 6, 6, 6, 6, 6, …
## $ gender       <chr> "Male", "Male", "Male", "Male", "Male", "Male", "Male", "…
## $ group        <chr> "Cases", "Cases", "Cases", "Cases", "Cases", "Cases", "Ca…
## $ test_id      <int> 272, 272, 446, 446, 684, 684, 684, 685, 685, 685, 781, 78…
## $ test_type    <chr> "LFT", "LFT", "LFT", "LFT", "Electrolytes", "Electrolytes…
## $ test_date    <date> 2025-02-18, 2025-02-18, 2025-01-15, 2025-01-15, 2025-02-…
## $ component    <chr> "AST", "GGT", "AST", "GGT", "NA", "K", "Glucose", "HB", "…
## $ value        <dbl> 51.800040, 80.631694, 85.141485, 10.422086, 138.213790, 4…

We might have an objective to prepare this data set for analysis but have inclusion or exclusion criteria that apply at different levels. We might have patients who need to be excluded as too young or old, or specific test results that were taken at the wrong time, or patients who have evidence of diabetes, or exclude specific test results that are out of range. All of this we need to do while stratified by the control group status.

To achieve this we use nesting to collapse the data frame into one row per patient, one row per test or one row per test result, depending on what we are trying to exclude. This allows dtrackr to dynamically change what it regards as a single countable thing, depending on the context of the pipeline.

processed = data %>%
  
  # the data is originally long format with one row per test result:
  track("{.count} test results") %>%
  mutate(maybe_diabetic = any(component == "Glucose" & value>130), .by = patient_id) %>%
  nest(test_panel = c(component,value), .messages="") %>%
  
  # Now the data is long format with one row per test:
  comment("{.count} tests") %>%
  nest(tests = starts_with("test_"), .messages="") %>%
  
  # and now long format with one row per patient:
  comment("{.count} patients") %>%
  group_by(group) %>%
  comment("{.count} patients") %>%
  
  # these exclusions are at the patient level
  exclude_all(
    .headline = "people",
    maybe_diabetic ~ "{.excluded} diabetics",
    age_category %in% age_cats[1:4] ~ "{.excluded} under 20"
  ) %>%
  
  # these are now back at the test level
  unnest(tests) %>%
  comment("{.count} tests",.headline = "") %>%
  exclude_all(
    .headline = "tests",
    test_date < "2025-01-07" ~ "{.excluded} with invalid dates"
  ) %>%
  count_subgroup(test_type, .headline = "") %>%
  
  # and finally at the granular test result level
  unnest(test_panel) %>%
  exclude_all(
    .headline = "results",
    component == "HB" & value < 14 ~ "{.excluded} invalid Hb results",
    component == "K" & value < 3.5 ~ "{.excluded} haemolysed K+"
  ) %>%
  group_by(test_type, .add=TRUE, .messages="By tests") %>%
  count_subgroup(component, .headline = "{test_type}") %>%
  ungroup(.messages = "{.count} eligible results") %>%
  nest(test_panel = c(component,value), .messages="") %>%
  comment("{.count} eligible tests") %>%
  nest(tests = starts_with("test_"), .messages="") %>%
  comment("{.count} eligible patients")
  

processed %>%
  flowchart()
%0 26:s->27 25:s->26 19:s->25 20:s->25 21:s->25 22:s->25 23:s->25 24:s->25 17:s->19 17:s->20 17:s->21 18:s->22 18:s->23 18:s->24 13:s->17 13:e->15 14:s->18 14:e->16 9:s->13 9:e->11 10:s->14 10:e->12 5:s->9 5:e->7 6:s->10 6:e->8 4:s->5 4:s->6 3:s->4 2:s->3 1:s->2 27 41 eligible patients 26 349 eligible tests 25 858 eligible results 19 Electrolytes Glucose: 57 items K: 50 items NA: 57 items 20 FBC HB: 35 items WCC: 65 items platelets: 65 items 21 LFT AST: 63 items GGT: 63 items 22 Electrolytes Glucose: 40 items K: 38 items NA: 40 items 23 FBC HB: 37 items WCC: 62 items platelets: 62 items 24 LFT AST: 62 items GGT: 62 items 17 By tests 18 By tests 13 Electrolytes: 57 items FBC: 65 items LFT: 63 items 14 Electrolytes: 40 items FBC: 62 items LFT: 62 items 15 results 30 invalid Hb results 7 haemolysed K+ 16 results 25 invalid Hb results 2 haemolysed K+ 9 205 tests 10 184 tests 11 tests 20 with invalid dates 12 tests 20 with invalid dates 5 Cases 53 patients 6 Controls 47 patients 7 people 20 diabetics 15 under 20 8 people 23 diabetics 13 under 20 4 stratify by group 3 100 patients 2 1000 tests 1 2654 test results

Maximum groupings

Going back to the original example data, in a slightly contrived example let’s assume we want to exclude age categories that don’t have a close gender match between cases and controls. We have to create a lot of small groups to count.

data %>% 
  group_by(age_category, gender, group) %>%
  summarise(
    n = n_distinct(patient_id)
  ) %>%
  pivot_wider(values_from = n, names_from = group) %>%
  filter(abs(Cases-Controls) <= 1) %>%
  glimpse()
## `summarise()` has grouped output by 'age_category', 'gender'. You can override
## using the `.groups` argument.
## Rows: 18
## Columns: 4
## Groups: age_category, gender [18]
## $ age_category <fct> 00-04, 00-04, 05-09, 10-14, 15-19, 25-29, 30-34, 35-39, 4…
## $ gender       <chr> "Female", "Male", "Male", "Male", "Female", "Female", "Fe…
## $ Cases        <int> 2, 2, 1, 3, 2, 1, 1, 3, 2, 1, 2, 1, 2, 1, 3, 2, 1, 2
## $ Controls     <int> 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 1, 3, 1, 2, 2, 1, 2

If we were to try and monitor this data frame through the pipeline there would be a problem with the flowchart because too many groups are generated. This causes performance and legibility issues for the resulting graph and is a result of an interim stage of the data pipeline where grouping is used to do fine scale summarisation operation. The most number of groups that dtrackr will attempt to keep track of is configurable but defaults to 16, and if the number of groups exceeds that it will pause tracking, until the number of groups is restored to a lower number, at which point it will start following again. A “< hidden steps >” message is inserted into the graph when this happens but this can be changed, or disabled altogether with options(dtrackr.hidden_steps = ""). dtrackr does not by default warn the user of this unless the options(dtrackr.verbose=TRUE) is set.

old = options(dtrackr.verbose=TRUE)

data %>% 
  track() %>%
  group_by(gender) %>%
  comment(c("{.count} items","before pause")) %>%
  
  # the tracking is paused on this next step as the number of groups becomes >16
  group_by(age_category, group, .add=TRUE) %>%
  comment("This message is not tracked") %>%
  summarise(
    n = n_distinct(patient_id)
  ) %>%
  pivot_wider(values_from = n, names_from = group) %>%
  filter(abs(Cases-Controls) <= 1) %>%
  
  # the tracking is automatically resumed at this point as the grouping has
  # returned to manageable levels.
  group_by(gender) %>%
  comment(c("{.count} summarised rows","after resume")) %>%
  flowchart()
## • This group_by() has created more than the maximum number of supported groupings (16) which will likely impact performance. We have paused tracking the dataframe.
## • To change this limit set the option 'dtrackr.max_supported_groupings'.
## • Automatically resuming tracking.
%0 6:s->7 6:s->8 5:s->6 3:s->5 4:s->5 2:s->3 2:s->4 1:s->2 7 Female 11 summarised rows after resume 8 Male 7 summarised rows after resume 6 stratify by gender 5 < hidden steps > 3 Female 1255 items before pause 4 Male 1399 items before pause 2 stratify by gender 1 2654 items 
options(old)

By default this behaviour is triggered if we get to 16 subgroups. This can be changed by setting the option:

options(dtrackr.max_supported_groupings = 16)

Pausing and unpausing the tracking can also be done manually by calling dtrackr::pause() and dtrackr::resume(). This is a fairly experimental feature, and I don’t expect it to be heavily used.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.