--- title: "Working with Loaders" description: > Turning tracking exports into Tracks: experiment manifests, read_tracks() and read_tracks_dir(), and registering a custom loader dialect or format. output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Working with Loaders} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) library(radiatR) ``` ## Overview `radiatR` provides a flexible loader framework that turns tracking exports from a wide variety of systems into `Tracks` objects. The helpers cover three typical tasks: 1. Reading experiment manifests (`import_info()` and `import_tracks()`). 2. Ingesting trajectory tables with `read_tracks()` or `read_tracks_dir()`. 3. Extending the loader registry with `register_loader_dialect()` for custom formats. This vignette walks through each step using the bundled millipede example data and finishes with a user-defined loader for a custom format. ## Loading a Real Experiment The package ships the trials from a *Cylindroiulus punctatus* (millipede) visual orientation experiment. Tracks are stored as paired tab-separated text files: `_point01.txt` holds two landmark rows (the origin + target location on the circumference); `_point02.txt` holds the full per-frame xy trajectory. ### Step 1 — discover files `import_tracks()` scans a directory for `_point01.txt` / `_point02.txt` pairs and returns a tibble of basenames. The example files ship as one archive, so unpack it into a temporary directory first. ```{r import-tracks} untar(system.file("extdata", "tracks.tar.gz", package = "radiatR"), exdir = tempdir()) track_dir <- file.path(tempdir(), "tracks") file_tbl <- import_tracks(track_dir) head(file_tbl) ``` ### Step 2 — read the manifest `import_info()` parses the trial manifest CSV. Use `cond_cols` to generate a compound condition column from multiple metadata fields. ```{r import-info} manifest_path <- system.file("extdata", "millipede_trials.csv", package = "radiatR") manifest <- import_info(manifest_path, cond_cols = c("type", "arc")) head(manifest) ``` ### Step 3 — extract and normalise trajectories `get_all_object_pos()` reads each file pair, uses the landmark rows to establish the unit-circle geometry, and returns a `Tracks` with normalised unit-circle coordinates. Passing `manifest` joins its columns (arc, type, obstacle, id, ...) onto every trial, matching on the `file` column; use `manifest_cols` to rename or restrict them. ```{r get-all} ts <- suppressWarnings(get_all_object_pos(file_tbl = file_tbl, track_dir = track_dir, manifest = manifest)) ts head(ts@meta$trial_limits[, c("video", "arc", "type", "obstacle")]) ``` ## dtrack format The bundled example data was tracked with **dtrack** (), a desktop tracking tool for video recordings of freely-moving animals. The data are from: > Kirwan J.D. & Nilsson D.-E. (2019). *A millipede compound eye mediating > low-resolution vision.* Vision Research 165, 36–44. > dtrack exports headerless, tab-separated text files with columns `frame`, `x`, `y`, and a fourth confidence/flag column. `radiatR` has no built-in dtrack dialect: the *Registering a Custom Dialect* section below writes one in a few lines and uses it to read these files. The `_point01` / `_point02` role split used in the bundled data — landmarks in one file, trajectory in the other — is specific to this experiment and is not a general dtrack convention. ## Tracking-tool format support `radiatR` reads tabular (CSV/TSV) exports from many tracking tools through named *dialects*. Each dialect maps a tool's column conventions onto the `Tracks` model of one position per individual per frame. Pass the dialect name to `read_tracks(path, dialect = "name")`; tool-specific options go through `dialect_args`. The table below summarises what is and is not currently handled. Two limitations are general: - **Tabular text only.** The built-in dialects read CSV/TSV (and, where the optional packages are installed, Excel and Parquet). Native binary formats — such as NumPy `.npz` or MATLAB `.mat` — are not read. - **Position, not posture.** `radiatR` models a single position per individual per frame. Per-frame *posture* data (variable-length midline or outline polylines) is outside this model. Where a tool tracks discrete body keypoints, those can be loaded as extra columns and turned into a body-axis heading with the `bodypart_axis` rule or `pose_to_headings()`; dense posture polylines are not ingested. | Dialect | Handled | Not handled | |---|---|---| | `trex` | Positional CSV: plain `X`/`Y`, the `#wcentroid` / `#centroid` / `#pcentroid` centroid variants, and TRex's `(cm)` / `(px)` unit annotations in headers. Per-individual `_fishN` files (id from filename) and aggregated files with an id column. A file with a track column (`Track`, `TRACK_ID`) is refused unless `id_col` names the id. | Native `.npz` export; posture export (`midline_points`, `outline_points`). | | `deeplabcut` | The predictions CSV: header rows `scorer` / `bodyparts` / `coords`, with `individuals` for multi-animal projects (one track per individual). A single bodypart, or a likelihood-weighted centroid of several; `likelihood_min` masks points and the count is kept in the metadata. Bodypart columns are preserved for `bodypart_axis`. | Native HDF5 (`.h5`) export; labelled training data (`CollectedData_*.csv`); 3-D output. | | `sleap` | Analysis CSV: `.x` / `.y` / `.score`; a single node or a score-weighted centroid; `track` and `frame_idx` (a table without `frame_idx` is refused unless `time_col` is given); `score_min` masks points and the count is kept in the metadata. | Native `.slp` / HDF5 export. | | `ethovision` | The text export: a first line giving the number of header lines, the trial details, a names row and a units row. One track per file, `Trial time` in seconds. The centre point, or the body points named in `bodypart` (`nose`, `tail`) and their mean; a sample EthoVision marks `-` has no position and is counted in the metadata. The header block and units are kept in `@meta$source_meta`. | Native `.xlsx` (export as text). A file cut off mid-row (delete the partial last line). | | `anymaze` | The CSV export: `Time` as a duration (`0:01:02.500`), `Centre position X` / `Centre position Y`. A sample with an empty position is counted in the metadata. Pass the file path, or read `Time` as text (`readr::read_csv(f, col_types = readr::cols(Time = "c"))`): readr's own time parsing drops the fraction of a second. | Head and tail positions; the `In ` occupancy columns. | | `trackmate` | Spot tables with `TRACK_ID` / `FRAME` / `POSITION_T` / `POSITION_X` / `POSITION_Y`, including the three extra header rows of version 7. Spots in no track (`TRACK_ID` `None`) stop the load unless `dialect_args = list(unlinked = "drop")`. | Native TrackMate `.xml`. | | `idtrackerai_wide` | `trajectories.csv`: one `x` / `y` pair per animal, `seconds` as time, `nan` for missing positions. | `.npy` trajectory files. | | `toxtrac` | `Tracking_RealSpace.txt` (mm, seconds) and the headerless `Tracking.txt` (pixels, frames; 6 or 11 columns). The id is `_`. | The statistics and project files. | Built-in dialects are kept only for tracking software that is widely used and still developed. When a tool's export is not directly supported, two escape hatches remain: convert to a tidy CSV and use `read_tracks()` with an explicit `mapping`, or register a custom dialect (see *Registering a Custom Dialect* below). ## Bundled single-file examples The package ships small real exports from two tools so the dialects can be tried without any external data. Each loads with one `read_tracks()` call, pointing the `dialect` argument at the matching tool. **DeepLabCut** — the predictions CSV written when a video is analysed (header rows `scorer`, `bodyparts`, `coords`, plus `individuals` for multi-animal projects). By default the position is the likelihood-weighted centroid of all bodyparts, the time is the frame index, and each bodypart's coordinates are kept for the `bodypart_axis` rule. ```{r dlc-example} dlc_path <- system.file("extdata", "dlc_single_wasp_predictions.csv", package = "radiatR") ts_dlc <- read_tracks(dlc_path, dialect = "deeplabcut", fps = 40) ts_dlc ``` A likelihood threshold masks the points below it. A frame with a masked point has no position, so it is dropped only when `on_invalid = "drop"` says so, and what was masked is reported and kept: ```{r dlc-threshold} ts_dlc_sure <- read_tracks(dlc_path, dialect = "deeplabcut", fps = 40, dialect_args = list(likelihood_min = 0.9), on_invalid = "drop") ts_dlc_sure@meta$attrition ``` **SLEAP** — an analysis CSV with `.x` / `.y` / `.score` columns, `track` as the individual and `frame_idx` as time (frames). The `score_min` and `partial` arguments work as `likelihood_min` and `partial` do for DeepLabCut. ```{r sleap-example} sleap_path <- system.file("extdata", "sleap_three_mice.analysis.csv", package = "radiatR") ts_sleap <- read_tracks(sleap_path, dialect = "sleap") ts_sleap ``` The bundled dtrack example is larger and is shown in full in the *Loading a Real Experiment* section above. ## Reading Tabular Trajectories For data already in a data frame or CSV, `read_tracks()` is the central entry point. ```{r basic-load} example_df <- data.frame( id = rep(c("A", "B"), each = 4), time = rep(seq(0, 3), times = 2), x = c(1, 1.2, 1.4, 1.5, -0.2, -0.1, 0, 0.15), y = c(0, 0.1, 0.3, 0.4, 1.0, 1.1, 1.3, 1.5) ) ts_df <- read_tracks(example_df, mapping = list(id = "id", time = "time", x = "x", y = "y")) ts_df ``` To batch a directory of CSV files, `read_tracks_dir()` calls `read_tracks()` for each match and row-binds the result. ```{r dir-load} csv_dir <- tempfile() dir.create(csv_dir) write.csv(example_df, file.path(csv_dir, "trial01.csv"), row.names = FALSE) write.csv(transform(example_df, id = paste0(id, "_2")), file.path(csv_dir, "trial02.csv"), row.names = FALSE) dset <- read_tracks_dir(csv_dir, pattern = "\\.csv$") length(dset) unlink(csv_dir, recursive = TRUE) ``` ## Manifest Helpers (Synthetic Example) For a quick demonstration without real files, `import_info()` works on any manifest CSV, and `import_tracks()` works on any directory following the `_point01` / `_point02` naming convention. ```{r manifest} manifest_path <- tempfile(fileext = ".csv") writeLines( c("file,type, arc", "trial01.csv, baseline,90", "trial02.csv,stimulus,135"), manifest_path ) import_info(manifest_path, cond_cols = c("type", "arc")) ``` ## Registering a Custom Dialect When no built-in dialect fits your exporter, register your own loader: a function that receives what was passed to `read_tracks()` (a file path or a data frame) and returns a data frame with `x` and `y` (or `angle`), and normally `id` and `time`. The dialect is the only reader of the file, so it can handle layouts the generic reader cannot: no header row, several header rows, or a format that is not a table at all. The bundled millipede tracks were made with dtrack, which writes a headerless, tab-separated file of `frame`, `x`, `y` and a flag column. A loader for it: ```{r custom-loader} dtrack_loader <- function(x) { # colClasses = "numeric" makes a malformed cell an error, not a silent NA d <- utils::read.delim(x, header = FALSE, colClasses = "numeric") if (ncol(d) < 3L) stop("dtrack: expected at least 3 columns (frame, x, y) in ", basename(x)) data.frame( id = tools::file_path_sans_ext(basename(x)), # one file, one track frame = d[[1]], x = d[[2]], y = d[[3]] ) } register_loader_dialect("dtrack", dtrack_loader) ``` After registration the dialect is used by name, like a built-in one: ```{r use-custom} track_file <- file.path(track_dir, "10_10_point02.txt") ts_raw <- read_tracks(track_file, dialect = "dtrack") head(ts_raw@data[, c("id", "frame", "x", "y")]) ``` A loader that reads files can take `read_tracks(read_opts = )` by declaring a `read_opts` argument: it receives the non-`NULL` entries (`decimal`, `delim`, `sheet`, `ext`) and must honour them or stop. Without that argument, any `read_opts` stop the load. Given several files, the loader is called once per file and the results are combined. Here `pattern` picks one session's trajectory files and leaves out the `_point01` landmark files, which share the same layout: ```{r use-custom-dir} ts_session <- read_tracks_dir(track_dir, pattern = "^10_.*_point02\\.txt$", dialect = "dtrack") ts_session ``` Dialects live in-memory for the current session. To persist them across projects, place the registration call in a package or in an `.Rprofile` that you source before using `radiatR`.