The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
What people say, and what they do.
saydo measures the say-do gap: whether the themes in what people say (interviews, open survey answers, feedback, support tickets) go with differences in what they do (session logs, errors, adherence, outcomes). It codes the text with an LLM, checks that coding against a human, links the codes to each participant’s quantitative data, and tests every theme against every metric, in one reproducible R workflow.
LLM coding on its own is already well served: quallmer codes text against a codebook and validates it thoroughly. What saydo adds is the step after coding, which MAXQDA offers as “qualitative themes by quantitative groups” but R has lacked: joining codes to external participant data, rolling segment codes up to participants, and testing theme-by-metric differences with adjusted p-values and effect sizes.
transcripts ──► llm_auto_code() ──► code_agreement() is the LLM coding reliable?
│
behavioural logs ──────┴──────────► cross_tab_themes() which themes go with metric shifts?
│
└──► plot_theme_impact()saydo is not on CRAN yet. Install the development version from GitHub:
# install.packages("remotes")
remotes::install_github("neu-data/saydo")You also need an API key for an LLM provider supported by ellmer, for example
OPENAI_API_KEY or ANTHROPIC_API_KEY in your
.Renviron.
The package includes a simulated dataset and an offline keyword coder, so you can try the whole workflow without an API key.
library(saydo)
users <- mock_mixed_data(n_users = 120, seed = 42)
users
#> # A tibble: 120 × 5
#> user_id profile session_duration_sec errors_encountered transcript_feedback
#> <chr> <chr> <dbl> <int> <chr>
#> 1 USR-1001 frustrated 434 5 The layout changed and now I can't f…
#> 2 USR-1002 frustrated 475 11 I got completely lost trying to save…
#> 3 USR-1003 frustrated 420 11 Why is the checkout process so confu…
#> 4 USR-1004 frustrated 477 4 Why is the checkout process so confu…
#> 5 USR-1005 efficient 165 0 Really clean UI. Navigating from the…
#> 6 USR-1006 efficient 105 0 The feature works exactly as expecte…
#> 7 USR-1007 efficient 104 1 Extremely efficient workflow. Saved …
#> 8 USR-1008 frustrated 472 12 This interface is incredibly frustra…
#> 9 USR-1009 casual 225 2 Standard performance. Nothing specta…
#> 10 USR-1010 casual 304 3 Logged in, checked the system status…
#> # ℹ 110 more rowsA codebook maps each code name to a definition. Clear, specific definitions give much more reliable coding than bare labels.
codebook <- c(
UI_Confusion = "The user reports being lost, confused, or unable to find something in the interface",
Positive_Experience = "The user expresses satisfaction, praise, or enjoyment of the product",
Feature_Request = "The user asks for a feature or capability that does not currently exist"
)coded <- llm_auto_code(
users, user_id, transcript_feedback, codebook,
chat = ellmer::chat_openai(model = "gpt-4.1-mini")
)Each text is coded in its own conversation, so earlier transcripts
never influence later ones. Requests run in parallel
(max_active), and the result has one
TRUE/FALSE column per code. Any ellmer chat
works, for example ellmer::chat_anthropic(),
ellmer::chat_google_gemini(), or a local model through
ellmer::chat_ollama().
To run the example offline, swap the LLM for the demo coder. The rest of this README uses its output:
coded <- llm_auto_code(users, user_id, transcript_feedback, codebook,
coder = mock_coder())
coded[, c("user_id", names(codebook))]
#> # A tibble: 120 × 4
#> user_id UI_Confusion Positive_Experience Feature_Request
#> <chr> <lgl> <lgl> <lgl>
#> 1 USR-1001 TRUE FALSE FALSE
#> 2 USR-1002 TRUE FALSE FALSE
#> 3 USR-1003 TRUE FALSE TRUE
#> 4 USR-1004 TRUE FALSE TRUE
#> 5 USR-1005 FALSE TRUE FALSE
#> 6 USR-1006 FALSE TRUE FALSE
#> 7 USR-1007 FALSE TRUE TRUE
#> 8 USR-1008 TRUE FALSE FALSE
#> 9 USR-1009 FALSE FALSE TRUE
#> 10 USR-1010 FALSE FALSE FALSE
#> # ℹ 110 more rowsBefore analysing the codes, have a person code a random sample of the texts and compare:
set.seed(1)
to_check <- coded[sample(nrow(coded), 40), c("user_id", "transcript_feedback")]
# ...a human adds UI_Confusion, Positive_Experience and Feature_Request columns...
code_agreement(coded, human_coded)For each theme you get percent agreement, Cohen’s kappa with a confidence interval, and the LLM’s precision, recall and F1 against the human codes. Low recall means the LLM misses the theme, and low precision means it over-applies it; either is usually fixed by sharpening the code’s definition. As a rough guide, kappa above 0.6 is substantial agreement and above 0.8 near-perfect.
results <- cross_tab_themes(coded, session_duration_sec)
print(results, width = Inf)
#> # A tibble: 3 × 12
#> theme n_absent n_present mean_absent mean_present difference conf_low conf_high
#> <chr> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 UI_Confusion 71 49 182. 451. 269. 244. 294.
#> 2 Positive_Experience 76 44 381. 136. -245. -271. -218.
#> 3 Feature_Request 84 36 299. 275. -23.8 -81.1 33.6
#> cohens_d p_value p_adjusted significant
#> <dbl> <dbl> <dbl> <lgl>
#> 1 4.04 3.73e-38 1.12e-37 TRUE
#> 2 -2.72 1.01e-31 1.52e-31 TRUE
#> 3 -0.160 4.12e- 1 4.12e- 1 FALSEUsers who describe UI confusion spend about 269 seconds longer per session. Feature requests, which the simulation adds at random, should show no effect.
For each theme, cross_tab_themes() runs Welch’s t-test
(present vs absent), reports the mean difference with a 95% confidence
interval and Cohen’s d, and adjusts p-values across themes for multiple
comparisons (Benjamini–Hochberg by default, see p_adjust).
Themes too rare to test get NA rather than a misleading
result.
plot_theme_impact(results)
Interview transcripts are often coded by segment, with several rows
per participant, while behavioural metrics have one row per participant.
Code the segments, then pass the participant table as
metrics:
coded_segments <- llm_auto_code(segments, participant_id, segment_text, codebook,
chat = ellmer::chat_openai())
cross_tab_themes(coded_segments, minutes_active, metrics = participant_logs)A theme counts as present for a participant if it appears in any of
their segments. The codes are then joined to
participant_logs by participant_id.
| Function | Purpose |
|---|---|
llm_auto_code() |
Apply a codebook to text with an LLM (or any custom coder) |
code_agreement() |
Validate LLM codes against human codes: kappa, precision, recall, F1 |
cross_tab_themes() |
Test whether a numeric metric differs when each theme is present |
plot_theme_impact() |
Plot metric differences per theme with confidence intervals |
mock_mixed_data() |
Simulated users with metrics and feedback text |
mock_coder() |
Offline keyword coder for demos and tests |
code_agreement() results alongside any analysis. Reviewers
will expect to see them.cross_tab_themes() describes group differences and does not
adjust for confounders. For that, fit your own model on the coded
data.chat and record it, because LLM coding can change between
model releases.ellmer::chat_ollama().MIT © Neudata Consulting Ltd
Neudata Consulting Ltd · Insight. Impact.
Innovation.
www.neu-data.com ·
contact@neu-data.com
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.