The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Analyzing a typical user study

library(colleyRstats)
#> Loading required package: ggplot2
#> Registered S3 methods overwritten by 'ggpp':
#>   method                  from   
#>   heightDetails.titleGrob ggplot2
#>   widthDetails.titleGrob  ggplot2

This vignette walks through the workflow colleyRstats was built for: a within-subjects user study, from raw data to the text and figures that go into the manuscript. Every number in the paper should come out of this script – no retyping, no copy-paste drift.

The study data

Twenty-four participants experienced three interface conditions and rated their mental demand (0–100, NASA-TLX style) and trust (1–7 Likert) after each. In your project this data frame comes from your logging or survey export; here we simulate it.

set.seed(42)
n <- 24

main_df <- data.frame(
  Participant = factor(rep(seq_len(n), each = 3)),
  ConditionID = factor(rep(c("Baseline", "HUD", "LED"), times = n))
)

cond_effect <- c(Baseline = 55, HUD = 46, LED = 44)[as.character(main_df$ConditionID)]
main_df$tlx_mental <- pmin(100, pmax(0, round(cond_effect + rnorm(nrow(main_df), sd = 10))))
main_df$trust <- pmin(7, pmax(1, round(3.5 +
  (main_df$ConditionID != "Baseline") * 0.9 + rnorm(nrow(main_df), sd = 1))))

Two things matter before any analysis:

The quick route: one call per dependent variable

analyze_and_report() runs the whole per-DV pipeline: it checks the assumptions (and phrases the justification for the methods section), builds the matching ggstatsplot figure with automatic parametric/non-parametric selection, reports the omnibus test, and – for three or more groups – the significant post-hoc comparisons.

res <- analyze_and_report(
  main_df,
  dv = "tlx_mental", iv = "ConditionID",
  design = "within", subject = "Participant",
  ylab = "Mental Demand (TLX)"
)
#> A Shapiro--Wilk test on the residuals of the additive model with participant and ConditionID as factors ($n = 72$) indicated no significant deviation from normality ($W = 0.98$, $p = 0.245$); therefore, parametric tests were used.
#> An ANOVA estimation for factorial designs using 'afex' found a significant effect of \ConditionID{} on tlx\_mental (\F{1.92}{44.2}{5.75}, \p{0.007}, $\omega_{p}^{2}$ = 0.11).
#> A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to HUD (\m{45.88}, \sd{9.76}; \padj{0.033}).
#> A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to LED (\m{45.21}, \sd{12.51}; \padj{0.015}).

The result carries everything separately, so you can place the pieces where they belong:

res$plot

res$methods  # for the Methods section
#> [1] "A Shapiro--Wilk test on the residuals of the additive model with participant and ConditionID as factors ($n = 72$) indicated no significant deviation from normality ($W = 0.98$, $p = 0.245$); therefore, parametric tests were used."
res$text     # the omnibus result
#> [1] "An ANOVA estimation for factorial designs using 'afex' found a significant effect of \\ConditionID{} on tlx\\_mental (\\F{1.92}{44.2}{5.75}, \\p{0.007}, $\\omega_{p}^{2}$ = 0.11). "
res$posthoc  # significant pairwise comparisons (NULL for 2 groups)
#> [1] "A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\\m{54.92}, \\sd{11.10}) in terms of tlx\\_mental compared to HUD (\\m{45.88}, \\sd{9.76}; \\padj{0.033}). " 
#> [2] "A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\\m{54.92}, \\sd{11.10}) in terms of tlx\\_mental compared to LED (\\m{45.21}, \\sd{12.51}; \\padj{0.015}). "

For a whole questionnaire battery, report_all() does this for every scale and adds a Holm-corrected summary across the dependent variables:

battery <- report_all(
  main_df,
  dvs = c("tlx_mental", "trust"),
  iv = "ConditionID",
  design = "within", subject = "Participant",
  labels = c(tlx_mental = "Mental Demand", trust = "Trust")
)
#> A Shapiro--Wilk test on the residuals of the additive model with participant and ConditionID as factors ($n = 72$) indicated no significant deviation from normality ($W = 0.98$, $p = 0.245$); therefore, parametric tests were used.
#> An ANOVA estimation for factorial designs using 'afex' found a significant effect of \ConditionID{} on tlx\_mental (\F{1.92}{44.2}{5.75}, \p{0.007}, $\omega_{p}^{2}$ = 0.11).
#> A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to HUD (\m{45.88}, \sd{9.76}; \padj{0.033}).
#> A Student's t post-hoc test (Holm-adjusted) found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to LED (\m{45.21}, \sd{12.51}; \padj{0.015}).
#> A Shapiro--Wilk test on the residuals of the additive model with participant and ConditionID as factors ($n = 72$) indicated no significant deviation from normality ($W = 0.97$, $p = 0.089$); therefore, parametric tests were used.
#> An ANOVA estimation for factorial designs using 'afex' found a significant effect of \ConditionID{} on trust (\F{1.77}{40.77}{7.37}, \p{0.003}, $\omega_{p}^{2}$ = 0.14).
#> A Student's t post-hoc test (Holm-adjusted) found that LED was significantly higher (\m{4.46}, \sd{0.78}) in terms of trust compared to Baseline (\m{3.42}, \sd{1.10}; \padj{0.002}).
battery$summary
#>           dv                                              method statistic
#> 1 tlx_mental ANOVA estimation for factorial designs using 'afex'  5.754671
#> 2      trust ANOVA estimation for factorial designs using 'afex'  7.365422
#>       p.value      p.holm
#> 1 0.006577594 0.006577594
#> 2 0.002635125 0.005270249

Step by step, if you prefer control

1. Check the assumptions

check_normality_by_group(main_df, "ConditionID", "tlx_mental", subject = "Participant")
#> [1] TRUE
#> attr(,"tests")
#>       group  n         W   p_value p_adjusted testable
#> 1 residuals 72 0.9781936 0.2448001  0.2448001     TRUE
#> attr(,"method")
#> [1] "residuals"
#> attr(,"p_adjust")
#> [1] "none"
#> attr(,"untestable")
#> character(0)
#> attr(,"dropped_subjects")
#> character(0)

With subject given, the check tests what a repeated-measures analysis assumes to be normal: the residuals once participant and condition effects are removed (or, for two conditions, the per-participant differences) – not the raw scores per condition, which also carry the spread between participants.

assumption_methods_text() turns the same checks into the sentence reviewers expect next to the choice of test:

assumption_methods_text(main_df, x = "ConditionID", y = "tlx_mental", subject = "Participant")
#> A Shapiro--Wilk test on the residuals of the additive model with participant and ConditionID as factors ($n = 72$) indicated no significant deviation from normality ($W = 0.98$, $p = 0.245$); therefore, parametric tests were used.

2. Plot with automatic test selection

The gg*WithPriorNormalityCheck* wrappers run the normality check and pick the parametric or non-parametric variant for you. The Asterisk versions annotate significant pairwise comparisons with APA-style stars instead of p-values:

plot_within_stats_asterisk(
  data = main_df,
  x = "ConditionID", y = "tlx_mental",
  ylab = "Mental Demand (TLX)",
  xlabels = c("Baseline", "HUD", "LED"),
  subject = "Participant"
)
#> Scale for x is already present.
#> Adding another scale for x, which will replace the existing scale.

(Every function also has a snake_case alias – plot_within_stats_asterisk() is ggwithinstatsWithPriorNormalityCheckAsterisk().)

3. Factorial designs: the Aligned Rank Transform

When there is more than one factor and normality is violated, the standard non-parametric route is the Aligned Rank Transform (ARTool). reportART() turns the ANOVA table into LaTeX sentences, and reportArtCon() reports the pairwise contrasts with rank-biserial effect sizes:

m <- ARTool::art(
  tlx_mental ~ ConditionID + Error(Participant / ConditionID),
  data = main_df
)
reportART(anova(m), dv = "mental demand")
#> The ART found a significant main effect of \ConditionID{} on mental demand (\F{2}{46}{5.56}, \p{0.007}, $\eta_{p}^{2}$ = 0.19, 95\% CI: [0.02, 0.38]).
ac <- ARTool::art.con(m, ~ ConditionID, adjust = "holm")
reportArtCon(
  ac,
  data = main_df, iv = "ConditionID", dv = "tlx_mental",
  paired = TRUE, id = "Participant"
)
#> An ART-C post-hoc test (Holm-adjusted) found that tlx\_mental for the \ConditionID{} Baseline was significantly higher (\mdn{56.00}, \iqr{12.75}) than for HUD (\mdn{46.50}, \iqr{11.75}; \padj{0.016}, \rankbiserial{0.54}) and LED (\mdn{47.00}, \iqr{12.50}; \padj{0.016}, \rankbiserial{0.64}).

4. Descriptives

reportMeanAndSD(main_df, iv = "ConditionID", dv = "tlx_mental")
#> Baseline: \m{54.92}, \sd{11.10}
#> HUD: \m{45.88}, \sd{9.76}
#> LED: \m{45.21}, \sd{12.51}

Into the manuscript

Every reporter accepts sink_to = "results/tlx.tex" to write its sentences to a file your manuscript can \input{} – re-run the analysis and the paper updates itself. Figures go through save_paper_figure(), which uses publication presets (ACM-style single-column 3.33 in, full-width 7 in):

fig_path <- file.path(tempdir(), "tlx-mental.pdf")
save_paper_figure(res$plot, fig_path, columns = 2)
#> Saved figure to 'C:\Users\Mark\AppData\Local\Temp\RtmpSsndDd/tlx-mental.pdf' (7 x 4.66666666666667 in, base font 9 pt).

The LaTeX macros used by the reporters (\F, \p, \m, …) are defined by latex_preamble() or the shipped colleyRstats.sty; see vignette("overleaf") for the full R-to-Overleaf pipeline, including emit_overleaf(), which bundles the entire analysis into a folder that compiles as-is.

Where to go next

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.