Gtheory4LLM: Generalizability Theory for LLM Subjective Tasks

Studies the reliability and generalizability of subjective judgments produced by large language models (LLMs), including annotation, rating, and structured LLM-as-a-judge tasks. Specifies evaluator, prompt, generation, and repeated-run facets through crossed or explicitly nested random sources with configurable item interactions. Fits univariate models, joint Gaussian models, and joint discrete models for binary, ordinal, and unordered categorical outcomes, with source-specific covariance. Gaussian models use exact balanced likelihood; discrete models use a dense first-order Laplace approximation with Gaussian latent random effects. Supported balanced decision studies compare evaluator, prompt, and replication allocations using observed Gaussian or explicitly requested latent binary and ordinal reliability, with random or fixed facets after Brennan (2001). Gaussian fits report asymptotic Wald standard errors for their variance components and delta-method intervals for the coefficients; discrete fits report point estimates only. Scalar nominal reliability and joint Gaussian-discrete fitting are not implemented. Includes three publicly archived LLM annotation datasets covering hate-speech, mental-health, and drug-review tasks. Discrete fitting is limited to small models; the preflight report describes supported designs and computational limits. Generalizability coefficients follow the variance-decomposition framework of Brennan (2001) <doi:10.1007/978-1-4757-3456-0>.

Version:	0.2.0
Depends:	R (≥ 4.5.0)
Imports:	Matrix (≥ 1.6.0), OpenMx (≥ 2.22.11), graphics, methods, stats, utils
Suggests:	lme4, ordinal, knitr, rmarkdown
Published:	2026-10-04
DOI:	10.32614/CRAN.package.Gtheory4LLM (may not be active yet)
Author:	Jin Liu [aut, cre, cph]
Maintainer:	Jin Liu <Veronica.Liu0206 at gmail.com>
BugReports:	https://github.com/Veronica0206/Gtheory4LLM/issues
License:	GPL-3
URL:	https://github.com/Veronica0206/Gtheory4LLM
NeedsCompilation:	no
Citation:	Gtheory4LLM citation info
Materials:	README, NEWS
CRAN checks:	Gtheory4LLM results

Documentation:

Reference manual:	Gtheory4LLM.html , Gtheory4LLM.pdf
Vignettes:	How many evaluators, prompts, and runs? (source, R code)

Downloads:

Package source:	Gtheory4LLM_0.2.0.tar.gz
Windows binaries:	r-devel: not available, r-release: not available, r-oldrel: not available
macOS binaries:	r-release (arm64): not available, r-oldrel (arm64): not available, r-release (x86_64): Gtheory4LLM_0.2.0.tgz, r-oldrel (x86_64): Gtheory4LLM_0.2.0.tgz

Linking:

Please use the canonical form https://CRAN.R-project.org/package=Gtheory4LLM to link to this page.