Package: evaluatellm
Title: Statistical Inference for Language Model Evaluations
Version: 0.1.0
Authors@R: person("Charles", "Coverdale", email = "charlesfcoverdale@gmail.com",
    role = c("aut", "cre", "cph"))
Description: Treats language model evaluations as statistical experiments and
    supplies the inference they require. Provides central limit theorem and
    cluster-robust standard errors for evaluation scores, paired and unpaired
    model comparisons, variance decomposition when several responses are drawn
    per question, control-variate variance reduction, multiplicity adjustment
    across benchmark suites, and power and minimum detectable effect
    calculations for planning evaluations, following Miller (2024)
    <doi:10.48550/arXiv.2411.00640>. For evaluations scored by a model judge,
    implements agreement statistics against a human gold standard and
    prediction-powered inference (Angelopoulos et al. 2023)
    <doi:10.1126/science.adi6000> with the power-tuned estimator of
    Angelopoulos, Bates and Jordan (2023) <doi:10.48550/arXiv.2311.01453>, so a
    small set of human labels debiases a large set of judge scores. Leaderboards
    are supported through bootstrap rank intervals and Bradley-Terry ratings
    (Bradley and Terry 1952) <doi:10.2307/2334029>. Accepts scores from any
    evaluation harness.
License: MIT + file LICENSE
Encoding: UTF-8
Language: en-GB
URL: https://charlescoverdale.github.io/evaluatellm/,
        https://github.com/charlescoverdale/evaluatellm
BugReports: https://github.com/charlescoverdale/evaluatellm/issues
RoxygenNote: 7.3.3
Depends: R (>= 4.1.0)
Imports: cli (>= 3.6.0), graphics, grDevices, stats, utils
Suggests: testthat (>= 3.0.0), knitr, rmarkdown, sandwich
VignetteBuilder: knitr
Config/testthat/edition: 3
NeedsCompilation: no
Packaged: 2026-09-08 07:19:25 UTC; charlescoverdale
Author: Charles Coverdale [aut, cre, cph]
Maintainer: Charles Coverdale <charlesfcoverdale@gmail.com>
Repository: CRAN
Date/Publication: 2026-09-15 13:40:02 UTC
Built: R 4.5.2; ; 2026-09-15 18:27:00 UTC; unix
