The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

aiEvalR 0.1.0

First release.

aiEvalR provides a statistical and psychometric framework for evaluating large language models and other AI systems, organized around seven modules.

Modules

• Psychometric quality (methodological core): multi-facet generalizability theory for AI evaluation. ai_generalizability() fits a crossed G-study (case x prompt x model x run) via lme4; ai_dstudy() projects the generalizability coefficient and dependability index (Phi) for any measurement design without refitting. Also ai_conditional_sem() and ai_decision_consistency().
• Reliability: ai_test_retest() (ICC), ai_internal_consistency() (Cronbach’s alpha), ai_bootstrap_reliability(), and the ai_reliability() wrapper.
• Robustness: stress_test(), prompt_sensitivity(), adversarial_stability(), and the ai_robustness() wrapper.
• Fairness: ai_group_disparity() (demographic parity, equalized odds), ai_dif_mh() (Mantel-Haenszel screening), and ai_fairness(), which integrates with the companion aiDIF package for item-level IRT-based AI-scoring bias when item MLEs are supplied.
• Calibration: ece(), brier_score(), calibration_curve(), empirical_output_interval(), and the ai_calibration() wrapper, with input validation throughout.
• Hallucination: hallucination_rate() (aggregation of adjudicated verdicts), lexical_overlap() (a lexical baseline, deliberately named to avoid implying factual-consistency detection), and citation_accuracy().
• Dashboard: ai_dashboard() combines rescaled module scores into a descriptive overall profile.

Testing

Comprehensive test suite (~93% line coverage), verifying statistical behavior against known-value cases rather than merely checking for non-null output.

Vignettes

• getting-started: a module-by-module tour with runnable examples.
• psychometric-core: a deeper treatment of the generalizability-theory workflow (G-studies and D-studies).

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.