Package {ebrahim.gof}

Type:	Package
Title:	Goodness-of-Fit and Calibration Tests for Logistic Regression
Version:	2.6.0
Date:	2026-08-26
Maintainer:	Ebrahim Khaled Ebrahim <ebrahimkhaled@alexu.edu.eg>
Description:	Provides a unified battery of goodness-of-fit and calibration tests for binary logistic regression, runnable in a single call via 'run.all.gof()'. Around twenty-five tests spanning five decades of literature are aggregated and grouped by the departure each is built to detect: global and standardized statistics, partition tests, directed and covariate-space tests, smoothing and resampling tests, and calibration tests. Each is obtained from its own package where installed and attributed to its authors. The package also implements the author's own procedures for sparse data, where the Hosmer-Lemeshow test loses power: the omnibus Ebrahim-Farrington test, the directed 'EDGE' test, 'DeepGOF-1' (a pretrained convolutional statistic whose level comes from the analyst's own parametric bootstrap rather than from the network), a Cauchy-combination ensemble, 'legoft()' (a pretrained combination whose weights are fixed offline and ship frozen, so two analysts obtain the same p-value), and 'shrink.gof()' for penalized (ridge) logistic regression, where shrinkage biases the fitted probabilities and invalidates the usual chi-squared references. For more details see Hosmer (1980) <doi:10.1080/03610928008827941> and Farrington (1996) <doi:10.1111/j.2517-6161.1996.tb02086.x>.
License:	GPL-3
URL:	https://github.com/ebrahimkhaled/ebrahim.gof
BugReports:	https://github.com/ebrahimkhaled/ebrahim.gof/issues
Depends:	R (≥ 3.5.0)
Imports:	parallel, stats
Suggests:	testthat (≥ 3.0.0), knitr, rmarkdown, ResourceSelection, ggplot2, CompQuadForm, statmod, mgcv, BAGofT, givitiR, callr
Encoding:	UTF-8
LazyData:	true
RoxygenNote:	7.3.2
VignetteBuilder:	knitr
NeedsCompilation:	no
Packaged:	2026-08-26 12:25:31 UTC; ebrah
Author:	Ebrahim Khaled Ebrahim [aut, cre]
Repository:	CRAN
Date/Publication:	2026-08-28 09:50:24 UTC

ebrahim.gof: Goodness-of-Fit and Calibration Tests for Logistic Regression

Description

A unified toolbox of goodness-of-fit and calibration tests for binary logistic regression, callable in a single line via run.all.gof. The package is aimed particularly at sparse data, where the classical Hosmer–Lemeshow test loses power.

The author's own tests

• ef.gof — the omnibus Ebrahim–Farrington (EF) test for binary data with automatic grouping.

• def.gof / edge.gof — the Directed EF (“EDGE”) test, which spends its few degrees of freedom on the smooth calibration-shape directions where structured misfit concentrates.

• def.ensemble.gof — a Cauchy-combination ensemble of the directed bases.

Aggregated tests (for comparison)

run.all.gof also runs, in one call, a wide range of classical and modern tests — Hosmer–Lemeshow, McCullagh, Osius–Rojek, le Cessie–van Houwelingen, Stute–Zhu, the binary-adaptive BAGofT test, and the givitiR calibration test. Each aggregated test is obtained from its own package (where installed) and is attributed to its authors; these are provided for head-to-head comparison, not claimed as original to this package.

Data

gof_demo is a bundled example dataset with a documented, reproducible misfit for illustrating the battery.

Author(s)

Maintainer: Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg (ORCID)

See Also

The vignette vignette("ebrahim-gof-toolbox", package = "ebrahim.gof").

Covariate-Space Directed Ebrahim-Farrington (CDEF) Goodness-of-Fit Test

Description

A directed goodness-of-fit test for binary logistic regression whose direction lives in covariate space (functions of the predictors) rather than in fitted-probability space like def.gof. It projects the standardized residuals onto a covariate-space basis (polynomials and pairwise products, natural splines, or a combination that also includes fitted-probability bends) and calibrates the quadratic form with the Farrington estimation-adjusted projection, exactly as in def.gof. This makes it sensitive to omitted interactions and to local / oscillatory departures that fitted-probability grouping can miss.

Usage

cdef.gof(
  object,
  predicted_probs = NULL,
  X = NULL,
  basis = c("poly", "spline", "combined"),
  method = c("satterthwaite", "imhof")
)

Arguments

Details

Let \tilde r_i=(y_i-\hat p_i)/\sqrt{\hat p_i(1-\hat p_i)} be the standardized residuals and Z a covariate-space basis matrix. The statistic is S=(Z'\tilde r)'(Z'Z)^{-1}(Z'\tilde r), whose null distribution is a weighted sum of \chi^2_1 variables with weights the eigenvalues of (Z'Z)^{-1}Z'\Omega Z, where \Omega=I-V^{1/2}X(X'VX)^{-1}X'V^{1/2} adjusts for estimating \hat\beta. The p-value uses a Satterthwaite scaled-\chi^2 approximation (default) or Imhof's method (CompQuadForm). Rank-deficient bases are reduced automatically.

Value

A one-row data.frame with Test, Basis, Test_Statistic, df, Method, and p_value.

References

Farrington, C. P. (1996). On Assessing Goodness of Fit of Generalized Linear Models to Sparse Data. JRSS-B 58(2), 349-360.

See Also

def.gof, ef.gof, run.all.gof.

Examples

set.seed(1)
n <- 600; x1 <- runif(n, -3, 3); x2 <- rnorm(n)
# truth has an omitted interaction; fit the additive model
y <- rbinom(n, 1, plogis(0.3 + 0.8 * x1 - 0.5 * x2 + 0.4 * x1 * x2))
fit <- glm(y ~ x1 + x2, family = binomial())
cdef.gof(fit)                    # covariate-space directed test (poly basis)
cdef.gof(fit, basis = "spline")  # for local / oscillatory misfit

DeepGOF-1: a pretrained goodness-of-fit test for logistic regression

Description

Tests whether a fitted binomial glm is correctly specified, using a convolutional network that was trained once, offline, on simulated departures and is shipped frozen with this package. The analyst never trains anything: the network reads the model's residual map and the p-value is the rank of the observed score within the analyst's own parametric bootstrap, so the level does not depend on what the network learned.

Usage

deepgof1(fit, B = 199L, K = 6L)

Arguments

Details

The residual map is a K by K grid over the empirical ranks of the two covariates with the largest |\hat\beta_j| \hat\sigma_j; each cell holds a standardized residual sum, approximately standard normal under a correct model. Misfit therefore has a location on the map – an omitted quadratic paints a stripe, an omitted interaction a saddle – which is what the convolutional statistic reads.

The test is a small-sample instrument. Against the classical partition tests it gains most at n of 50 to 200 and the gain decays as n grows; because the grid uses only two covariates, misfit that lives off those axes is harder for it to see than for covariate-space or smoothing tests. See run.all.gof to run it alongside the classical battery.

Value

An object of class "deepgof1": a list with statistic (the observed score), p.value, B, K, axes (the two covariates the grid was built on), boot (the B bootstrap scores) and method.

Reproducibility

A bootstrap refit that fails to converge is scored +Inf, so it counts against rejection – the conservative direction. Set a seed before calling for a reproducible p-value.

References

Ebrahim, E.K. (2026). DeepGOF-1: a pretrained convolutional goodness-of-fit test for logistic regression with a computable consistency certificate.

Besag, J. and Clifford, P. (1989). Generalized Monte Carlo significance tests. Biometrika 76, 633–642. doi:10.1093/biomet/76.4.633

See Also

run.all.gof, ef.gof, legoft

Examples

set.seed(1)
n  <- 150
x1 <- runif(n, -3, 3); x2 <- rnorm(n)
# a model with an omitted quadratic term
y  <- rbinom(n, 1, plogis(0.3 + 0.8 * x1 - 0.5 * x2 + 0.9 * (x1^2 - mean(x1^2))))
fit <- glm(y ~ x1 + x2, family = binomial())
deepgof1(fit, B = 49)   # B = 49 to keep the example fast; use the default in practice

Combine Directed GOF Tests into One Decision (Ensemble)

Description

Combines the three Directed Ebrahim-Farrington (DEF) basis tests ("poly2", "poly3", "stukel") into a single goodness-of-fit decision, so the user does not have to choose a basis. By default the p-values are combined with the Cauchy Combination Test (CCT), which controls the error rate under the strong dependence between tests computed on the same fitted model. The omnibus EF test can optionally be added to the vote.

Usage

def.ensemble.gof(
  object,
  predicted_probs = NULL,
  X = NULL,
  components = c("poly2", "poly3", "stukel"),
  add_ef = FALSE,
  combine = c("cct", "minp", "fisher"),
  G = 10,
  extra_pvalues = NULL
)

Arguments

Details

Because the component tests are computed on the same fit, their p-values are strongly dependent. The CCT (combine = "cct") has an asymptotic standard-Cauchy null whose tail is robust to this dependence, so it needs no calibration. The "minp" (Sidak) and "fisher" rules assume independence and are offered for comparison only; under positive dependence "minp" is conservative and "fisher" is anti-conservative, so they should be calibrated by simulation before use (not done here).

Value

A one-row data.frame with columns Test, Combiner, Components, k, and p_value.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

Liu, Y. and Xie, J. (2020). Cauchy combination test. JASA, 115(529), 393-402.

See Also

def.gof, ef.gof.

Examples

set.seed(1)
n <- 500
x <- runif(n, -3, 3)
y <- rbinom(n, 1, 1 / (1 + exp(-(0.6 * x))))
fit <- glm(y ~ x, family = binomial())
def.ensemble.gof(fit)                 # CCT of the three DEF bases
def.ensemble.gof(fit, add_ef = TRUE)  # add the omnibus EF

Directed Ebrahim-Farrington (DEF) Goodness-of-Fit Test

Description

Performs the Directed Ebrahim-Farrington (DEF) goodness-of-fit test for a fitted binary logistic regression model. DEF concentrates its power on a small set of calibration-curve "shape" directions by projecting the grouped standardized residuals onto a low-dimensional basis and testing the squared length of that projection.

Naming note: this test is published under the name EDGE (Ebrahim Directed Goodness-of-fit Evaluation), and edge.gof is the primary interface going forward. def.gof() is retained, unchanged, as a fully supported legacy name.

Usage

def.gof(
  object,
  predicted_probs = NULL,
  X = NULL,
  G = 10,
  basis = c("poly3", "poly2", "stukel", "ensemble"),
  method = c("satterthwaite", "imhof")
)

Arguments

Details

The observations are sorted by predicted probability and split into G equal-frequency groups; the standardized grouped residual vector r is projected onto a basis matrix Z of smooth shapes, giving S = (Z'r)'(Z'Z)^{-1}(Z'r). Its null distribution is a weighted sum of \chi^2_1 variables with weights equal to the eigenvalues of (Z'Z)^{-1}Z'\Omega Z, where \Omega = I - U(X'WX)^{-1}U' is the estimation-adjusted covariance of the grouped residuals. The p-value uses a Satterthwaite scaled-\chi^2 approximation (default) or Imhof's method (if the CompQuadForm package is installed). Bases: "poly2", "poly3" (default), "stukel"; "ensemble" runs all three and combines them via def.ensemble.gof.

Value

A one-row data.frame with columns Test, Basis, Test_Statistic (the statistic S), df, Method, and p_value. When basis = "ensemble", the return is that of def.ensemble.gof.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

Ebrahim, K. E. and El-Kotory, A. Omnibus versus Directed Goodness-of-Fit Tests for Sparse Data in Binary Logistic Regression (companion paper).

See Also

ef.gof, def.ensemble.gof.

Examples

set.seed(1)
n <- 500
x <- runif(n, -3, 3)
y <- rbinom(n, 1, 1 / (1 + exp(-(0.6 * x))))
fit <- glm(y ~ x, family = binomial())
def.gof(fit)                       # default poly3 basis
def.gof(fit, basis = "stukel")     # tail-shape basis
def.gof(fit, basis = "ensemble")   # combine all three (CCT)

Deployable learned-ensemble GOF test via parametric bootstrap

Description

Turns a pre-trained ensemble meta into a deployable goodness-of-fit test for any fitted model: it scores the model, then calibrates the p-value by a per-dataset parametric bootstrap from the fitted model (so no knowledge of the truth or the data-generating design is required). Validity comes from the bootstrap, independent of how meta was trained.

Usage

deploy.gof(object, meta, B = 99, feature_fn = gof.features)

Arguments

Value

A one-row data.frame with the score, B, and the bootstrap p_value.

See Also

gof.features, cdef.gof.

EDGE: Directed Goodness-of-Fit Test for Binary Logistic Regression

Description

edge.gof() is the primary interface to the EDGE test (Ebrahim Directed Goodness-of-fit Evaluation): a grouped, directed goodness-of-fit test for binary logistic regression under sparse data. EDGE projects the grouped standardized residuals onto a small pre-specified basis of calibration shapes (cubic "poly3" by default) and refers the resulting quadratic form to its closed-form weighted chi-squared null distribution – no refit, no resampling, no tuning.

edge.gof() computes exactly the same statistic as the legacy name def.gof (retained for backward compatibility); the returned Test label is "EDGE".

Usage

edge.gof(
  object,
  predicted_probs = NULL,
  X = NULL,
  G = 10,
  basis = "poly3",
  method = "satterthwaite"
)

Arguments

Value

A one-row data.frame with columns Test ("EDGE"), Basis, Test_Statistic, df, Method, and p_value, as documented in def.gof.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

Ebrahim, E. K. and El-Kotory, A. (2026). EDGE: a closed-form directed goodness-of-fit test for sparse logistic regression. Manuscript.

Ebrahim, E. K. and El-Kotory, A. (2026). A directional Hosmer-Lemeshow goodness-of-fit test for sparse logistic regression. arXiv:2607.15454.

See Also

def.gof (legacy name), ef.gof, def.ensemble.gof, run.all.gof.

Examples

set.seed(1)
x <- runif(500, -3, 3)
y <- rbinom(500, 1, plogis(0.6 * x))
fit <- glm(y ~ x, family = binomial())
edge.gof(fit)                      # default cubic basis, G = 10
edge.gof(fit, basis = "stukel")    # Stukel-shape basis

EDGES: Cauchy-Combination Ensemble of Directed GOF Tests (Alias)

Description

Alias for def.ensemble.gof(); see the EDGES paper. edges.gof() is the brand name (EDGES = the Cauchy-combination ensemble of the EDGE directed bases) used in the manuscript. It takes exactly the same arguments as def.ensemble.gof and returns exactly the same value; the legacy name def.ensemble.gof() is retained unchanged for back-compatibility.

Usage

edges.gof(...)

Arguments

Value

A one-row data.frame with columns Test, Combiner, Components, k, and p_value.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

Liu, Y. and Xie, J. (2020). Cauchy combination test. JASA, 115(529), 393-402.

See Also

def.ensemble.gof (legacy name), edge.gof, def.gof.

Examples

set.seed(1)
x <- runif(500, -3, 3)
y <- rbinom(500, 1, plogis(0.6 * x))
fit <- glm(y ~ x, family = binomial())
edges.gof(fit)                 # identical to def.ensemble.gof(fit)

Ebrahim-Farrington Goodness-of-Fit Test for Logistic Regression

Description

Performs the Ebrahim-Farrington goodness-of-fit test for logistic regression models. This test is particularly effective for binary data and sparse datasets, providing an improved alternative to the traditional Hosmer-Lemeshow test.

Usage

ef.gof(
  y,
  predicted_probs = NULL,
  model = NULL,
  m = NULL,
  G = 10,
  method = c("chisq", "normal")
)

Arguments

Details

The Ebrahim-Farrington test is based on Farrington's (1996) theoretical framework but simplified for practical implementation with binary data. The test uses a modified Pearson chi-square statistic with data-dependent grouping, where observations are grouped by their predicted probabilities.

For binary data (when G is specified), the test automatically groups observations into G groups based on predicted probabilities and applies the simplified Ebrahim-Farrington statistic:

Z_{EF} = \frac{T_{EF} - (G - 2)}{\sqrt{2(G-2)}}

where T_{EF} is the modified Pearson chi-square statistic, and G is the number of groups.

For grouped data (when m is provided), the test applies the original Farrington test with full variance calculations.

Value

A data frame with the following columns:

Note

• For binary data with automatic grouping (G specified): Use the Ebrahim-Farrington test which is computationally efficient and doesn't require the model specification.

• For grouped data (m provided): Use the original Farrington test which requires the fitted model object.

• The test statistic follows a standard normal distribution under the null hypothesis of adequate model fit.

• For binary data with m=1 for all observations and no grouping, the test is not applicable and will return a p-value of 1.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

Farrington, C. P. (1996). On Assessing Goodness of Fit of Generalized Linear Models to Sparse Data. *Journal of the Royal Statistical Society. Series B (Methodological)*, 58(2), 349-360. Ebrahim, K. E. (2025). Goodness-of-Fits Tests and Calibration Machine Learning Algorithms for Logistic Regression Model with Sparse Data. *Master's Thesis*, Alexandria University. Hosmer, D. W., & Lemeshow, S. (1980). A goodness-of-fit test for the multiple logistic regression model. *Communications in Statistics - Theory and Methods*, 9(10), 1043–1069. https://doi.org/10.1080/03610928008827941

See Also

hoslem.test for the Hosmer-Lemeshow test

Examples

# Example 1: Binary data with automatic grouping (Ebrahim-Farrington test)
set.seed(123)
n <- 500
x <- rnorm(n)
linpred <- 0.5 + 1.2 * x
prob <- 1 / (1 + exp(-linpred))
y <- rbinom(n, 1, prob)

# Fit logistic regression
model <- glm(y ~ x, family = binomial())
predicted_probs <- fitted(model)

# Perform Ebrahim-Farrington test with 10 groups
result <- ef.gof(y, predicted_probs, G = 10)
print(result)

# Example 2: Compare with different number of groups
result_4 <- ef.gof(y, predicted_probs, G = 4)
result_20 <- ef.gof(y, predicted_probs, G = 20)

# Example 3: Grouped data (original Farrington test)
# Note: This requires actual grouped data with trials > 1
## Not run: 
# Simulated grouped data
n_groups <- 50
m_trials <- sample(5:20, n_groups, replace = TRUE)
x_grouped <- rnorm(n_groups)
linpred_grouped <- -0.5 + 1.0 * x_grouped
prob_grouped <- 1 / (1 + exp(-linpred_grouped))
y_grouped <- rbinom(n_groups, m_trials, prob_grouped)

# Fit model for grouped data
data_grouped <- data.frame(successes = y_grouped, trials = m_trials, x = x_grouped)
model_grouped <- glm(cbind(successes, trials - successes) ~ x, 
                     data = data_grouped, family = binomial())
predicted_probs_grouped <- fitted(model_grouped)

# Original Farrington test
result_grouped <- ef.gof(y_grouped, predicted_probs_grouped, 
                         model = model_grouped, m = m_trials)
print(result_grouped)

## End(Not run)

Goodness-of-fit evidence features for a fitted model

Description

Builds the evidence vector used by the learned-ensemble goodness-of-fit test: one-sided z-scores \Phi^{-1}(1-p) from a panel of GOF tests plus the covariate-space directed tests. Larger values mean stronger evidence of misfit.

Usage

gof.features(
  object,
  tests = c("HL", "HL-equalwidth", "Pigeon-Heyse", "Tsiatis", "Xie", "EF", "DEF.poly2",
    "DEF.poly3", "DEF.stukel")
)

Arguments

Value

A named numeric vector of evidence features.

See Also

deploy.gof, cdef.gof, run.all.gof.

Synthetic binary outcome data with a smooth calibration misfit

Description

A small, fully synthetic dataset for demonstrating the goodness-of-fit and calibration battery. It was generated reproducibly (see data-raw/make_gof_demo.R) from a logistic data-generating process whose true linear predictor includes a quadratic term in (standardized) age. A model that regresses outcome on age linearly (together with bmi, sex and treatment) is therefore mildly misspecified, through a smooth, low-dimensional calibration distortion. This is the regime in which the directed Ebrahim–Farrington / EDGE test (edge.gof, def.gof) is designed to have more power than classical omnibus tests such as Hosmer–Lemeshow.

Usage

gof_demo

Format

A data frame with 800 rows and 5 variables:

outcome

binary response, 0/1 (event rate about 0.27).

age

continuous covariate, years (range about 20–70). The true model depends on age quadratically.

bmi

continuous covariate, body mass index in kg/m^2.

sex

binary covariate, 0 = female, 1 = male.

treatment

binary covariate, 0 = control, 1 = treated.

Details

The true data-generating linear predictor is

\eta = -0.6 + 0.8 z_a - 0.7 z_a^2 + 0.5 z_b + 0.4\,\mathrm{sex} - 0.3\,\mathrm{treatment},

where z_a = (\mathrm{age} - 45)/14 and z_b = (\mathrm{bmi} - 27)/4, and \Pr(\mathrm{outcome} = 1) = \mathrm{plogis}(\eta).

Source

Simulated; see data-raw/make_gof_demo.R in the package sources.

Examples

data("gof_demo", package = "ebrahim.gof")
fit <- glm(outcome ~ age + bmi + sex + treatment,
           data = gof_demo, family = binomial)
edge.gof(fit)

Grouped-covariate companion to gof_demo (replicated covariate patterns)

Description

A companion dataset to gof_demo built from the same data-generating process and seed discipline (see data-raw/make_gof_demo.R), except that the covariates are coarsened before the linear predictor is computed: age is rounded to 10-year bins (20, 30, ..., 70) and bmi to whole integers. The recorded covariates are therefore exactly the covariates the outcome was generated from, and many observations share a covariate pattern (328 distinct patterns among 800 observations, versus one pattern per observation in gof_demo).

Usage

gof_demo_grouped

Format

A data frame with 800 rows and 5 variables:

outcome

binary response, 0/1 (event rate about 0.27).

age

age in years, rounded to 10-year bins (20, 30, ..., 70). The true model depends on (binned) age quadratically.

bmi

body mass index in kg/m^2, rounded to whole integers.

sex

binary covariate, 0 = female, 1 = male.

treatment

binary covariate, 0 = control, 1 = treated.

Details

Its purpose is to demonstrate the sparse-versus-grouped distinction that run.all.gof surfaces: the battery reports the per-observation ("sparse") and per-covariate-pattern ("grouped") forms of the pattern-sensitive tests (Pearson, deviance, McCullagh) side by side, and on replicated-pattern data such as this the two forms can disagree on the same fitted model. On the all-continuous gof_demo (every observation its own pattern) the two forms coincide – the degenerate case.

The true data-generating linear predictor has the same form as for gof_demo,

\eta = -0.6 + 0.8 z_a - 0.7 z_a^2 + 0.5 z_b + 0.4\,\mathrm{sex} - 0.3\,\mathrm{treatment},

with z_a = (\mathrm{age} - 45)/14 and z_b = (\mathrm{bmi} - 27)/4 computed from the binned age and bmi, and \Pr(\mathrm{outcome} = 1) = \mathrm{plogis}(\eta).

Source

Simulated; see data-raw/make_gof_demo.R in the package sources.

See Also

gof_demo, run.all.gof

Examples

data("gof_demo_grouped", package = "ebrahim.gof")
fit <- glm(outcome ~ age + bmi + sex + treatment,
           data = gof_demo_grouped, family = binomial)
# sparse and grouped forms reported side by side:
run.all.gof(fit, include_slow = FALSE, install = "no")

Install the optional packages used by run.all.gof()

Description

The slow tests in run.all.gof rely on optional packages that live in Suggests (givitiR and callr for the GiViTI calibration test, mgcv for the GAM tests, BAGofT for the adaptive test, and ResourceSelection for the Lai-Liu test). Per CRAN policy the package never installs them on its own; this helper installs the missing ones for you, asking first.

Usage

gof_install_suggests(pkgs = NULL, ask = interactive(), update = FALSE)

Arguments

Value

Invisibly, the character vector of packages that were installed or updated (empty if nothing was needed).

See Also

run.all.gof

Examples

## Not run: 
# install whatever optional packages are missing, after confirming:
gof_install_suggests()

# also update any that are out of date:
gof_install_suggests(update = TRUE)

# just the GiViTI dependencies, no prompt:
gof_install_suggests(c("givitiR", "callr"), ask = FALSE)

## End(Not run)

Pretrained goodness-of-fit test for binary logistic regression

Description

Combines eleven classical and directed goodness-of-fit statistics with weights that were fixed offline and ship frozen, and calibrates the combination by a parametric bootstrap at the fitted parameters. Nothing is retrained when you call it.

Usage

legoft(object, B = 199, seed = NULL, weights = NULL)

Arguments

Details

The p-value is exact in finite samples when the null parameters are known. With the parameters estimated – the case here – the calibration is asymptotic; simulation at n = 500 put the empirical size at 0.037 against a nominal 0.05.

Value

an object of class "legoft": the statistic, its bootstrap p-value, the member p-values, and B_used.

See Also

legoft.localize for which domain of evidence carries the misfit.

Examples

set.seed(1)
x1 <- runif(300, -3, 3); x2 <- rnorm(300)
y  <- rbinom(300, 1, plogis(0.3 + 0.8 * x1 - 0.5 * x2 + 0.25 * x1^2))
fit <- glm(y ~ x1 + x2, family = binomial())
legoft(fit, B = 99, seed = 1)

Localize misspecification with familywise error control

Description

Reports which of two domains of evidence carries the misfit, using closed testing over the domains. A domain is implicated only when every intersection containing it rejects, so the probability of implicating any collection of domains whose pooled evidence is jointly exchangeable with the reference draws' is at most alpha.

Usage

legoft.localize(object, B = 199, seed = NULL, alpha = 0.05)

Arguments

Details

The two domains are defined by what the members can see, not by taxonomy. INDEX holds the seven statistics that read the linear index – the grouping tests, which stratify on fitted risk, and the directed tests, which examine bends in that same index. COV holds the four that read directions orthogonal to it. The calibration and link readings are pooled deliberately: they are not separately identifiable, because fitted risk is a monotone transform of the index.

A verdict says where the evidence lies. It does not say which part of the model to repair: a departure of one kind can move members assigned to the other domain, and an unimplicated domain is not thereby certified correct.

Value

an object of class "legoft_localize": the verdict, the intersection p-values, and the member p-values.

See Also

legoft

Examples

set.seed(1)
x1 <- runif(400, -3, 3); x2 <- rnorm(400)
y  <- rbinom(400, 1, plogis(0.3 + 0.8 * x1 - 0.5 * x2 + 0.4 * x1 * x2))
fit <- glm(y ~ x1 + x2, family = binomial())
legoft.localize(fit, B = 99, seed = 1)

Plot the GiViTI calibration belt from a goodness-of-fit battery

Description

Draws the GiViTI calibration belt stored on a run.all.gof result that was produced with calibration_plot = TRUE. The belt shows the fitted calibration curve with a confidence region against the 45-degree line.

Usage

## S3 method for class 'gof_battery'
plot(x, ...)

Arguments

Value

x, invisibly.

Print a goodness-of-fit battery

Description

Formats the run.all.gof result as a compact, readable table: rows grouped by test family, p-values shown to four decimals (or scientific for very small values, "-" when not available), and a significance flag. The object is still a plain data.frame underneath, so all the raw columns remain available for programmatic use.

Usage

## S3 method for class 'gof_battery'
print(x, ...)

Arguments

Value

x, invisibly.

Run a Battery of Goodness-of-Fit Tests at Once

Description

Runs several goodness-of-fit tests for a binary logistic regression in one call and returns one tidy data.frame, one row per test. Pass a fitted glm to run the whole battery; pass (y, predicted_probs) to run the tests that need only predictions. Each test is wrapped so that a failure of one test never aborts the whole run.

Usage

run.all.gof(
  object,
  predicted_probs = NULL,
  X = NULL,
  tests = "all",
  G = 10,
  include_slow = TRUE,
  parallel = FALSE,
  ncores = NULL,
  calibration_plot = FALSE,
  install = c("ask", "no", "yes"),
  control = list()
)

Arguments

Details

How to read the battery. Every test here answers the same question – does the fitted model describe the data it was fitted to – but they differ in the departure each is built to notice. That is why the panel is more informative than any single p-value: agreement across families is evidence of fit, and disagreement tells you what kind of misfit is present. A test that rejects points at the departure its own construction is sensitive to.

The one thing the panel cannot do is rescue an invalid reference distribution. Under sparse data – almost every covariate pattern unique – the classical chi-square references fail, which is the situation this package was written for.

Global and standardized statistics. These compare observed and fitted responses over the whole sample without grouping.

• Pearson – the sum of squared Pearson residuals. Its chi-square reference assumes many observations per covariate pattern. On sparse data that assumption fails and the test is unreliable in both directions; it is reported for completeness and comparison, not for use.

• Deviance – the likelihood-ratio statistic against the saturated model. It carries the same sparse-data problem as Pearson, and in practice is the more conservative of the two.

• Osius-Rojek – rescues the Pearson statistic by standardizing it with its asymptotic mean and variance computed under increasing sample size rather than increasing cell counts, giving a normal reference that stays valid when patterns are unique. Fast, parameter-free, and a sensible default. It can be anti-conservative at small samples and conservative under a strongly skewed covariate.

• McCullagh – standardizes the Pearson statistic by its exact conditional moments rather than asymptotic ones (computed by the Kuss 2002 algorithm). Typically the most powerful of the global statistics, at a cost that grows steeply with sample size.

• Copas-RSS – the unweighted sum of squares of the raw residuals, referred to its own moments. Weighting each residual equally rather than by its variance makes it comparatively sensitive to misfit in the middle of the risk range.

• Information-Matrix – the White/Orme test. It compares two estimators of the information matrix that agree only if the model is correctly specified, so it is an omnibus check on the whole specification rather than on calibration alone.

Partition tests. These sort observations by fitted risk, group them, and compare observed with expected counts group by group.

• HL – the Hosmer-Lemeshow test, the field's default: G groups cut at percentiles of fitted risk (the "deciles of risk" when G = 10), referred to chi-square on G - 2 degrees of freedom. Note that this reference was established by simulation, not derivation. Familiar and cheap, but modest in power, and its result depends on the grouping.

• HL-equalwidth – the same statistic with groups cut at fixed probability intervals instead of percentiles. When fitted risks are concentrated in a narrow range, intervals can come out empty and the test may not be computable at all, which is why percentile grouping is usually preferred.

• Pigeon-Heyse – a variance-corrected Hosmer-Lemeshow statistic that accounts for the variability of fitted probabilities within each group. The correction is conservative in sparse designs, so a non-rejection carries less weight than the nominal level suggests.

• F-test – deviance residuals compared across the risk groups by a one-way analysis of variance. It is markedly liberal under sparsity and grows more so with sample size; it is included for comparison.

• Lai-Liu-HL – Lai and Liu's procedure for using the Hosmer-Lemeshow test in large samples, where any test eventually rejects. It standardizes power to a reference sample size n0 and so returns no p-value: the statistic is the standardized power and the accept/reject decision appears in the Note column. Tune with control = list("Lai-Liu-HL" = list(n0 = ..., k = ...)).

Directed tests. Rather than asking whether anything is wrong, these ask whether a particular shape of departure is present, which buys power when the guess is right.

• EF, EF-normal – the omnibus Ebrahim-Farrington test, with the chi-square and normal references respectively. Built for sparse data, where the classical grouped statistics lose their reference.

• DEF.poly2, DEF.poly3, DEF.stukel – the directed forms, each aiming the test at a smooth departure in the shape of the calibration curve: a quadratic or cubic drift in the linear predictor, or Stukel's asymmetry-and-tail family. Powerful when the misfit resembles the chosen basis, weaker when it does not.

• Stukel – a two-degree-of-freedom score test against Stukel's generalized logistic link, which nests the logit and lets the two tails bend independently. It is aimed squarely at link misspecification. Note that the combined two-parameter form does not always hold its nominal level in sparse designs; the one-sided components are better behaved.

Covariate-space tests. These partition the covariates themselves rather than the fitted risk, so they can see structure that risk-ordering averages away – an omitted interaction, for instance, need not disturb the marginal calibration at all.

• Tsiatis – a score test that adds indicator variables for regions of the covariate space and asks whether they improve the fit.

• Xie – clusters the covariate space and compares observed with expected counts per cluster, using the corrected degrees of freedom G - k/2 - 1 with k the number of predictors.

• Pulkstenis-Robinson – crosses a categorical covariate with risk groups, so it needs at least one categorical predictor. The package auto-detects one (any factor, character or logical, or a numeric with few distinct values, controlled by getOption("ebrahim.gof.pr.maxlev", 6)) and returns NA with a note when there is none.

Smoothing tests. These replace grouping with a smoother, so nothing is lost to an arbitrary choice of bin edges.

• le-Cessie – the le Cessie-van Houwelingen score test, which smooths the residuals over the covariate space with a kernel and asks whether the smoothed surface is further from zero than chance allows. Sensitive to local structure that omnibus statistics average away, and one of the most expensive tests here, with cost growing sharply in the sample size.

• HL-GAM, PR-GAM, Xie-GAM – variants that fit a deliberately overfitted generalized additive model and use it to define the grouping, letting the data rather than the analyst choose where the boundaries fall. These need mgcv.

Resampling tests. When a statistic has no usable closed-form reference, these build one by simulation.

• Stute-Zhu – a cumulative-residual test: residuals are accumulated along the fitted linear predictor and the largest excursion of that path is compared with a parametric bootstrap. It needs no binning and no bandwidth, and in practice it is the best-behaved test in the battery on size. Set the number of resamples with control = list("Stute-Zhu" = list(B = ...)).

• BAGofT – the binary adaptive test, which splits the data, uses one part to learn a partition that separates fitted from observed, and tests on the other. Its behaviour depends strongly on how many splits and resamples it is given; set them with control = list(BAGofT = list(nsim = ...)) and be aware that the published default is far more expensive than a single split. Needs the BAGofT package.

Calibration tests. These come from clinical prediction, and ask directly whether predicted risks match observed frequencies.

• GiViTI, GiViTI-external – the GiViTI polynomial calibration test, which fits a polynomial of the fitted risk and tests whether it departs from the identity, under the internal and external development assumptions respectively. The two are not interchangeable: on data used to fit the model the internal form is the appropriate one. It also produces the calibration belt, which shows where on the risk scale a model drifts; see calibration_plot. Wraps givitiR in an isolated callr subprocess, so a failure inside its compiled dependencies returns NA instead of ending your session. Select with control = list(GiViTI = list(devel = "internal")).

• eHL – an e-value form of the Hosmer-Lemeshow test, reported as p = \min(1, 1/e). E-values are safe under optional stopping, which conventional p-values are not, but the conversion shown here is conservative.

Combinations. Rather than choosing one test, these pool several.

• Ensemble.Vote(3DEF) and Ensemble.Univ(3DEF+EF) – Cauchy combinations of the directed tests, and of those plus the omnibus EF. The Cauchy combination is valid without knowing how the members correlate, which is what makes pooling dependent tests possible at all. The point is to avoid having to guess the departure in advance, at the cost of being slightly less powerful than the single best member would have been.

• See also legoft, a pretrained combination whose weights are fixed offline and ship frozen, so two analysts running it on the same data obtain the same p-value.

Implementation notes. Tsiatis and Xie cluster the covariate space with k-means using a fixed internal seed, so results are reproducible and your own random stream is left untouched. Every bundled test reproduces the implementation used in the original simulation study: Osius-Rojek and Stukel follow LogisticDx's gof.glm (Stukel via statmod::glm.scoretest when statmod is installed), Copas-RSS follows the rms gof residual, and HL follows ResourceSelection::hoslem.test.

For goodness of fit after penalized fitting, where none of the above references are valid because the coefficients are shrunk, see shrink.gof.

Value

A data.frame (of class gof_battery) with columns Test, Family, Statistic, df, p_value, and Note, one row per test. A dedicated print method shows the rows grouped by family with formatted p-values and significance flags; the underlying columns remain available for programmatic use.

Note

Grouped vs sparse forms. Pearson, Deviance and McCullagh are reported in two forms: the default (sparse / one-trial) form and a "(grouped)" form computed on the distinct covariate patterns (each a Binomial(m_g, P_g)). The two are identical when every covariate pattern is unique (fully sparse data, as in the simulation) and differ only when patterns repeat (m_g > 1). To avoid clutter, the "(grouped)" row is shown only when it actually differs from the sparse form (i.e., when some pattern repeats); on fully sparse data it is a duplicate and is omitted. Osius-Rojek is always computed on covariate patterns, matching its classical (LogisticDx) definition.

Farrington vs EF. The original Farrington (1996) test is a grouped (covariate-pattern) test. The Ebrahim-Farrington (EF) test is its sparse-data counterpart: it does not group by covariate pattern but forms G data-dependent bins of the predicted risk, so it applies directly to fully sparse data. Use EF for sparse binary data; the grouped Farrington form is appropriate only when covariate patterns repeat.

Reproducibility of the parallel path. With parallel = TRUE the cluster's random-number streams are initialized with clusterSetRNGStream, seeded deterministically from the session's current RNG state. Two runs from the same set.seed state (and the same ncores) therefore give identical bootstrap p-values. Note that the parallel L'Ecuyer-CMRG streams necessarily differ from the serial RNG stream, so parallel = TRUE results differ (within Monte-Carlo error) from parallel = FALSE results at the same seed; this is standard and both are valid. Results also depend on ncores, because the replicates are split across workers.

Author(s)

Ebrahim Khaled Ebrahim ebrahimkhaled@alexu.edu.eg

References

The aggregated tests are due to their original authors; they are provided here for comparison and credited as follows.

Farrington CP (1996). "On Assessing Goodness of Fit of Generalized Linear Models to Sparse Data." Journal of the Royal Statistical Society B, 58(2), 349–360. doi:10.1111/j.2517-6161.1996.tb02086.x

Hosmer DW, Lemeshow S (1980). "Goodness of Fit Tests for the Multiple Logistic Regression Model." Communications in Statistics – Theory and Methods, 9(10), 1043–1069. doi:10.1080/03610928008827941

McCullagh P (1985). "On the Asymptotic Distribution of Pearson's Statistic in Linear Exponential Family Models." International Statistical Review, 53(1), 61–67. doi:10.2307/1402880

Osius G, Rojek D (1992). "Normal Goodness-of-Fit Tests for Multinomial Models with Large Degrees of Freedom." Journal of the American Statistical Association, 87(420), 1145–1152. doi:10.1080/01621459.1992.10476271

le Cessie S, van Houwelingen JC (1991). "A Goodness-of-Fit Test for Binary Regression Models, Based on Smoothing Methods." Biometrics, 47(4), 1267–1282. doi:10.2307/2532385

Stukel TA (1988). "Generalized Logistic Models." Journal of the American Statistical Association, 83(402), 426–431. doi:10.1080/01621459.1988.10478613

Stute W, Zhu LX (2002). "Model Checks for Generalized Linear Models." Scandinavian Journal of Statistics, 29(3), 535–545. doi:10.1111/1467-9469.00304

Tsiatis AA (1980). "A Note on a Goodness-of-Fit Test for the Logistic Regression Model." Biometrika, 67(1), 250–251. doi:10.1093/biomet/67.1.250

Xie XJ, Pendergast J, Clarke W (2008). "Increasing the Power: A Practical Approach to Goodness-of-Fit Test for Logistic Regression Models with Continuous Predictors." Computational Statistics & Data Analysis, 52(5), 2703–2713. doi:10.1016/j.csda.2007.09.027

Pulkstenis E, Robinson TJ (2002). "Two Goodness-of-Fit Tests for Logistic Regression Models with Continuous Covariates." Statistics in Medicine, 21(1), 79–93. doi:10.1002/sim.943

Nattino G, Finazzi S, Bertolini G (2014). "A New Calibration Test and a Reappraisal of the Calibration Belt for the Assessment of Prediction Models Based on Dichotomous Outcomes." Statistics in Medicine, 33(14), 2390–2407. doi:10.1002/sim.6100

Zhang J, Ding J, Yang Y (2021). "Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning." Journal of the American Statistical Association. doi:10.1080/01621459.2021.1979010

Pigeon JG, Heyse JF (1999). "An Improved Goodness of Fit Statistic for Probability Prediction Models." Biometrical Journal, 41(1), 71–82. doi:10.1002/(SICI)1521-4036(199903)41:1<71::AID-BIMJ71>3.0.CO;2-O

Copas JB (1989). "Unweighted Sum of Squares Test for Proportions." Journal of the Royal Statistical Society C, 38(1), 71–80. doi:10.2307/2347682

White H (1982). "Maximum Likelihood Estimation of Misspecified Models." Econometrica, 50(1), 1–25. doi:10.2307/1912526

Orme C (1988). "The Calculation of the Information Matrix Test for Binary Data Models." The Manchester School, 56(4), 370–376. doi:10.1111/j.1467-9957.1988.tb01339.x

Kuss O (2002). "Global Goodness-of-Fit Tests in Logistic Regression with Sparse Data." Statistics in Medicine, 21(24), 3789–3801. doi:10.1002/sim.1421

Lai X, Liu L (2018). "A Simple Test Procedure in Standardizing the Power of Hosmer-Lemeshow Test in Large Data Sets." Journal of Statistical Computation and Simulation, 88(13), 2463–2472. doi:10.1080/00949655.2018.1467912

Nattino G, Finazzi S, Bertolini G (2014). "A New Calibration Test and a Reappraisal of the Calibration Belt for the Assessment of Prediction Models Based on Dichotomous Outcomes." Statistics in Medicine, 33(14), 2390–2407. doi:10.1002/sim.6100

Zhang J, Ding J, Yang Y (2021). "Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning." Journal of the American Statistical Association, 118(541), 194–206. doi:10.1080/01621459.2021.1979010

Liu Y, Xie J (2020). "Cauchy Combination Test: A Powerful Test with Analytic p-Value Calculation under Arbitrary Dependency Structures." Journal of the American Statistical Association, 115(529), 393–402. doi:10.1080/01621459.2018.1554485

Hosmer DW, Hosmer T, le Cessie S, Lemeshow S (1997). "A Comparison of Goodness-of-Fit Tests for the Logistic Regression Model." Statistics in Medicine, 16(9), 965–980. doi:10.1002/(sici)1097-0258(19970515)16:9<965::aid-sim509>3.0.co;2-o

The methods introduced by this package, and the studies that evaluate them, are reported in the following. Reproduction materials for each are archived and citable.

Ebrahim EK, El-Kotory A (2026). "A Directional Hosmer-Lemeshow Goodness-of-Fit Test for Sparse Logistic Regression." arXiv:2607.15454 [stat.ME]. doi:10.48550/arXiv.2607.15454

Ebrahim EK, El-Kotory A (2026). "Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data." arXiv:2607.16344 [stat.ME]. doi:10.48550/arXiv.2607.16344 Reproduction materials: doi:10.5281/zenodo.21286171

Ebrahim EK (2026). "Goodness-of-Fit Tests and Calibration Machine-Learning Algorithms for Logistic Regression with Sparse Data." M.Sc. thesis, Alexandria University. arXiv:2608.11140 [stat.ME]. doi:10.48550/arXiv.2608.11140

Ebrahim EK (2026). "EDGE: a directed goodness-of-fit test for sparse logistic regression." Reproduction materials. doi:10.5281/zenodo.21247541

Ebrahim EK (2026). "EDGES: A Selection-Free Ensemble Goodness-of-Fit Test." Reproduction materials. doi:10.5281/zenodo.21320865

Ebrahim EK (2026). "Detection Subspaces: A Theory of Goodness-of-Fit Tests." Reproduction materials. doi:10.5281/zenodo.21687498

Ebrahim EK (2026). "Shrinkage Invalidates the Hosmer-Lemeshow Test." Reproduction materials for shrink.gof. doi:10.5281/zenodo.21900114

See Also

ef.gof, def.gof, def.ensemble.gof, gof_install_suggests.

Examples

set.seed(1)
n <- 500
x <- runif(n, -3, 3)
y <- rbinom(n, 1, 1 / (1 + exp(-(0.6 * x))))
fit <- glm(y ~ x, family = binomial())

## The fast tests. Every covariate pattern here is unique, so this is the
## sparse case the package is written for.
res <- run.all.gof(fit, include_slow = FALSE)
res

## The return value is a plain data.frame, so the panel can be read
## programmatically as well as printed.
res[res$p_value < 0.05, c("Test", "Family", "p_value")]   # what rejected
table(res$Family)                                        # coverage by family

## A correctly specified model: the panel should mostly agree, and any
## isolated rejection is the false positive you expect at the 5 percent level.
mean(res$p_value < 0.05, na.rm = TRUE)

## Now a model that is genuinely wrong -- the quadratic term is omitted.
y2  <- rbinom(n, 1, 1 / (1 + exp(-(0.6 * x + 0.5 * x^2))))
bad <- glm(y2 ~ x, family = binomial())
run.all.gof(bad, include_slow = FALSE)

## Pick specific tests, for instance one per family, which is the pairing the
## package recommends over relying on any single statistic.
run.all.gof(fit, tests = c("McCullagh", "HL", "Stukel", "Tsiatis"))

## Grouping tests take the number of groups; the choice is a convention
## rather than a derived optimum, so it is worth varying.
for (g in c(5, 10, 20))
  print(run.all.gof(fit, tests = "HL", G = g))


## The full battery (include_slow = TRUE by default). The slow tests need the
## suggested packages mgcv, BAGofT, givitiR and callr; in an interactive
## session run.all.gof() offers to install any that are missing
## (install = "ask"). See also gof_install_suggests().
run.all.gof(fit, install = "no", control = list("Stute-Zhu" = list(B = 50)))

## The GiViTI calibration belt shows WHERE on the risk scale a model drifts,
## which a single p-value cannot.
res2 <- run.all.gof(fit, tests = c("McCullagh", "GiViTI"),
                    calibration_plot = TRUE)
plot(res2)   # redraw the stored belt

## Run the bootstrap loops on a PSOCK cluster. Seeds are handled internally,
## so a parallel run reproduces a serial one.
set.seed(1)
run.all.gof(fit, tests = "Stute-Zhu", parallel = TRUE, ncores = 2,
            control = list("Stute-Zhu" = list(B = 50)))

The shrinkage-corrected Hosmer-Lemeshow test

Description

The shrinkage-corrected Hosmer-Lemeshow test

Usage

shrink.gof(
  X,
  y,
  lambda,
  G = 10,
  basis = c("edge", "decile"),
  B = 999,
  seed = NULL,
  penalize = NULL,
  uncorrected = FALSE
)

Arguments

Details

The correction subtracts the estimated shrinkage non-centrality and prepivots against \pi(\tilde\beta) rather than \pi(\hat\beta) (Beran prepivoting), so the reference world matches the null being tested.

Reference implementation for: Ebrahim, E. K. (2026), "Shrinkage invalidates the Hosmer–Lemeshow test: goodness of fit for penalized logistic regression, with an application to glaucoma diagnosis."

Depends only on base R and stats, so results do not move with package versions.

Value

a list with the statistic and p-value for each basis requested. Goodness of fit for penalized (ridge) logistic regression

The Hosmer–Lemeshow test is not valid when the coefficients are shrunk: penalization biases the fitted probabilities, the grouped residuals acquire a non-centrality, and the usual chi-squared reference is wrong. shrink.gof() removes that non-centrality and refers the corrected statistic to a bootstrap built from the debiased generator.

a list with the corrected statistics and their bootstrap p-values on each requested basis.

See Also

run.all.gof for the unpenalized battery.

Examples

set.seed(1)
X <- matrix(rnorm(400 * 5), 400, 5)
y <- rbinom(400, 1, plogis(0.3 + X %*% c(0.8, -0.5, 0.3, 0, 0)))
shrink.gof(X, y, lambda = 40, basis = "decile", B = 99, seed = 1)