Package {statcheck}

Title:	Extract Statistics from Articles and Recompute P-Values
Version:	1.6.1
Date:	2026-10-23
Description:	A "spellchecker" for statistics. It checks whether your p-values match their accompanying test statistic and degrees of freedom. statcheck searches for null-hypothesis significance tests (NHST) in APA style (e.g., t(28) = 2.2, p < .05), using pattern-matching ("regular expressions"). It recalculates the p-value using the reported test statistic and degrees of freedom. If the reported and computed p-values don't match, 'statcheck' will flag the result as an error. If the reported p-value is statistically significant and the computed one is not, or vice versa, the result will be flagged as a decision error. You can use 'statcheck' directly on a string of text, but you can also scan a PDF or HTML file, or even a folder of PDF and/or HTML files. PDF files are converted to text using the R package 'pdftools', so no additional software installation is necessary. You can find more information on what 'statcheck' can and cannot do and a point-and-click web interface to scan PDF or HTML or DOCX articles on https://statcheck.io.
License:	GPL-3
Depends:	R (≥ 3.6.0)
Imports:	DT, ggplot2, graphics, pdftools, plyr, rlang, rmarkdown, stats, stringi, stringr, tcltk, utils
Suggests:	testthat
ByteCompile:	yes
Encoding:	UTF-8
LazyLoad:	yes
URL:	https://github.com/MicheleNuijten/statcheck
BugReports:	https://github.com/MicheleNuijten/statcheck/issues
Config/roxygen2/version:	8.1.1
NeedsCompilation:	no
Packaged:	2026-10-26 08:55:58 UTC; mnuijten
Author:	Michele B. Nuijten [aut, cre], Sacha Epskamp [aut], Willem Sleegers [ctb], Edoardo Costantini [ctb], Paul van der Laken [ctb], Sean Rife [ctb], John Sakaluk [ctb], Chris Hartgerink [ctb], Steve Haroz [ctb]
Maintainer:	Michele B. Nuijten <m.b.nuijten@tilburguniversity.edu>
Repository:	CRAN
Date/Publication:	2026-10-26 13:00:02 UTC

statcheck: Extract statistics from articles and recompute p-values

Description

The package statcheck can extract Null Hypothesis Significance Test (NHST) results from articles (or plain text) and recomputes p-values to check whether a reported NHST result is internally consistent or not.

Details

statcheck can be used for multiple purposes, including:

• Self-checks: you can use statcheck to make sure your manuscript doesn't contain copy-paste errors or other inconsistencies before you submit it to a journal.

• Peer review: editors and reviewers can use statcheck to check submitted manuscripts for statistical inconsistencies. They can ask authors for a correction or clarification before publishing a manuscript.

• Research: statcheck can be used to automatically extract statistical test results from articles that can then be analyzed. You can for instance investigate whether you can predict statistical inconsistencies (see e.g., Nuijten et al., 2017, doi:10.1525/collabra.102), or use it to analyze p-value distributions (see e.g., Hartgerink et al., 2016, doi:10.7717/peerj.1935).

Using statcheck on a string of text

The most basic usage of statcheck is to directly extract NHST results and check for inconsistencies in a string of text. See statcheck for details and an example of how to do this.

Using statcheck on an article

Another option is to run statcheck on an article (PDF or HTML). This is a useful option if you want to check for inconsistencies in a single article (e.g., as a final check before you submit it). Depending on whether you want to check an article in HTML or PDF, you can use checkHTML or checkPDF, respectively. Note: it is recommended to check articles in HTML, as converting PDF files to plain text sometimes results in some conversion errors.

Using statcheck on a folder of articles

Finally, it is possible to run statcheck on an entire folder of articles. This is often useful for meta-research. To do so, you can use checkPDFdir to check all PDF articles in a folder, checkHTMLdir to check all PDF articles in a folder, and checkdir to check both PDF and HTML articles in a folder.

Accuracy of the algorithm in detecting inconsistencies

It is important to note that statcheck is not perfect. Its performance in detecting NHST results depends on the type-setting and reporting style of an article and can vary widely. However, statcheck performs well in classifying the retrieved statistics in different consistency categories. We found that statcheck’s sensitivity (true positive rate) and specificity (true negative rate) were high: between 85.3 respectively, depending on the assumptions and settings. The overall accuracy of statcheck ranged from 96.2 can be found in Nuijten et al., 2017, doi:10.31234/osf.io/tcxaj.

Manual

Details on what statcheck can and cannot do can be found in the online manual.

Web app

statcheck is also available as a free, online web app at https://statcheck.io.

Author(s)

Maintainer: Michele B. Nuijten m.b.nuijten@tilburguniversity.edu (ORCID)

Authors:

• Michele B. Nuijten m.b.nuijten@tilburguniversity.edu (ORCID)

• Sacha Epskamp mail@sachaepskamp.com (ORCID)

Other contributors:

• Willem Sleegers (ORCID) [contributor]

• Edoardo Costantini [contributor]

• Paul van der Laken (ORCID) [contributor]

• Sean Rife (ORCID) [contributor]

• John Sakaluk (ORCID) [contributor]

• Chris Hartgerink (ORCID) [contributor]

• Steve Haroz (ORCID) [contributor]

References

Hartgerink, C. H. J., Van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., Van Assen, M. A. L. M. (2016). Distributions of p-values smaller than .05 in psychology: What is going on? PeerJ, 4, e1935. doi:10.7717/peerj.1935

Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2017). Journal data sharing policies and statistical reporting inconsistencies in psychology. Collabra: Psychology, 3(1), 1-22. doi:10.1525/collabra.102

Nuijten, M. B., Van Assen, M. A. L. M., Hartgerink, C. H. J., Epskamp, S., & Wicherts, J. M. (2017). The validity of the tool "statcheck" in discovering statistical reporting inconsistencies. Preprint retrieved from doi:10.31234/osf.io/tcxaj.

See Also

Useful links:

• https://github.com/MicheleNuijten/statcheck

• Report bugs at https://github.com/MicheleNuijten/statcheck/issues

Extract statistics from PDF/HTML articles and recalculate p-values

Description

These functions search for NHST results in PDF and/or HTML articles and send the extracted statistics to statcheck.

Usage

checkHTML(files, ...)

checkPDF(files, ...)

Arguments

Details

On macOS, file selection dialogs used in related functions (e.g., via tk_choose.files) can be unreliable. If you encounter problems, it is recommended to type the full file path manually instead of relying on the dialog.

Also, make sure that the files you provide are actual HTML or PDF documents. Passing other file types may result in errors.

Value

A statcheck data frame with the extracted statistics. See statcheck for details.

Note

This package has been primarily tested on Windows. Mac users may experience issues with multi-file selection dialogs.

Examples

## Not run: 
# Check a single HTML file
checkHTML("path/to/file.html")

# Check multiple PDF files
checkPDF(c("file1.pdf", "file2.pdf"))

# If dialogs fail on macOS, type the full file path manually:
checkHTML("/Users/username/Documents/file.html")

## End(Not run)

Extract statistics from folders with PDF/HTML articles and recalculate p-values

Description

These functions search for NHST results in all PDF and/or HTML articles in a certain folder and send the extracted statistics to statcheck.

Usage

checkHTMLdir(dir, subdir = TRUE, extension = TRUE, ...)

checkPDFdir(dir, subdir = TRUE, ...)

checkdir(dir, subdir = TRUE, ...)

Arguments

Details

On macOS, the file selection dialog can sometimes be unreliable or fail to open when using 'tk_choose.files()'. If you encounter this issue, it is recommended to manually type the filename or path instead of using the dialog.

Value

A statcheck data frame with the extracted statistics. See statcheck for details.

Note

This package was primarily tested on Windows. Mac users may experience issues with multi-file selection dialogs.

Examples

## Not run: 
# Scan a folder of PDFs
checkPDFdir("path/to/pdfs")

# Scan a folder of HTML articles
checkHTMLdir("path/to/html")

# Scan a folder of mixed PDFs/HTML
checkdir("path/to/articles")

## End(Not run)

Identify specific points in a statcheck plot.

Description

With this function you can simply point and click on the datapoints in the plot to see the corresponding statcheck details, such as the paper from which the data came and the exact statistical results.

Usage

## S3 method for class 'statcheck'
identify(x, alpha = 0.05, ...)

Arguments

Examples

## Not run: 

# First we need a statcheck object
# Here, we create one by running statcheck on some raw text

txt <- "This test is consistent t(28) = 0.2, p = .84, but this one is 
inconsistent: F(2, 28) = 4.2, p = .01. This final test is even a
gross/decision inconsistency: z = 1.23, p = .03"

result <- statcheck(txt)

# Now, we can run identify.statcheck(), or shorter, simply identify():
identify(result)

# Further instructions:
# click on one or multiple points of interest
# press Esc
# a dataframe with information on the selected points will appear


## End(Not run)

Plot method for statcheck

Description

Function for plotting of statcheck objects. Reported p values are plotted against recalculated p values, which allows the user to easily spot if articles contain miscalculations of statistical results.

Usage

## S3 method for class 'statcheck'
plot(x, alpha = 0.05, APAstyle = TRUE, group = NULL, ...)

Arguments

Details

If APAstyle = FALSE, inconsistencies between the reported and the recalculated p value are indicated with an orange dot. Recalculations of the p value that render a previously non significant result (p >= .5) as significant (p < .05), and vice versa, are considered decision errors, and are indicated with a red dot. Exactly reported p values (i.e. p = ..., as opposed to p < ... or p > ...) are indicated with a diamond.

Acknowledgements

Many thanks to John Sakaluk who adapted the plot code to create graphs in APA style.

See Also

statcheck

Examples

# First we need a statcheck object
# Here, we create one by running statcheck on some raw text

txt <- "This test is consistent t(28) = 0.2, p = .84, but this one is 
inconsistent: F(2, 28) = 4.2, p = .01. This final test is even a
gross/decision inconsistency: z = 1.23, p = .03"

result <- statcheck(txt)

# We can then plot the statcheck object 'result' by simply calling plot() on 
# "result". R will know what kind of plot to make, because "result" is of 
# class "statcheck"
plot(result)

Extract statistics and recompute p-values

Description

statcheck extracts Null Hypothesis Significance (NHST) results from strings and returns the extracted values, reported p-values and recomputed p-values.

Usage

statcheck(
  texts,
  stat = c("t", "F", "cor", "chisq", "Z", "Q"),
  OneTailedTests = FALSE,
  alpha = 0.05,
  pEqualAlphaSig = TRUE,
  pZeroError = TRUE,
  OneTailedTxt = FALSE,
  AllPValues = FALSE,
  messages = TRUE
)

Arguments

Details

statcheck roughly works in three steps.

1. Scan text for statistical results

statcheck uses regular expressions to recognizes statistical results from t-tests, F-tests, \chi2-tests, Z-tests, Q-tests, and correlations. statcheck can only recognize these results if the results are reported exactly according to the APA guidelines:

• t(df) = value, p = value

• F(df1, df2) = value, p = value

• r(df) = value, p = value

• \chi2 (df, N = value) = value, p = value (N is optional)

• Z = value, p = value

• Q(df) = value, p = value (statcheck can distinguish between Q, Qw / Q-within, and Qb / Q-between)

statcheck takes into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.

2. Recompute p-value

statcheck uses the reported test statistic and degrees of freedom to recompute the p-value. By default, the recomputed p-value is two-sided

3. Compare reported and recomputed p-value

This comparison takes into account how the results were reported, e.g., p < .05 is treated differently than p = .05. Incongruent p values are marked as an error. If the reported result is significant and the recomputed result is not, or vice versa, the result is marked as a decision_error.

Correct rounding is taken into account. For instance, a reported t-value of 2.35 could correspond to an actual value of 2.345 to 2.354 with a range of p-values that can slightly deviate from the recomputed p-value. statcheck will not count cases like this as errors.

Note that when statcheck flags an error or decision_error, it implicitly assumes that the p-value is the inconsistent value, but it could just as well be the case that the test statistic or degrees of freedom contain a reporting error. statcheck merely detects wether a set of numbers is consistent with each other.

Value

A data frame containing for each extracted statistic:

source

Name of the file of which the statistic is extracted

test_type

Character indicating the statistic that is extracted

df1

First degree of freedom (if applicable)

df2

Second degree of freedom

test_comp

Reported comparison of the test statistic, when importing from pdf this will often not be converted properly

test_value

Reported value of the statistic

p_comp

Reported comparison, when importing from pdf this might not be converted properly

reported_p

The reported p-value, or NA if the reported value was n.s.

computed_p

The recomputed p-value

raw

Raw string of the statistical reference that is extracted

error

The computed p value is not congruent with the reported p-value

decision_error

The reported result is significant whereas the recomputed result is not, or vice versa.

one_tailed_in_txt

Logical. Does the text contain the string "sided", "tailed", and/or "directional"?

apa_factor

What proportion of all detected p-values was part of a fully APA reported result?

See Also

For more details, see the online manual.

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
statcheck(txt)

Generate HTML report for statcheck output

Description

This function uses R Markdown to generate a nicely formatted HTML report of statcheck output.

Usage

statcheckReport(statcheckOutput, outputFile, outputDir = getwd())

Arguments

Details

This function temporarily saves the inserted statcheck output as an .RData file in the "output" folder in the statcheck package directory. This file is then called by the .Rmd template that is saved in the folder "rmd", also in the statcheck package directory. After the HTML report is generated, the .RData file is removed again.

Value

An HTML report, saved in the directory specified in the argument "output_dir".

Examples

## Not run: 

# first generate statcheck output, for instance by using the statcheck() 
function

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)

# next, use this output to generate a formatted HTML report of the results
statcheckReport(stat, outputFile = "statcheckHTMLReport", 
                outputDir = "C:/mydocuments/results")

# you can now find your HTML report in the folder 
# "C:/mydocuments/results" under the name "statcheckHTMLReport.html".


## End(Not run)

Summary method for statcheck

Description

Gives the summaries for a statcheck object.

Usage

## S3 method for class 'statcheck'
summary(object, ...)

Arguments

Value

A data frame containing for each source of statistics:

source

Name of the file/origin of which the statistics are extracted

nr_p_values

The number of extracted reported p values per article

nr_errors

The number of errors per article

nr_decision_errors

The number of decision errors per article

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)
summary(stat)

Trimming method for statcheck output

Description

Returns a subset of columns of a statcheck object.

Usage

trim(object, ...)

Arguments

Value

A data frame containing for each source of statistics:

source

Name of the file/origin of which the statistics are extracted

raw

Raw string of the statistical reference that is extracted

computed_p

The recomputed p-value

error

The computed p value is not congruent with the reported p-value

decision_error

The reported result is significant whereas the recomputed result is not, or vice versa.

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)
trim(stat)