The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Calculating Confidence Intervals and P-values for Various ICCs

Kilem L. Gwet, Ph.D.

2019-09-16

Abstract

irrICC is an R package that provides several functions for calculating various Intraclass Correlation Coefficients (ICC). This package follows closely the general framework of inter-rater and intra-rater reliability presented by Gwet (2014).

In this document, I like to show you how to obtain the confidence interval and the p-value associated with a particular Intraclass Correlation Coefficient (ICC) your previously obtained. To learn how to obtain the various ICCs implemented in this package, please refer to the User Guide.

Note that the package scales is used in this document for formating some numbers. But it not needed to run the irrICC package.

ICC under Model 1A

The following code gives us the inter-rater reliability coefficient of 0.252, the associated 95% confidence interval of (0.015,0.784). As for the p-value, the function pval.ICC1a() produces several p-values associated with various values of the null parameters rho.zero given by (0,0.1,0.3,0.5,0.7,0.9). Typically, researchers will calculate the p-value associated with 0 null value to test for statistical significance.
You can provide your own null values using the function pvals.ICC1a() whose default null value is 0. It’s why pvals.ICC1a(iccdata1) yields 0, 0.015.

  icc1a.fn(iccdata1)
#>      sig2s    sig2e     icc1a n r max.rep min.rep Mtot ov.mean
#> 1 1.761312 5.225529 0.2520899 5 4       3       1   40     5.2
  ci.ICC1a(iccdata1)
#>          lcb       ucb
#> 1 0.01495456 0.7841262
  pval.ICC1a(iccdata1)
#>   rho.zero       pval
#> 1      0.0 0.01533516
#> 2      0.1 0.13458658
#> 3      0.3 0.53041794
#> 4      0.5 0.81006323
#> 5      0.7 0.94659101
#> 6      0.9 0.99530941
  pvals.ICC1a(iccdata1)
#>   rho.zero       pval
#> 1        0 0.01533516

The 95% confidence level is what is offered by default. Can we obtain a 90% confidence interval instead? The answer is yes. This is achieved as follows:

  ci.ICC1a(iccdata1,conflev = 0.90)
#>          lcb       ucb
#> 1 0.04185384 0.7082716

Now, suppose you want to compute p-values based on Model 1A for the null values 0.15,0.25, and 0.45. You would proceed as follows:

  pvals.ICC1a(iccdata1,rho.zero = c(0.15,0.25,0.45))
#>   rho.zero      pval
#> 1     0.15 0.2298508
#> 2     0.25 0.4351265
#> 3     0.45 0.7556120

ICC under Model 1B

The following code gives us the intra-rater reliability coefficient of 0.562, the associated 95% confidence interval of (0,1). The function pval.ICC1b() gives you p-values for a predermined vector of null values (0.0,0.1,0.3,0.5,0.7,0.9). For example, the p-value associated with the null value 0.3 is given by 7.075e-02. If you want to supply your own null values you will need to use function pvals.ICC1b(), the default null value being 0.. Remember that Model 1B can only give you an intra-rater reliability coefficient. If you need an inter-rater reliability then you must use a different model.

  icc1b.fn(iccdata1)
#>     sig2r    sig2e     icc1b n r max.rep min.rep Mtot ov.mean
#> 1 4.32087 3.365846 0.5621217 5 4       3       1   40     5.2
  ci.ICC1b(iccdata1)
#>   lcb ucb
#> 1   0   1
  pval.ICC1b(iccdata1)
#>   rho.zero         pval
#> 1      0.0 4.686342e-06
#> 2      0.1 1.386945e-03
#> 3      0.3 7.074659e-02
#> 4      0.5 3.142024e-01
#> 5      0.7 6.484932e-01
#> 6      0.9 9.301346e-01
  pvals.ICC1b(iccdata1)
#>   gam.zero         pval
#> 1        0 4.686342e-06

Again, instead of the default 95% confidence interval, you may request a 90% confidence interval as follows:

  ci.ICC1b(iccdata1,conflev = 0.90)
#>   lcb ucb
#> 1   0   1

P-values associated with an arbitrary vector of null values (0.15,0.25,0.45) are calculated as follows:

  pvals.ICC1b(iccdata1,gam.zero = c(0.15,0.25,0.45))
#>   gam.zero        pval
#> 1     0.15 0.002962246
#> 2     0.25 0.020746475
#> 3     0.45 0.166885491

It follows that for the null value 0.25 you get p-value = 2.075e-02.

ICC under Model 2

Model 2 With Interaction

Under Model 2 with interaction, the confidence intervals and p-values are calculated as follows:

  icc2.inter.fn(iccdata1)
#>      sig2s    sig2r    sig2e    sig2sr    icc2r     icc2a n r max.rep
#> 1 2.018593 4.281361 1.315476 0.4067361 0.251627 0.8360198 5 4       3
#>   min.rep Mtot ov.mean
#> 1       1   40     5.2
  ci.ICC2r.inter(iccdata1)
#>          lcb       ucb
#> 1 0.02191927 0.7792666
  ci.ICC2a.inter(iccdata1)
#>         lcb       ucb
#> 1 0.5478536 0.9645078
  pval.ICC2r.inter(iccdata1)
#>   rho.zero         pval
#> 1      0.0 0.0009601902
#> 2      0.1 0.1191859085
#> 3      0.3 0.5526437466
#> 4      0.5 0.8194818940
#> 5      0.7 0.9480505063
#> 6      0.9 0.9953334341
  pvals.ICC2r.inter(iccdata1)
#>   rho.zero         pval
#> 1        0 0.0009601902
  pvals.ICC2a.inter(iccdata1)
#>   gam.zero         pval
#> 1        0 2.306507e-05

The function ci.ICC2r.inter(iccdata1) produced the 95% confidence interval (0.022,0.779) associated with the inter-rater reliability coefficient ICCr = 0.252. If needed, change the confidence level to 90% for example ci.ICC2r.inter(iccdata1,conflev=0.90) to get (0.044,0.702).
The function ci.ICC2a.inter(iccdata1) produced the 95% confidence interval (0.548,0.965) associated with the intra-rater reliability coefficient ICCa = 0.836. If needed, change the confidence level to 90% for example ci.ICC2a.inter(iccdata1,conflev=0.90) to get (0.609,0.953).
The function pval.ICC2r.inter(iccdata1) produced a series of p-values associated with the inter-rater reliability for the 6 arbitrarily selected null values (0,0.1,0.3,0.5,0.7,0.9).
The function pvals.ICC2r.inter(iccdata1) can produce p-values associated with the inter-rater reliability for an arbitrary input vector of null values, the default value being 0. However if you want to compute the p-values for the 2 null values 0.25 and 0.45, it could be achieved as follows:

  pvals.ICC2r.inter(iccdata1,rho.zero = c(0.25,0.45))
#>   rho.zero      pval
#> 1     0.25 0.4589435
#> 2     0.45 0.7689111

The function pvals.ICC2a.inter(iccdata1) was used to compute the p-value associated with the intra-rater reliability for the default null value of 0. It be used with an arbitrary vector of null values. For the 2 null values 0.25 and 0.45, the associated p-values are computed as follows:

  pvals.ICC2a.inter(iccdata1,gam.zero = c(0.25,0.45))
#>   gam.zero         pval
#> 1     0.25 0.0006793227
#> 2     0.45 0.0079884253

Model 2 Without Interaction

Under Model 2 without interaction, the confidence intervals and p-values are calculated as follows:

  icc2.nointer.fn(iccdata1)
#>      sig2s   sig2r    sig2e     icc2r    icc2a n r max.rep min.rep Mtot
#> 1 2.090769 4.34898 1.598313 0.2601086 0.801157 5 4       3       1   40
#>   ov.mean
#> 1     5.2
  ci.ICC2r.nointer(iccdata1)
#>          lcb       ucb
#> 1 0.02869092 0.7805637
  ci.ICC2a.nointer(iccdata1)
#>         lcb       ucb
#> 1 0.5505474 0.9639793
  pvals.ICC2r.nointer(iccdata1)
#>   rho.zero         pval
#> 1        0 5.413829e-06
  pvals.ICC2a.nointer(iccdata1)
#>   gam.zero         pval
#> 1        0 7.887974e-09

All the above procedures work in the same way as those described in the case of Model 2 with interaction.

ICC under Model 3

Under Model 3 with interaction, the confidence intervals and p-values are calculated as follows:

  icc3.inter.fn(iccdata1)
#>      sig2s    sig2e    sig2sr     icc2r     icc2a n r max.rep min.rep Mtot
#> 1 2.257426 1.315476 0.2238717 0.5749097 0.6535279 5 4       3       1   40
#>   ov.mean
#> 1     5.2
  ci.ICC3r.inter(iccdata1)
#>         lcb       ucb
#> 1 0.2162702 0.9222461
  ci.ICC3a.inter(iccdata1)
#>        lcb       ucb
#> 1 0.142446 0.9091092
  pvals.ICC3r.inter(iccdata1)
#>   rho.zero         pval
#> 1        0 0.0009601902
  pvals.ICC3a.inter(iccdata1)
#>   gam.zero        pval
#> 1        0 0.008717435

Under Model 3 without interaction, the confidence intervals and p-values are calculated as follows:

  icc3.nointer.fn(iccdata1)
#>      sig2s    sig2e     icc2r     icc2a n r max.rep min.rep Mtot ov.mean
#> 1 2.241792 1.470638 0.6038611 0.6038611 5 4       3       1   40     5.2
  ci.ICC3r.nointer(iccdata1)
#>         lcb       ucb
#> 1 0.2494563 0.9248785
  pvals.ICC3r.nointer(iccdata1)
#>   rho.zero         pval
#> 1        0 5.413829e-06

For a more detailed discussion of the use of these functions, please refer to the specific function documentation.

References:

Gwet, K.L. (2014, ISBN:978-0970806284). “Handbook of Inter-Rater Reliability,” 4th Edition. Advanced Analytics, LLC

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.