Model
The linear mixed model used for the calculations is specified as the
following:
\[
Y_{i} \sim\ N \left(\alpha_{j[i],k[i]}, \sigma^2 \right)
\]
\[
\alpha_{j} \sim\ N \left(\mu_{\alpha_{j}}, \sigma^2_{\alpha_{j}}
\right)
\text{, for id j = 1,} \dots \text{,J}
\]
\[
\alpha_{k} \sim\ N \left(\mu_{\alpha_{k}}, \sigma^2_{\alpha_{k}}
\right)
\text{, for items k = 1,} \dots \text{,K}
\]
Components of Variance
Mean Squared Error (MSE)
\[
MSE = \sigma^2
\]
Variance Between Subjects
\[
MSB = n_k \cdot \sigma^2_{\alpha j} + \sigma^2
\]
Variance Between Items/Judges
\[
MSJ = n_{j} \cdot \sigma^2_{\alpha_{k}} + \sigma^2
\]
Variance Within Subjects/Participants
\[
MSW = \sigma^2 + \sigma^2_{\alpha_{k}}
\]
Intraclass Correlation Coefficients
\[
ICC_{(1,1)} = \frac{MSB - MSW}{MSB + (n_j-1) \cdot MSW}
\] \[
ICC_{(2,1)} = \frac{MSB - MSE}{MSB+(n_j -1) \cdot MSE + n_j \cdot (MSJ -
MSE) \cdot n^{-1}}
\] \[
ICC(_{3,1)} = \frac{MSB - MSE}{MSB + (n_j -1) \cdot MSE}
\] \[
ICC_{(1,k)} = \frac{MSB - MSW}{MSB}
\] \[
ICC_{(2,k)} = \frac{MSB - MSW}{MSB + (MSJ - MSE) \cdot n_j^{-1}}
\] \[
ICC_{(3,k)} = \frac{MSB - MSE}{MSB}
\]
ICC Confidence Intervals
ICC(1,1)
\[
F = \frac{MSB}{MSW}
\]
\[
df_{n} = n_j - 1
\]
\[
df_{d} = n_j \cdot (n_k - 1)
\]
\[
F_{L} = \frac{F}{F_{(1 - \alpha, \space df_{n}, \space df_{d})}}
\]
\[
F_{U} = F \cdot F_{(1 - \alpha, \space df_{d}, \space df_{n})}
\]
\[
Lower \space CI = \frac{(F_L - 1)}{(F_L + (n_j - 1))}
\]
\[
Upper \space CI = \frac{(F_U - 1)}{(F_U + n_j - 1)}
\]
ICC(2,1)
\[
F = \frac{MSJ}{MSE}
\]
\[
vn = (n_k - 1) \cdot (n_j - 1) \cdot [(nj \cdot ICC_{(2,1)} \cdot F +
n_j \cdot
(1 + (n_k - 1) \cdot ICC_{(2,1)}) -
n_k \cdot ICC_{(2,1)})]^2
\]
\[
vd = (n_j - 1) \cdot n_k^2 \cdot ICC_{(2,1)}^2 \cdot F^2 + (n_j \cdot
(1 +
(n_k - 1) \cdot
ICC_{(2,1)}) - n_k \cdot ICC_{(2,1)})^2
\] \[
v = \frac{vn}{vd}
\]
\[
F_{L} = F_{(1 - \alpha, \space n_j-1, \space v)}
\]
\[
F_{U} = F_{(1 - \alpha, \space v, \space n_j - 1)}
\]
\[
Lower \space CI = \frac{n_j \cdot (MSB - F_U \cdot MSE)}{(F_U \cdot
(n_k \cdot MSJ + (n_k \cdot n_j - n_k - n_j) \cdot MSE) + n_j \cdot
MSB)}
\]
\[
Upper \space CI = \frac{n_j \cdot (MSB \cdot F_L - MSE)}{(n_k \cdot
MSJ + (n_k \cdot n_j - n_k - n_j) \cdot MSE + n_j \cdot F_L \cdot MSB)}
\]
ICC(3,1)
\[
F = \frac{MSJ}{MSE}
\] \[
df_{n} = n_j - 1
\] \[
df_{d} = (n_j-1) \cdot (n_k - 1)
\]
\[
F_{L} = \frac{F}{F_{(1 - \alpha, \space df_{n}, \space df_{d})}}
\] \[
F_{U} = F \cdot F_{(1 - \alpha, \space df_{n}, \space df_{d})}
\] \[
F3L <- F31/qf(1 - alpha, df21n, df21d)
\] \[
F3U <- F31 * qf(1 - alpha, df21d, df21n)
\] \[
Lower \space CI = (F3L - 1)/(F3L + n_k - 1)
\]
\[
Upper \space CI = (F3U - 1)/(F3U + n_k - 1)
\]
ICC(1,k)
\[
F = \frac{MSB - MSW}{MSB}
\] \[
df_{n} = n_j - 1
\]
\[
df_{d} = n_j \cdot (n_k - 1)
\]
\[
F_{L} = \frac{F}{F_{(1 - \alpha, \space df_{n}, \space df_{d})}}
\] \[
F_{U} = F \cdot F_{(1 - \alpha, \space df_{d}, \space df_{n})}
\]
\[
Lower \space CI = 1-\frac{1}{F_L}
\] \[
Upper \space CI = 1-\frac{1}{F_U}
\]
ICC(2,k)
\[
F = \frac{MSB - MSW}{MSB}
\]
\[
vn = (n_k - 1) \cdot (n_j - 1) \cdot [(nj \cdot ICC_{(2,k)} \cdot F +
n_j \cdot
(1 + (n_k - 1) \cdot ICC_{(2,k)}) -
n_k \cdot ICC_{(2,k)})]^2
\]
\[
vd = (n_j - 1) \cdot n_k^2 \cdot ICC_{(2,k)}^2 \cdot F^2 + (n_j \cdot
(1 +
(n_k - 1) \cdot
ICC_{(2,k)}) - n_k \cdot ICC_{(2,k)})^2
\]
\[
v = \frac{vn}{vd}
\]
\[
F_{L} = F_{(1 - \alpha, \space n_j-1, \space v)}
\] \[
F_{U} = F_{(1 - \alpha, \space v, \space n_j - 1)}
\]
\[
L3 = \frac{n_j \cdot (MSB - F_U \cdot MSE)}{(F_U \cdot (n_k \cdot MSJ +
(n_k \cdot n_j - n_k - n_j) \cdot MSE) + n_j \cdot MSB)}
\]
\[
U3 = \frac{n_j \cdot (MSB \cdot F_L - MSE)}{(n_k \cdot MSJ + (n_k \cdot
n_j - n_k - n_j) \cdot MSE + n_j \cdot F_L \cdot MSB)}
\]
\[
Lower \space CI = \frac{L3 \cdot n_k}{(1 + L3 \cdot (n_k - 1))}
\]
\[
Upper \space CI = \frac{U3 \cdot n_k}{(1 + U3 \cdot (n_k - 1))}
\]
ICC(3,k)
\[
F = \frac{MSB}{MSE}
\]
\[
df_n = n_j - 1
\]
\[
df_d = (n_j - 1) \cdot (n_k - 1)
\]
\[
F_L = \frac{F}{F_{(1 - \alpha, \space df_n, \space df_d})}
\]
\[
F_U = F \cdot F_{(1 - \alpha, \space df_d, \space df_n)}
\]
\[
Lower \space CI = 1-\frac{1}{F_L}
\]
\[
Upper \space CI = 1-\frac{1}{F_U}
\]
Standard Error Calculations
The standard error of the measurement (SEM), standard error of the
estimate (SEE), and standard error of prediction (SEP) are all estimated
with the following calculations.
The default SEM calculation is the following:
\[
SEM = \sqrt{MSE}
\] Alternatively, the SEM can be calculated as the following and
the ICC is determined through the se_type argument:
\[
SEM = \sqrt{\frac{SS_{total}}{(N-1)}} \cdot \sqrt{1-ICC}
\] The other measures default to using the following equations.
The default is to use \(ICC_{3,1}\),
but can be modified with the se_type argument.
\[
SEE = \sqrt{\frac{SS_{total}}{(N-1)}} \cdot \sqrt{ICC \cdot (1-ICC)}
\]
\[
SEP = \sqrt{\frac{SS_{total}}{(N-1)}} \cdot \sqrt{1-ICC^2}
\]
Coefficient of Variation
The CV is calculated 3 potential ways within reli_stats.
I highly recommend reporting the default version of CV.
- From the MSE (default)
\[
CV = \frac{ \sqrt{MSE} }{ \bar y}
\]
- From the SEM
\[
CV = \frac{SEM}{ \bar y}
\]
- From the model residuals (most liberal)
\[
CV = \frac{\sqrt{\frac{\Sigma^{N}_{i=1}(y_i - \hat y_i)^2}{N_{obs}}}}{
\bar y}
\]
Other Confidence Intervals
If the other_ci argument is set to TRUE then confidence
intervals will be calculated for the CV, SEM, SEP, and SEE.
Chi-square
The default method is type = 'chisq. This method
utilizes the chi-square distribution to approximate confidence
intervals for these measurements. The accuracy of these estimates is
likely variable, and is probably poor for small samples.
For the CV, the calculation is as follows:
\[
Lower \space CI =
\frac{CV}{\sqrt{(\frac{\chi^2_{1-\alpha/2}}{df_{error}+1}-1) \cdot CV^2
\cdot \frac{\chi^2_{1-\alpha/2}}{df_{error}}}}
\]
\[
Upper \space CI =
\frac{CV}{\sqrt{(\frac{\chi^2_{\alpha/2}}{df_{error}+1}-1) \cdot CV^2
\cdot \frac{\chi^2_{\alpha/2}}{df_{error}}}}
\] For the variance based measures (s) the calculation is as
follows:
\[
Lower \space CI = s \cdot \sqrt{\frac{df_{error}}{\chi^2_{1-\alpha/2}}}
\]
\[
Upper \space CI = s \cdot \sqrt{\frac{df_{error}}{\chi^2_{\alpha/2}}}
\]
Bootstrap
If type is not set to chisq then bootstrapping is
performed.
reli_aov utilizes a non-parametric ordinary
bootstrap.reli_stats utilizes a parametric bootstrap.
The number of resamples can be set with the replicates argument
(default is 1999; increase for greater accuracy or lower for greater
speed). The reported confidence intervals are estimated using the
percentile method type = 'perc', the normal
type = 'norm', or basic methods
type = 'basic'.
To ensure reproducibility, please use set.seed() when
these confidence intervals are calculated.
