3. Rasch against 2PL model
In psychology, researchers measure mental attributes of people whose values of e.g. mathematical ability or personality traits are not directly observable in this way. Hence psychological tests are constructed to measure latent properties. When a test taker answers a maths test, researchers want to conclude as reliably as possible about the person’s maths competence. Methods from IRT are used for psychological test construction and evaluation. A well-known approach is the Rasch model. The Rasch model assumes that the probability of solving an individual item from the test is determined by the characteristics of the item (item parameters) and of the person (person parameters). The more pronounced the math ability \(\theta\) of person \(p\) the higher the probability of answering an item correctly. Another observation regarding the Rasch model is that as the easiness \(d\) of an item \(i\) increases, the probability of answering the item correctly also increases. The easiness is known as the intercept-parameter. We can transfer this assumption into a formula:
\[ \text{Pr}(U_{pi} = 1|\theta_{p}, d_{i}) = \frac{e^{\theta_p + d_{i}}}{1 + e^{\theta_p + d_{i}}} \]
One strict assumption of the Rasch model is that all items share the same slope parameter. In this context, the slope plays an important role in accurately distinguishing between abilities slightly above and abilities slightly below its easiness. In other literature the slope parameter is also called discrimination. Two people with an math ability difference of 1 would in all items have the same probability difference of solving an item correctly. The Rasch model equation from above does not contain an extra parameter for modeling the slope. For items that are not well described by the Rasch model, other more flexible models can be applied.
The two-parameter logistic (2PL) model from Birnbaum (1968) is a more adaptable solution and can be seen as an extension of the Rasch model. When the assumption of equal slopes among all items is violated, this model can be utilized. Therefore the 2PL model introduces besides the intercept parameter or easiness \(d_{i}\) a second item parameter called the slope parameter \(a\) of an item \(i\). Mathematically the 2PL model can be described with:
\[ \text{Pr}(U_{pi} = 1|\theta_{p},a_{i}, d_{i}) = \frac{e^{a_i \cdot \theta_p + d_{i}}}{1 + e^{a_i \cdot \theta_p + d_{i}}} \] If \(a\) is high, it indicates that the item possesses a strong ability to distinguish among test taker. This means that the probability of giving a correct response increases more rapidly as the ability \(\theta\) increases.
When it comes to the evaluation of psychological and educational tests researcher face the question which IRT model is better to describe the data. There are various statistical methods such as Wald, LR, score and gradient tests that compare two models in terms of their data fit. In this example, our specific hypothesis test aims to address the question of whether the inclusion of the additional slope parameter in the 2PL model is essential for describing the data, or if the slope parameters are sufficiently similar for the Rasch model to adequately describe the data. Therefore it becomes relevant to test the null hypothesis of equal slope parameters as this may allow the rejection of an incorrectly assumed Rasch model. With a power analysis we can answer the question what sample size is minimally needed to discard the simpler Rasch model. In the following we first conduct a priori power analysis and then a posteriori power analysis for testing the Rasch model against the 2PL model.
3.1. A priori power analysis
We are in the process of creating a novel test designed to assess the
mathematical ability of test takers. We plan a test with 5 items that
measure the same construct. After an intensive incorporation with the
subject of math tests and the 5 already formulated items we discusses
with other authors the expected item parameter for each item.
Specifically, we hypothesize the presence of unequal slope parameters.
We know that after collecting data we want to evaluate the test using
the Rasch or the 2PL model. We aim to ensure that if a model comparison
favors the 2PL model, we can confidently assert that this model choice
provides the best fit for our expected data. This is only with a
sufficiently large power possible. To conduct a power analysis we use
the R package irtpwr.
To setup a hypothesis, we use the altpars argument of
the setup.hypothesis function to define the parameters
under the alternative hypothesis of unequal slope parameters. We do not
have to determine the null hypothesis of equal parameters because they
are implicitly defined by the alternative hypothesis. Firstly, we
generate distinct slope and intercept parameters for each item. Then we
pass them into the altpars argument from the function
setup.hypothesis. Additionally, due to the underlying
hypothesis test, we set the type to
"1PLvs2PL".
# parameters under the alternative hypothesis
altpars <- list(a = rlnorm(5, sdlog = 0.4),
d = rnorm(5))
hyp <- setup.hypothesis(type = "1PLvs2PL", altpars = altpars)The hypothesis is prepared for subsequent power analysis. We want to
know the required sample size for a test of the Rasch against 2PL model
for a power of 0.8 while using an alpha level of 0.05. Therefore, we
transfer the previously defined hypothesis "hyp" into the
irtpwr function:
res <- irtpwr(hyp = hyp, power = 0.8, alpha = 0.05)
summary(res)
#> Sample sizes for power = 0.8 (alpha = 0.05):
#>
#> Statistic N
#> Wald 503
#> LR 504
#> Score 520
#> Gradient 505
#>
#> Method: AnalyticalPrior to the commencement of the study we now planned the sample size. The results show that to reach our desired power of 0.8 we need a sample size of 503 to 520. The Wald Test would in this case be the most efficient choice. In addition, we have the option to generate power curves through power analysis. This allows us to obtain a more comprehensive understanding of the associations between sample size and power:
A second reflection about our study design revealed that we want a
higher statistical power. The alternative and null hypothesis remain
unchanged. We take the saved irtpwr result object
res and modify it within the generic summary
function:
summary(res, power = 0.9)
#> Sample sizes for power = 0.8 (alpha = 0.05):
#>
#> Statistic N
#> Wald 649
#> LR 650
#> Score 671
#> Gradient 652
#>
#> Method: AnalyticalAs expected we need more test taker to answer the test to reach a power of 0.9. We minimum sample size would be 649.
3.2. A posteriori power analysis
In this chapter we already have constructed a 5 item math test but
without a prior conducted power analysis. Exactly 1000 participants
answered the test. We aim to assess the test by employing both the Rasch
model and the 2PL model. Later on, we would compare both IRT models and
conclude which one would provide the best model fit. To allow a reliable
model comparison we need a minimum of 80% statistical power. To do this,
we conduct a posteriori power analysis. First of all we need the dataset
from our math test. We use the LSAT 7 dataset which is included in the
mirt package. The dataset includes 5 columns (for each item
one) and 1000 rows (for each participant one). One row shows for each
item if the participant answered this item correctly (coded with 1) or
not (coded with 0). For the first time we now have to install and load
the mirt package:
We load the data with:
For the power analysis we again have to define the type
and parameters under the alternative hypothesis. In this situation where
the data is collected we can use a fitted mirt model.
Hence, we fit a 2PL model to analyze our data and obtain the parameters
associated with the alternative hypothesis of unequal slopes. To fit a
2PL model one can execute the following code:
The fitted 2PL model mirtfit is then used for the
altpars argument from the function
setup.hypothesis. As above we set
type = "1PLvs2PL". We write the following code:
To calculate the power for a sample size of 1000 using our hypothesis from above we run:
res <- irtpwr(hyp = hyp, N = 1000, alpha = 0.05)
summary(res)
#> Power for N = 1000 (alpha = 0.05):
#>
#> Statistic Power
#> Wald 0.7411
#> LR 0.8513
#> Score 0.8308
#> Gradient 0.8987
#>
#> Method: AnalyticalFor a sample size of 1000 we reach a power between 0.74 to 0.89 depending the underlying test statistic. For our further analysis we should use the gradient test. If the gradient test favors the 2PL model, we can confidently claim that the 2PL model is a better fit than the Rasch model with a probability of 89%. We would conclude that given our data it would be better to assume unequal slope parameter.
A detailed look on the relationships between sample size and power can be plotted with:
The irtpwr function builds on one of two available
methods that are described in our paper in more detail (Zimmer et
al. (2022), https://doi.org/10.1007/s11336-022-09883-5):
- an analytical approach (
method = “analytical”, this is the default option) - a sampling-based approach (
method = “sampling”)
To use the sampling based approach, we can execute:
res <- irtpwr(hyp = hyp, N = 600, alpha = 0.05, method = "sampling")
summary(res)
#> Power for N = 1000 (alpha = 0.05):
#>
#> Statistic Power
#> Wald 0.4946
#> LR 0.6073
#> Score 0.5957
#> Gradient 0.6578
#>
#> Method: Sampling-BasedThe sampling-based approach offers two parameters that can be
increased from their default values to obtain a more exact result. These
are the sample size of the sampling-based approach
(sampling.npers) and the sample size for the approximation
of the Fisher expected matrix (approx.npers). These may be
tweaked upwards if the result from repeated executions of the
irtpwr function is not stable.
Since the analytical approach can be too time-intensive for larger
numbers of items, we might want to know beforehand how long it will take
approximately. We do so using the calc.time function. In
our example hypothesis, we used 5 items. To estimate the computation
time (in seconds) for 10 items we can use:
Note that the estimated time can sometimes deviate from the actual
time. As a result from our test runs, we suggest to expect an increase
of 80% as a worst case scenario. Is the estimated computational load
when using the analytical default setting too large, we can use the
sampling-based approach for these applications. The interpretation of
the irtpwr output remains the same.
