The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

Creating constraints in TestDesign package

Sangdon Lim


Introduction

This document explains how to create constraints data for loadConstraints(). Automated test assembly in practice is often desired to assemble a test so that its contents adhere to a test blueprint, which asserts various requirements the assembled test should satisfy. As of TestDesign version 1.1.0, constraints can be read in from data.frame objects or .csv spreadsheet files. The input data is expected to be in the following structure:

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C1 Number Item 30 30
C2 Number Item LEVEL == 3 10 10
C3 Number Item LEVEL == 4 10 10
C4 Number Item LEVEL == 5 10 10
C5 Number Item STANDARD == 1 17 20

Constraints data must have seven columns, named as CONSTRAINT_ID, TYPE, WHAT, CONDITION, LB, UB, ONOFF on the first row. Beginning from the second row, each row must have corresponding values for each column. A convenient way for working with constraints is to use a spreadsheet application (e.g. Excel) and work on the content from there.

Readers are also encouraged to tinker with example constraints included in the package:


Content balancing with shadow-test approach

This section aims to provide context on why the constraints input format does not have a column for weights.

The TestDesign package performs content balancing using the shadow-test approach (van der Linden & Reese, 1998). This means that the test will be assembled in a way that strictly satisfies all constraints with no violations. The reader may be familiar with the use of weights in test blueprints for indicating which constraints should be prioritized. These constraint-wise weights are mainly needed when traditional content balancing methods are used, where items are selected one by one. When items are selected one by one, there is a fundamental limitation that there is no guarantee that the resulting test will satisfy all constraints. For this reason, weights are used as supplements to traditional content balancing to work around this limitation, to guide the item selection process in a way that the number of violated constraints is minimized.

Unlike with traditional content balancing methods, the shadow-test approach operates without needing weights. This is because the shadow-test approach directly finds a combination of items that satisfies all constraints, and therefore has no need to prioritize certain constraints to satisfy, as would be needed in traditional content balancing methods that select items one by one.


Required columns

Column 1: CONSTRAINT_ID

This column specifies the identifier of each constraint. Character values can be used as long as the values are unique.

Column 2: TYPE

This column specifies the type of constraint. Following values are expected: Number, Order, Enemy, Include, Exclude, AllorNone.

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C1 Number Item 30 30
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C2 Sum Item WORDS 500 600
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C32 Order Item LEVEL
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C33 Enemy Item ID %in% c(“SC00001”, “SC00002”)
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C34 Include Item ID %in% c(“SC00003”, “SC00004”)
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C35 Exclude Item PTBIS < 0.15
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C36 AllOrNone Item ID %in% c(“SC00005”, “SC00006”)

Column 3: WHAT

This column specifies the unit of assembly the constraint uses. Expected values are Item or Stimulus.

Column 4: CONDITION

This column specifies the condition of the constraint. An R expression returning logical values (TRUE or FALSE) is expected. The variables supplied in item attributes can be used in the expression as variable names.

Some examples are:

For TYPE == SUM, using a variable name imposes the constraint on the sum of the variable. The following row tells the solver to keep the sum of WORDS between 500–600.

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C2 Sum Item WORDS 500 600

For TYPE == SUM, constraints on conditional sums can be imposed by using a variable name, placing a comma, and then giving an R expression returning logical values. The following row tells the solver to keep the sum of WORDS within DOK == 1 items between 50–80.

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C3 Sum Item WORDS, DOK == 1 50 80

In set-based assembly, Per Stimulus can be used to specify the number of items to select in each stimulus. For example, the following row tells the solver to select 4 to 6 items per stimulus:

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C3 Number Item Per Stimulus 4 6

Column 5-6: LB and UB

These two columns specify lower and upper bounds on the number of selected items. These must be specified when TYPE is Number, and otherwise must be left empty.

Some example rows are provided.

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C1 Number Item 12 12
CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C17 Number Item DOK >= 2 15 30

Column 7: ONOFF

Set this to OFF to turn off the constraint from being applied. ON or leaving it blank applies the constraint. The following example specifies the order constraint to be not applied.

CONSTRAINT_ID TYPE WHAT CONDITION LB UB ONOFF
C18 Order Passage CONTENT OFF


References

van der Linden W. J., Reese L. M. (1998). A model for optimal constrained adaptive testing. Applied Psychological Measurement, 22(3), 259-270. https://doi.org/10.1177/01466216980223006


These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.