Off-line run with a shadow CAT design

The following code sets up a shadow CAT design (i.e., a CAT where a number of constraints must be satisfied throughout the test administration). This uses integer programming methods to find the most optimal solution given a (potentially large) number of constraints. The following is a relatively simple in its inception, but demonstrates the generality of the setup and the use of the test_properties input.

library('mirtCAT')

set.seed(1)
bank <- 300 #bank size
N <- 500 #calibration sample size

a <- rlnorm(bank, .2,.3)
d <- rnorm(bank)
pars <- data.frame(a1=a, d=d)
mod <- generate.mirt_object(pars, itemtype = '2PL')
head(coef(mod, simplify=TRUE)$items)

##               a1          d g u
## Item.1 1.0121369  0.8936737 0 1
## Item.2 1.2905816 -1.0472981 0 1
## Item.3 0.9505746  1.9713374 0 1
## Item.4 1.9710852 -0.3836321 0 1
## Item.5 1.3483105  1.6541453 0 1
## Item.6 0.9549078  1.5122127 0 1

This next section defines a number of test-level properties, such as

1) The type of item response stimuli (e.g., true-false, multiple-choice, fill-in-the-blank, etc), 1) The content category (e.g., addition or subtraction), and 1) Number of words in each item stem

The purpose of this shadow CAT is to find an optimal measurement test, according to the maximum information criteria, subject to the following constraints:

exactly 40 total items administered
between 3 to 5 fill-in-the-blank items only
items 5 and 6, as well as 10 and 11, should not appear in the same test
item 17 and 18 should appear in the test no matter what
item 15 should not appear
the same number of addition and subtraction questions should be included
the average number of words in the item stems should be less than 20

## generate random response pattern
set.seed(1)
pattern <- generate_pattern(mod, Theta = matrix(-2:2))
pattern[, 1:6]

##      [,1] [,2] [,3] [,4] [,5] [,6]
## [1,]    0    0    0    0    1    0
## [2,]    0    1    0    0    0    0
## [3,]    1    0    1    1    1    1
## [4,]    1    1    1    1    1    1
## [5,]    1    0    1    1    1    1

test_properties <- data.frame(stimuli=rep(c('TF', 'MC', 'fill'), each=bank/3),
                              content=rep(c('add', 'sub'), times=150),
                              words=sample(10:30, bank, TRUE))
head(test_properties)

##   stimuli content words
## 1      TF     add    11
## 2      TF     sub    28
## 3      TF     add    14
## 4      TF     sub    21
## 5      TF     add    12
## 6      TF     sub    18

# constraint generating function
constr_fun <- function(person, test, design){
  mo <- extract.mirtCAT(test, 'mo')
  nitems <- extract.mirt(mo, 'nitems')
  tp <- extract.mirtCAT(design, 'test_properties')

  lhs <- matrix(0, 9, nitems)
  lhs[1,] <- 1
  lhs[2,tp$stimuli == 'fill'] <- 1
  lhs[3,tp$stimuli == 'fill'] <- 1
  lhs[4, c(5, 6)] <- 1
  lhs[5, c(10,11)] <- 1
  lhs[6, c(17,18)] <- 1
  lhs[7, 15] <- 1
  lhs[8, tp$content == 'add'] <- 1
  lhs[8, tp$content == 'sub'] <- -1
  lhs[9, ] <- tp$words

  # relationship direction
  dirs <- c("==", ">=", '<=', '==', '==', '==', '==', '==', '<=')

  #right hand side
  rhs <- c(40, 3, 5, 1, 1, 2, 0, 0, 40*20)

  #all together
  constraints <- data.frame(lhs, dirs, rhs)
  constraints
}

# next item selection function
customNextItem <- function(person, design, test){
    objective <- computeCriteria(person=person, design=design, test=test,
                                 criteria = 'MI') 
    item <- findNextItem(person=person, design=design, test=test,
                         objective=objective)
    item
}

result <- mirtCAT(mo=mod, start_item = 30, local_pattern = pattern, 
                  design = list(customNextItem=customNextItem, 
                              constr_fun=constr_fun, 
                              test_properties=test_properties,
                              min_SEM=0))
plot(result[[1]])

plot of chunk unnamed-chunk-2

(items <- summary(result[[1]])$items_answered)

##  [1]  30 206  55 283 142  50 278 185 192 186 126 102 255   5  22  51 109
## [18] 162  21 235 163 120 179  85 169  93 113  34  62 170  11 151 175  16
## [35]  82  80  91 140  18  17

head(test_properties[items, ]) #items answered

##     stimuli content words
## 30       TF     sub    30
## 206    fill     sub    13
## 55       TF     add    21
## 283    fill     add    18
## 142      MC     sub    25
## 50       TF     sub    23

table(test_properties[items, 1:2])

##        content
## stimuli add sub
##    fill   3   2
##    MC     8   9
##    TF     9   9

colSums(table(test_properties[items, 1:2]))

## add sub 
##  20  20

mean(test_properties[items, 3])

## [1] 20

plot(result[[3]])

plot of chunk unnamed-chunk-2

(items <- summary(result[[3]])$items_answered)

##  [1]  30 274 166 206 178 171  56 160 106 202 265  70  95 147 110   4 187
## [18] 216 197  31  68 180  11  92 122  39 113  18 157 108 194  83 177 107
## [35]  93  43  25  63  17   5

head(test_properties[items, ]) #items answered

##     stimuli content words
## 30       TF     sub    30
## 274    fill     sub    14
## 166      MC     sub    30
## 206    fill     sub    13
## 178      MC     sub    28
## 171      MC     add    28

table(test_properties[items, 1:2])

##        content
## stimuli add sub
##    fill   1   4
##    MC     8   9
##    TF    11   7

colSums(table(test_properties[items, 1:2]))

## add sub 
##  20  20

mean(test_properties[items, 3])

## [1] 19.175

Notice that each of these response patterns satisfy all of the proposed constraints. This is the general benefit of shadow CATs: optimal item selection (according to the item selection criteria) while simultaneously considering a wide and complicated set of item-design constraints.