This example demonstrates how to collect response data with mirtCAT for new items in an item bank. The constraint is that these new items should not influence the selection process, nor should they influence the computation of the latent trait estimates. Hence, we will require setting up a customized item selection function in order to control how many/when these unscored items should be administered, and also include various flags to mirtCAT() to indicate that they should not be included in the scoring process.
Say that we had the following bank of existing items.
library('mirtCAT')## Loading required package: mirt## Loading required package: stats4## Loading required package: lattice## Loading required package: shinyoptions(stringsAsFactors = FALSE)
# define population IRT parameters
set.seed(1234)
nitems <- 100
itemnames <- paste0('Item.', 1:nitems)
a <- matrix(rlnorm(nitems, .2, .3))
d <- matrix(rnorm(nitems))
pars <- data.frame(a1=a, d=d, g=0.2)
mod <- generate.mirt_object(pars, '3PL')
# math items definitions
# addition for one factor and multiplication for the other
questions <- answers <- character(nitems)
choices <- matrix('a', nitems, 5)
spacing <- floor(d - min(d)) + 1 #easier items have more variation
for(i in 1:nitems){
n1 <- sample(1:100, 1)
n2 <- sample(101:200, 1)
ans <- n1 + n2
questions[i] <- paste0(n1, ' + ', n2, ' = ?')
answers[i] <- as.character(ans)
ch <- ans + sample(c(-5:-1, 1:5) * spacing[i,], 5)
ch[sample(1:5, 1)] <- ans
choices[i,] <- as.character(ch)
}
df <- data.frame(Questions=questions, Answer=answers, Option=choices, Type='radio')
head(df)## Questions Answer Option.1 Option.2 Option.3 Option.4 Option.5 Type
## 1 69 + 142 = ? 211 211 219 207 195 231 radio
## 2 63 + 142 = ? 205 214 220 205 193 208 radio
## 3 52 + 178 = ? 230 230 224 233 236 221 radio
## 4 94 + 120 = ? 214 199 208 205 214 226 radio
## 5 44 + 194 = ? 238 229 241 238 244 235 radio
## 6 82 + 189 = ? 271 283 267 291 271 275 radioNow, we wish to collect responses for two new items.
new2 <- data.frame(Questions = c('Q101', 'Q102'),
Answer = c('A1', 'A1'),
Option = matrix(paste0('A', 1:5), nrow=2, ncol=5),
Type = 'radio')
new2## Questions Answer Option.1 Option.2 Option.3 Option.4 Option.5 Type
## 1 Q101 A1 A1 A3 A5 A2 A4 radio
## 2 Q102 A1 A2 A4 A1 A3 A5 radioThe conceptual difficulty here is that there are no estimated IRT parameters available for these items yet, because we have no sample data from which to estimate them.
In order to include these two new items in the CAT we must include some provisional IRT parameters to, in essence, ‘trick’ mirtCAT() into thinking these items have valid IRT parameters. However, we’ll flag which of these items contain arbitrary parameters by using the constrain input list, specifically the not_scored element.
First, let’s build our new objects to crate a CAT bank with 102 items (compared to the original 100 items above).
# new stimuli information
newdf <- rbind(df, new2)
tail(newdf)## Questions Answer Option.1 Option.2 Option.3 Option.4 Option.5 Type
## 97 71 + 147 = ? 218 206 198 234 202 218 radio
## 98 41 + 116 = ? 157 153 167 155 151 157 radio
## 99 67 + 131 = ? 198 214 202 194 198 182 radio
## 100 39 + 160 = ? 199 202 199 195 197 204 radio
## 101 Q101 A1 A1 A3 A5 A2 A4 radio
## 102 Q102 A1 A2 A4 A1 A3 A5 radio# new parameter data.frame and model
newpars <- rbind(pars, data.frame(a1 = c(0,0), d = c(0,0), g=0))
tail(newpars) # parameter values for unscored items are arbitrary## a1 d g
## 97 0.8690258 0.6202102 0.2
## 98 1.5895661 -0.9659032 0.2
## 99 1.6353797 0.1626547 0.2
## 100 2.3078933 -2.0782375 0.2
## 101 0.0000000 0.0000000 0.0
## 102 0.0000000 0.0000000 0.0newmod <- generate.mirt_object(newpars, '3PL')
coef(newmod, simplify=TRUE)## $items
## a1 d g u
## Item.1 0.850 0.415 0.2 1
## Item.2 1.327 -0.475 0.2 1
## Item.3 1.691 0.066 0.2 1
## Item.4 0.604 -0.502 0.2 1
## Item.5 1.389 -0.826 0.2 1
## Item.6 1.422 0.167 0.2 1
## Item.7 1.028 -0.896 0.2 1
## Item.8 1.037 0.168 0.2 1
## Item.9 1.031 0.355 0.2 1
## Item.10 0.935 -0.052 0.2 1
## Item.11 1.058 -0.196 0.2 1
## Item.12 0.905 -0.649 0.2 1
## Item.13 0.968 -1.110 0.2 1
## Item.14 1.245 0.849 0.2 1
## Item.15 1.629 0.022 0.2 1
## Item.16 1.182 0.831 0.2 1
## Item.17 1.048 -1.244 0.2 1
## Item.18 0.929 0.169 0.2 1
## Item.19 0.950 0.673 0.2 1
## Item.20 2.521 -0.026 0.2 1
## Item.21 1.272 -0.191 0.2 1
## Item.22 1.054 -0.782 0.2 1
## Item.23 1.070 2.058 0.2 1
## Item.24 1.402 0.751 0.2 1
## Item.25 0.992 1.824 0.2 1
## Item.26 0.791 0.080 0.2 1
## Item.27 1.451 -0.631 0.2 1
## Item.28 0.898 -1.513 0.2 1
## Item.29 1.216 -0.636 0.2 1
## Item.30 0.922 0.226 0.2 1
## Item.31 1.700 1.014 0.2 1
## Item.32 1.059 0.253 0.2 1
## Item.33 0.987 -1.172 0.2 1
## Item.34 1.051 0.669 0.2 1
## Item.35 0.749 -1.650 0.2 1
## Item.36 0.860 -0.366 0.2 1
## Item.37 0.635 -0.316 0.2 1
## Item.38 0.817 -1.948 0.2 1
## Item.39 1.118 0.920 0.2 1
## Item.40 1.062 -0.623 0.2 1
## Item.41 1.887 -0.334 0.2 1
## Item.42 0.886 1.395 0.2 1
## Item.43 0.945 0.637 0.2 1
## Item.44 1.123 -0.108 0.2 1
## Item.45 0.906 0.514 0.2 1
## Item.46 0.913 0.399 0.2 1
## Item.47 0.876 1.663 0.2 1
## Item.48 0.839 0.276 0.2 1
## Item.49 1.044 0.506 0.2 1
## Item.50 1.052 0.348 0.2 1
## Item.51 0.710 -0.377 0.2 1
## Item.52 1.026 0.098 0.2 1
## Item.53 0.876 1.639 0.2 1
## Item.54 0.901 -0.876 0.2 1
## Item.55 1.163 0.122 0.2 1
## Item.56 1.446 1.362 0.2 1
## Item.57 2.002 -0.235 0.2 1
## Item.58 0.969 -1.053 0.2 1
## Item.59 1.977 -0.870 0.2 1
## Item.60 0.863 -0.390 0.2 1
## Item.61 1.487 -0.847 0.2 1
## Item.62 2.624 -0.261 0.2 1
## Item.63 1.209 -0.414 0.2 1
## Item.64 0.999 -0.183 0.2 1
## Item.65 1.219 0.407 0.2 1
## Item.66 2.082 0.625 0.2 1
## Item.67 0.868 1.678 0.2 1
## Item.68 1.841 -0.069 0.2 1
## Item.69 1.820 -0.321 0.2 1
## Item.70 1.351 1.471 0.2 1
## Item.71 1.224 1.704 0.2 1
## Item.72 1.065 0.043 0.2 1
## Item.73 1.094 -0.333 0.2 1
## Item.74 1.484 -1.822 0.2 1
## Item.75 2.273 1.411 0.2 1
## Item.76 1.166 -0.838 0.2 1
## Item.77 0.805 -1.124 0.2 1
## Item.78 0.983 3.044 0.2 1
## Item.79 1.320 0.235 0.2 1
## Item.80 1.111 -0.033 0.2 1
## Item.81 1.158 -2.732 0.2 1
## Item.82 1.161 -0.100 0.2 1
## Item.83 0.809 0.976 0.2 1
## Item.84 1.159 0.414 0.2 1
## Item.85 1.576 0.912 0.2 1
## Item.86 1.506 1.984 0.2 1
## Item.87 1.441 1.169 0.2 1
## Item.88 1.082 -0.509 0.2 1
## Item.89 1.153 0.704 0.2 1
## Item.90 0.854 -0.198 0.2 1
## Item.91 1.202 -0.538 0.2 1
## Item.92 1.319 -2.856 0.2 1
## Item.93 2.038 -0.790 0.2 1
## Item.94 1.649 0.488 0.2 1
## Item.95 1.053 2.168 0.2 1
## Item.96 1.359 0.501 0.2 1
## Item.97 0.869 0.620 0.2 1
## Item.98 1.590 -0.966 0.2 1
## Item.99 1.635 0.163 0.2 1
## Item.100 2.308 -2.078 0.2 1
## Item.101 0.000 0.000 0.0 1
## Item.102 0.000 0.000 0.0 1
##
## $means
## F1
## 0
##
## $cov
## F1
## F1 1With these new objects built, we’re ready to use mirtCAT(). However, if we are interested in how/when these new items should be administered, while also desiring that the items selection method remain adaptive (e.g., with the maximum-information criteria), then we’ll need a customNextItem() function definition to control the item selection scheme.
In the following code, items are selected using the MI criteria. However, if item 101 or 102 have not yet been administered then they will be randomly selected 10% of the time until both of these items appear in the CAT session. To get a better feel for how this item selection function behaves, simply uncomment the browser() line and walk-through the object state using R’s internal debugging functions.
customNextItem <- function(design, person, test){
# browser()
items_answered <- na.omit(extract.mirtCAT(person, 'items_answered'))
new_items <- 101:102
# if there are any new items to be administered, then 10% of
# the time randomly pick one to administer
if(!all(new_items %in% items_answered) && runif(1) < .1){
pick <- new_items[!(new_items %in% items_answered)]
if(length(pick) == 1) return(pick)
else return(sample(pick, 1))
}
# othewise, pick item with the MI criteria
item <- findNextItem(person=person, design=design, test=test,
criteria = 'MI')
item
}Finally, we have all the components necessary to pass to mirtCAT() in order to control these new special items. Let’s run a test case to see how this selection process behaves.
# test pattern
set.seed(1)
pat <- generate_pattern(newmod, Theta = 0, df = newdf)
result <- mirtCAT(newdf, newmod, method = 'EAP', criteria = 'MI', start_item = 'random',
local_pattern = pat,
design = list(min_SEM = .2,
# ensure these items are not scored
constraints = list(not_scored = 101:102),
customNextItem=customNextItem))Notice the design element constraints = list(not_scored = 101:102). This tells mirtCAT() to never use item 101 or 102 in the scoring of the \(\theta\) components. Hence, when you inspect the properties of the CAT session whenever item’s 101 and 102 are administered the previous \(\hat{\theta}\) and associated \(SE\) estimates are identical to the previous response pattern state (because nothing changes in the CAT, only the responses for these items are collected). For ease of location, the thetas_SE_history retains the labels of these new items so that they can be easily located from the estimation history.
summary(result)## $final_estimates
## Theta_1
## Estimates 0.2653299
## SEs 0.2062403
##
## $raw_responses
## [1] "3" "1" "3" "5" "2" "4" "2" "2" "5" "5" "1" "5" "4" "1" "1" "5" "3"
## [18] "3" "5" "3" "4" "1" "1" "1" "5" "3" "3" "4" "2" "1" "5" "1" "4" "1"
## [35] "4" "4" "4" "2" "2" "1" "4" "4" "2" "4" "5" "3" "5" "5" "3" "2" "5"
## [52] "3" "2" "3" "1" "5" "1" "1" "3" "5" "2" "1" "5" "2" "2" "1" "3" "4"
## [69] "4" "2" "2" "1" "1" "2" "5" "2" "4" "2" "5" "2" "2" "5" "4" "1" "4"
## [86] "4" "3" "1" "2" "2" "2" "3" "3" "2" "1" "5" "1" "5" "4" "2" "5" "2"
##
## $scored_responses
## [1] 0 1 1 1 0 1 1 0 1 1 1 0 1 1 0 1 0 1 0 1 1 1 1 0 1 1 1 1 0 0 0 1 1 1 0
## [36] 1 0 0 0 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 0
## [71] 1 1 1 1 1 1 0 0 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 0
##
## $items_answered
## [1] 62 75 66 20 57 68 41 93 69 59 3 15 99 94 98 31 6
## [18] 85 27 96 79 61 101 100 24 2 5 21 63 65 102 87 55 82
## [35] 91 29 84 14 56 16 44 80 89 70 72 32 73 50 8 11 39
## [52] 49 9 52 86 34 88 76 64 40 18 10 22 30 19 43 46 45
## [69] 7 74 71 90 1 48 97 12 36 60 17 58 26 54 13 33 83
## [86] 42 51 25 23 53 77 47 67 37 95 28 4 35 38 92 78 81
##
## $thetas_history
## Theta_1
## [1,] 0.000000000
## [2,] -0.621601266
## [3,] -0.342265947
## [4,] -0.118285616
## [5,] 0.102724421
## [6,] -0.135465547
## [7,] 0.008212652
## [8,] 0.136972907
## [9,] 0.001182598
## [10,] 0.105174077
## [11,] 0.210466438
## [12,] 0.277790579
## [13,] 0.148955294
## [14,] 0.207744033
## [15,] 0.252736938
## [16,] 0.188257534
## [17,] 0.219217501
## [18,] 0.131153423
## [19,] 0.161702056
## [20,] 0.106440907
## [21,] 0.138421785
## [22,] 0.171682003
## [23,] 0.219261742
## [24,] 0.219261742
## [25,] 0.183569627
## [26,] 0.206654703
## [27,] 0.242809368
## [28,] 0.281827318
## [29,] 0.311662347
## [30,] 0.268141771
## [31,] 0.208929062
## [32,] 0.208929062
## [33,] 0.224342399
## [34,] 0.247698177
## [35,] 0.272508684
## [36,] 0.236214508
## [37,] 0.265525880
## [38,] 0.214491607
## [39,] 0.153452211
## [40,] 0.074913641
## [41,] 0.091802979
## [42,] 0.116034705
## [43,] 0.079182758
## [44,] 0.096112970
## [45,] 0.107416764
## [46,] 0.072421562
## [47,] 0.091370485
## [48,] 0.062493345
## [49,] 0.080219017
## [50,] 0.098632514
## [51,] 0.069619542
## [52,] 0.082721445
## [53,] 0.044771085
## [54,] 0.061379026
## [55,] 0.030552600
## [56,] 0.038610997
## [57,] 0.052614177
## [58,] 0.074249070
## [59,] 0.097818342
## [60,] 0.115862508
## [61,] 0.094931858
## [62,] 0.109541845
## [63,] 0.125264161
## [64,] 0.106894040
## [65,] 0.120577481
## [66,] 0.131543415
## [67,] 0.142506114
## [68,] 0.154417124
## [69,] 0.165376377
## [70,] 0.184139426
## [71,] 0.171511459
## [72,] 0.177453615
## [73,] 0.190990271
## [74,] 0.201273803
## [75,] 0.212047771
## [76,] 0.221158527
## [77,] 0.236472671
## [78,] 0.218526347
## [79,] 0.200999609
## [80,] 0.218728012
## [81,] 0.205606477
## [82,] 0.216344370
## [83,] 0.231506489
## [84,] 0.218992510
## [85,] 0.235413584
## [86,] 0.208344145
## [87,] 0.213512216
## [88,] 0.224539671
## [89,] 0.228458296
## [90,] 0.231838799
## [91,] 0.235902510
## [92,] 0.249043580
## [93,] 0.252956537
## [94,] 0.256764797
## [95,] 0.244258409
## [96,] 0.247181607
## [97,] 0.260998833
## [98,] 0.250361679
## [99,] 0.261425644
## [100,] 0.256199405
## [101,] 0.268225102
## [102,] 0.269427689
## [103,] 0.265329867
##
## $thetas_SE_history
## Theta_1
## 1.0000000
## 0.7548353
## 0.6722111
## 0.6085983
## 0.5589631
## 0.5253759
## 0.4842760
## 0.4549861
## 0.4345534
## 0.4086428
## 0.3933434
## 0.3764879
## 0.3696571
## 0.3545867
## 0.3431957
## 0.3349382
## 0.3258627
## 0.3226753
## 0.3146059
## 0.3103663
## 0.3039357
## 0.2986620
## 0.2957117
## theta_SE 0.2957117
## 0.2874217
## 0.2829751
## 0.2797514
## 0.2774798
## 0.2746487
## 0.2718502
## 0.2706582
## theta_SE 0.2706582
## 0.2675727
## 0.2649813
## 0.2627104
## 0.2604413
## 0.2586356
## 0.2576300
## 0.2578025
## 0.2599520
## 0.2569882
## 0.2542633
## 0.2539774
## 0.2512933
## 0.2490171
## 0.2488317
## 0.2464954
## 0.2459742
## 0.2437226
## 0.2416602
## 0.2412598
## 0.2392847
## 0.2397172
## 0.2376866
## 0.2377494
## 0.2359365
## 0.2340835
## 0.2325220
## 0.2313024
## 0.2298529
## 0.2291498
## 0.2277964
## 0.2265310
## 0.2258159
## 0.2245711
## 0.2234115
## 0.2223018
## 0.2212498
## 0.2202730
## 0.2196677
## 0.2183117
## 0.2174459
## 0.2166732
## 0.2158923
## 0.2151498
## 0.2144272
## 0.2139073
## 0.2133364
## 0.2128122
## 0.2126050
## 0.2119369
## 0.2112999
## 0.2109037
## 0.2102435
## 0.2100055
## 0.2100374
## 0.2095240
## 0.2090744
## 0.2086326
## 0.2082196
## 0.2078254
## 0.2076548
## 0.2072842
## 0.2069293
## 0.2066749
## 0.2063319
## 0.2063853
## 0.2061497
## 0.2062183
## 0.2059148
## 0.2067795
## 0.2066423
## 0.2062403