-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathREADME.Rmd
214 lines (175 loc) · 10.6 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
---
output:
github_document:
html_preview: TRUE
---
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
fig.path = "README-",
tidy = TRUE
)
```
# caMST
[](http://cran.r-project.org/package=caMST)
[](http://travis-ci.org/AnthonyRaborn/caMST)
[](https://codecov.io/gh/AnthonyRaborn/caMST)
[](https://cran.r-project.org/package=caMST)
[](https://cran.r-project.org/package=caMST)
[](https://doi.org/10.5281/zenodo.3532057)
## Installation
```{r gh-installation, eval = FALSE}
install.packages('caMST') # CRAN-tastic
# install.packages("devtools")
devtools::install_github("AnthonyRaborn/caMST") # the stable developmental version; add the argument `ref = 'devel'` for the developmental version with no stability guarantee
```
## Usage
Here are some examples demonstrating how to use this package for various adaptive test frameworks.
```{r loading functions, echo = FALSE, eval = FALSE}
source("R/mixed_adaptive_test_function.R")
source("R/multistage_test_function.R")
source("R/module_selection_function.R")
source("R/routing_item_selection.R")
source("R/iterative_maximum_likelihood_theta_estimation.R")
source("R/computerized_adaptive_test_function.R")
```
### Computer Adaptive Multistage Testing (CMT)
```{r CMT example}
# load the package
library(caMST)
# using simulated test data
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responses data
data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design
data(mst_only_items) # the CMT item bank
data(mst_only_matrix) # the matrix specifying how the item data frame relates to the modules
# run the CMT model using MFI for module selection
resultsMFI <-
multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix,
transition_matrix = example_transition_matrix,
method = "BM", response_matrix = example_responses,
initial_theta = 0, model = NULL, n_stages = 3, test_length = 18)
resultsMFI # print a summary of the results
# run the CMT model using MLWMI for module selection
resultsMLWMI <-
multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix,
transition_matrix = example_transition_matrix,
method = "BM", response_matrix = example_responses,
initial_theta = 0, model = NULL, n_stages = 3,
module_select = "MLWMI", test_length = 18)
resultsMFI # print a summary of the results
# how good were the MFI estimates?
data.frame("True Theta" = example_thetas,
"Estimated Theta" = [email protected],
"CI95 Lower Bound" = [email protected] -
1.96*[email protected],
"CI95 Upper Bound" = [email protected] +
1.96*[email protected])
# how good were the MLWMI estimates?
data.frame("True Theta" = example_thetas,
"Estimated Theta" = [email protected],
"CI95 Lower Bound" = [email protected] -
1.96*[email protected],
"CI95 Upper Bound" = [email protected] +
1.96*[email protected])
```
The theta estimates under CMT using the default module selection are close as all the estimates fall within the 95% confidence interval. Using the "MLWMI" module selection produces different, but similarly accurate estimates.
### Mixed Computerized Adaptive Multistage Testing (Mca-MST)
This is a method that follows the traditional multistage testing framework with one exception: the routing stage utilizes item-level adaptation to try to better estimate individual examinees' abilities before administering the additional stages. The current implementation allows for any three-stage design with a single routing module (e.g., 1-2-3, 1-5-3, etc.), though this will be extended for any arbitrary number of stages in future updates.
Here is an example with a 1-3-3 design that has 18 items, where each stage has the same number of items, maximum Fisher information is used for all item- and module-level adaptations, and the expected a posteriori estimator is used for the provisional and final ability estimates.
```{r McaMST example}
# using simulated test data
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responses data
data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design
data(cat_items) # the items designated for use in the routing module with item-level adaptation
data(mst_items) # the items designated for use in the second and third modules with module-level adaptation
data(example_module_items) # the matrix specifying how the item data frame relates to the modules
# what does the data look like?
example_thetas
example_responses[,1:5] # first 5 items
example_transition_matrix
head(cat_items) # 564 items to choose from for the first module
head(mst_items) # 18 items in the three 2nd stage modules and another 18 in the three 3rd stage modules
head(example_module_items, 10) # notice that there are only 6 items in the first module!
# run the Mca-MST model
results <- mixed_adaptive_test(response_matrix = example_responses,
cat_item_bank = cat_items, initial_theta = 0,
method = "EAP", item_method = "MFI",
cat_length = 6, cbControl = NULL, cbGroup = NULL,
randomesque = 1, mst_item_bank = mst_items,
modules = example_module_items,
transition_matrix = example_transition_matrix,
n_stages = 3)
results # prints a summary of the results
# How good was our estimate of the individual's abilities?
data.frame("True Theta" = example_thetas,
"Estimated Theta" = [email protected],
"CI95 Lower Bound" = [email protected] -
1.96*[email protected],
"CI95 Upper Bound" = [email protected] +
1.96*[email protected])
```
For these five individuals, the estimated ability level appears fairly close at worst and almost precisely correct at best. In all cases, the 95% confidence interval includes the simulated ability level, indicating that the Mca-MST method works well for these individuals.
### Number-Correct (NC) Scoring with CMT
In some cases, applied testing demand the use of NC scoring (even if it's not psychometrically valid). To facilitate NC tests, the `multistage_test()` function accepts a list with the cutoff score for each route specified. Otherwise, the rest of the inputs are the same.
```{r NC multistage_test example}
data(mst_only_items)
data(example_thetas)
data(example_module_items)
data(example_transition_matrix)
data(example_responses)
# these are the same as in the previous example
# however, for NC scoring, the lsit of cutoff values needs to be specified
# create nc_list as explained in 'details'
nc_list = list(
module1 = c(4, 5, 7),
module2 = c(8, 14, Inf),
module3 = c(8, 14, 20),
module4 = c(-Inf, 14, 20)
)
# this is the ONLY difference currently! Everything else remains the same
# run the example
nc.results <- multistage_test(
mst_item_bank = mst_only_items,
modules = example_module_items,
transition_matrix = example_transition_matrix,
method = "BM",
response_matrix = example_responses,
initial_theta = 0,
model = NULL,
n_stages = 3,
test_length = 18,
nc_list = nc_list
)
# printing a MST using NC scoring also shows the NC scoring method used
nc.results
# How well does NC scoring estimate the individual's abilities?
# Using the estimation procedure from Baker for theta
data.frame("True Theta" = example_thetas,
"Estimated Theta" = [email protected],
"CI95 Lower Bound" = [email protected] -
1.96*[email protected],
"CI95 Upper Bound" = [email protected] +
1.96*[email protected])
```
With this example data, the NC scoring does about as well as the previous methods in recovering the ability levels of these individuals.
### Computer Adaptive Testing (CAT)
Using the same individuals, we can administer a computerized adaptive test with item-level adaptation. Generally, CAT performs better than the other methods at the cost of algorithm complexity and potential difficulties maintaining item security and content balance. However, it is the golden standard for computer-based test and serves as a great test for 'best-case scenarios'.
```{r CAT example}
set.seed(12345) # to keep randomesque selecting the same items
# using simulated test data
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responsesdata
data(cat_items) # using just the CAT-only routing items for the entire CAT test
catResults <- computerized_adaptive_test(cat_item_bank = cat_items, response_matrix = example_responses, randomesque = 5, maxItems = 18,
nextItemControl = list(criterion = "MFI", priorDist = "norm", priorPar = c(0, 1), D = 1, range = c(-4, 4), parInt = c(-4, 4, 33), infoType = "Fisher", random.seed = NULL, rule = "precision", thr = .3, nAvailable = NULL, cbControl = NULL, cbGroup = NULL))
catResults
data.frame("True Theta" = example_thetas,
"Estimated Theta" = [email protected],
"CI95 Lower Bound" = [email protected] -
1.96*[email protected],
"CI95 Upper Bound" = [email protected] +
1.96*[email protected])
```
The CAT method, using the precision rule with a value of .3 (i.e., stopping when the SEM is less than .3) and a maximum test length of 18, produces theta confidence intervals that encompass the true theta in each case. It is important to note, however, that only one individual saw less than 18 items (and he only saw 17), and this individual is the only one with an SEM of less than .3, so in reality we would probably want to increase the maxItems each individual can see in order to get a better estimate of their abilities. However, the theta estimates are accurate and all fall within the 95% CIs.