How the Length and Characteristics of Routing Module Affect Ability Estimation in ca-MST?

In this study, how the length and characteristics of routing module in different panel designs affect measurement precision is examined. In the scope of the study, six different routing module length, nine different routing module characteristics, and two different panel design are handled. At the end of the study, the effects of conditions on measurement precision are examined by estimating correlation, RMSE and SEE for final theta estimation values. According to the results, it is seen that as the routing module length increases, lower RMSE and higher correlation are obtained. Nevertheless; good results were obtained in both panel designs even if the a and c distributions were outside the range suggested in the literature where routing module length is 15 and above. In addition, the three-stage panel design generally yielded better results in terms of measurement accuracy.


Introduction
Computer adaptive test (CAT) which is conducted on a computer and the process of which depends on the test-taker's answers has a wide range of application internationally. While CAT, in which item level based individualization is done, is a highly popular method in terms of providing fast and reliable results in proficiency estimation, ca-MST (computerized multistage testing), in which module level based individualization is done, has recently become more popular.
ca-MST is a good combination of linear test forms and traditional item-level CAT since it has both sides' advantages. One thing is that with its quality of being adaptive, ca-MST is more effective than linear tests. Another advantage is that in ca-MST test developers can review test forms before application and test takers are allowed to review and revise answers [26].
As it can be seen in Figure 1 below, the algorithm based approach of a ca-MST refers to the tests in which the algorithm selects the items which are previously combined and called modules, and this test is constructed in stages. At the first stage of testing (module I in Figure 1) in a ca-MST an initial set of items, which sometimes can be named as routing module, is administered to the test takers. Based on their performance, then at the second stage, test takers are scattered to one of the many different modules whose adaptation is based on the test taker's estimated proficiency level. The design of the ca-MST defines the number of stages and the number of available modules in each stage [22].
An example of a ca-MST application constructed according to 1-2-3 panel design is represented in Figure 1 below.
Here, one module (routing module) in the first stage, two modules (easy and difficult) in the second stage, and three modules (easy, medium, and difficult) in the third stage take place. The six possible pathways for test takers who take the test of that panel design are Routing-Easy-Easy, Routing -Easy-Medium, Routing-Easy-Hard, Routing-Hard-Easy, Routing-Hard-Medium, Routing-Hard-Hard. These pathways have an important role in test design, test assembly and application [13].
The first individualization in ca-MST application is provided by the scores that the individuals got from the routing module. It is expected that the routing module is usually constructed with enough items and average difficulty [22,23]. Besides, a long routing module causes less fault in deciding on the module that an individual will take in the next stage [11,16].
During the administration of an MST, several factors such as stage number, module numbers per stage, length of modules and item difficulty distribution, may affect the precision of the measurement [23]. The longer the test lengths are, the more accurate ability estimation can be achieved. This is true especially when a certain trait level cannot be produced by the routing module which is not long enough. In this case, the examinees' routing module could be wrongly assigned [20]. Normally in a fixed test length, unless the routing module is not well adjusted in terms of length, second and third stages have the shortage of items and also the test cannot be adapted anymore. On the contrary, when the length of the routing module is not long enough, the measurement of ability for the second-stage adaptation can be not good enough [16]. Moreover, the measurement precision of the ability estimation is greatly affected by the statistical characteristics of the routing module [11]. For these reasons, the length and characteristics of the routing module has an important role both in ability estimation and for new researches.
In the literature, it is seen that there are also different studies which show how basic components affect the proficiency estimation when they are exposed to different conditions [2,3,4,12,14,17,21,22] in addition to the fact that different applications have been tried to decrease the disadvantages of ca-MST application [9,25].
In this study, the effect of routing module which is one of the basic components of ca-MST on measurement precision in condition that it has different lengths and item characteristics. While b parameters of the items in the routing module is kept constant in every condition, it is examined whether the distribution of a and c parameters are in between the value ranges suggested in the literature. In routing module length, on the other hand, the effect of the values which are above or below the suggested values of module length in literature on the ability estimation is investigated. Both variables are examined in both two and three-stage panel designs.

Method
In this study, it is aimed to examine the effect of routing module on proficiency estimation when it has different length and item characteristics.
Data construction: MSTGen [8] simulation software tool is used to construct simulee and item data within the scope of the study. While 5000 simulees are simulated in the study, these individuals' proficiency distribution is derived from a normal distribution (N(0,1)), the average of which is 0 and the standard deviation of which is 1. In the study, for ca-MST two different panel designs, one of which is 1-3 [1,15,16,18] and the other is 1-3-3 [5,7,14], are simulated. In ca-MST, module and test length display characteristics depending on the characteristic being measured [24]. Studies in the literature reveal that while module lengths vary between 5 and 30, 20-item module is regarded as the most popular length [11,24]. Besides, in previous MST studies, the dichotomous items ranging from 33 to 60 were used for test lengths [3]. In the study, routing module having different lengths is considered as the independent variable, and its effect on the proficiency estimation of different lengths is examined. For this reason, the length of routing module varies between 5 and 30 items. In the second stage of panel design 1-3, modules length is 20 items, whereas in the second and third stage of panel design 1-3-3 modules consist of 10 items. According to that, item numbers in the stages and the test length of ca-MST are presented in Table 1.  As seen in Table 1, while the lengths of routing module vary as 5, 10, 15, 20, 25, and 30 items, the test lengths an simulee takes are made up of 25, 30, 35, 40, 45, and 50 items.
b parameters of the items taking place in the modules are chosen as to create normal distribution and theta values which are defined in the panel designs of the study are chosen as to be the average; and standard deviation is defined as 0.3. The routing module is constructed to reflect one TIF center (theta points of 0.0) and two/three stage modules are constructed to reflect three TIF centers (theta points of -1.0, 0.0 and 1.0). The items in all the modules except the ones in the routing module are formed so that parameter a has a uniform distribution of [0.8; 2.0] and parameter c has a uniform distribution of [0.05; 0.2] [6,19]. While parameter a and c in routing module are formed to have a uniform distribution, value ranges differentiated. According to that, parameter a and c of the items in routing module are presented in Table 2. As seen in Table 2, it is seen that the items in routing module number 3 have the most distinctiveness and the least guessing parameters, while the items in routing module number 7 have the least distinctiveness and the most guessing parameters.
As a result, in this study the effect of 108 different simulation conditions (= 2 panel designs x 6 routing module lengths x 9 routing module characteristics) on proficiency estimation is examined.
ca-MST Components: For ca-MST application, MSTGen [8] simulation software tool is used. In the scope of the study, Minimum Fisher Information method (MFI) is used as the module selection method. In temporary proficiency level and ultimate proficiency estimation, Expected Posteriori Distribution is used as the estimation method. As the test start rule, random item selection method whose item difficulty level varies between -0.

Findings
Correlation values obtained in the study are presented in Figure 2, those of RMSE are in Figure 3, and those of SEE for final theta are presented in Figure 4.
In Figure 2, it is seen that the highest correlation value in different lengths of routing modules in 1-3 panel design is obtained in RMC3, and the highest RMSE value is in RMC7. While RMC3 condition has the best distribution of a and c parameters in the routing module, RMC7 has the worst. As the length of the routing module increases, correlation values have also increased. When looking at the values in hand, it is clear that in different RMC conditions the lowest correlation values are in RML5 and the highest ones are in RML30. It is also seen that correlation values obtained in RMC2 and RMC3 conditions are very close to each other. However, the difference between correlation values increases in RMC8 and RMC9 conditions in which the distribution of a parameter is the same, but that of c parameter worsens.
The distribution of values obtained in 1-3-3 panel design also resemble to the one in 1-3 panel design. According to that, the highest correlation values in different lengths of routing modules which have the same routing module characteristics are obtained in RMC3, and the lowest ones are received in RMC7. The correlation values increase as the length of routing module in the same RMC condition increases. When the routing module length condition is considered, the lowest correlation values are derived in RML5, and the highest ones are in RML30. When 1-3 and 1-3-3 panel designs are compared, it is observed that correlation values get closer to each other as the panel design switches from two-stage to three-stage structure. Increasing situation of correlation values can be more clearly seen in routing modules which are short and have bad distribution of a and c parameters. In RMC3 condition which has good routing module length and distribution of a and c parameters, the length of RML30 has almost the same correlation value; however, the biggest difference is observed in the correlation value of the length of RML5 in RMC7 condition.
In Figure 3, it is seen that the lowest RMSE values in different lengths of routing modules in 1-3 panel design are obtained in RMC3, and the highest RMSE values are in RMC7. While RMC3 condition has the best distribution of a and c parameters in the routing module, RMC7 has the worst. As the length of the routing module increases, RMSE values have decreased. When looking at the values in hand, it is clear that the highest RMSE values are in RML5 and the lowest ones are in RML30. In RMC1, RMC4, and RMC7 conditions, a parameters in routing modules have the worst distribution, and for this reason RMSE values have been higher compared to other conditions.   The distribution of values obtained in 1-3-3 panel design also resemble to the one in 1-3 panel design. According to that, the lowest RMSE values in different lengths of routing modules are obtained in RMC3, and the highest ones are received in RMC7. Whereas RMC3 condition has the best distribution of a and c parameters in the routing module, RMC7 has the worst. As the length of the routing module increases, RMSE values have decreased. When looking at the values in hand, it is seen that the highest RMSE values are in RML5 and the lowest ones are in RML30. In RMC1, RMC4, and RMC7 conditions, a parameter in routing modules have the worst distribution, and for this reason RMSE values have been higher compared to other conditions. When 1-3 and 1-3-3 panel designs are compared, it is observed that RMSE values have decreased as the panel design switches from two-stage to three-stage structure. This decrease can be more clearly seen in routing modules which are short and have bad distribution of a and c parameters. While in RMC3 condition which has good routing module length and distribution of a and c parameters, the length of RML30 has almost the same RMSE value, the biggest difference is observed in the RMSE value which RML5's length has in RMC7 condition.
When Figure 4 is examined, in 1-3 panel design, it is seen that in the peak-end proficiency distributions of different routing module lengths SEE values generally come closer to each other and they have high level of standard error. However, in RM7 and RM9 routing module conditions of RML5 condition there is a higher SEE in [2,4] proficiency parameter distribution. With the increase of the length of the routing module, this situation has disappeared. As the length of the routing module increases, the lowest SEE values are obtained in different RMC conditions. Especially in the [-2,2] range of the proficiency parameter distribution, close SEE values are obtained in the routing module of the same lengths. In 1-3-3 panel design, too, it is seen that in the peak-end proficiency distributions of different routing module lengths SEE values generally come closer to each other and they have high level of standard error. However, when the panel design is 1-3-3, SEE difference that is received in the same RMC condition but different lengths of the routing module. Especially in RMC1, RMC4, and RMC6 conditions this situation is more obvious. As the panel design becomes three-staged, the high SEE values in the distribution of [2,4]

Conclusions
In this study, the effect of routing module length and routing characteristics in different panel designs on measurement precision is investigated. According to the results gathered in the end, it is observed that as the routing module length increases correlation values also increase, whereas RMSE values decrease. This situation occurs in both two-stage and three-stage panel designs. However, especially in conditions which have short routing module length, better results are reached in three-stage panel design.
For instance, when routing module length in RMC7 condition is 5 items, correlation value is .94 in two-stage panel design, while it is .96 in three-stage panel design. This value received in three-stage panel design is higher than many correlation values obtained in two-stage panel design. In RMC7xRLM5 condition which has the lowest value in three-stage panel design, in while a and c parameters have distributions other than the ones suggested in the literature in terms of routing module characteristic, it is very close to the correlation value gathered in RMC9xRML10 condition in two-stage panel design despite being the shortest routing module. However, in two-stage panel design RMC9xRML10 condition, different from RMC7xRML5 condition, is simulated in a better range than the range of a parameter and has the test length of 5 more items. Similar results are observed in examining RMSE values. It is seen that RMSE value in RMC7xRML5 condition is the highest in two-stage panel design. However, RMSE value obtained in three-stage panel design is seen to be closer to RMSE value in RMC9xRML10 condition in two-stage panel design.
While the changes of correlation and RMSE values obtained with the stage increase do not cause a significant difference in conditions whose long routing modules (RML25 and RML30), especially a and c parameters, are in ranges suggested in the literature (RMC2, RMC3, RMC5 and RMC6), they are very obvious in conditions where the ranges of short routing modules (RML5 and RML10), especially a and c parameters, are outside the ranges suggested in the literature (RMC1, RMC4, RMC7, RMC 8 and RMC9). These differences are much more in short routing module conditions (RML5 and RML10). In short, the distribution of a and c parameters affect panel designs which have short routing module more.
SEE values for the final estimation get closer to each other in almost every condition in peak-end proficiency distribution and received high values. In the distribution in [-2, 2] range of proficiency parameter, SEE values in different routing module lengths in two-stage panel design are close to the ones in the same RMC condition. The difference among routing module lengths is more obvious in conditions where a and c parameters are outside the range suggested in the literature (RMC1, RMC4, RMC7, RMC 8 and RMC9) compared to the others that are inside. This difference is observed to decrease as the stage number increases in panel design.
Consequently, in conditions where routing module length is 15 and more, even though a and c distribution values of items in routing module are outside the range suggested in the literature, a better measurement precision is received in both two-staged and three-staged panel designs. According to that, if a short panel design is to be done, it is suggested that routing module length should not be under 15 items. If a module whose distribution values of a and c parameters in routing module are outside the suggested range and is to be used and the length is to be less than 15 items, it is strongly suggested that the stage number should be increased in panel design. As the item number in routing module increases, correlation values in each stage have increased but RMSE values have decreased. With the increase in routing module length, although there is a continuous improvement in measurement precision, it is necessary that routing module length should not be too much. In this case, researchers should consider the aims of their study and decide the routing module length accordingly.
The results obtained from the study generally support the findings in the literature [3,11,16]. According to the studies on routing module length in MST, Kim and Plake (1993) reported that the length of the routing module was important in reducing the size of the proficiency estimation errors, whereas Patsula (1999) and Kim, Tseng, Chung, and Dodd (2008) indicated that the length of the routing test has a little effect on the ability estimation. Considering the number of stage studies; it is seen that increasing the number of stages from two to three increases the measurement precision (10,16).
Despite the encouragement for the evidence from the computer simulations, generalizing it to the real data can be questioned; that is why the findings received from computer simulations should be tested via real data. In addition to that, because the obtained results will be affected by the test characteristics and the item pool, generalizability of the certain results can be confined [16]. For this reason, conditions examined in the study should also be tested with the real values. In this study, routing module length and characteristics in different panel designs which are derived from the same item pool are examined by the use of one panel. Moreover, item exposure control and content balancing are not included in the study. Therefore, for future studies, researchers may conduct similar studies in different number of panels and in different panel designs. Different interpretations may be done with different outputs including item exposure control and content balancing strategies. In this study, item number is fixed to 10 in three-staged design, so in different studies researchers may want to change the number of items in stages.