Here, we present a protocol to develop the computerized adaptive testing system of the functional assessment of stroke (CAT-FAS). The CAT-FAS can simultaneously assess four functions (two motor functions [upper and lower extremities], postural control, and basic activities of daily living) with sufficient reliability and administrative efficiency.
The computerized adaptive testing system of the functional assessment of stroke (CAT-FAS) can simultaneously assess four functions (motor functions of the upper and lower extremities, postural control, and basic activities of daily living) with sufficient reliability and administrative efficiency. CAT, a modern measurement method, aims to provide a reliable estimate of the examinee’s level of function rapidly. CAT administers only a few items whose item difficulties match an examinee’s level of function and, thus, the administered items of CAT can provide sufficient information to reliably estimate the examinee’s level of function in a short time. The CAT-FAS was developed through four steps: (1) determining the item bank, (2) determining the stopping rules, (3) validating the CAT-FAS, and (4) establishing a platform of online administration. The results of this study indicate that the CAT-FAS has sufficient administrative efficiency (average number of items = 8.5) and reliability (group-level Rasch reliability: 0.88 – 0.93; individual-level Rasch reliability: ≥70% of patients had Rasch reliability score ≥0.90) to simultaneously assess four functions in patients with stroke. In addition, because the CAT-FAS is a computer-based test, the CAT-FAS has three additional advantages: the automatic calculation of scores, the immediate storage of data, and the easy exporting of data. These advantages of the CAT-FAS will be beneficial to data management for clinicians and researchers.
Dysfunctions of the upper and lower extremities (UE and LE), postural control, and basic activities of daily living (BADL) are major sequelae of stroke1,2,3. The assessment of these four functions in patients with stroke is fundamental for clinicians to evaluate patients’ levels of dysfunctions, set treatment goals and plans, and monitor the longitudinal trajectories of these functions.
The Fugl-Meyer Assessment (FM),4 the Postural Assessment Scale for Stroke patients (PASS),5 and the Barthel Index (BI)6 have good psychometric properties to assess the UE/LE motor functions, postural control, and BADL, respectively, in patients with stroke7,8,9. However, the total of 72 items from these three measures impedes the feasibility of assessing all three measures within a time-limited therapeutic session. A more efficient testing method is warranted. Computerized adaptive testing (CAT) is a modern measurement method. Compared with conventional measurement methods, CAT provides a more reliable estimate of the examinee’s level of function in much less time10,11,12. In conventional measurement methods, each examinee receives the same test form (or item sets), in which many items are too difficult or too easy for the examinee. These items provide limited information for estimating the examinee’s level of function and are time-intensive for examinees. In contrast, in CAT, each examinee gets a tailored item set, in which the difficulty level of the selected items meets the function level of the examinee. Because these items are tailored for that particular examinee, CAT can provide a more reliable estimate of the examinee’s level of function with fewer items and, thus, in much less time. The steps of CAT development are shown in Supplementary File 1: Appendix 1.
Because CAT promises reliable and efficient assessments, the CAT-FAS was developed to improve the administrative efficiency of the three measures previously used (FM, PASS, and BI)13. This paper describes the development and administration of the CAT-FAS. This protocol provides information for researchers to develop their CATs and for prospective users of the CAT-FAS to administer it. We also address the strengths and weaknesses of the CAT-FAS.
This study protocol was approved by a local institutional review board, and all patients gave informed consent.
1. Development of the CAT-FAS
2. Administration of the CAT-FAS
The results of the simulation showed that the 10 candidate sets of stopping rules had sufficient average Rasch reliability (0.86 – 0.95) and varied administrative efficiency (the average number of items = 6.4 – 17.5). Considering the trade-off between reliability and administrative efficiency, the set of LRI < 0.010 was selected as the optimal set of stopping rules for the CAT-FAS because of its sufficient average Rasch reliability (0.88 – 0.93, see Table 1), individual-level Rasch reliability (≥70% of the patients had a Rasch reliability of ≥0.90), and administrative efficiency (the average number of items = 8.5, see Table 2).
Figure 1: Process of exploring the performance of the CAT-FAS via simulation analysis. This figure shows the process of exploring the measurement reliability and efficiency (number of items needed for administration) of the CAT-FAS with 10 candidate sets of stopping rules. Please click here to view a larger version of this figure.
Average | % of the patients with reliability ≥ 0.90 | |
CAT-FAS | ||
UE motor function | 0.88 | 69.8 |
LE motor function | 0.9 | 76.2 |
Postural control | 0.93 | 88.6 |
BADL | 0.9 | 78.9 |
Item bank (58 items) | ||
UE motor function | 0.9 | 69.4 |
LE motor function | 0.92 | 77.4 |
Postural control | 0.96 | 96 |
BADL | 0.94 | 93.4 |
UE: upper extremity; LE: lower extremity; BADL: basic activities of daily living |
Table 1: Rasch reliability of the CAT-FAS. For the CAT-FAS, the average Rasch reliability of the four functions ranged from 0.88 to 0.93, and the individual-level Rasch reliability shows ≥70% of the participants with a Rasch reliability of ≥0.90.
Average | Range | % of the patients using 5–10 items | % of the patients using | |
> 10 items | ||||
CAT-FAS | 8.5 | ~4-13 | 66.4 | 19.5 |
Table 2: Efficiency (number of items) of the CAT-FAS. The average number of items needed for administration is 8.5. Most participants (66.4%) were assessed using 5 – 10 items.
Supplementary File 1. Please click here to download this file.
The results presented here showed that the CAT-FAS administered about 10% of the items in the original tests (the average number of items used in the CAT-FAS: 8.5 items vs. the original tests: 72 items). These findings indicate that the CAT-FAS has good administrative efficiency. The results were in line with previous studies, which reported that a CAT administered only about 10 items or less to assess social function, balance, or activities of daily living in patients with stroke10,11,20. The CAT-FAS, having good administrative efficiency, has great potential to reduce the time and burden for both patients and clinicians.
The average Rasch reliability of the CAT-FAS was 0.88 – 0.93, and more than 70% of the patients had a Rasch reliability of ≥0.90. These results reveal a good Rasch reliability of the CAT-FAS in patients with stroke. The good Rasch reliability of the CAT-FAS can be ascribed to two factors: a sound item bank and the feature of multidimensionality. First, the item bank of the CAT-FAS contains 58 items that cover a wide range of functional level for each domain14. The sufficient item coverage of the item bank can provide sufficient information to reliably estimate the examinee’s level of function. Second, the CAT-FAS is a multidimensional CAT (i.e., four domains of the CAT-FAS), in which a patient’s item response on any domain can be used to simultaneously estimate the patient’s abilities (scores) of all four domains by considering the correlations among all domains. This feature of a multidimensional CAT has been proven to improve the Rasch reliability in previous studies on developing multidimensional CATs21,22. The CAT-FAS with good Rasch reliability can be used to precisely calibrate the patients’ levels of the four functions (UE/LE motor function, postural control, and BADL) with limited random measurement error.
In addition, because the CAT-FAS is a computer-based test, the CAT-FAS has three additional advantages: an automatic calculation of scores, an immediate storage of data, and the easy exporting of data. The automatic calculation of scores saves examiners’ time and reduces mistakes in scoring. The immediate storage of data improves the efficiency of monitoring an examinee’s longitudinal changes in the four functions. The easy exporting of data enhances the efficiency of processing electronic medical records, sharing administration results between/within clinicians and patients, and analyzing data for research. These advantages of the CAT-FAS improve the overall efficiency of data management for clinicians and researchers.
The results presented here revealed that the CAT-FAS, with different sets of stopping rules, showed different performances on administrative efficiency and reliability. In general, a trade-off relationship was found between administrative efficiency and reliability. For example, the set of LRI < 0.001 had a higher reliability and lower administrative efficiency compared to the set of Rasch reliability ≥ 0.90 or LRI < 0.020. The set of LRI < 0.010 had both sufficient administrative efficiency and sufficient reliability, so it was selected as the final set of stopping rules for the CAT-FAS. If prospective users need the CAT-FAS to have a higher administrative efficiency or reliability, they can select another set of stopping rules for administering the CAT-FAS.
The first four items of the CAT-FAS were selected within each of the four domains. This design can prevent an unexpected situation that may occur in a multidimensional CAT. The unexpected situation is that a domain’s score of a multidimensional CAT might be estimated without administering any items from that domain. The unexpected situation occurs because a multidimensional CAT can use (1) the scores of the other domains and (2) the correlations among domains to estimate the scores of the domain without any items being administered15. In contrast, the CAT-FAS’s item selection rule of the first four items promises that at least one item from each domain is administered. Thus, the CAT-FAS can provide more representative information to estimate patients’ four functions.
Three limitations of the CAT-FAS are noticed. First, the training time for administration may be long because prospective users have to become familiar with the 58 items in the item bank, as well as with the instructions and rating criteria. Second, the four domains of the CAT-FAS cannot be administered separately. Third, the results presented here were from a simulated study instead of actual administrations of the CAT-FAS in patients with stroke. Therefore, the results may be somewhat different from those of an actual administration. Field tests of the CAT-FAS are warranted in the future.
The authors have nothing to disclose.
This study was supported by research grants from the Ministry of Science and Technology (105-2314-B-002 -015 -MY3).
Computer | Any | Compatible with software listed below | |
MATLAB software | The MathWorks Inc. | http://www.mathworks.com/products/matlab/ | Numerical computing software, which is used in the Protocol Section 1 (Step 1.3) |
Java Development Kit | Oracle | https://www.oracle.com/java/ | Programming language, which is used in the Protocol Section 1 (Step 1.5) |