File
McCabe E1, Gross D1, Bulut O2
1University of Alberta, Department of Physical Therapy, Edmonton, Canada, 2University of Alberta, Centre for Research in Applied Measurement and Evaluation, Edmonton, Canada
Background: Patient-reported outcome measures (PROMs) are considered valuable sources of information. A clinically useful PROM will balance efficiency (quick administration), while maintaining precision (capturing individual variation) in measuring health outcomes. Item response theory (IRT) and computerized adaptive testing (CAT) can achieve these aims. In CAT, patients receive a unique set of items from a large item bank targeted towards their own health status. The individualized test produces a reliable measurement with far fewer items than traditional questionnaires.
Developing a new CAT can be a resource-intensive process. A CAT requires a bank of well-written items, which then must be calibrated using real patient response data. One approach is to use secondary analysis of patient responses to items from existing PROMs (i.e., legacy measures) to calibrate the instrument. This approach requires fewer resources because it capitalizes on previous work to develop high-quality PROMs, and repurposes PROM data collected in prior clinical research.
Purpose: The purpose of this study was to develop a computerized adaptive patient-reported outcome measure of physical functioning using secondary analysis of a dataset and items from legacy measures. Specifically, we aimed to
(1) create and test an operational CAT for physical functioning and
(2) evaluate the efficiency and precision of various CAT designs through real data and simulations.
Methods: We conducted secondary analysis of data from a Canadian provincial workers' compensation database on claimants undergoing work assessment. We used a subset of data from 1429 participants who completed three legacy measures: two scales from the 'Medical Outcomes Study Short-Form 36 Health Survey' and the 'Lower Extremity Functional Scale'. We first verified that the data meets the assumptions for IRT and the IRT model fit. We then calibrated the items from the legacy measures based on the generalized partial credit model to create a CAT item bank of 31 items. Efficiency and accuracy of various CAT designs were evaluated using computer simulations, manipulating the maximum test length and/or conditional standard error of measurement (cSEM) (i.e., precision-based) CAT stopping rules.
Results: IRT and confirmatory factor analyses support combining the items from the three legacy measures. Computer simulations showed that by using CAT, the length of testing could be reduced from 31 items to 8 items without a significant loss of information (r = 0.95, indicating 95% correlation with legacy measure scores). When a precision-based stopping rule of cSEM = 0.22 (corresponding to traditional reliability coefficient of 0.95) was tested, an average 16 items were used by the CAT. We found that combining the precision-based (cSEM = 0.31) and maximum test length (8 items) stopping rules was highly effective in improving efficiency, while maintaining a desirable level of precision (r = 0.94).
Conclusion(s): We developed an operational CAT for patient-reported physical functioning and demonstrated the efficiency of using CAT for PRO assessment.
Implications: We published the information necessary for practitioners to implement the CAT in work rehabilitation settings. In addition, the procedures we outlined are straightforward and feasible. They can be applied to the development of CATs for other health constructs.
Keywords: Computerized Adaptive Testing, Physical Functioning, Item Response Theory
Funding acknowledgements: The Workers' Compensation Board of Alberta funded the original study from which the data for this research were obtained.
Developing a new CAT can be a resource-intensive process. A CAT requires a bank of well-written items, which then must be calibrated using real patient response data. One approach is to use secondary analysis of patient responses to items from existing PROMs (i.e., legacy measures) to calibrate the instrument. This approach requires fewer resources because it capitalizes on previous work to develop high-quality PROMs, and repurposes PROM data collected in prior clinical research.
Purpose: The purpose of this study was to develop a computerized adaptive patient-reported outcome measure of physical functioning using secondary analysis of a dataset and items from legacy measures. Specifically, we aimed to
(1) create and test an operational CAT for physical functioning and
(2) evaluate the efficiency and precision of various CAT designs through real data and simulations.
Methods: We conducted secondary analysis of data from a Canadian provincial workers' compensation database on claimants undergoing work assessment. We used a subset of data from 1429 participants who completed three legacy measures: two scales from the 'Medical Outcomes Study Short-Form 36 Health Survey' and the 'Lower Extremity Functional Scale'. We first verified that the data meets the assumptions for IRT and the IRT model fit. We then calibrated the items from the legacy measures based on the generalized partial credit model to create a CAT item bank of 31 items. Efficiency and accuracy of various CAT designs were evaluated using computer simulations, manipulating the maximum test length and/or conditional standard error of measurement (cSEM) (i.e., precision-based) CAT stopping rules.
Results: IRT and confirmatory factor analyses support combining the items from the three legacy measures. Computer simulations showed that by using CAT, the length of testing could be reduced from 31 items to 8 items without a significant loss of information (r = 0.95, indicating 95% correlation with legacy measure scores). When a precision-based stopping rule of cSEM = 0.22 (corresponding to traditional reliability coefficient of 0.95) was tested, an average 16 items were used by the CAT. We found that combining the precision-based (cSEM = 0.31) and maximum test length (8 items) stopping rules was highly effective in improving efficiency, while maintaining a desirable level of precision (r = 0.94).
Conclusion(s): We developed an operational CAT for patient-reported physical functioning and demonstrated the efficiency of using CAT for PRO assessment.
Implications: We published the information necessary for practitioners to implement the CAT in work rehabilitation settings. In addition, the procedures we outlined are straightforward and feasible. They can be applied to the development of CATs for other health constructs.
Keywords: Computerized Adaptive Testing, Physical Functioning, Item Response Theory
Funding acknowledgements: The Workers' Compensation Board of Alberta funded the original study from which the data for this research were obtained.
Topic: Outcome measurement; Occupational health & ergonomics
Ethics approval required: Yes
Institution: University of Alberta
Ethics committee: the Health Research Ethics Board
Ethics number: Pro00073119
All authors, affiliations and abstracts have been published as submitted.
