File
Hall M1, Gierl M2, Paslawski T3, Zarski C1, Roduta Roberts M4, Hall J5, McFarlane L3, Flemming C6, McCabe E6, Bostick G1
1University of Alberta, Physical Therapy, Edmonton, AB, Canada, 2University of Alberta, Educational Psychology, Edmonton, AB, Canada, 3University of Alberta, Communication Sciences and Disorders, Edmonton, AB, Canada, 4University of Alberta, Occupational Therapy, Edmonton, AB, Canada, 5University of Alberta, Pharmacy and Pharmaceutical Sciences, Edmonton, AB, Canada, 6University of Alberta, Rehabilitation Science, Edmonton, AB, Canada
Background: Multiple-choice question (MCQ) development is an arduous task for faculty who typically have little training in test development. Faculty tend to develop tests in an inefficient manner, writing one item at a time. Writing high quality items requires content expertise, and use of design principles and guidelines to identify the knowledge and skills required to solve problems in a specific context. Cognitive models may improve this process by providing a framework that describes the knowledge, content, and reasoning skills needed to create a problem-solving task (i.e. an item). By varying the sources of information and content elements, an automated-item generation computer program can produce a large number of high-quality items from the basis of one cognitive model
Purpose: To
1) describe automated item generation and test assembly in physical therapy (PT) education,
2) compare item quality of MCQs generated through a computerized process (CG items) to traditional MCQs generated by hand (HG items), and
3) determine if experts could accurately distinguish between CG and HG items.
Methods: Faculty members with relevant expertise in musculoskeletal spinal disorders developed cognitive models using three types of required information:
(1) problem and scenarios specific to the item,
(2) sources of information needed to solve the problem, and
(3) content elements within the sources of information that can be manipulated for item generation.
Then, items were generated using automated item generation software. Finally, an exam blueprint was applied to an automated test assembly program to develop parallel test forms. The item quality scores and accuracy in identifying the type of items was assessed by four PT educators with expertise in musculoskeletal spinal disorders and experience writing MCQs. They rated the quality of a sample of CG and HG items, and indicated whether the items were of the HG or CG type.
Results: Nine cognitive models were developed resulting in 23,653 items available for use in content assessment. Application of an automated test assembly program to develop parallel test forms resulted in 60 tests. A sample of 15 CG and 15 HG MCQs were rated for quality with overall mean item quality scores (95% confidence intervals) of 3.3 (3.2,3.3) and 3.3 (3.3,3.4) for CG and HG items, respectively. Mann-Whitney U tests revealed no statistically significant differences between each of the 8 quality indicators between CG and HG items. Educators correctly identified CG items 20-53% of the time and HG items 47-67% of the time.
Conclusion(s): These data suggest that item quality does not differ between CG and HG items. Content experts with experience writing MCQs are not able to accurately distinguish between CG and HG items. It is now possible to generate thousands of items and produce multiple, parallel exams while writing a much smaller number of cognitive models.
Implications: Writing items and creating tests is a time and resource-consuming task for PT educators. Automated item generation enables PT educators to efficiently generate large banks of MCQs that provide flexibility in test administration and enhanced exam security.
Keywords: Assessment, Automated item generation, multiple choice questions
Funding acknowledgements: This project was supported by the Teaching and Learning Enhancement Fund from the University of Alberta
Purpose: To
1) describe automated item generation and test assembly in physical therapy (PT) education,
2) compare item quality of MCQs generated through a computerized process (CG items) to traditional MCQs generated by hand (HG items), and
3) determine if experts could accurately distinguish between CG and HG items.
Methods: Faculty members with relevant expertise in musculoskeletal spinal disorders developed cognitive models using three types of required information:
(1) problem and scenarios specific to the item,
(2) sources of information needed to solve the problem, and
(3) content elements within the sources of information that can be manipulated for item generation.
Then, items were generated using automated item generation software. Finally, an exam blueprint was applied to an automated test assembly program to develop parallel test forms. The item quality scores and accuracy in identifying the type of items was assessed by four PT educators with expertise in musculoskeletal spinal disorders and experience writing MCQs. They rated the quality of a sample of CG and HG items, and indicated whether the items were of the HG or CG type.
Results: Nine cognitive models were developed resulting in 23,653 items available for use in content assessment. Application of an automated test assembly program to develop parallel test forms resulted in 60 tests. A sample of 15 CG and 15 HG MCQs were rated for quality with overall mean item quality scores (95% confidence intervals) of 3.3 (3.2,3.3) and 3.3 (3.3,3.4) for CG and HG items, respectively. Mann-Whitney U tests revealed no statistically significant differences between each of the 8 quality indicators between CG and HG items. Educators correctly identified CG items 20-53% of the time and HG items 47-67% of the time.
Conclusion(s): These data suggest that item quality does not differ between CG and HG items. Content experts with experience writing MCQs are not able to accurately distinguish between CG and HG items. It is now possible to generate thousands of items and produce multiple, parallel exams while writing a much smaller number of cognitive models.
Implications: Writing items and creating tests is a time and resource-consuming task for PT educators. Automated item generation enables PT educators to efficiently generate large banks of MCQs that provide flexibility in test administration and enhanced exam security.
Keywords: Assessment, Automated item generation, multiple choice questions
Funding acknowledgements: This project was supported by the Teaching and Learning Enhancement Fund from the University of Alberta
Topic: Education; Musculoskeletal
Ethics approval required: Yes
Institution: University of Alberta
Ethics committee: Research Ethics Board
Ethics number: Pro00065004
All authors, affiliations and abstracts have been published as submitted.
