File
Miciak M1,2, Woodhouse LJ3, Mardis N4, Graham K1, Brown A5, Chorzempa H1
1Alberta Innovates, Performance Management and Evaluation, Edmonton, Canada, 2University of Alberta, Faculty of Rehabilitation Medicine, Edmonton, Canada, 3University of Alberta, Department of Physical Therapy, Edmonton, Canada, 4Canadian Institutes of Health Researh, Institute of Health Services and Policy Research, Ottawa, Canada, 5University of Toronto, Dalla Lana School of Public Health, Toronto, Canada
Background: Assessing inter-professional practice performance at a system level provides valuable information regarding accountability and program quality for decision-makers. In physiotherapy, we equate evaluation of physiotherapy programs with clinical outcome measures. However, assessing system-level inter-professional performance requires a systematic approach to develop and select key performance indicators (KPIs) that are relevant to multiple stakeholders and begin to measure impact. Unlike indicators for specific health conditions (e.g., urinary incontinence) or procedures (e.g., joint replacements), system KPIs are not as obvious, which makes identifying KPIs challenging.
Purpose: A pan-Canadian, multi-stakeholder group of health services and policy and research impact experts was convened to determine a core set of KPIs using an impact framework in order to assess the impact of health services and policy research on decision-making in the healthcare system.
Methods: A systematic 3-phased approach was used to develop and select KPIs: Phase 1: an Indicator Task Force developed a broad set of indicators; Phase 2 - an Impact Analysis Working Group refined and vetted the Phase 1 indicators; and Phase 3 - an Indicator Review Panel determined the core set indicators using a modified Delphi approach that collected quantitative and qualitative data in all rounds.
Results: In Phase 1, a 7-member Indicator Task Force of research impact practitioners identified 67 indicators informed by an impact assessment framework, systematized literature review, practical research impact assessment experience, and discussion of face validity, importance, and feasibility. In Phase 2, the IAWG and chair of the Indicator Review Panel refined wording and reduced 67 indicators to 46 based on discussion of a priori criteria including clarity, validity, and feasibility. In Phase 3, a 15-member pan-Canadian Indicator Review Panel, purposively recruited from diverse stakeholder groups (e.g., research funders, government) and representing 5 Canadian regions, applied a 3-round (survey - in-person meeting - survey) modified Delphi method to leverage expert knowledge. They reached consensus on a set of 23 indicators using 5 criteria: validity, reliability, importance, feasibility, and actionability. Themes from the qualitative data were related to indicator rating and refinement (e.g., indicators requiring qualitative assessment) as well as indicator implementation and future development (e.g., organizational culture of research use).
Conclusion(s): Establishing a process for assessing inter-professional practice is crucial for evaluation of the value and impact of new models of care on our healthcare system. KPIs used for this purpose go well beyond traditional clinical outcome measures to consider the wider health eco-system by tracing measures across the pathways from health services and policy research to impact.
Implications: The physiotherapy profession must move beyond traditional evaluation of outcome measures and participate in the development of robust KPIs to evaluate inter-professional modes of care. Physiotherapists need to participate in these impact assessments to build credibility with stakeholders and inform and bring a rehabilitation perspective to the use of evidence in decision-making at the system-level. In order to evaluate and demonstrate the value of physiotherapy services in all healthcare contexts (e.g., public/private, developed/developing nations), physiotherapists must engage in these multi-stakeholder initiatives that evaluate systems-level healthcare performance.
Keywords: Impact assessment, performance measures, inter-professional practice
Funding acknowledgements: Canadian Institutes of Health Research - Institute of Health Services and Policy Research; Alberta Innovates
Purpose: A pan-Canadian, multi-stakeholder group of health services and policy and research impact experts was convened to determine a core set of KPIs using an impact framework in order to assess the impact of health services and policy research on decision-making in the healthcare system.
Methods: A systematic 3-phased approach was used to develop and select KPIs: Phase 1: an Indicator Task Force developed a broad set of indicators; Phase 2 - an Impact Analysis Working Group refined and vetted the Phase 1 indicators; and Phase 3 - an Indicator Review Panel determined the core set indicators using a modified Delphi approach that collected quantitative and qualitative data in all rounds.
Results: In Phase 1, a 7-member Indicator Task Force of research impact practitioners identified 67 indicators informed by an impact assessment framework, systematized literature review, practical research impact assessment experience, and discussion of face validity, importance, and feasibility. In Phase 2, the IAWG and chair of the Indicator Review Panel refined wording and reduced 67 indicators to 46 based on discussion of a priori criteria including clarity, validity, and feasibility. In Phase 3, a 15-member pan-Canadian Indicator Review Panel, purposively recruited from diverse stakeholder groups (e.g., research funders, government) and representing 5 Canadian regions, applied a 3-round (survey - in-person meeting - survey) modified Delphi method to leverage expert knowledge. They reached consensus on a set of 23 indicators using 5 criteria: validity, reliability, importance, feasibility, and actionability. Themes from the qualitative data were related to indicator rating and refinement (e.g., indicators requiring qualitative assessment) as well as indicator implementation and future development (e.g., organizational culture of research use).
Conclusion(s): Establishing a process for assessing inter-professional practice is crucial for evaluation of the value and impact of new models of care on our healthcare system. KPIs used for this purpose go well beyond traditional clinical outcome measures to consider the wider health eco-system by tracing measures across the pathways from health services and policy research to impact.
Implications: The physiotherapy profession must move beyond traditional evaluation of outcome measures and participate in the development of robust KPIs to evaluate inter-professional modes of care. Physiotherapists need to participate in these impact assessments to build credibility with stakeholders and inform and bring a rehabilitation perspective to the use of evidence in decision-making at the system-level. In order to evaluate and demonstrate the value of physiotherapy services in all healthcare contexts (e.g., public/private, developed/developing nations), physiotherapists must engage in these multi-stakeholder initiatives that evaluate systems-level healthcare performance.
Keywords: Impact assessment, performance measures, inter-professional practice
Funding acknowledgements: Canadian Institutes of Health Research - Institute of Health Services and Policy Research; Alberta Innovates
Topic: Professional issues; Globalisation: health systems, policies & strategies; Professional issues
Ethics approval required: No
Institution: Not applicable
Ethics committee: Not applicable
Reason not required: This was not a research study. It was foundational work to develop the key performance indicators for program assessment. Experts were providing opinions on and selecting indicators. The indicators will now be used in research impact assessments.
All authors, affiliations and abstracts have been published as submitted.
