985_20230124_ce_educators_toolkit

EDUCATION TOOLKIT Acknowledgements Authors The Society for Academic Continuing Medical Education The authors wish to thank the following individuals for their valuable contributions to this project. Project Team: Sharon Ambata-Villanueva, MA, CTDP Ashleigh Jaggars, MPH Tharshini Jeyakumar, MHI Inaara Karsan, MHI Morag Paton, PhD, MEd Nathaniel Williams Sarah Younus, MPH Advisory Committee Members: Mik Bauer Craig Campbell, MD, FRCPC Asha Maharaj, MBA Tymothi Peters William Rayburn, MD, MBA Suzan Schneeweiss, MD, MEd, FRCPC Bita Zakeri, PhD Reviewers: Rich Frankel, MBA Gurpreet Grewal, B. Ed (ADED) Ginny Jacobs, PhD, MEd, MLS, CHCP Rebecca Kolb, MA Patricia O'Sullivan, EdD Ivan Silver, MD, MEd, FRCPC Jane Tipping, MADEd, PCC (ICF) ACCME: Graham McMahon, MD, MMSc Betsy Williams, PhD, MPH bwilliams@prckansas.org David Wiljer, PhD David.wiljer@uhn.ca Joyce Fried, FSACME joycemfried@gmail.com Gabrielle Kane, MB, EdD, FRCPC kaneg@uw.edu © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 3 985_20230124

6 81 82 83 84 85 86 87 90 91 93 97 100 101 103 104 106 Debrief checklist for small group learning 78 Implementation checklist 79 3.1.4 Key considerations 80 Tips for effective delivery of small group learning Challenges of facilitating small group learning Mitigating challenges in small group learning Planning canvas for small group learning Evaluation consideration for small group learning Additional resources 3.2 Intervention two: Case-based learning 3.2.1 Introduction to case-based learning 89 Why facilitate case-based learning Case scenario Guiding questions for case-based learning 92 Key enablers for case-based learning Importance of case-based learning Formula for success 98 3.2.2 Framing the problem and preparation 99 How to develop a case Types of cases Steps to writing a case study Methods for presenting a case Framing the problem and preparation checklist 105 3.2.3 Implementation guidelines EDUCATION TOOLKIT Table of contents © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

9 170 171 176 177 178 180 184 168 Kirkpatrick-Barr framework 169 Moore's framework Evaluate if learners achieve outcomes Evaluation approaches Evaluation checklist Additional resources Glossary of terms Reassess CE planning preparedness References EDUCATION TOOLKIT Table of contents © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

1.1 Introduction Healthcare is complex and rapidly evolving; with this in mind, interdisciplinary care and professional development continues to be of great importance. There is a critical need for healthcare professionals to maintain and continuously update their knowledge and skills, and to reflect on changes that impact their practice. Accredited continuing education (CE) promotes a lifelong learning mindset of continuous professional development to gain new knowledge and skills. Key Terms: Continuing education (CE): CME, CPD, accredited CE are educational activities that serve to maintain, develop, or increase the knowledge, skills, and professional performance and relationships health professionals use to provide services for patients, the public, or the profession [1]. Continuing professional development (CPD): includes all activities that any health professional undertakes, formally and informally, including CE, in order to maintain, update, develop, and enhance their knowledge, skills, and attitudes in response to the needs of their patients [1]. 12 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Preliminary search for educational strategies: Effective education strategies for continuing professional development were identified following a review of the academic literature and discussions with experts in continuing education. Identification of best practices: Best practices in instructional design were identified through a rapid review of the literature and an eDelphi approach for each education intervention [3]. The rapid reviews of academic literature provided the foundation for developing evidence-based best practice recommendations for the design, implementation, and evaluation of the educational interventions. The findings from the rapid review were validated by an eDelphi panel consisting of academic professionals, CE experts, and healthcare professionals. The eDelphi approach involved a virtual focus group and survey to further identify best practice recommendations and practical implementation strategies for each education intervention. Development of a toolkit: An evidence-based toolkit was developed based on the recommendations drawn from the literature and continuing education professionals. The toolkit design focused on incorporating recommendations for each education intervention that will help CE leaders, educators, and healthcare professionals plan, implement, and evaluate their intervention to promote learner engagement, knowledge uptake, and practice change. Review, validate, and revise: The toolkit was reviewed for clarity, utility, and relevance to the target audience by a diverse group of stakeholders with varied experiences in CE. An iterative process was used to incorporate feedback from stakeholders throughout the toolkit design process. 1 2 3 4 15 TOOLKIT INTRODUCTION How the toolkit was developed © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Choose and integrate education interventions into your CE session CE Session Small group learning Promote self-esteem and participation Introduce learner to a range of perspectives and gain feedback from peers Assist in the development of social, communication, and leadership skills Case-based learning Foster critical thinking through the use of real-world scenarios Promote collaboration and interprofessional learning through inquiry-based approaches The selected strategies presented in this toolkit can be used in combination when designing a CE session. However, your CE session design is not limited to these education interventions. You can use them together or other learning approaches to deliver educational content. The following diagram outlines unique features of each intervention discussed in this toolkit that can be incorporated within your CE session: Reflective learning Foster the development of self-awareness Facilitate lifelong learning Identify individual gaps and needs TOOLKIT INTRODUCTION 18 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Self-assessment for CE planning preparedness If you are planning a CE session, use the following self-assessment tool to identify areas in the toolkit that will be beneficial to you. Focus your time on sections in the toolkit where you report low confidence to refine your skillset in CE planning and design. If you report confidence from levels 1 to 3, you may want to go through the content thoroughly and complete the activities. If you report a confidence level from 4 to 5, use the toolkit as a reference resource and use the worksheets to help with the design of your CE session. 21 Please rate your level of confidence with each of the CE planning components, learning formats, and evaluation activities on a scale of 1 (not at all familiar) to 5 (extremely familiar). Confidence level: 1 (Not at all) - 5 (Extremely) CE Component Initiating planning of CE session Structuring a longitudinal and multimodal education intervention Conducting a needs assessment 1 2 3 4 5 1 2 3 4 5 Developing learning objectives Integrating principles of inclusion, diversity, equity, and accessibility into activities Delivering CE sessions virtually Facilitating small group learning Planning and encouraging small group discussion Creating an inclusive, safe, and supportive learning space Maintaining group dynamics Mitigating challenges and conflicts in groups 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

22 Please continue to rate your level of confidence with each of the CE planning components, learning formats, and evaluation activities on a scale of 1 (not at all familiar) to 5 (extremely familiar). Confidence level: 1 (Not at all) - 5 (Extremely) CE Component Case-based learning Developing a case for activities Presenting the use diverse methods and modalities 1 2 3 4 5 1 2 3 4 5 Identifying strategies for examining and engaging learners in case analysis Defining the facilitator's role in case-based learning discussion Conducting activities to disseminate case analysis and debrief the case Reflective Learning Determining learning goals and when to integrate reflective learning Defining facilitator's role in reflective learning Applying and integrating the critical reflective inquiry model in activities 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Assessing knowledge comprehension of learners Evaluating outcomes of intervention using structured frameworks Selecting evaluation approaches to assess performance change or skill development 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Assessing and Evaluating CE Interventions Providing and integrating feedback Self-assessment for CE planning preparedness © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

How to begin planning your CE session SECTION TWO 23 985_20230124

In this section: First steps to education intervention design Structure a longitudinal and multimodal education intervention Use the PDSA cycle to structure your CE intervention Conduct a needs assessment to understand the audience and their learning needs Learning objectives Develop targeted learning objectives How to write a learning objective How to make a learning objective actionable IDEA principles Designing for inclusivity Designing for diversity Designing for equity Designing for accessibility Checklist Section 2 Virtual delivery Key considerations for virtual delivery 24 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2.1 Structure a longitudinal and multimodal education intervention Plan Study Do Act P S D A To increase the likelihood that learning will change practice, the CE planner should ideally create learning experiences that facilitate rehearsal, practice, and feedback using what was learned through a longitudinal and multimodal approach [9]. Follow an iterative PDSA cycle to design and develop educational interventions that foster meaningful learning experiences and allow healthcare professionals to stay abreast of advances in clinical knowledge and skills while also developing a continuous learning mindset [9]. Use a quality improvement approach to structure a longitudinal education intervention that will allow learners to apply their knowledge, receive appropriate feedback, and have opportunities to enhance their learning. 25 INITIAL PLANNING QUALITY IMPROVEMENT © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

A P D S Step 1: Plan Identifying professional practice gaps and underlying educational needs Step 2: Do Engaging in the intervention Step 3: Study Assessing the intervention Step 4: Act Integrate assessment insights and adjust Use the PDSA cycle to structure your CE intervention INITIAL PLANNING Following an iterative PDSA cycle to frame education interventions will provide added value to learners and enhance practice improvement and performance [10,11]. Use the PDSA cycle to structure and continually seek ways to improve your CE interventions. 26 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

PDSA cycle Identify current professional practice gaps and underlying educational needs. Design an education intervention that addresses learner needs to close gaps [10]. Step 1: Plan Assess whether it achieved the outcomes from the quintuple aim framework (experience of care, population health, care team well-being, per capita cost, and equity) and seek data about the effectiveness of the intervention's impact [9]. Step 3: Study Iterative process of modifying and adapting the education intervention based on the insights about the effectiveness of the intervention to address learner needs and close gaps [10]. Step 4: Act 27 Implement an educational intervention that is designed address learners' educational needs and promote learner change and/or care outcomes. [9]. Step 2: Do INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Understand your audience and their educational needs by conducting a needs assessment An important first step when designing content for a CE session is to determine who the target audience is and what their educational needs are. Identifying these needs can be achieved through conducting a needs assessment. Needs assessments help define the gap between current and desired professional practices and/or care outcomes for health professional learners and their patients [12-13]. Data derived from the needs assessment can be used to develop educational objectives and inform methods of delivery [12-13]. The following are different types of learning needs that can be addressed through a needs assessment: 28 Self-recognized or perceived needs I know what I want and need to know Unknown or unperceived needs I don't know what I don't know INITIAL PLANNING Miscalculated or misperceived needs I think I know something that I don't Emergent needs Now I have some new information, I realize I want or need to learn something else instead of or in addition to what I am learning now © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

The best way to know what learners perceive as their learning needs it is to ask them directly [13]. However, learners are not always aware of their learning needs. Unperceived needs can be identified through experts in the field, patients and clinical data. The following are common methods of conducting needs assessments: Methods for conducting a needs assessment 29 INITIAL PLANNING Determining perceived needs- "I know what I want and need to know" Determining unperceived needs- "I don't know what I don't know" Survey A questionnaire is delivered to the target audience to determine what potential participants may want or need to know [13]. Focus groups An interview conducted with several representatives of the target audience led by a facilitator to gain insight into participants' views and opinions [13]. Key informant A delegate from the target audience talks with their colleagues, gathers information on perceived learning needs and then reports to the CE planner [13]. Additional methods Interviews Meetings with colleagues (formal or informal) Evaluations of previous CE activity Expert advisory group Chart audit Input from patients Additional methods Process where experts in CE or clinical practice are asked questions about their program and content preferences [13]. A method that systematically examines patient charts and EMR/EHR data looking for patterns of care that can be appropriate or inappropriate [14]. Data on patient satisfaction and experience can be captured to identify learning needs. This information can be gathered through interviews, surveys, and including patients on planning committees [14]. Knowledge tests Critical incident reports Direct observations of practice performance © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Practice: Identify the gap Current state (what learners know and do) Ideal state (what learners should know and do) Needs assessments help define the gap between current and desired education practices for your target audience [12-13]. Identification and analysis of educational needs can provide the foundation for developing educational objectives. In this exercise, identify the educational or professional gap that the CE session will address. 30 Explain how the CE session can help address the identified gap: How prevalent is the need among healthcare professionals? How many different assessment sources indicated this need? Will this need have a significant impact on the delivery of optimal patient care? How likely is it that a CE session will result in a change in practice behavior? Are there sufficient resources available within your organization to address this topic? How receptive will healthcare professionals be to a session on this topic? Guiding questions for the planning committee to consider [15]: 1. 2. 3. 4. 5. 6. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2.2 Develop targeted l earning objectives Clear Measurable Identify behavior or attitude change Learning objectives should clearly state, in measurable terms, the behavior or attitude the learner is expected to adopt on completion of the activity. Use verbs based on Blooms’ Taxonomy to structure clear learning objectives (See page 34) [16]. Employ the TACT (target, action, context, and time) principles to articulate the expected behavior or attitude change in clinical practice and how it can be measured (See page 36) [16]. 31 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Learning objectives should reflect the desired knowledge, skills, and abilities that learners should develop as a result of participating in your CE session. Bloom’s Taxonomy describes the types of knowledge and cognitive processes used by learners. When writing a learning objective, first establish which type of knowledge (left side of the table on page 33) you wish your learner to acquire. Using this type of knowledge, next determine which cognitive process (top of the table) you wish to focus on in your instruction. Bloom’s Taxonomy increases in rigor from left to right [17a]. Learning objectives should be written using action verbs that target the desired combination of knowledge and cognitive process to make it clear to learners what changes they should expect upon completion of the CE activity [17]. Framing learning objectives in this way has the added benefit of providing objective measurable indicators of behavior that can support the demonstration of learner change. Use the following table to create targeted learning objectives. The examples provided are not exhaustive but provide a starting point for developing your learning objectives. Bloom's Taxonomy has continuously evolved to meet the changing needs of educators. As you become more familiar with learning objectives, you may wish to explore the different versions. Please see page 35 in the toolkit for additional resources. How to write a learning objective 32 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

33 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved Factual Knowledge Conceptual Knowledge Procedural Knowledge The learner will list symptoms of juvenile diabetes The learner will recall the differences between Crohn's Disease and Ulcerative Colitis The learner will list the key factors in effective team management The learner will summarize the functions and features of a new medical device The learner will explain why they would recommend a particular anti- depressant drug for a patient The learner will explain how to complete a successful kidney transplant The learner will complete a surgical safety checklist The learner will apply the four pillars of medical ethics to a patient situation The learner will perform a tracheal intubation The learner will be able to differentiate the clinical presentations of acute rhinosinusitis vs acute bacterial rhinosinusitis to develop treatment plans The learner will distinguish which patients to prioritize in a triage situation The learner will attribute a successful surgical operation to the specific methods used The learner will rank risk factors in terms of severity for COVID-19 The learner will critique the effectiveness of different managerial styles The learner will check their implementation of a vascular stent The learner will create a personal quick guide for identifying patients with a higher risk for stroke The learner will design a treatment plan for a person living with cancer Remember Understand Apply Analyze Evaluate Create The Procedural Dimension The Knowledge Dimension The Taxonomy Table [17a] The learner will create step by step plan onboarding new team members 985_20230124

Commonly used verbs [17b] The following are commonly used verbs for each level of Bloom's Taxonomy. While there are many others, this chart is a helpful jumping-off point [17b]. List Explain Use Analyze Judge Construct Recall Describe Apply Compare Appraise Design State Paraphrase Demonstrate Distinguish Rearrange Arrange 34 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved Knowledge Understand Apply Analyze Evaluate Create Name Discuss Act Differentiate Compare Organize Label Translate Solve Categorize Assess Plan Define Summarize Illustrate Contrast Evaluate Compare Repeat Classify Operate Examine Synthesize Create Identify Express Sketch Relate Defend Develop Cite Interpret Dramatize Infer Estimate Formulate Select Infer Employ Test Manage Write 985_20230124

Additional Resources 35 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved Learning Objectives Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: The classification of educational goals, Handbook II: Affective domain. New York: David McKay Co. Dave, R.H. (1970). Psychomotor levels in Developing and Writing Behavioral Objectives, pp.20-21. R.J. Armstrong, ed. Tucson, Arizona: Educational Innovators Press. Harrow, Anita J. (1972). A Taxonomy of the Psychomotor Domain: A Guide for Developing Behavioral Objectives. New York: David McKay Co. Simpson, E. J. (1972). The classification of educational objectives in the psychomotor domain: The psychomotor domain. Vol. 3. Washington, DC: Gryphon House. 985_20230124

1 2 3 4 TACT (target, action, context, and time) is a mnemonic that refers to principles that can be used to structure your learning objective so that it clearly articulates the expected clinical behavior or attitude change [16]. Creating an actionable learning objective is important to direct learners through the CE intervention and for facilitators to stay focused on the goals of the CE intervention [16]. Please refer to the following example on the right when you are creating your learning objectives. How to make a learning objective actionable Primary care physicians TARGET ACTION CONTEXT TIME Whom is the learning objective directed toward? What is the behavior or attitude change required? Where is the behavior or attitude change taking place? What is the time frame to demonstrate behavior or attitude change? To implement recommendations on screening for colorectal cancer among the average-risk population aged 50 to 74 years In their clinical practice Within the next three months Example Learning Objective: Implement recommendations on colorectal cancer screening among the population between the ages of 50 to 74 years in family physician's clinical practice within the next three months. As part a CE session on colorectal cancer screening awareness ... 36 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Practice: Write a learning objective TARGET Whom is the learning objective targeted toward? TACT Principle ACTION What is the behavior or attitude change required? CONTEXT Where is the behavior or attitude change taking place? TIME What is the time frame to demonstrate behavior or attitude change? 37 Write your learning objective based on the TACT principles. Learning Objective © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2.3 IDEA principles Inclusion, diversity, equity, accessibility The IDEA principles of inclusion, diversity, equity, and accessibility acknowledge learners' identities, demographics, learning preferences and needs, experiences, and professional backgrounds that should be considered and applied when providing a learning opportunity [18]. These principles should be applied in the preparation, implementation, and evaluation stages. In order to ensure the plans are aligned with (and responsive to) the diverse needs of the learning community, it is imperative that representative voices be heard and incorporated into the educational planning. The IDEA principles should also be incorporated when considering planning committee composition, location, and/or context. They should also be applied toward understanding the relationships and histories embedded within these systems that may affect the success of a learning intervention. 38 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Designing for inclusivity Using inclusive and respectful language is an important component of program design as this influences the development of collaborative relationships in both learning and clinical spaces [18]. Planning committees, if you have one, optimally should seek representation and/or feedback from diverse stakeholders (including those with lived experience and their families) as learning interventions are developed to ensure inclusive language and attentiveness to constructs such as gender, race, ethnicity, age, ability, sexual orientation, and profession or discipline [18]. Culturing an inclusive learning environment will be an important goal of facilitation. Setting ground rules early on in a program is helpful [18]. 39 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Designing for diversity Developing authentic learner-centered experiences includes paying attention to the diversity of learner needs and the diversity within real-life examples or group dynamics. Representing authenticity in examples or scenarios will be an important piece in achieving impact [18]. Acknowledging that learners (and planners or facilitators) have diverse lived experiences and learning preferences can inspire planners to develop a broad variety of learning activities. For example, planners may wish to seek out examples from under- represented/marginalized populations, ensuring that stereotypes are eradicated, and biases are removed, minimized, and/or declared [18]. In program delivery, for example, visual or audio tasks can be accompanied by written transcripts to account for learners with diverse learning needs or preferences [18]. 40 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Designing for equity Designing for equity means that developers work to address the differences between individuals and population groups that are systematic, unnecessary, unfair and avoidable [18]. When facilitating learning opportunities, consider broadening perspectives from the individual to the institutional level, taking into consideration advantages some may access or burdens some may be unable to avoid [18]. Consider designing activities that support equity-based improvements in program or service delivery, decision making, or resource allocation [18]. 41 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Designing for accessibility Accessibility is an important consideration in both content development and education delivery. Bringing examples related to accessibility into small group, reflective, or case-based activities allows for the learner to include accessibility as a consideration [18]. The increased use of virtual platforms to deliver case-based learning can promote accessibility by reaching a larger target audience. However, it is critical to consider access to network bandwidth and the burden of cost, especially for remote and marginalized areas [18]. Additional considerations include assistive devices for learners with audiovisual challenges, strategies to facilitate physical limitations, and language translation when appropriate. Furthermore, the technology used for virtual, blended, or hybrid learning sessions needs to be perceived as easy to use and accessible to encourage learners to actively participate [18]. 42 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

IDEA considerations when designing your CE intervention [18] Does your intervention plan use inclusive and respectful language? INCLUSION DIVERSITY Are the facilitators using inclusive language when addressing learners? Do you seek representation or feedback about your activities and materials from diverse stakeholders? Do the facilitators have a clear guideline on how to set ground rules and expectations? Do your materials represent and respect the diversity in your population? Do you engage diverse stakeholders when vetting your CE content? Is your content free of stereotypes and bias associated with gender, race, ethnicity, culture, religion, age, sexual orientation, ability, and other identities? Are the context, backgrounds, experiences, and needs of learners considered in the CE session design? 43 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Do your learning objectives target systemic and structural issues or individual-level issues? EQUITY ACCESSIBILITY Does your content prompt equity-based improvements to clinical care? Does your CE content consider the experiences and needs of vulnerable or marginalized groups? Are your facilitators unbiased, objective, and inclusive? Will your CE session be conducted in-person or virtually? Do your CE session activities address scenarios pertaining to access? Are there different methods to access the CE session materials to participate? If you are using a virtual platform, is it easy to use and understand? 44 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved IDEA considerations when designing your CE intervention [18] 985_20230124

Minimize disruptions to the flow of the session by providing technology requirements prior to the session. 1 As virtual delivery methods become more prominent, consider the following strategies when designing your CE intervention: 2.4 Key considerations for virtual CE delivery Assign a second facilitator to help learners offline with technical issues. 2 Encourage learners to stay on camera and prompt them to change their screen name so that it reflects their preferred name. 3 Breakout rooms can employ self- managed facilitation to guide discussion or a facilitator can be present in each breakout room to monitor the discussion. 4 Facilitators should provide clear guidelines and task-oriented instructions when learners are engaging in virtual breakout rooms. 5 45 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2.5 Additional resources Needs assessment Mann KV. Not another survey! Using questionnaires effectively in needs assessment. Journal of Continuing Education in the Health Professions. 1998;18(3):142-9. doi: 10.1002/chp.1340180303 McCawley PF. Methods for conducting an educational needs assessment. University of Idaho. 2009;23: 6-14. https://www.extension.uidaho.edu/publishing/pdf/bul/bul0870.pdf Michener Institute of Education at UHN: A Guide to Performing Needs Assessments Tip sheet on methods and tools for performing needs assessments among learners Grant J. Learning needs assessment: assessing the need BMJ 2002; 324 :156 doi: 10.1136/bmj.324.7330.156 Learning objectives IDEA principles Association of American Medical Colleges (AAMC): Writing Learning Objectives Tip sheet outlining the s teps for writing a learning objective and how to use Bloom's Taxonomy American College of Surgeons Division of Education: Tips for Writing Learning Objectives Tip sheet with explanation and examples of how to write learning objectives Centre for Addiction and Mental Health (CAMH): Health Equity and Inclusion Framework for Education and Training Report outlining a framework for integrating equity and inclusion into the planning, development, and implementation of educational initiatives ACCME: Advancing Social Justice Resources Webinar and additional resources about how to integrate equity, diversity, and inclusion principles into CE sessions 46 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Additional resources Virtual delivery considerations ACCME: Best practices for planning and leading virtual meetings - tips for faculty Tip sheet for facilitators using online learning tools Harvard University: Best practices for teaching remotely Website outlining course types and tips for teaching through virtual and blended methods Royal College of Physicians and Surgeons of Canada: Ten tips for virtual teaching Tip sheet identifying strategies for virtual teaching Temerty Faculty of Medicine University of Toronto CPD: Tips for use of ZOOM technology for digital learning Tip sheet for using Zoom to facilitate a videoconference session Temerty Faculty of Medicine University of Toronto CPD: Virtual synchronous teaching using Zoom (interactive webinar) Webinar on tips for synchronous teaching and applying interactor features on Zoom Dr. Heather MacNeill: Synchronous teaching and learning Youtube channel about online learning, effective teaching principles, co-facilitation, and interactivity using Zoom The University of Minnesota: Online course design resources An online course that instructs how to develop remote teaching plans, assignments, and assessments 47 INITIAL PLANNING © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Educational interventions for CE sessions SECTION THREE 48 985_20230124

3.1 Facilitation of small group learning INTERVENTION ONE 49 Small group learning: an instructional approach that encompasses active participation, purposeful activities, and face-to-face interaction. It is a concerted and collaborative effort in learning new knowledge and skills, and attaining a mutual objective [19]. 985_20230124

In this subsection: Intervention 1 50 Introduction to small group learning Why small group learning? Case scenario Guiding questions Key enablers Formula for success Framing the problem and preparation Plan out discussion Mind map Strategies for creating an inclusive, safe and supportive space Implementation guidelines How to facilitate small group learning Maintaining group dynamics Role of facilitator Approaches to encourage discussion Questioning strategies Debrief checklist Key considerations Tips for in-person and virtual delivery Challenges with facilitating small group learning sessions Mitigating challenges © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.1.1 Introduction to facilitating small group learning In this module, you will learn how to prepare and implement facilitation strategies for small group learning interventions. These facilitation guidelines can be used in combination with other educational interventions to deliver integrated CE sessions. Key Terms: Group Dynamics: The behavior and attitude patterns involved when learners interact with each other [20]. Mind Map: A creative and logical method of note-taking and note-making that maps out your ideas [21]. 51 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Why facilitate small group learning ? Small group learning is an advantageous education intervention for CE sessions. Through promoting a collaborative and team-oriented environment, learners develop skills in problem-solving, team building, decision making, and critical thinking [20,22]. Learners engaged in small group learning activities constructively work together to attain a mutual objective and build their own understanding in conjunction with peers [20]. The main goal of small group learning is to have a learner-driven session rather than an instructor-led one, however, the facilitator plays a crucial role in the success of a small group learning intervention by fostering a supportive environment to encourage group engagement and collaboration [19,20]. Effective facilitation strategies are an indicator of productive CE sessions [20]. 52 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

An organization has been developing a collaborative care model for patients with both mental and physical health challenges. This model is a new way of working for the organization and poses several significant challenges for the health professions team. Katrina, a CE leader, has observed the need for educational support in achieving their clinic's new goals. In Katrina's planning for a CE session, several committee members have suggested that they include small group learning to improve collaborative goal setting and increase completion rates of patient-reported outcome measures. These areas are essential for implementing a collaborative care model. Case scenario © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 53 985_20230124

As Katrina is planning her small group learning session, she encourages facilitators to consider the following questions: What steps are necessary to create a safe, inclusive, and collaborative space for learners? How confident are you in achieving this? How would you manage group dynamics and prompt discussion? How would you identify and mitigate barriers to participation? Could you manage unexpected events that come up? What are the important principles and considerations for small group learning experiences you should consider? How comfortable are you with this type of learning format? 1 2 3 4 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 54 985_20230124

Targeted planning Question effectively Balanced facilitation Key enablers INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 55 985_20230124

Targeted planning Planning is crucial to the success of a small group learning activity. It is sometimes seen as the most important role for facilitators. During planning, facilitators should outline learning objectives and have a plan to foster a safe and productive learning environment [23-25]. Additionally, factors such as time, space, and recruiting faculty should be considered in the planning phase [23]. Learning objectives that guide a small group learning intervention should be clear, measurable, and identify a behavior or attitude change [24]. INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 56 985_20230124

Question effectively Questions allow the learners to evaluate their understanding of key concepts [24]. The facilitators play a pivotal role by asking thoughtful questions, actively listening, and responding positively to learners. Questioning allows facilitators to understand the learner's needs, monitor progress, and provide support [20,24]. Questioning is crucial in establishing a meaningful learning experience. INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 57 985_20230124

Balanced facilitation Effective facilitation is a balance between encouraging learners to lead the conversation and understanding when it is appropriate for the facilitator to provide guidance [20,25-26]. Effective planning through targeted learning objectives and mind mapping can help facilitators balance when to guide and when to listen. This can help learners achieve the desired mastery of the material and support gaps in knowledge that arise during group discussions [25]. Note: See page 62 for a definition of mind mapping. INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 58 985_20230124

1 2 3 Formula for success Facilitators should create a small group to focus on individualized learning needs and encourage active learning. Facilitators should ensure group dynamics that foster inclusion, fairness, and critical reflection on constructive feedback. Facilitators should prompt discussion, understand learners’ progress, stimulate curiosity, sustain and initiate conversation among learners, and outline follow-up action plans. INTERVENTION INTRODUCTION Identify facilitators, select time, space, and delivery modality for the CE session. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 59 985_20230124

3.1.2 FRAMING THE PROBLEM AND PREPARATION Facilitating small group learning 60 985_20230124

Planning your small group discussion HOW TO PREPARE Just as it is important for learners to arrive in a small group well prepared, it is equally important for the facilitator(s) to have properly planned and anticipated ways to engage learners in discussion [19-20,22-29]. Consider creating group sizes between 4 and 5 learners in a virtual setting or 6 to 8 learners in an in-person setting. These group sizes allow the facilitator to focus on individual learning needs, tailor content, and encourage active participation [19-20,23-24,29]. Mind mapping can be used as an informal technique to organize a facilitator's discussion points for a small group session and during the learning session to summarize key points [21]. 61 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Mind map: A teaching resource HOW TO PREPARE Mind mapping Mind mapping is a creative and logical method of note-taking and note-making that maps out your ideas [21]. Facilitators can use it as a teaching resource to: Prepare and review CE sessions Prepare questions Facilitate small group discussions Summarize information from different sources into key topics and keywords [21]. Map them to identify how they are interrelated [21]. Prepare and review CE sessions Prepare questions Facilitate small group discussions Write the session topic down in the center of the page and follow up with subsequent sub- topics or questions around the topic [21]. Re-draw or tidy up the map and group similar topics together [21]. Use this method to identify key questions for discussion [21]. Summarize key points and action items discussed during a learning session [21]. Visually depict how the discussion evolved [21]. 62 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

How to create a mind map Collaborative care model for patients and clinicians For Katrina's CE session on the novel collaborative care model at her organization, she creates the following mind map to structure the group's discussion: Common challenges Socio-economic reasons Cultural reasons Skills required Collaborative goal setting Shared decision- making Compassionate care Patient-reported outcomes Use in clinical decision-making Easy-to-use platform Communication HOW TO PREPARE 63 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Practice: Create a mind map Topic: Mind maps can be a useful tool in planning out your session's discussion and identifying key concepts that you want learners to take away from the CE session. Use the following template to create a mind map for your CE session. HOW TO PREPARE 64 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Creating an inclusive, safe, and supportive space An important step in planning your CE session is to consider how the facilitator will foster an inclusive, safe, and supportive environment . Creating a space where learners can be themselves and feel respected and not judged is necessary to encourage engagement amongst learners. Integrating strategies for inclusion within your CE sessions provide opportunities for learners to have meaningful participation in discussions. For developing a CE session on the novel collaborative care model, Katrina considers strategies for creating an inclusive space to ensure that learners are comfortable, actively engaged, and participatory. The next page explores strategies Katrina can use to foster a safe learning environment. HOW TO PREPARE © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 65 985_20230124

Strategies for creating an inclusive, safe, and supportive space HOW TO PREPARE CONDUCT INTRODUCTIONS Allow time at the beginning of the CE session for brief group introductions. This is an important step to allow the group to become familiar with each other [20]. AVOID USE OF FORMAL TITLES Facilitators should avoid the use of titles to promote an inclusive space and reduce potential disengagement due to hierarchy [20]. BALANCE LEARNER TYPES When possible, ensure groups represent an equal balance of learner types (dominant, enthusiastic, quiet) [20]. WEAR NAME TAGS AND USE TENT CARDS Having everyone wear name tags allows facilitators to address learners by their names and builds rapport with the learners [20]. ROLE MODEL APPROPRIATE BEHAVIOR SET GROUND RULES Creating and monitoring expectations throughout the session can promote shared responsibility and respect [20]. As a facilitator, demonstrate good interpersonal and communication skills. Hold yourself accountable to the same ground rules as the learners [20]. MONITOR WHO IS CONTRIBUTING Involve every learner in the discussion by moving the conversation away from dominant individuals while encouraging less dominant, quiet individuals to contribute [20]. INTERVENE WHEN NECESSARY Encourage learners to ask and answer each other's questions before intervening or providing guidance [20]. PROVIDE POSITIVE FEEDBACK Positive reinforcement of constructive contributions promotes a supportive environment and encourages learners to continue participating [18]. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 66 985_20230124

Be on time Be empathetic and considerate of others Be respectful of others' ideas and opinions Be involved in the discussion Be patient and wait for your turn to share your ideas Be objective and share only factual information Ensure confidentiality Avoid posting or sharing inappropriate materials Be careful with humor and sarcasm Critique ideas, not people Example of ground rules 1 3 2 4 5 6 8 7 9 10 Ground rules help learners understand acceptable behaviors during the CE intervention. They make the role of the learner clear in order to stimulate the best environment for learners [30]. If time permits, it can be a useful exercise to engage the group in creating the ground rules. As Katrina is designing the CE session, she identifies the following ground rules that will be introduced at the beginning of the small group learning session: HOW TO PREPARE 67 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Check to confirm that the content is relevant and suitable for attendees. Develop targeted learning objectives that are concise, measurable, and identify a specific behavior change. Create group sizes between 4 and 5 (virtual environment) learners or 6 to 8 (in-person) learners. HOW TO PREPARE Framing the problem and preparation checklist THINGS TO THINK ABOUT WHEN PREPARING FACILITATION OF SMALL GROUP LEARNING Plan out discussion points using a mind map prior to the small group learning intervention. Consider ground rules that you want to incorporate into your CE intervention. Consider how to create a safe, inclusive, and supportive environment. 68 If you are conducting the intervention virtually, refer to best practices for virtual delivery on page 45. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.1.3 IMPLEMENTATION GUIDELINES Facilitating small group learning 69 985_20230124

1 3 4 How to facilitate small group learning Briefing Introduce concepts and learning objectives [20] Leading activities Prompt discussion and gauge learner progress [20] Debriefing Close session and deliver knowledge assessment [20] HOW TO IMPLEMENT 70 2 Foster a safe, inclusive, and supportive space [20] Group dynamics © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

1 Set the stage for the session and activity. Facilitate introductions and icebreakers. Outline the objectives and provide overview of the session. Review ground rules and expectations of behavior. Briefing checklist for small group learning HOW TO IMPLEMENT 71 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Understanding and maintaining group dynamics are important skills for a facilitator. A facilitator should start the CE intervention by outlining guidelines and ground rules that will help create a positive group learning environment. The facilitator should then continue monitoring the group to ensure mutual respect is being upheld and there is equal participation among group members. Facilitators need to be aware of and then mitigate disagreements if they are escalated [26,31]. Tuckman's stages of small group development explain how group dynamics evolve through the duration of a small group learning intervention. Facilitators play an important role in guiding each stage and can use Tuckman's stages to structure their facilitation technique as well as manage group dynamics [20,32-33]. 2 Importance of maintaining group dynamics HOW TO IMPLEMENT 72 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

How to guide learners through Tuckman's stages of small group development 2 Lead introductions and icebreakers, outline objectives, and provide an overview for the session [20,32-33]. Facilitator needs to be directive to provide structure for the team and clarify expectations [20,32-33]. 1 2 3 4 5 FACILITATOR ROLE TUCKMAN'S STAGE FORMING The initial formation of the group and getting acquainted [20,32-33]. STORMING Group actively performs the task; however, some conflict may emerge [20,32-33]. NORMING Develop consensus and begin to focus on team goals [20,32-33]. PERFORMING The team is mature, well functioning, and focused on problem-solving [20,32-33]. ADJOURNING Group successfully worked together and documented results [20,32-33]. Facilitator coaches the learners by helping them focus on goals and expectations, managing the process and conflict, generating ideas, and explaining decisions [20,32-33]. Facilitator provides encouragement, helps to build consensus and gives feedback to the group [20,32-33]. Facilitator has a non-interfering role because tasks and objectives are delegated among the group [20,32-33]. Facilitator provides alternate perspectives and uses questioning techniques to engage learners' critical thinking [20,32-33]. Facilitator engages learners in the reflection process by asking questions and providing feedback to the learner group [20,32-33]. Facilitator is receptive to feedback from the learner group and uses feedback to adjust facilitation approach in future sessions [20,32-33]. HOW TO IMPLEMENT 73 985_20230124

2 Role of facilitator during small group learning Prepare learning materials Support the activities and learning process [20]. Outline guidelines and expectations [20,25]. Provide a structure Keep the group on task and focused on the topic [25]. Ensure the session runs on time by monitoring discussion and tabling discussions that run long [19]. Engage learners in discussion Ensure active participation of learners [19-20,25]. Monitor and support learners' progress [20,25]. Summarize discussion Explain and provide guidance when necessary [20]. Highlight the group's progress and identify key points that were discussed [25]. Develop learners' thinking Prompt learners with open-ended questions [20,25]. Encourage learners to move beyond memory recall and mobilize critical thinking skills [20,25]. Safe and supportive space Maintain group dynamics by upholding guidelines [19-20]. Take steps to ensure learners feel comfortable and safe participating [19-20]. Consider the following tasks when conducting a small group learning intervention to maintain group dynamics and encourage the flow of discussion: HOW TO IMPLEMENT 74 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 Cross-over groups Useful in larger groups and if the topic has multi- facets. In the first stage, subgroups discuss one facet [16-17]. Then in stage 2, the subgroups are reformed so that each new subgroup contains representatives from all the stage 1 groups [20,25]. Facilitator techniques to encourage small group discussions Seating arrangements When providing large activities, avoid using large desks, which can inhibit interaction among learners [16-17]. A circular arrangement can promote eye contact among learners and improve engagement [20,25]. Buzz groups Brief discussions in small groups where learners talk among themselves. The facilitator monitors, drops by and listens but does not actively participate in the discussion [20,25]. Snowball groups Learners form pairs and discuss a topic, then increase group size to four learners to discuss the same topic [16-17]. This technique increases the comfort of learners by starting off in a smaller pair [20,25]. Write/Pair/Share When posed with a question, learners formulate their thinking in writing before engaging in a discussion with a partner. The pair then shares their ideas with the group [20,25]. Role play Group members act out a scenario. This technique allows group members to directly apply content [16-17]. It allows for real-time feedback and for group members to develop a sense of self-efficacy [20,25]. HOW TO IMPLEMENT 75 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 Importance of questioning for small group discussions Aim of questioning Understand the needs of learners and provide adequate support [20,25]. Stimulate curiosity; sustain and initiate conversation among learners [20,25]. OPEN-ENDED QUESTIONS Capture in-depth responses, allowing the facilitator to ascertain the learners' comprehension of topic, problem-solving ability , and thinking skills [20,25]. CLOSED-ENDED QUESTIONS Yield a specific answer to the question, enabling the facilitator to check the knowledge of learners but not their level of understanding [20,25]. HOW TO IMPLEMENT 76 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 PAUSE POSE Questioning technique for facilitators Use the following questioning technique to prompt discussion and gauge learner progress. Silence can feel uncomfortable but should be welcomed as it allows learners time to process the question, think, and formulate a response [20]. With this approach, avoid immediately rephrasing the question or answering the question yourself [20]. Pose an open-ended question to the group; e.g., What challenges of the collaborative care model for patients and clinicians should we consider? Pause, allowing learners to reflect, and formulate a response. DISCUSS Engage learners to discuss the question with the group or in pairs. POSITIVE FEEDBACK Provide a positive response to encourage engagement. HOW TO IMPLEMENT 77 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

4 Debrief checklist for small group learning Summarize key points from discussion, unresolved questions, and important clinical links that have been made throughout the session. Provide feedback to learners and encourage reflection among learners. Be receptive to facilitation feedback from learners and adapt facilitation techniques for subsequent sessions. Thank learners for their contributions. 78 Consider integrating with Reflective Learning Intervention 3 HOW TO IMPLEMENT © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Identify learning objectives, expectations, and activities. Modify your involvement as a facilitator based on the group's progression through Tuckman's 5 Stages of small group development. Utilize different activities and questioning techniques to encourage discussion. Provide learners with support, materials, and structure to guide them through the CE intervention. Encourage discussion and support differing perspectives; however, mitigate conflicts within groups if they escalate. 79 Implementation checklist THINGS TO THINK ABOUT WHEN FACILITATING SMALL GROUP LEARNING HOW TO IMPLEMENT © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.1.4 KEY CONSIDERATIONS Facilitating small group learning 80 985_20230124

In-person delivery Virtual delivery Develop ground rules to foster a safe environment for learning and engagement [20,26-27]. Outline your expectations of the learner and have a plan to foster a safe and productive learning environment [20,26-27]. Communicate agenda with timelines, learning objectives, and learning materials [20,26-27]. Communicate technical requirements and set-up. Test video and audio quality as this can impact the level of interactivity within the group. Ensure an active and engaging learning environment for all participants [20,26-27]. Tips for effective delivery of small group learning KEY CONSIDERATIONS 81 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Challenges of facilitating small group learning sessions LACK OF A LEARNER- CENTRIC DISCUSSION In some cases, the group is overly reliant on the facilitator to answer questions and lead the discussion [19]. The facilitator should encourage the group to work through the questions collaboratively and ask prompting questions to help guide the group's thinking [19]. MINIMAL PARTICIPATION Small group learning is dependent on active participation to sustain discussion; however, poor preparation and disengagement can hinder critical discussion [19]. Facilitators should try engaging learners by asking direct questions to learners who look disengaged [19]. INEFFECTIVE FACILITATOR QUESTIONING If the facilitator's questions do not rise above the level of recall, this will impact the learner's development of skills to critically analyze and problem-solve clinical issues [19]. Poor questioning techniques could also impact the flow of discussion [19]. KEY CONSIDERATIONS 82 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Mitigating challenges in small group learning A short knowledge test at the beginning of the session ensures the accountability of the group [20]. Facilitators can share their clinical experience to draw connections to the topic where appropriate [23]. Divide groups into pairs for the task and positively reinforce contributions [27]. Summarize the main points discussed and divert the discussion to others by addressing the learners by their names. Provide groups with different tasks and roles to ensure equal involvement [20]. Healthy discussion and differing opinions are good in order to have critical analysis ; however, the facilitator should intervene when disagreements escalate [20]. Learners unprepared Learners are reluctant to participate Individual learners monopolize discussion Heated discussions Consider integrating with Case-Based Learning Intervention 2 83 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Sample: Planning canvas for facilitators What techniques and activities will you use to encourage discussion? What strategies can you use to create an inclusive, safe and supportive space for the learners? How will you work through conflicts that arise in groups? What group size will you choose? How will you support groups through Tuckman's Stages of group development? What challenges do you anticipate and how will you mitigate them? KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 84 985_20230124

Evaluation is an essential component of a CE session to assess knowledge uptake, identify how learners interacted with the activities, and examine whether learning objectives were effectively translated into practice. It provides an opportunity for both the facilitator as well as the learner to reflect on the session and provide constructive feedback to each other. Check out Section Four on page 155, which provides detailed strategies on how to assess and evaluate your CE interventions. Consider how you will evaluate your CE session KEY CONSIDERATIONS 85 How can I assess and evaluate the outcomes of a CE intervention by engaging stakeholders? What are some evaluation approaches I can integrate into my CE session ( knowledge tests, semi-structured interviews, feedback surveys)? What frameworks can be used to inform my evaluation questions and process ( RE-AIM, Kirkpatrick-Barr, Moore's framework)? How can I provide and utilize feedback to improve the CE session? Consider the following questions when evaluating and assessing a CE session: © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Additional resources Further reading Kumar S., Deshmukh V., and Adhish VS. Building and leading teams. Indian journal of community medicine: official publication of Indian Association of Preventive & Social Medicine. 2014;39(4), 208–213. doi: 10.4103/0970- 0218.143020 Edmunds S, Brown G. Effective small group learning: AMEE Guide No. 48. Med Teach. 2010;32(9):715-26,. doi: 10.3109/0142159X.2010.505454 Temerty Faculty of Medicine University of Toronto CPD: Choosing instructional methods and integrating active learning Tip sheet outlining instructional methods for CME/CPD design and integrating active learning 86 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.2 Case-based learning Case-based learning: an educational intervention that describes when learners learn by solving real world problems [34]. INTERVENTION TWO 87 985_20230124

In this subsection: Intervention 2 88 Introduction to case-based learning Why case-based learning? Case scenario Guiding questions Key enablers Formula for success Framing the problem and preparation How to develop a case Steps to writing a case Methods for presenting a case Implementation guidelines How to facilitate case- based learning When to present the case Importance of facilitator's role Working through the cases Questioning Dissemination Key considerations Group learning delivery methods Challenges: delivery approaches © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

The previous module provided you with best practices and implementation guidelines to facilitate a small group learning CE intervention. In this module, you will learn how to prepare and deliver case-based learning, which can also be delivered effectively in a small group setting. Key Terms: Active learning: an instructional design approach that fosters the application of knowledge, analysis, and synthesis by engaging learners through activities such as case scenarios and problem-solving [35]. Inquiry-based learning: enables learners to actively participate and construct knowledge through a self- directed learning process and problem-solving skills [36]. Problem-based learning: fostering an interactive learning experience that establishes a context focused on clinical practice through presenting and solving real-world cases [37]. 3.2.1 Introduction to case-based learning 89 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Why facilitate case-based learning ? Case-based learning (CBL) encourages learners to foster a deep approach to learning by moving from the acquisition and reproduction of knowledge to seeking meaning through the application of knowledge [38]. In contrast to the traditional didactic lecture-based teaching methods, CBL engages learners to analyze problems presented in authentic cases, make inferences based on information provided, and make decisions to simulate real-world professional context [39]. CBL education interventions for CE in healthcare require learners to build on their knowledge, collect clinical information, and synthesize complex information to formulate and test diagnostic hypotheses [34]. 90 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

The COVID-19 pandemic has contributed to a shift toward a virtual care delivery model; however, this model of care poses new challenges for healthcare providers and organizations. Many healthcare providers have found it difficult to engage with patients and deliver compassionate care virtually. The organization has asked John, a director of CE, to create a case-based learning intervention to help healthcare professionals adopt virtual care for chronic pain management. Case-based learning promotes interprofessional collaboration by sharing experiences and receiving feedback from other care providers. The case-based learning intervention is intended to foster critical thinking, problem-solving, and decision- making skills among clinicians to support this new model of care. Case scenario 91 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

As John is developing a case-based learning session, he considers the following questions: What strategies would I employ to encourage critical thinking and reflection? How would I assess learners’ knowledge comprehension throughout the session? How would I deliver the case to engage learners and/or how will I facilitate the sharing and problem-solving of current and active patient problems, and learning from those discussions? What stakeholders would I involve in the design and implementation of a realistic case? 1 2 3 4 92 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Develop with authenticity Center on an active learner experience Provide autonomy and opportunity for improvement Key enablers 93 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Develop with authenticity Case-based learning provides an opportunity to bring authentic experiences into the learning environment. It is important to include representative stakeholders in the case-study writing process [38]. When creating cases, authenticity is best achieved by staying true to a real-world example: include the main character with a problem that needs to be solved, describe the problem, and provide supporting data that will lead learners to ask questions [38]. 94 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Center on an active learning experience 20 Case-based learning centers around an active learner experience [38,40]. Learners are provided with information throughout the case that requires them to synthesize and analyze data. The process allows learners to make inferences based on the information provided and make decisions to simulate a real-world professional context [38,40]. Concepts of team-based and inquiry-based learning are important to integrate into case-based learning opportunities [38]. Note: See definition of inquiry-based learning on page 89 . 95 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Provide autonomy and opportunity for improvement Through effective facilitation, learners gain autonomy in their learning process. Case-based lear ning relies on strong facilitation to enable learners in their journey [38]. Prompts and questions that guide participants to describe ("what"), analyze ("so what") and propose an action ("now what") are effective facilitator tools in case-based learning that enable autonomy [38]. Knowledge tests allow learners to identify gaps and areas for improvement [38]. 96 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Clinicians learn best when they are presented with scenarios that are relevant to their type of practice and represent realistic problems that are just at the threshold of their capability. Cases are ideal at allowing an individual learner to identify a key challenge for them that might be different than the challenges recognized by their peers. The stories of clinical cases make them compelling and memorable. Getting feedback on sample cases is akin to using a flight simulator to learn to fly: it's important, safe, and creates lots of opportunity for practice and feedback. 97 Importance of case-based learning INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

1 Consult with stakeholders to help inform case writing and structure learning objectives. 2 3 Formula for success Case studies should be reflective of authentic patient cases. Select an optimal delivery method (i.e., written case, video case, didactic lecture) based on your learning objectives. 98 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.2.2 FRAMING THE PROBLEM AND PREPARATION Case-based learning 99 985_20230124

HOW TO PREPARE Types of Cases 100 Case Type Directed Case Facilitator presents a case scenario to the learners, followed by a discussion. The discussion is structured based on close-ended questions [41]. Description Competency Dilemma or Decision Case Learners are presented with a problem to work through and discuss [41]. Knowledge comprehension of foundational concepts Learners work on analyzing a retrospective case and its outcomes. They reflect on the scenario presented and engage in a discussion [41]. Analysis or Issue Case Learners are presented with the case in multiple parts and are encouraged to make a decision prior to working on the next component [41]. Learners present a scenario or problem in their current practice setting and engage in a discussion with peers to solve at the moment. Problem-solving and decision- making skills Interrupted Case Spontaneous Case Problem-solving skills Analytical skills Problem-solving and decision- making skills There are many different types of cases that can be used as the foundation for a CE session. John considers the following approaches for facilitating a case-based learning activity: © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Enables healthcare professionals to apply their learning in subsequent clinical practice [34]. Structured approach that can be adapted to meet the needs of various CE programs [34]. CASE METHODOLOGY How to develop a case HOW TO PREPARE Collaborative approaches to development Succinct clinical scenarios Authentic clinical scenarios Embedding the best available evidence C S A E 101 Below is a common methodology for developing a case [34]; however, this is just one way to conduct a case-based learning session. Sometimes cases can be spontaneously generated by learners or the CE providers or facilitators may identify a helpful case from personal experience that helps illustrate key points. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

How to develop a case HOW TO PREPARE Description Principle Actions Collaborative/ Interdisciplinary Authentic Succinct Evidence-based Integrate an interdisciplinary approach when developing a case [34]. Form an interdisciplinary team that includes subject matter experts, instructional designers, and clinicians [34]. Engage healthcare professionals and patients in the development process and use real case scenarios [34]. Ensure the cases are reviewed by experts and end- users [34]. Examine current guidelines and literature before developing a case materials. Develop cases that are realistic to healthcare professionals and patients [34]. Cases should be written clearly and concisely [34]. Learning materials and resources presented to resolve the case should be grounded on evidence. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 102 985_20230124

Steps to writing a case study HOW TO PREPARE D E V E L O P A G U I D E The guide will include a summary of the literature, gap analysis, and case examples [34]. P R E P A R E A N D D R A F T C A S E S Write cases based on real scenarios that will be relevant to care providers. Keep cases short and use pseudonyms to protect identities [34]. R E V I E W C A S E S W I T H C O N T E N T E X P E R T S It is important to review the cases for consistency and adequacy in assessing the learning objectives [34]. O R G A N I Z E A W O R K S H O P Review literature and gap analysis to identify learning outcomes. Members should have a common understanding of the case process [34]. John considers the following steps when writing his case study on virtual care model. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 103 985_20230124

Written cases include an in-depth description and analysis of the clinical scenario [42]. It provides novel insights to support care providers in optimizing the delivery of care. Cases include details such as assessment findings and test reports [42]. Written case HOW TO PREPARE Methods for presenting a case Video scenarios are filmed based on cases created by subject matter experts [43]. Learners receive the video case study during the CPD session and are also provided with a manual containing the written case, the learning objectives, and relevant information necessary for group discussions [43]. Video case A subject matter expert presents the topic and then allows learners to present cases on a voluntary basis [44]. Participants can informally discuss cases that are related to the topic and are engaged in a discussion that can directly impact their practice [44]. Didactic lecture © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 104 985_20230124

Engage stakeholders to help inform case study writing and structure learning objectives. Conduct a literature review and gap analysis. Framing the problem and preparation checklist THINGS TO THINK ABOUT WHEN PREPARING CASE-BASED LEARNING Organize a workshop to identify priority setting and learning outcomes. HOW TO PREPARE Prepare and draft short cases relevant to the target audience. Pilot test the cases with a cohort of learners. 105 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.2.3 IMPLEMENTATION GUIDELINES Case-based learning 106 985_20230124

1 2 3 How to facilitate case-based learning Present the case Debrief the case Provide learners with adequate time to read the case. Set goals and identify the problem. Brainstorm approaches. Synthesize solutions and reach a consensus. Deliver solutions and engage in knowledge exchange. Discuss the case HOW TO IMPLEMEMT 107 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

When to present the case Learner role Learners read, analyze and collaboratively propose a solution to the problem identified in the case [44]. Learners use concepts from the CE session to read, analyze, and complete a case individually [45]. Provided and discussed during the CE session Provided after t he CE session and completed individually Learners work through the case together and can use interdisciplinary perspectives to create a solution [44]. Provides a method to test knowledge uptake during the CE session and learners can work at their own pace [45]. Advantages Requires an increased level of learner engagement to direct this process and may require longer sessions to complete the case [44]. Absence of interdisciplinary perspectives to help analyze the case. Learners will not receive the support of a facilitator [45]. Disadvantages 1 Learners are required to review t he case and prepare a solution to the problem identified. During the session, learners work with their peers to come to cons ensus on a solution [38]. Provided prior to the CE session and discussed during the CE session Learners come to sessions with solutions rather than working together to come up with a solution [38]. Allows for shorter sessions because learners come to the session prepared with points [38]. 108 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Examine the case through social learning 2 Cases can be examined through social learning. Social learning is a cognitive process, which occurs through the interactions with peers in a social context [46-47]. Observing the behaviors of others and direct reinforcement are important elements in the transfer of knowledge [46-47]. Role-playing is a valuable strategy to facilitate perspective-taking in solving cases. However, there can be challenges to learner engagement, which could have an impact on the learning process [47]. When developing a case-based learning session to help healthcare professionals adopt virtual care for chronic pain management, John considers strategies for mitigating barriers to learner engagement. The next page explores strategies for how John can develop cases to foster a meaningful learning experience. 109 HOW TO IMPLEMEMT © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Barriers to learning engagement Environmental distractions Minimal attention span Overconfidence or uncertainty Lack of interest and motivation 1 3 2 4 5 Selecting a case that is relevant to your target audience [47] Developing cases that provide a unique learning experience and build on existing knowledge [47] Establishing trust, which creates a safe space for discussion and critical feedback among peers [40] Integrating group activities and providing opportunities for learner interaction [43] Utilizing questioning techniques to prompt discussion [20] Providing guidance when necessary [20] Barriers to learning engagement could be addressed: Mitigation Strategies 2 110 HOW TO IMPLEMEMT Case considered irrelevant or obtuse by the learner 6 Fatigue © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2 Facilitate and moderate discussions Provide stimulus to the process by asking leading questions, challenging thinking and raising issues that might need to be considered [43]. Encourage learners to set the pace of the activity [43]. Ask questions that are open (and avoid questions that have single correct answers). Support and monitor participants Support learners to relate their work on the case to the learning objectives of the course or session [43]. Attend to the needs of the learner group and aid when needed [43]. Demonstrate knowledge of topic Discuss with learners to ensure they can connect the topic to their broader clinical practice [43]. Share professional experience to connect the case with practical context [43]. Create safe environments Promote a safe and inclusive environment to encourage and engage learner participation [43]. Foster an atmosphere in which open exchange of ideas is facilitated [40]. Outline expectations of respect and inclusion to the group [43]. Engage learners Encourage interactivity through the use of various features in video-conferencing platforms (e.g., chats, polls). Ask for feedback from learners to address their learning needs [43]. Actively listen and validate contributions from group learners [43]. Facilitator's role in case-based learning discussion Consider integrating with Facilitation of Small Group Learning Intervention 1 111 HOW TO IMPLEMEMT © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2 Working through the cases The environment sets the stage for collaboration; therefore, the setup should be meaningful to promote discussion [38,44]. In-Person: Create an open space by moving tables and chairs around in a conference room Virtual: Breakout rooms for small discussions Easy to access video conferencing links Intentional environment setup 112 HOW TO IMPLEMEMT Engage learners to perform open inquiry through guiding questions and sharing their varying multidisciplinary expertise [38,44]. When the learning objectives include team-based or interprofessional competencies, create groups that are representative of diverse disciplines and demographics to help plug knowledge gaps [38,44]. Promote collaboration © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2 Working through the cases Case-based learning operates most effectively when it is learner-directed and facilitator-monitored [40]. Encourage learners to: Choose how to approach the case Delegate tasks and roles Use their experiences to inform the case analysis Learner-directed 113 HOW TO IMPLEMEMT Facilitators play a role in ensuring that learners understand the tasks and topic of the case. If learners have questions, they should first seek answers within their group [40]. However, if the group continues to have difficulty, facilitators can provide some clarification or open-ended questions to guide them [40]. Ask "Why" or "How" questions to prompt discussion. Check for understanding © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Questioning in case-based learning 3 Module 3 HOW TO IMPLEMEMT 114 John uses the following questioning framework to prompt discussion in his session. DESCRIPTIVE ANALYSIS PROPOSED ACTION Help learners dissect what has happened in the case and identify what information is available [45]. "What" Detail what is expected of the group and what are different ways they can proceed with the case [45]. "So what" Come up with a plan of action based on the details of the case and discussion [45]. "Now what" © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 Dissemination of case analysis Submit the report during or after the session [42]. Explains the solution and how it relates to the curriculum [42]. Outlines the process of case analysis [42]. Learners: Facilitator: Written report Create a slide deck to present the solution [43]. Highlights the steps taken to analyze the case and connect it to the curriculum [30]. Interacts with session participants by asking for feedback and for any questions they might have [43]. Learners: Facilitator: Formal presentation Build on each other's ideas to explain possible solutions and address challenges [44]. Collectively discusses with larger group the process the smaller groups took to approach the case [44]. Learners: Facilitator: Post their solutions to an online discussion board and openly comment on their peers' so lutions [37]. Peers provide feedback and critique post-session [37]. Learners: Informal open-discussion Online discussion board The outcomes of case analysis can be varied and will be dependent on the curriculum. 115 John can consider the following examples to engage learners in synthesizing their case results. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 How to debrief the case Debriefing is a direct and intentional conversation among the learner group to summarize the session and come to conclusions about the case [37]. It usually takes place after the group has worked through the case and has synthesized the results [37]. John uses the following steps to engage the learners in debriefing the case during his CE session: Prompt learners with questions to get them to reflect on the key takeaways from the case [48]. Give individual learners or groups an appropriate amount of time to generate their responses and record answers either on paper or through an electronic poll [48]. Share ideas or thoughts about the case between groups or in pairs [48]. Engage each pair or group to share their responses with the larger group and participate in an in-depth discussion [48]. The facilitator shares their own key takeaways and relates them to the points raised by the learners. At this point, the facilitator fills any gaps and clarifies concepts covered in the case [48]. 116 HOW TO IMPLEMEMT Consider integrating with Reflective Learning Intervention 3 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Consider when to present the case to learners (before, during, or after the session). Foster an inclusive and safe learning atmosphere. Implementation checklist THINGS TO THINK ABOUT WHEN DELIVERING CASE-BASED LEARNING Incorporate questioning strategies as part of the instructional design to encourage critical thinking and fruitful discussion. Maximize the transfer of learning by creating a collaborative learning space. HOW TO IMPLEMEMT Select the best approach to disseminate the case analysis and debrief. 117 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.2.4 KEY CONSIDERATIONS Case-based learning 118 985_20230124

Group learning delivery methods Small group delivery Help guide and monitor discussion [49]. Only assist when the group requires help or asks for help [49]. Provide feedback to the group and ask open-ended questions to prompt discussion [49]. From four to eight participants work on completing the case [49]. Description Facilitator role Present case to the group [44]. Active role in facilitating and leading discussion [44]. Groups larger than 20 learners work on analyzing a case. The session usually integrates a didactic lecture to complement discussion [44]. Large group delivery Mitigate technology difficulties [50]. Create discussion groups/breakout rooms [50]. Manage learner discussion and encourage participation from all learners [50]. Inform participants of video conferencing tools that can be used [50]. The use of video conferencing platforms to conduct case-based discussions and activities [50]. Virtual/blended delivery Here are some considerations when selecting the delivery methods: 119 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved Engage in active participation [49]. Create self-directed plan on how to approach case [49]. Be attentive and respectful of group’s ideas [49]. Actively participate and contribute to the case discussion [44]. Integrate simple large group collaborative approaches such as think-pair-share. Leverage tools on video conferencing platforms to participate [50]. Actively join the discussion with the camera on or participate through the chat box function [50]. Learner role 985_20230124

Challenges: Delivery approaches KEY CONSIDERATIONS In-person Virtual or blended 01 02 Access to technology Lack of digital literacy Challenge in monitoring virtual learner progress [49]. Difficulty monitoring active participation and learner engagement if the camera is off [49]. Mitigation: Set expectations at the beginning of the session; reinforce throughout the session. The malfunctioning microphone may limit participation [49]. Mitigation: Encourage the use of the chat box if the microphone is not working. Unequal or lack of participation can affect case analysis and group discussion [49]. A facilitator who too actively participates takes away important decision-making opportunities from learners [49]. Mitigation: The facilitator stays within the role and only monitors. Physical Setting: availability of an open and movable setting may not always be an option, which could detract from a case discussion and analysis [49]. 120 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Sample: Planning canvas 121 Who would you engage to develop the case? How would you present the case (before, during or after the session)? When would you present the case? What delivery approach would you select? How would you debrief the case? How would you work through the case? KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Evaluation is an essential component of a CE session to assess knowledge uptake, identify how learners interacted with the activities, and examine whether learning objectives were effectively translated into practice. It provides an opportunity for both the facilitator as well as the learner to reflect on the session and provide constructive feedback to each other. Check out Section Four on page 155, which provides detailed strategies on how to assess and evaluate your CE interventions. Consider how you will evaluate your CE session KEY CONSIDERATIONS 122 How can I assess and evaluate the outcomes of a CE intervention by engaging stakeholders? What are some evaluation approaches I can integrate into my CE session ( knowledge tests, semi-structured interviews, feedback surveys)? What frameworks can be used to inform my evaluation questions and process ( RE-AIM, Kirkpatrick-Barr, Moore's framework)? How can I provide and utilize feedback to improve the CE session? Consider the following questions when evaluating and assessing a CE session: © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Additional resources Further reading McLean SF. Case-Based Learning and its Application in Medical and Health-Care Fields: A Review of Worldwide Literature. J Med Educ Curric Dev. 2016;3. doi: 10.4137/JMECD.S20377 Leung JS, Brar M, Eltorki M, Middleton K, Patel L, Doyle M, et al. Development of an in situ simulation-based continuing professional development curriculum in pediatric emergency medicine. Advances in Simulation. 2020;5(1):12. doi: 10.1186/s41077-020-00129-x 123 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.3 Reflective learning Reflective learning: a type of learning where learners critically reflect upon their own thoughts, behaviors and actions in practice scenarios [51-52]. INTERVENTION THREE 124 985_20230124

In this subsection: Introduction to reflective learning Framing the problem and preparation Implementation guidelines Intervention 3 Key considerations Determine learning goals for reflective learning When to use reflective learning Importance of facilitator's role Brave space for learning Stages of critical reflective inquiry model Applications of the critical reflective inquiry model Question prompts for reflective learning Key principles for facilitation of reflective learning Considerations for facilitating reflective learning Challenges Using reflective practice within a virtual context Why reflective learning? Case scenario Guiding questions Key enablers Formula for success 125 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.3.1 Introduction to reflective learning The previous modules provided you with the necessary guidelines and tools to implement small group and case-based learning sessions. This module will prepare you to implement reflective learning in a CE session. Note that reflective learning can be integrated as a component within small group or case- based learning sessions. Key Terms: Brave Space: enable a safe learning environment, where learners engage authentically with their peers in challenging discourses [53-54] . Critical Reflective Inquiry (CRI): this model enables care providers to learn from their experiences and evaluate by exploring the knowledge, beliefs, and attitudes ingrained in clinical practice settings [51]. 126 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Why facilitate reflective learning ? In a rapidly evolving healthcare system with continued importance on lifelong learning, it is critical for clinicians to continuously update their knowledge and reflect on their clinical practice [56]. Reflective learning provides an opportunity for healthcare practitioners to deliberately think about their clinical environment; subsequently, this educational intervention plays an influential role in promoting practice change [55]. Reflection encourages healthcare practitioners to examine their daily actions and the nature of their work routines, which is imperative in ensuring the quality of healthcare provision . Reflective learning offers a pragmatic avenue to advance CE practices [52]. 127 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Due to the COVID-19 pandemic, there has been an increased burden on the healthcare system and a rapid shift in clinician workload. At a hospital, there has been a shortage of staff and high turnover. The staff are very stressed with all of the required work and are reporting high levels of burnout. Aliyah, an education coordinator, has been tasked with creating a CE program to promote wellness and principles of resilience among clinicians. The planning committee is considering incorporating ways to reflect on practice to encourage self-awareness and a commitment to lifelong learning. Case scenario 128 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

5.1.2 As Aliyah is developing a reflective learning session, she considers the following: How would I ensure a safe and inclusive learning environment for discussion? How would I encourage learners to participate in reflective learning? How would I integrate reflective learning within a virtual context? How would I integrate reflective learning into the CE session to promote the wellbeing of clinicians? 1 3 4 5 129 Why should I consider adding reflection into the learning plan? 2 6 How do I assess reflections, follow-up and give feedback to ensure the integrity of the activity? © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Key enablers Identify learning needs and practice gaps Create a supportive and continuous learning process Establish a brave space for learning 130 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

It is critical to identify learning needs and practice gaps as reflection is based on reconceptualization of past experiences that lead to further reflection and inform new experiences [57]. Using a structured approach helps to transform the practice experiences of care providers into tacit—rather than only explicit—knowledge. This leads to increased self-awareness and identification of gaps in knowledge and skills, thereby improving practice change and patient care delivery [51]. Identify learning needs and practice gaps 131 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Supportive and continuous learning process Reflective learning is a process rather than a single-time event. To do this well, facilitators should act as guides to learners, supporting them through the process of reflection [57-59]. Framing and reframing, open dialogue, and helpful feedback encourage learners (and facilitators) to identify gaps, reflect on new material, situate it into their own personal contexts, and encourage continual learning and practice change [57-59]. 132 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Create a brave space for learning Brave spaces enable a safe learning environment, where learners engage authentically with their peers in challenging discourses [53- 54]. Although brave spaces can create discomfort for learners, they provide resources and support to those who are most sensitive and facilitate a deep and extensive conversation [53-54]. Support is intended to enhance and not draw away from participation and learning opportunities [53-54]. The best reflections come from being in a position of discomfort; thus, educators who wish to stimulate more meaningful reflections need to be willing to be provocative, sometimes profoundly so. Thus, they need to create not only safe spaces, but brave spaces. 133 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

1 2 3 Develop contextual and situated reflective learning activities. Formula for success Facilitate with clear guidance and practical goals, ensuring continuous integration into context. Evaluate with the learner's end goals in mind, providing feedback in a safe and productive way to help identify gaps and integrate new knowledge into practice. 134 INTERVENTION INTRODUCTION © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.3.2 FRAMING THE PROBLEM AND PREPARATION Reflective learning 135 985_20230124

HOW TO PREPARE Determine learning goals for reflective learning When determining learning goals, CE leaders or educators should consider the following guiding questions: Are there any gaps in knowledge, skills, and attitudes that should be addressed [57]? How can reflective learning be used to help integrate care providers' new learning with existing knowledge and past experiences and incorporate it into current or future practice [57]? Is the aim of reflective learning to identify practice needs or focus on learning [57]? 136 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

HOW TO PREPARE When to use reflective learning Reflection can take place at any point of an event or situation: before, during and after [59]. Reflection prior to an encounter can contribute to self-development and lifelong learning. This allows learners to challenge their perceptions or approach a situation with a specific learning objective [59]. Reflective learning can be integrated within a larger CE intervention to maximize the learning experience [57]. 137 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

The presence of a brave space in place of a safe space encourages a positive view of critical reflection. This can shift learner mindset in seeing reflection as a necessary part of small-group learning [53-54]. 02 Create Brave Spaces 03 Facilitate Discussion Importance of facilitator's role in reflective learning Promote Deeper Reflection 01 Encourage learners to consider their scenario beyond what was initially presented. This level of self-awareness will enable learners to understand how they can make positive changes to practice [52,55,60]. Learners can present their narratives to analyze in a small group using the critical reflective inquiry model [51]. Learners can act as peer facilitators in reflective analysis and discussion. HOW TO PREPARE 138 Consider integrating with Facilitation of Small Group Learning Intervention 1 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

ENGAGE LEARNERS OPEN DIALOGUE CREATE A SPACE 1 2 3 Facilitate an open dialogue by sharing diverse viewpoints respectfully [61]. Engage the learners in co-creating ground rules or amending existing rules [61]. Provide an equitable and inclusive space to share stories and thoughts [61]. What is brave space Brave spaces are necessary for creating reflective learning since the learning stimulus can be necessarily provocative. Brave spaces can be established through creating classroom guidelines and etiquette [61]. An emphasis is placed on social justice, thereby creating a space that is equitable and inclusive of all social identity groups [61]. Key principles that promote brave space include controversy with civility, owning intentions and impact, challenge by choice, and respect [61]. BRAVE SPACE HOW TO PREPARE 139 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

CONTROVERSY WITH CIVILITY Diffe rent views are encouraged. Learners work to understand the cause for conflicting perspectives and collaboratively work together to identify a common solution [53-54]. Create a brave space for learning OWNING INTENTIONS AND IMPACT Facilitators foster a safe learning environment that encourages learners to engage with less fear and greater honesty [53-54]. CHALLENGE BY CHOICE Learners have the choice to determine the activities in which they would like to engage. During a challenging conversation, facilitators can probe learners to stimulate discussion [53-54]. RESPECT It is imperative to show respect to one another and understand the various cultural values and beliefs individuals bring with them [53-54]. HOW TO PREPARE 140 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Determine learning goals for reflective learning. Determine when to integrate reflective learning with your CE intervention. Framing the problem and preparation checklist THINGS TO THINK ABOUT WHEN PREPARING REFLECTIVE LEARNING Create a brave space for learning. HOW TO PREPARE 141 Plan the stimulus that will activate the learners, create engagement and generate reflective thinking. Plan for how to record and retain the reflection(s) from each learner. Plan for how to use the recorded reflections to facilitate and encourage behavior or practice change. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.3.3 IMPLEMENTATION GUIDELINES Reflective learning 142 985_20230124

Assist learners in examining their narratives using reflective analysis to compare their feelings, thoughts and behaviors to [51]: scientific knowledge/ethical standards their own intentions This process will facilitate self- awareness among learners on how they deliver care and how they can improve their own practices [51]. REFLECTIVE Encourage learners to critique their own practices to remove inefficiencies and incorporate new developments where necessary [51]. Facilitate dialogue for learners to examine disparities between their own [51]: beliefs vs practice, intentions vs actions, patient needs vs learner actions. CRITICAL Consider whether to ask learners to draft a written narrative of a practical situation they wish to analyze using reflection [51]. These narratives should include all characteristics of the scenario, including how the learner felt, thought and acted [51]. DESCRIPTIVE HOW TO IMPLEMENT 143 Stages of critical reflective inquiry model (CRI) Use the CRI model to facilitate reflective learning [51]. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Applications of the critical reflective inquiry model Knowledge creation: Reflective analysis of narratives can add to current best practices and applications of knowledge to practice. It can be used to guide learners' own practice [51]. Improving practice: Encourages continuing professional development of learners [51]. Knowledge transfer: This model can be used in group settings such as conferences to analyze and discuss practice scenarios. Learnings can be applied to individual practice [51]. Personal action plan: Action plans can be created by learners to help them reach their goals. HOW TO IMPLEMENT 144 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Encourage learners to create a plan of action based on their practice situation. Encourage learners to compare their situation to standard practices. Implementation checklist THINGS TO THINK ABOUT WHEN USING REFLECTIVE LEARNING Support peer facilitation and creation of brave spaces for dialogue. HOW TO IMPLEMENT 145 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3.3.4 KEY CONSIDERATIONS Reflective learning 146 985_20230124

1 2 3 Key principles for facilitation of reflective learning Clear guidance Integration Provide learners with clear guidance on critical reflection [57]. Frame reflection as a practical goal (e.g., enhancement of patient care) [62]. Integrate reflective learning into existing curricula and design of competency-based assessments [63]. Practical goals 147 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Considerations for facilitating reflective learning Identifying learning needs based on a critical reflection of one’s experience and practice is vital [64]. Focus on integrating new learning with existing knowledge and stimulating cognitive experience [57]. Identify learning needs It is critical for practitioners to understand their personal beliefs, attitudes, and values. Reflection provides an explicit approach to their integration [64]. Understand values An active approach to learning is important for creating an integrated knowledge base that leads to the consolidation of new and existing knowledge [64]. Active learning Module 1 148 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

LACK OF TIME Acquiring time away from clinical practice can pose a challenge to participation in reflective learning sessions. Additionally, time to complete reflection exercises and portfolios may potentially hinder participants from actively participating [51,56,64]. MINIMAL ENGAGEMENT Learners may be guarded and reluctant to be transparent when asked to share their gaps in knowledge and/or deficiency in skills. Reflective learning was perceived to decline among clinicians who have been practicing for several years [59]. DIFFICULTY WITH ASSESSMENT Assessing reflective learning is challenging due to the absence of validated tools, insufficient resources, and the uncertainty of the learning objectives. In addition, assessment length can contribute to fatigue and learners may not be honest during reflective activities [56]. Challenges with reflective learning 149 KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Using reflective practice within a virtual learning context Digital storytelling Virtual expert panel 01 02 Activity is presented with an image and guiding questions [65]. Chat functionality can be used to facilitate and engage in discussions with panelists [65]. In this way, participants can reflect on diverse perspectives of the same situation [65]. 1. 2. 3. Learners take photographs to express emotions before, during, and after a clinical scenario [59]. Learners reflect on experiences [59]: How they dealt with the situation. What they would do differently next time. Learners present their digital story (e.g., MS Powerpoint/MS Photostory) [59]. Learners can also create an ePortfolio with a collection of stories [59]. 1. 2. a. b. 3. 4. 150 KEY CONSIDERATIONS Aliyah considers the following examples for delivering the CE session virtually: © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Sample: Reflective practice Question 1: Provide a brief description of the situation. How did you respond? How did you feel during and after the event? 1 2 3 What happened Looking back Looking forward Topic you are reflecting on: How would you describe this event (describe your feelings)? 151 Question 2: Do you think you have achieved your learning goals? Why or why not? Are you satisfied with how you handled the situation? Question 3: What lessons did you learn from this scenario? What is one thing you want to remember or change from this activity? What do you plan to do differently now? © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Sample: Personal action plan 1. What changes do you want to make as a result of attending this CE session? Aim to define at least two specific and measurable changes. 2. How important is it for you to achieve your stated learning objectives for this session on a scale from 1-5? (1=least important, 5=most important) 1 2 3 4 5 3. How confident are you that you can achieve your goal(s) on a scale from 1-5 (1=least confident, 5=most confident) 1 2 3 4 5 4. Define a timeline for your strategy. When do you plan to start, assess, and finish the process? Start Date: Assess Date: Finish Date: 5. How will you know that you have reached your goal? What and how will you measure it? 6. What barriers will you encounter? What strategies could you employ to address them? 7. What resources do you have or need to achieve your goal? 152 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Evaluation is an essential component of a CE session to assess knowledge uptake, identify how learners interacted with the activities, and examine whether learning objectives were effectively translated into practice. It provides an opportunity for both the facilitator as well as the learner to reflect on the session and provide constructive feedback to each other. Check out Section Four on page 155, which provides detailed strategies on how to assess and evaluate your CE interventions. Consider how you will evaluate your CE session KEY CONSIDERATIONS 153 How can I assess and evaluate the outcomes of a CE intervention by engaging stakeholders? What are some evaluation approaches I can integrate into my CE session ( knowledge tests, semi-structured interviews, feedback surveys)? What frameworks can be used to inform my evaluation questions and process ( RE-AIM, Kirkpatrick-Barr, Moore's framework)? How can I provide and utilize feedback to improve the CE session? Consider the following questions when evaluating and assessing a CME/CPD session include: © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Additional resources 154 Further reading NHS Training for AHP Support Workers: Workbook 10: Reflective Practice and Continuing Professional Development (CPD) Portfolios Workbook that outlines how health care professionals can engage in reflective practice and continuing professional development Royal College of Radiologists: Reflective learning template Template for personal reflection on CPD activities Health and Care Professions Council: Reflective practice template Template outlining prompts to engage individual learners or groups in reflection KEY CONSIDERATIONS © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Quick guide to assessing and evaluating CE interventions SECTION FOUR 155 985_20230124

In this section: Section 4 4.1 Assessing knowledge comprehension Knowledge comprehension assessment 4.2 Session feedback 156 Identify key stakeholders Logic model Outcome frameworks for evaluation Evaluation approaches Providing feedback on the CE session 4.3 How to evaluate the intervention © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Continuous evaluation is important in understanding the efficacy of educational interventions and determining whether the learning has been translated into practice. The evaluation aims to improve the longitudinal and sustainable adoption and implementation of evidence-based CE interventions for healthcare professionals [7]. Assessment, feedback, and evaluation HOW TO EVALUATE SESSION FEEDBACK 01 KNOWLEDGE COMPREHENSION 02 EVALUATION OF INTERVENTION 03 157 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

4.1 Assessing knowledge comprehension of learners Goals Assess the learner's knowledge and/or attitude retention. Requires post- session assessment [66]. Modality Knowledge, skill, and attitude assessments are largely performed using multiple-choice questions and case-based vignettes in the form of tests or surveys. Questions should be short and framed around the learning outcomes of the session [66]. Knowledge comprehension assessments, while helpful, are not a measure of actual or anticipated practice performance [66]. Purpose Goals of CE interventions include assessing learners’ problem- solving, decision-making skills, and knowledge uptake. However, knowledge comprehension assessments can be used to identify the extent of knowledge uptake within the group [66]. Although comprehension assessments are not mandatory, they can serve as a form of reflection and allow learners to set benchmark goals in areas they want to work on more [66]. Assess the change in the learner's knowledge and/or attitude. Requires a pre- and post-session assessment [66]. HOW TO EVALUATE 158 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

4.2 Providing feedback on the session Provision of feedback promotes learning by informing learners of their progress, highlighting areas of improvement, and providing learners with an opportunity to evaluate their own performance [27]. It also assists in self-reflective practices for facilitators to identify areas to improve on and strategies that worked well with the learners [27]. The following diagram outlines the Pendleton feedback model to engage learners in providing instant facilitator feedback [27]. Positive Improvement Action Plan Facilitator Learner Summary Highlights what went well Highlights what could have gone better Suggests action plan Summarizes key points Comments on what went well Comments on what could have gone better Describes and agrees on action plan Wraps up and implements in next section HOW TO EVALUATE 159 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

Create a logic model to guide the evaluation process CREATE A LOGIC MODEL 01 Identify key stakeholders and write them down. IDENTIFY STAKEHOLDERS 03 Consider outcome frameworks when selecting outcomes. SELECT OUTCOMES 02 4.3 How to evaluate the intervention Evaluation is important in ensuring your intervention meets the intended goals and whether it was implemented as intended. This section will provide you with the tools to appropriately and effectively evaluate CE education interventions. Below is a three-step process to help guide you through the evaluation and identify potential opportunities for improvement. 160 HOW TO EVALUATE © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

1 PATIENTS AND FAMILIES Identify the purpose and impact of the education intervention. Identify key stakeholders ADMINISTRATORS HEALTHCARE PROFESSIONALS HEALTH SYSTEM LEADERS AND EDUCATORS Who are your stakeholders? Write them down. 161 Stakeholder engagement is a continuous process. PURPOSE IMPACT STAKEHOLDERS Stakeholders include those who are directly or indirectly impacted. HOW TO EVALUATE © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2 STAKEHOLDERS PURPOSE INPUTS ACTIVITIES OUTPUTS OUTUTS SHORT OUTUTS MEDIUM OUTUTS Create a logic model OUTCOMES Who are your stakeholders? What is the problem? What resources are required? What are the specific actions that make up the intervention? What specific activities will be produced? What changes do you anticipate as a result of the intervention? LONG EDUCATION INTERVENTION DELIVERED EDUCATION INTERVENTION RESULTS ASSUMPTIONS What are some assumptions and factors not under your control? 162 The logic model is intended to be a process that illustrates the pathway of change from developing the evidence-based interventions to the outcomes measured in the evaluation. © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

2 STAKEHOLDERS INPUTS ACTIVITIES OUTPUTS OUTUTS SHORT OUTUTS MEDIUM OUTUTS Example of a logic model OUTCOMES Administrators Healthcare professionals Health system leaders Educators CE providers Goal: To develop a case-based learning session for CE that drives sustained mastery and performance of professional capability. Time Financial resources Staff CE experts Admin support Conduct a needs assessment Needs assessment report LONG ASSUMPTIONS 163 Determine learning objectives Prepare and draft cases Conduct the session Evaluate Case studies Education intervention Evaluation report Increased self- awareness Increased skills and abilities to advance their own CE Increased motivation and confidence in knowledge accquired Change in attitude towards lifelong learning Increased critical thinking and decision- making skills Empowerment, self-sufficiency, and self- directing Optimize and provide safe and compassionate care for better patient outcomes Knowledge used in and developed from practice Working in partnerships with key stakeholders - serve as catalysts for change HOW TO EVALUATE © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

STAKEHOLDERS INPUTS ACTIVITIES OUTPUTS OUTUTS SHORT OUTUTS MEDIUM OUTUTS Practice: Create your own logic model OUTCOMES Goal: LONG ASSUMPTIONS HOW TO EVALUATE 164 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

3 Select an outcome framework for evaluation CE leaders can leverage the following frameworks to develop an education intervention that is learner-centric. Outcomes are changes in learners' knowledge, skills. and attitudes. The frameworks below can be used to form your evaluation questions (e.g., surveys and/or interview guides). Please see pages 166-176 to guide you through this process. The choice of the evaluation framework is often dependent on the context of the intervention. HOW TO EVALUATE The RE-AIM framework could be utilized as a guiding framework for evaluating the intervention [67-68]. Reach Effectiveness Adoption Implementation Maintenance RE-AIM Framework The Kirkpatrick-Barr framework serves as a useful guide in assessing the learners' educational outcomes [69- 70]. Level 1: Reaction Level 2: Learning Level 3: Behavior Level 4: Performance Kirkpatrick-Barr Framework Moore's Framework The Moore's framework could also be used to assess the learners' educational outcomes [71] . Level 1: Participation Level 2: Satisfaction Level 3: Learning (declarative and procedural knowledge) Level 4: Learning (competence) Level 5: Performance Level 6: Patient Health Level 7: Community Health Note: Both Kirkpatrick-Barr and Moore's frameworks assess the learners' educational outcomes. You could use either framework. 165 © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124

RE-AIM framework Use the RE-AIM framework to assess whether the CE education intervention is effective in driving current and sustained mastery and performance of professional capability. Below are some guiding questions to help you evaluate [3]: Reach Element Description Guiding Question Number, proportion, and representativeness of healthcare professionals taking part in the CE intervention [3]. Effectiveness Adoption Implementation Maintenance Impact of the CE intervention on key outcomes [3]. How do I reach the intended population with the educational intervention [3]? How do I know my educational intervention is effective [3]? Number, proportion, and representativeness of organizations and leaders/educators willing to initiate a CE intervention [3]. How do I support organizations to deliver this educational intervention successfully [3]? Delivery of the CE intervention (i.e., consistency of delivery, time, and cost) [3]. How do I ensure the educational intervention is delivered effectively [3]? How do I sustain the educational intervention in the long term [3]? Long-term impact of the CE intervention on outcomes [3]. Tools Pre/post quantitative assessments Semi-structured interviews Surveys Feedback or debriefs at the end of the session Usability metrics Practice audit Self-reported data Observations 166 HOW TO EVALUATE © 2022 by the Accreditation Council for Continuing Medical Education (ACCME®) – All Rights Reserved 985_20230124