Assessment and Evaluation Methods Flashcards

A simulation educator runs a 'mock code' scenario for residents halfway through their rotation. The educator provides immediate, targeted feedback on performance using a checklist but does not assign a grade. The primary goal is to identify areas for improvement before the end of the rotation. This is an example of which type of assessment?

Answer: Formative assessment

Formative assessment is conducted during the learning process to monitor progress and provide ongoing feedback to improve learning. [13, 21, 28] Since the goal is to identify areas for improvement without assigning a final grade, it is formative. Summative assessment evaluates learning at the end of an instructional unit. [21, 27]

An educator is designing an assessment for a complex skill, such as breaking bad news, which involves empathy, rapport-building, and adapting to emotional cues. Which assessment tool is most appropriate for capturing the nuances of a learner's performance in these domains?

Answer: A global rating scale with descriptive anchors

A global rating scale (GRS) is better suited for assessing complex skills and behaviors like communication and professionalism. [8, 12, 17] Unlike a binary checklist that focuses on discrete actions, a GRS with descriptive anchors allows raters to make a holistic judgment about the quality of a performance across a continuum. [12]

A simulation program is implementing a high-stakes Objective Structured Clinical Examination (OSCE) to credential learners for a specific procedure. To ensure fairness and consistency in scoring across all participants, which of the following is the most critical element to implement?

Answer: Conducting robust rater training to establish inter-rater reliability

For any performance-based assessment, especially a high-stakes one, ensuring that different raters score the same performance consistently is paramount for fairness and validity. [18, 19, 23] Robust rater training is the primary method to achieve high inter-rater reliability, minimizing variability that comes from examiner judgment. [19, 23]

A simulation program director is using the CIPP (Context, Input, Process, Product) model to conduct a comprehensive evaluation of a new curriculum. Which of the following questions would be addressed during the 'Input' evaluation phase?

Answer: Are the available faculty, funding, and equipment adequate to implement the curriculum as planned?

The 'Input' phase of the CIPP model focuses on assessing the adequacy of resources, the feasibility of the plan, and the potential strategies. [2, 11, 16] It answers the question, 'How should it be done?' [11] 'Product' evaluation addresses outcomes (A), 'Process' evaluation addresses implementation fidelity (C), and 'Context' evaluation addresses needs (D). [2, 16]

A CHSE convenes a panel of subject matter experts to establish a defensible pass/fail score for a new summative assessment. The experts review each item on the scoring tool and estimate the probability that a minimally competent practitioner would perform the item correctly. This process is a hallmark of which standard-setting method?

Answer: The Angoff method

The Angoff method is a widely used, evidence-based procedure where a panel of experts evaluates assessment items to determine a cut-score based on the expected performance of a 'minimally competent' individual. [1, 4, 6] This approach bases the passing standard on the content of the exam, not on the performance of the group taking it. [4, 5]

An educator develops a new assessment checklist for a central venous catheter insertion scenario. To help establish the tool's validity, the educator asks a group of experienced intensivists to review the checklist to ensure it includes all critical steps and excludes any irrelevant ones. This activity is primarily focused on gathering what type of validity evidence?

Answer: Content validity

Content validity refers to the extent to which an assessment tool adequately represents all facets of the concept or skill being measured. [29] Using subject matter experts to review the instrument's items for relevance and completeness is a fundamental step in establishing evidence of content validity. [3, 25]