John H. Choe, MD
University of Washington
Program Size: More than 101 residents
Academic Setting: University-Based
Clinical Setting: Inpatient Wards
Overview
Rotation-specific EPA-based evaluations are used for residents on rotations at the University of Washington (UW) Medical Center; two evaluations are provided here (night medicine and hematology-oncology wards) as examples.
Evaluation questions are based upon representative entrustable professional activities (EPAs) specific to the work of each rotation. Faculty members attest to the level of supervision required (as a proxy for degree of entrustment given) for the interns and senior residents in each of these rotation-specific clinical contexts. For each evaluation question, the UW Internal Medicine Residency Program has mapped the potential responses to "milestone elements", the narrative descriptions within the columns of each of the 22 subcompetencies. When a faculty member selects a given degree of entrustment for one of these rotation activities, the milestone elements mapped to that response are counted as "confirmed" for that resident by the program. The program tracks the number of times each milestone element has been confirmed for each resident; once confirmed a predetermined number of times, individual milestone elements are recorded as having been completed to program satisfaction. Completing a milestone element to program satisfaction generally requires assessment by multiple faculty members across different clinical rotations. By tracking the completion of milestone elements, the program can quickly and efficiently determine progress in the 22 subcompetencies reported semiannually to the ACGME. This process efficiently bridges EPA-based evaluation questions to ACGME milestone reporting for the >170 UW internal medicine residents.
Download the Hematology-Oncology Evaluation »
Download the Night Medicine Evaluation »
Development
To create each UW medicine rotation, key faculty select eight to ten rotational EPAs representative of the essential work expected of a resident on that rotation. Program faculty members review the 22 ACGME subcompetencies to identify those related to the work of each rotational EPA. The narrative descriptions (“milestone elements”) in the columns of each related subcompetency are mapped to the different response categories for the evaluation item. Once the different response categories of each EPA have been mapped to milestone elements in several of the ACGME subcompetencies, program staff members link these items together in the computer evaluation database.
Lessons Learned
- Initial investment of time to refine this evaluation system was significant; however, this proved invaluable in coming to a shared vision of the evaluation system across the institution.
- The discussions about which EPAs were most representative of the work of that rotation helped stimulate conversations about which resident activities were critical to the program’s educational mission.
- Regular faculty (and resident) education is necessary to avoid evaluation “inflation”.
- Creation of a system that explicitly focuses on the needs and limitations of each “constituency” (resident; faculty member; program leadership; ACGME/ABIM) can reduce some of this downstream effort.
Faculty Development and Training
Faculty development is required to help reduce the tendency to over-generously evaluate interns. Faculty are prompted frequently that evaluations are not level-specific, and greater supervision is expected of residents at the beginning of training. Faculty members that choose “independent” or “aspirational” when evaluating interns are prompted to fill in a text box to explain their reasoning; future effort to identify “outlier” faculty members is possible through the evaluation software. Reminders are provided to residents, who often are used to attaining “high grades”. They are frequently reminded that the evaluations are meant for coaching and feedback, rather than for summative grading.
How Used to Inform Decisions about a Learner's Milestone
This tracking of confirmed and completed milestone elements can be graphically displayed using evaluation software. At UW, MedHub has been set up to indicate with different colors those milestone elements that have been completed to program satisfaction. This graphical display allows program CCC members to quickly scan milestone progress for the >170 residents in the UW program. CCC members are able to complete this review and semiannual reporting for ACGME in a single meeting using this system.
For more information, please contact johnchoe@uw.edu.