Dale S. Vincent, MD
Tripler Army Medical Center
Program Size: 31-100 residents
Academic Setting: Community-Based
Clinical Setting: All
Background
Faculty at our institution expressed frustration at being unable to aggregate and display the competencies of our trainees in an easy-to-understand format using a well-known commercial product. We developed a unique product using Google Forms, a free software program that collects, aggregates, and displays the acquisition of competencies chronologically throughout the academic year. Setting: An internal medicine residency program with 36 residents in a tertiary-care military medical center. Method: All existing evaluation forms and tools were modified to include these elements: 1) a single global assessment of resident capability using an anchored 9-point Likert scale (Learner 1-4 ; Manager 5-6; Teacher 7-8; Scholar 9), and 2) an assessment of specific competencies using a 5-point scale (Beginning, Developing, Competent, Proficient, and Exemplary). Faculty identified key competencies for each clinical rotation (mid-rotation, end-of-rotation evaluations), directly observed clinical events (e.g. history taking, physical examination, counseling, performing a procedure), and specific events (e.g. leading a journal club, making a presentation, submitting a safety report, receiving a patient comment card). Evaluations are mid-rotation, end-of-rotation, and ad hoc. The evaluation forms are accessible via a link sent to or saved by evaluators.
Results
Color-coded competencies are displayed for each trainee on legal paper in landscape mode. The Y-axis is a list of core competencies grouped and ordered by ACGME competencies and order of month of training that competency should be achieved. The X-axis displays competency evaluations chronologically. Faculty report that milestone progress is easily ascertained and aggregated using this visual representation of resident evaluations.
Download the Tool
Development
Faculty reviewed existing end-of-rotation and Mini-CEX evaluations and changed them using competency-based language. Evaluations were then reviewed to ensure that key Entrustable Professional Activities were addressed (e.g. conducting a family meeting). New directly-observed evaluations were then developed, such as leading a journal club, teaching, engaging in teamwork, conducting a family meeting, completing a discharge summary, and performing a hand-off. In year one, 36 residents were evaluated in 148 sub-competencies, and faculty had submitted an average of 9 end-of-rotation evaluations and 9 directly observed evaluations per resident.
Lessons Learned
Advantages: 1) ease of access to the evaluation tools; 2) ability to add or change evaluation instruments at any time; 3) rapid aggregation of competencies by resident in an interpretable format for the Clinical Competency Committee; 4) ability of the CCC to graph the trajectory of the resident using the Learner-Manager-Teacher-Scholar rubric; and 5) aggregation of results by sub-competency, for programmatic feedback.
Disadvantages: 1) no built-in reminder system for absent or late evaluations—this has to be done by an administrator; 2) inability of learners to see their evaluations without an administrator.
Faculty Development and Training
Faculty needed instruction on using the 9-point Likert scale (Learner-Manager-Teacher-Scholar), since it is competency based and not based on a peer comparison. Instructions were subsequently embedded into the evaluation tool. The system has no built-in reminder to complete evaluations; this has to be done by manually by email. However, this was compensated for by ease of use of the system. Faculty can save the link to the evaluation system, and enter evaluations at appropriate times. An administrator will require training on formatting the results in Excel.
How Used to Inform Decisions about a Learner's Milestone
The tool enables the graphical depiction of a learner's trajectory through the academic year. Easy-to-read, color-coded assessments also demonstrate progression toward achieving milestones in a left-to-right, chronological manner. An entire academic year can be displayed for most residents on three legal size papers.
For more information, please contact dale.s.vincent.civ@mail.mil.