Curated Milestones Evaluation Exhibit

Challenges in Internal Medicine Resident Procedure Training and the ACGME's PBLI Milestone System

Laura M. Hampton, MD
St. Vincent Hospital

Program Size: 31-100 residents
Academic Setting: Community-Based
Clinical Setting: Procedures

Overview

With changes in ABIM requirements for procedure performance in recent years, inpatient and outpatient procedure evaluation of internal medicine residents has become more challenging. This is particularly important with the ACGME’s outcomes-based milestone system as a framework for determining resident and fellow performance within the six ACGME Core Competencies. The ACGME’s Practice-Based Learning and Improvement (PBLI) milestone system and the ABIM require programs to rate and attest to residents’ procedural competency and knowledge. Currently, the ABIM requires performance of only a select few procedures, with knowledge and understanding of many others. However, no clear method of evaluating residents’ performance and knowledge has been identified.

At our program, residents are strongly encouraged to be active participants in common internal medicine procedures, as recommended by the ABIM. They log their procedures on New Innovations. Once they successfully perform a prerequisite number and feel confident, they may request certification by direct observation with an attending. Our program has created certification cards (see attachment) signed by an attending that our residents are required to carry on their person at all times, usually by taking a picture on their smart phone. A direct observation evaluation is also completed on New Innovations by the attending. In order to be certified, an attending must observe the resident perform all aspects of the procedure and attest to their readiness to perform independently.

Residents are also provided with educational resources and required to complete online tests created by our program to assess comprehensive knowledge of common IM procedures. These tests include anatomy, indications, contraindications, and management of procedural complications. A minimum 80% correct is required to pass, and explanations/feedback are provided at the end of the test. Our observation has been that some of our residents who are technically very skilled in procedures may not pass the tests on first attempt. Our tests ensures that residents fulfill the ABIM knowledge requirement, and provides an objective evaluation measure to assign PBLI PC-4 milestone ratings at our semi-annual CCC meetings (see attached chart). Our program coordinator and a faculty member specializing in procedures monitor our residents’ progress and update their status in New Innovations when all requirements are completed for a given procedure.

Download the Tool

Development

  1. Determination of which procedures we felt were important for our residents to perform. Our program requires certification in internal jugular central lines, arterial lines, and intubations by the end of intern year. We feel performance of these procedures is essential to our residents’ role as senior ICU resident. They are also required to obtain certification in pap smears (as required by the ABIM) before graduation.
  2. Development of procedure knowledge tests, a collaborative effort with our academic faculty. These were easy to host on an inexpensive external website.
  3. Correlation of our program goals to PBLI PC-4 milestone ratings (see attachment)

Lessons Learned

  1. After implementation, we discovered that many residents were not tracking procedures or carrying evidence of their certification when performing procedures independently. This is absolutely essential for JCAHO and ACGME expectations. Implementation of our policy has demonstrated immense improvement in this regard.
  2. We also discovered that technical expertise did not necessarily correlate with comprehensive procedure knowledge, as evaluated by our tests. The tests also serve as a teaching tool that have helped our residents gain knowledge of procedures, as required by the ABIM.

Faculty Development and Training

  1. Naturally, our faculty must be experienced in performing the procedures they teach in order to supervise and evaluate residents. Our core academic faculty and sub-specialists teach only the procedures they commonly perform and have knowledge in. We also emphasize the importance of direct observation and evaluation completion to all of our teaching faculty.
  2. We have one core faculty member with interest and experience in procedures designated as our procedure tracker who monitors our residents' progress. This faculty member also teaches procedure training on simulation models during intern orientation week, with additional individual sessions as needed throughout residency.

How Used to Inform Decisions about a Learner's Milestone

Our tool correlates performance of required procedures and knowledge tests to a PC-4 rating. This chart could certainly be applied to other medicine-based training programs who may struggle with some of the same procedure evaluation issues that we did. I would also envision that it could be customized to other surgical-based training programs with more extensive procedure requirements.

For more information, please contact lmhampto@gmail.com.