Even Better Together.
The quality and quantity of tools such as workshops, presentations and curriculum support is magnified when we work together.
Effective Feedback and Evaluation in Clinical Medicine:
A Faculty Development Workshop
General Internal Medicine Faculty Development Meeting
December 4-5, 1999
Catherine Lucey, MD
Program Director and Associate Chair
Department of Medicine
Washington Hospital Center
The goals of this workshop are:
· to help participants assess, provide effective feedback to and accurately evaluate their learners.
· to provide participants with insight into problems other faculty may have with evaluation and feedback, thus facilitating the development of successful faculty development programs.
At the completion of this workshop, participants should:
· Understand the barriers to effective feedback and evaluation in clinical medical education;
· Appreciate the importance of goal setting, learning climate, and observation for high quality feedback and evaluation ;
· Understand the differences between feedback, formative evaluation and summative evaluation;
· Compare and contrast the use of micro and macro feedback;
· Recognize the need for a multifaceted evaluation system in any clinical setting;
· Identify tools available to enhance current evaluation methods.
General Concepts in Evaluation and Feedback:
Blanchard K, Johnson S. The One Minute Manager. William Morrow and Company, Inc. New York, New York, 1982.
(A quickly read and appreciated book that emphasizes goal setting, observation, instant reinforcing and corrective feedback and the overriding philosophy that the manager views the employee as valuable.)
Ende J. Feedback in clinical medical education. JAMA. 1983;250:777-81.
(A classic article drawing on business and psychology literature to form our current paradigm of effective feedback.)
Hewson MG, Little ML. Giving feedback in medical education. Verification of recommended techniques. J Gen Intern Med. 1998;13:111-116.
(This article finds that faculty in development courses verify that feedback that is specific, nonjudgemental and focuses on behaviors rather than personalities is preferable when they are in the learner’s role.)
Miller G. The assessment of clinical skills/competence/performance. Acad Medicine. 1990; 65(9);S63-S67.
(Another classic article outlining the "knows, knows how, shows how, does" hierarchy of competence and practice.)
Pangaro L. A new vocabulary and other innovations to improve intraining evaluation. Acad Med. 1999 November (in press)
(A formal article outlining the RIME paradigm of medical student competency .)
Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. J Gen Intern Med. 1998; 13:317-23.
(A very thorough and helpful article outlining the steps necessary in a reliable and valid evaluation program.)
Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in Internal Medicine: a review. Ann Intern Med. 1998; 129:42-48.
(A very comprehensive review of the current literature on evaluation tools and methods.)
Hemmer PA, Pangaro L. The effectiveness of formal evaluation sessions in better identifying students with marginal funds of knowledge. Acad Med . 1997; 72(7):641-643.
(One program’s approach to the lack of specificity and discriminatory ability of written evaluations– make the evaluators talk about the student!)
Kroenke K. Attending rounds: guidelines for teaching on the wards. J Gen Intern Med. 1992;7:68-75.
(An excellent review article to hand out to faculty beginning attending rounds. The section on feedback is particularly helpful.)
Food for Thought
Gordon MJ. Cutting the Gordian Knot: A two part approach to the evaluation and professional development of residents. Acad Med. 1997; 72 (10): 876-880.
(A thought provoking article which suggests that we use separate evaluation systems for ensuring clinical competence [i.e. minimal standard setting] and fostering professional growth. The author suggests that the reason we are frequently dissatisfied with current evaluation methods is that no system can be effective in both arenas.)
Lazare A. Shame and humiliation in the medical encounter. Arch Intern Med 1987; 147:1653-1658.
(A thought provoking article on shame, guilt, and humiliation in the doctor-patient relationship and how to avoid inducing these feelings. Clearly, there are some messages for the teacher-learner dyad as well.)
Resources for Faculty Development
Anderson W, Malacrea R. Giving Constructive Feedback. Office of Medical Education Research and Development, Michigan State University. ACP-ASIM Community Based Teaching Educational Clearinghouse. # 326
(an excellent faculty development package containing instructor manuals, videotapes, overheads and handouts.)
Wipf J, Pinsky L. Resident as Teacher. University of Washington 1994.
(A VHS tape which goes through a number of teaching skills, including feedback and evaluation, important to residents as teachers. Could also be used by faculty wishing to review the principles discussed here.)
Specific Evaluation Techniques and Tools:
MEDICAL RECORD AUDITS:
· Holmboe ES, Scranton R, Sumption K, Hawkins R. Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Acad Med. 1998;73:901-3.
(Medical record audits using explicit criteria can improve resident’s adherence to guidelines not only in the audited areas but in other preventive health areas.)
· Kern DE, Harris WL, Boekeloo BO, Barker LR, Hogeland P. Use of an outpatient medical record audit to achieve educational objectives: changes in residents’ performance over six years. J Gen Intern Med. 1990; 5:218-224.
(Systematic medical record audits can change not only individual resident behaviors but can foster changes in the culture of a medicine residency.)
CHART STIMULATED RECALL:
· Munger BS. "Oral Examinations" in Recertification: New Evaluation Methods and Strategies. American Board of Medical Specialties. Mancall and Bashook, eds. Evanston, Ill. 1994.
(Chart stimulated recall differs from simple medical record audit in that an evaluator reviews the chart documentation with the evaluatee present and asks questions regarding decision making. CSR was found to be valid but too cumbersome to use in the Emergency Medicine certification process.)
ABIM RATING SCALES:
· Gray JD. Global rating scales in residency education. Acad Med. 1996; 71:S55-63.
(This author reviews the general area of rating scales and also methodologic considerations for future research into rating scales and raters.)
· Haber RJ and Avins AL. Do ratings on the American Board of Internal Medicine resident evaluation form detect difference in clinical competence? J Gen Intern Med. 1995;9:140-145.
(This article points out that while the standard ABIM form does detect global differences in clinical competence, it is unable in common use to give information that leads to specific feedback in individual domains of competence.)
· Rand VE, Hudes ES, Browner WS, Wachter Rm, Avins Al. Effect of evaluator and resident gender on the American Board of Internal Medicine evaluation scores. J Gen Intern Med. 1998; 13: 60-674.
(A sobering article which points out the possibility that gender bias may impact resident evaluations.)
CLINICAL EVALUATION EXERCISES:
· Noel GL, Herbers JE, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do Internal Medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992; 117:757-765.
(These authors note that a structured form with explicit criteria improves the accuracy but not the interrater reliabiliaty of clinical skill assessment by faculty.)
· Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-cex (clinical evaluation exercise): A preliminary investigation. Ann Intern Med. 1995; 123: 795-799.
(Preliminary explorations into an abbreviated CEX form to foster better faculty compliance with observation and documentation of clinical skills. This led to a recent extensive revision of the CEX concept, currently in a pilot study.)
STANDARDIZED PATIENTS AND OSCE:
· Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills. Acad Med. 1993; 6: 443-53,
(The author traces the development of standardized patients and summarizes working group assessment of strengths, weaknesses and future research opportunities for this technique. He includes a helpful table of physical findings which can be simulated by standardized patients.)
· King AM, Perkowski-Rogers LC, Pohl HS. Planning standardized patient programs: case development, patient training and costs. Teaching and Learning in Medicine. 1994; 6(1): 6-14.
(A very practical article helping those who wish to set up standardized patient programs.)
USE OF NURSES OR PATIENTS IN CLINICAL EVALUATION OF PHYSICIANS AND MEDICAL STUDENTS
· Weaver MJ, Ow CL, Walker DJ, Degenhardt EF. A questionnaire for Patients’ evaluation of their physicians’ humanistic behaviors. J Gen Intern Med. 1993;8:135-139.
(The authors develop and validate a patient survey used to assess humanism in residents. Interestingly, the questionnaire correlates with patient satisfaction but not with attending assessment of humanistic behaviors of their trainees.)
· Kaplan CB, Center RM. The use of nurses to evaluate houseofficers’ humanistic behavior. J Gen Intern Med 1990; 5:410-414.
(These authors identify that nurses can discriminate between houseofficers when they assess complete cards assessing humanistic behavior. Substantial variability exists focused around gender of the physician and unit type (general medical vs. critical care) of the evaluating nurse. Unfortunately, no validation of the accuracy of nursing ratings is offered.))
· Tamblyn R, Benaroya S, Snell L, McLeod P, Schnarch B, Abrahamowicz M. The feasibility and value of using patient satisfaction ratings to evaluate internal medicine residents. J Gen Intern Med 1994; 9: 146-152.
(Patient ratings do provide helpful information about resident’s performance in the doctor patient relationship; however many evaluations are necessary to draw accurate conclusions. Interestingly, only women residents demonstrated improvement in ratings over this residency’s three year course.)
Sample Evaluation Form:
1. Standard ABIM Form: behavioral anchors at extremes of performance
2. USUHS "RIME" Form: Behavioral anchors at each level of performance