AAIM


Return to Volume 3, Issue 4 Table of Contents

Assessing Clinical Reasoning Skills

The Association of Program Directors in Internal Medicine Publications Committee has created this column to review current topics in medical education that may not come to the attention of faculty and staff in departments of internal medicine. This column will feature current trends in education theory and research as well as provide a bibliography of journal articles and texts that supply stimulating theory and innovative curriculum ideas for medical educators.

As another recruitment season approaches, program directors scan applications looking for the ideal combination of characteristics out of which will grow a competent, skilled internist. Ingredients include non-cognitive traits, such as altruism, maturity, and humanism, balanced with evidence of cognitive skills, such as knowledge and problem solving abilities. Each program director may have his or her own recipe for success. Few would argue that problem-solving skills are essential, especially in the field of internal medicine, but the ability to seperate knowledge from reasoning skills is problematic.

Given the importance of clinical reasoning skills in medicine, what if part of medical school selection criteria could be predictive of students’ future ability to skillfully apply what they learn in a clinically relevant context? These criteria should then also be predictive of future performance on the United States Medical Licensing Examination (USMLE) Step 2 exam.

According to the National Board of Medical Examiners, the USMLE Step 2 assesses an individual’s ability to apply medical knowledge in the context of clinical problem solving (1). In order to assess reasoning skills, a learner must be presented with a problem to solve; in medicine, reasoning skills are assessed with medical problems. If a learner is unable to solve a problem, it is unclear whether this inability represents insufficient background knowledge in the specific subject area or a more fundamental issue in how the individual is approaching the problem, hence the dilemma of content specificity of expertise. This limitation is overcome in USMLE Step 2 by including a large number of questions in a broad range of clinical areas. It does stand to reason that the exam would fall short of its goal of assessing the clinical reasoning skills of a medical student who has broad knowledge deficiencies as documented by performance on USMLE Step 1 and preclinical class work. Taking this into account, USMLE Step 2 is the best available objective measure of the clinical reasoning skills of graduating medical students.

Using retrospective data, Karl Roth, MD, and colleagues developed an equation for predicting performance on USMLE Step 2 and prospectively validated this equation on future classes (2). Their final equation included points for the verbal scores on the Scholastic Aptitude Test (SAT), grade point average for premedical science classes, and science major. Interestingly, they also found that the highest correlation was between Medical College Admission Test (MCAT) reading comprehension score and USMLE Step 2. Unfortunately, this study was conducted during the time period when the MCAT exam was reformulated and the authors were unable to include this data.

Claudio Violato, PhD, and colleagues published an article reviewing performance on the MCAT exam and the Medical Council of Canada (MCC) qualifying examination parts one and two (3). Part two of the MCC qualifying examination is structured similarly to both parts of the USMLE Step 2, emphasizing clinical reasoning skills. The authors found that of the four subsections of the MCAT (verbal reasoning, physical sciences, biological sciences, and writing sample), only verbal reasoning significantly correlated with performance on the MCC qualifying examination part two; undergraduate grade point average (GPA) significantly correlated with both parts (3).

The MCAT has been shown in other studies to correlate positively with performance on the USMLE Steps 1 and 2. In fact, Ellen Julian, PhD’s, study adds to this literature, but does not look at the subsection correlations to USMLE Step 2 performances, only the MCAT as a whole (4).

Assessments of verbal skills on most standardized examinations utilize a familiar format: a brief passage is followed by multiple choice questions designed to assess the test-taker’s comprehension, and often require application of the information to various problems. So it is not surprising that medical examinations utilize a similar format to correlate positively with verbal assessment examinations. Is the skill set an individual must possess in order to score well on such a test important to the practice of medicine or just important to success on the next standardized examination? Are innate problem-solving skills or some marker of these being measured? If so, then the verbal section of the MCAT deserves a closer look and, if necessary, refinement.

Residency program directors do not have access to an applicant’s MCAT scores or undergraduate GPA, even though these results have been shown repeatedly to be strong predictors of future performance in medical school and presumably beyond. This data, together with medical school performance and USMLE exams, improve the ability to predict the trajectory of graduating medical students. Scores from the verbal subsection of the MCAT add to the collage of information available about graduating medical students; however, further studies are needed to determine if it deserves special status.

Frederick Williams, MD
Program Director
Department of Medicine
Franklin Square Hospital Center

References and Recommended Reading
  1. United States Medical Licensing Examination. Step 2. Online. http://www.usmle.org/step2/default.htm. Accessed 11/14/05.
  2. Roth K, et al. Prediction of Students’ USMLE Step 2 Performances Based on Premedical Credentials Related to Verbal Skills. Acad Med. 1996;71(2):176-180.
  3. Violato C, Donnon T. Does the Medical College Admission Test Predict Clinical Reasoning Skills? A Longitudinal Study Employing the Medical Council of Canada Clinical Reasoning Examination. AcadMed. 2005;80(10):S14-16S.
  4. Julian E. Validity of the Medical College Admission Test for Predicting Medical School Performance. Acad Med. 005;80(10):910-917.
  5. Case S, et al. The Relationship between Clinical Science Performance in 20 Medical Schools and Performance on Step 2 of the USMLE Licensing Examination. Acad Med. 1996;71(1):S31-33S.
  6. Eva K, Reiter H. Where Judgement Fails: Pitfalls in the Selection Process for Medical Personnel. Adv Health Sci Educ. 2004;9:161-174.
  7. Berner E, Brooks C, Erdmann J. Use of the USMLE to Select Residents. Acad Med. 1993;68(10):753-759.
  8. Cuddy M, et al. Assessing the Validity of the USMLE Step 2 Clinical Knowledge Examination through an Evaluation of its Clinical Relevance. Acad Med. 2004;79(10):S43-45S.
  9. Elstein A. Beyond Multiple-choice Questions and Essays: The Need for a New Way to Assess Clinical Competence. Acad Med. 1993;68(4):244-249.