00:11:56 Eric Holmboe: Sadness for Ethan and for the impact of COVID-19 00:11:57 Susan Lane: worried that my Zoom is not going to cooperate. Sad for Ethan, our friend. Inspired by the candidates I met today. 00:12:10 Kim Bates, MD: Content for me, sadness for Dr. Fried’s loss 00:12:19 Sal Pindiprolu: Sad for all the losses, optimistic that things will improve soon 00:12:38 priyanka sultania dudani: compassionate nd caring for my patients, content for me and family, sadness to hear death and tragedy around me. 00:15:19 Eric Holmboe: Who is the guy with the moustache??? 00:15:27 Kelli Corning: John Cleese 00:16:13 John H. Choe: Roger Bush! 00:25:59 John H. Choe: Was on the writing group for some of the PBLI harmonized milestones. Was incredibly fruitful to compare notes about what PBLI means when talking to surgeons, psychiatrists, and non-IM people. Learned a lot from talking to people outside my "silo" 00:26:50 Neil Kothari: Thanks John - you guys did a great job with the harmonized milestones! 00:26:52 Alvin Calderon (he/him/his): Interested, although it broke our eval system 00:27:20 Susan Lane: yup, had to redo evals. but it's all good. slowly refining them. 00:27:41 Laura Paletta-Hobbs: Overall a positive response to the new clearer language. I am still suspecting that our residents will be overrated 00:27:41 Halle Sobel: Yes, redid evals, re-mapped 00:27:50 Laura Paletta-Hobbs: We redid evals as well 00:27:51 Philip Verhoef, MD, Hawaii, PNHP Board: we are struggling a little with the faculty evaluations too, but the milestones are so much better. 00:28:06 priyanka sultania dudani: it is comprehensive , but I felt it to be very linear. struggled with some areas especially well being? 00:28:08 Ravi Gupta: Well designed, agree that more time to re-map would really have helped, very little time to acclimate faculty... 00:28:16 Sal Pindiprolu: Using it as a platform to reform evaluations and also simplifying evaluations 00:28:24 Susan Lane: I am finding that there is a "left shift" and my faculty have not yet caught on...overscoring :( 00:28:32 Philip Verhoef, MD, Hawaii, PNHP Board: we are still seeing overrating too of residents, but will iteratively have to educate the faculty even though I tried to emphasize that it's okay for interns to be a 1! 00:28:50 Raj Jagarlamudi: Can anyone share how they simplified the evals? 00:29:05 Susan Lane: I did try to simplify - will show in a bit 00:29:06 Alvin Calderon (he/him/his): Raj, hopefully we can share that 00:29:08 priyanka sultania dudani: what would be objective areas to say the resident is at some level and ready to graduate ? 00:29:11 Laura Paletta-Hobbs: Our interns had a shock to system too 00:29:55 Philip Verhoef, MD, Hawaii, PNHP Board: we limited any single evaluation to looking at only a max of 6 milestones. Since there are lots of different rotations, many of them use different milestones, so we end up getting a comprehensive sense of the trainees. 00:30:05 Jon Lim: Agree with everyone. I think it has been an opportunity not only to incorporate a 2.0 version, but to re-evaluate assessment tool and system overall. I like what others have said about this being an ongoing iterative process for continuous quality improvement 00:30:11 Susan Lane: we advised our residents that their scores may start looking lower and not to be worried. 00:30:17 Susan Lane: (c/w last year) 00:30:22 Jon Lim: not easy work for some important changes 00:31:57 Eric Holmboe: Please check out the 2 open access (i.e. free) assessment tools available at Learn@ACGME: https://dl.acgme.org/pages/assessment 00:32:50 Philip Verhoef, MD, Hawaii, PNHP Board: This is how we are handling this too: assessed primarily by PD at the semi-annuals. 00:33:28 Jon Lim: would also note this JGME special issue focused on Milestones https://meridian.allenpress.com/jgme/issue/13/2s 00:33:33 Ken Steinberg (he/him): We use EPAs at University of Washington 00:33:33 Alvin Calderon (he/him/his): EPA 00:33:37 Kim Bates, MD: EPAs 00:33:40 Neil Kothari: We use EPAs at Rutgers 00:33:41 Fred Buckhold: We use EPAs at SLU 00:33:47 Susan Lane: yes we use direct obs of EPAs. modified the wording of milestones for faculty. 00:33:50 Laura Paletta-Hobbs: milestones directly - a subcompency row = an item 00:33:59 Melina Awar: Modified evals to reflect new milestones and evaluate off of that 00:34:03 Philip Verhoef, MD, Hawaii, PNHP Board: we have CEX's and i linked several of those questions to milestones. MiniCEX as well as what we call Targeted CEXs that assess specific skills, with mapping to milestones. 00:34:04 Ravi Gupta: milestones directly 00:34:17 priyanka sultania dudani: milestones directly 00:34:58 Susan Lane: This slide is epic and has guided my use of milestones ever since Kelly presented it! 00:35:08 Eric Holmboe: And here is a link to the Assessment Guidebook and other Guidebooks: https://www.acgme.org/what-we-do/accreditation/milestones/resources/ 00:38:01 Judy Blair-Elortegui: We found we got more "N/A" responses on evaluations using EPAs 00:38:57 Jon Lim: ^^I think one of our goals in revisiting our system is to ensure construct alignment--making sure the assessment EPAs selected match with what is happening during the rotation or learning activity 00:39:45 Kim Bates, MD: I also think the explanation section for the EPA is key (and I’m going to take some of that from John’s portion!). 00:41:59 John H. Choe: @Judy Blair-Elortegui- We actually very much direct our evaluators to liberally mark items as "N/A" if not directly observed. We don't want the evaluators to infer, but rather only comment on truly directly observed activities. Our secret mission is to try to encourage more direct observations among faculty 00:43:58 Melina Awar: we did capitals and BOLD too! 00:44:52 Meredith Eicken: It would be so amazing if NI could develop a shared directory where we could search and use other peoples' evaluations if they gave permission to share! 00:46:04 Susan Lane: I have shared DOTs to other programs - so can do 00:46:08 Kim Bates, MD: @mereditheicken Agreed! 00:46:29 John H. Choe: Great Coffee Talk reference! 00:46:40 Laura Paletta-Hobbs: Meredith - we were able to share with Stephanie Call's new program (I'm at her old program) with NI - I am not sure how we did it but can find out. 00:46:55 Meredith Eicken: Thank you Susan and Laura! I had no idea!! 00:47:28 David Wininger: Our CCC is very hung up on Prof 2 - the issue of how residents address ethical dilemmas 00:47:33 Julie Machen (she/her): How are other’s assessing PROF2 (ethics)? We assess on Palliative Care rotation, but residents don’t have this q6 months 00:47:36 Julie Machen (she/her): ^^ yes 00:48:05 Philip Verhoef, MD, Hawaii, PNHP Board: PROF2 on ICU and hem/onc rotations too 00:48:14 Melina Awar: Used the supplement guide and reviewed at PEC 00:49:57 Susan Lane: we are in the same boat Jaya! 00:51:13 Khalid Gadir: this is a tough one: we shared it with our palliative care team; all r2's rotate thru it 00:51:33 priyanka sultania dudani: on this new milestone ,we felt it was very linear , how do we know that resident should be at some level for that particular level of training or we just see his/her progress over years 00:51:42 Susan Lane: CCC taking longer as I don't yet have a good sense of where people "are" in the milestones 00:52:40 Khalid Gadir: PC6 is one that I like and I believe it is important 00:52:55 John H. Choe: Nothing like a global pandemic to force change faster than our usual glacier pace.... 00:53:40 Neil Kothari: Absolutely, John 00:53:54 Julie Machen (she/her): Do others have a goal for how many data points they’d like CCC to have for each sub-competency? I recognize that some areas will have more data than others, but is there a minimum target you shoot for? 00:53:57 Raj Jagarlamudi: Did anyone replace old Evals to Milestones 2.0- exactly how it is in New innovations; if you did- what problems did you face? 00:54:40 Uma Medapati: where would you use SBP3 -Physician role in health care system. 00:55:26 Halle Sobel: We have q’s to evaluate resident teaching (peers, interns) but I don’t know what to map it to 00:56:05 Julie Machen (she/her): @Uma- we assess the 2nd part of SBP3 (the part about payment systems) in continuity clinic 00:56:32 Eric Holmboe: AAMC Telehealth competencies can be found here: https://www.aamc.org/data-reports/report/telehealth-competencies 00:56:45 Melina Awar: @Uma we use SBP3 in primary care electives, continuity and our Value Based Care Elective 00:59:27 Susan Lane: I am also finding it difficult to properly assess a resident's overall understanding of the health care system. 00:59:38 Fred Buckhold: I’m curious for those that utilize EPAs what would map to this? Do a safe discharge of a hospitalized patient? 01:00:06 Kim Bates, MD: Not just assessment of SBP3 but also the education of residents on this subcompetency 01:00:15 Alvin Calderon (he/him/his): Yes Fred, DC and DC planning for us. 01:00:37 Eric Holmboe: I think the ACP Cost consciousness and Choosing Wisely might be helpful for SBP3 01:00:39 Philip Verhoef, MD, Hawaii, PNHP Board: we measure SBP3 on our geri and heme/onc evals since both use really coordinated care and complex drugs. we also use in cont clinic and inpatient evals, and we have it in our PD semi-annual eval (which also includes the wellness one) 01:00:51 Melina Awar: Can I attend your faculty development session? ;) 01:01:04 priyanka sultania dudani: we gave few residents average 2.5 because I do agree what Uma is saying , we have not had that culture of discussion more in inpatient side , more in OP setting I fee. I feel even we as attending we struggle with payment models . 01:01:06 Fred Buckhold: Of course - no promises that it will be any good! 01:04:00 Kim Bates, MD: What do you think you need from your departments/divisions to help you navigate the transition? 01:05:20 Susan Lane: Would be really great for our software systems to enable sharing across programs in an easy manner 01:05:44 Laura Paletta-Hobbs: I think I may need to reword some of our evals further for easier assessment. AND more DOT 01:05:55 Erin Maxwell: We need to better utilize New Innovations! 01:06:05 priyanka sultania dudani: better utilize NI. 01:06:11 Halle Sobel: I will further discuss with my CCC how to measure some of the new concepts in 2.0 and plan to take plenty of time at our upcoming meeting; would like to share evals with others on NI 01:06:21 Ken Steinberg (he/him): Bye everyone! Thank you! 01:06:25 Jon Lim: @ErinMaxwell I definitely need to work on my own competencies for assessment tech