AAAAI Modules

Resources Relevant to the Implementation
of Core ACGME-Required Competencies

Module IV
Professionalism
Assessment

		The measures for the assessment of professionalism chosen for AAAAI were selected to evaluate fellows’ professional behavior. Components of professionalism may be assessed in a variety of ways, including testing knowledge of ethical issues and asking learners to indicate their likely response to hypothetical ethical cases. While these types of assessments can be useful components of comprehensive evaluation, focusing on behavior provides the most authentic and relevant assessments when time and resources are limited.
		TOP

Commentary on the need for 360 degree assessment		In its table of toolbox methods, the ACGME suggests that 360 degree ratings are a desirable way to assess professionalism. 360-degree evaluations are assessments completed by multiple others with whom one interacts. The notion of 360-degrees means that supervisors, peers, and subordinates, and in the case of health care, patients and families, all provide information about an individual’s performance. 360-degree feedback has been used extensively in business organizations. It is beginning to be used in the health care administration arena. At this point in time, however, there is no evidence in the literature of the use of 360-degree evaluation in graduate medical education. The notion of such a type of assessment is appealing for the wide-ranging information it has the potential to bring to GME, and the ACGME recommends its use in a number of competency domains, including professionalism. The ACGME, in its draft Toolbox of Assessment Measures, provides some information about the properties and uses of 360-degree evaluation for assessment. It is recommended that the AAAAI consider working to develop a useful 360-degree evaluation tailored for health care practice. Such a measure would not only be a valuable resource in the assessment of fellows in training but also could fulfill the same role for maintenance of certification. Until new measures are available, collecting evaluative information from a variety of constituents - patients, faculty, and other professionals - can provide a rudimentary 360 degree view.
		TOP

Assessment of fellow by patients
Patient survey: American Board of Internal Medicine (ABIM) [adapted]		The ABIM‘s Continuous Professional Development Program uses patient assessment as one measure of physician skill. Designed to be completed by patients who have been contacted by mail some time after the visit, the ten-item instrument focuses on the physician-patient interaction that occurred during the medical encounter. Because respectful, altruistic behavior is demonstrated to a great degree during physician-patient interaction, assessment measures that are useful for evaluating interpersonal and communication skills may also be used to evaluate this component of professionalism. An adaptation of the ABIM instrument was chosen for the AAAAI toolbox for a number of reasons. First, the questions focus solely on aspects of the physician-patient interaction. Instruments that look at the concept of satisfaction more globally often include questions about other aspects of the medical visit, such as access to appointments, wait time, and convenience of parking. While this information is useful, a more targeted assessment of respectful, altruistic behavior is needed both to assure compliance with the ACGME mandate and also to provide useful, targeted formative feedback to fellows. Ten questions are enough to provide information about important specific components of the medical interaction, but are few enough not to be onerous for the patient to complete. Two adaptations to the instrument are suggested. First, the items should be shortened so that each reflects a single behavior. Research has shown that when multiple behaviors are assessed in one question, neither those doing the assessment nor those being assessed can be sure which components are being measured. The University of Pennsylvania Health System, in conjunction with Independence Blue Cross, created truncated single items from the ABIM instrument for reporting purposes. This adapted version of the items is suggested for use by A/I training programs. The second recommendation concerns how the instrument is administered. Since response to mailed surveys is low and may tend to reflect the evaluation of patients who are either very happy or very unhappy with their care, we suggest that the questions be handed out to patients by clinic staff at the end of the visit and completed on the spot. Having all of a fellow’s patients complete the forms during some period of time—one week every six months, for example— is likely to provide more complete and useful information. Care should be taken to ensure that the time period used is representative. The ABIM has granted permission for use of its core competencies by A/I training programs. The adapted ABIM form is an example of assessment via patient survey. The ACGME, in its draft Toolbox of Assessment Methods, provides some information about the properties and uses of patient surveys for assessment.
		TOP

Assessment of fellow by faculty
Checklist: SEGUE Framework		The SEGUE Framework was developed by Gregory Makoul at the Program in Communication and Medicine at the Northwestern University Medical School. Designed to be completed by either a faculty observer or a standardized patient, the SEGUE Framework focuses on specific components of communication and assesses whether or not they were present during an encounter. Development and beginning validity studies of the instrument are documented in: Makoul, G. (2001). The SEGUE Framework for teaching and assessing communication skills. Patient Education and Counseling, 45(1), 23-34. * This site requires free registration. It is extremely slow to load. You may choose to request a print copy of the article from your library.* Dr. Makoul has granted permission for use of the SEGUE Framework by A/I training programs. Because sensitivity to patients’ cultural, age, gender, and disability is demonstrated to a great degree during physician-patient interaction, assessment measures that are useful for evaluating interpersonal and communication skills may also be used to evaluate this component of professionalism. The SEGUE Framework was chosen for the AAAAI Toolbox for a number of reasons. First, it focuses on both important elements of interpersonal and communication skill, the content and the process of the medical encounter. While this assessment tool was originally developed for use at the undergraduate medical education level, the components of communication assessed by the tool are applicable at all levels of training. For example, although the level of content expected of a fellow is different from that expected of a medical student, the inclusion of this component in the medical interview is expected of both. The elements included in the Framework were selected based on review of two literatures – education and research – and on work in two contexts. The items each describe a single task and, as a result, are clear and unambiguous. The Framework is coded dichotomously, thus simplifying the scoring and making the attainment of a high degree of inter-rater reliability more likely. Finally, information gained from use of the Framework can provide residents with feedback targeted very specifically at content and process skills. While standardized patient-based assessment can be useful for assessing sensitivity, it is not very practical for A/I training programs, which are typically small in size. Observation by faculty of an actual patient encounter, especially if conducted on an unannounced basis, is likely to provide a more realistic snapshot of resident behavior than scheduled time with a standardized patient. The literature on assessment indicates that, to be generalizable, a relatively large number of observations need to be made. At this point, since the purposes of assessing interpersonal and communication skills in A/I training programs are to demonstrate compliance with the ACGME mandate and to provide formative feedback to fellows, occasional observations conducted over the course of the fellowship should suffice. The SEGUE Framework is an example of assessment via checklist. The ACGME, in its draft Toolbox of Assessment Methods, provides some information about the properties and uses of checklists for assessment.
		TOP

Assessment of fellows by other professionals
Rating scale American Board of Internal Medicine (ABIM)		As a part of its Project Professionalism, the ABIM provides a professional associate rating form designed for health care professionals to use in rating subspecialty physicians on a number of areas relative to professionalism, including responsiveness to patients, respect, integrity, and compassion. The professional associate rating form was chosen for the AAAAI toolbox for a number of reasons. First, the questions focus on aspects of professionalism that may be manifested in behavior. Second, the rater is instructed to compare the target of the evaluation to similar others, thus providing a context for comparison. One caveat is that some of the questions ask the rater to assess multiple behaviors. Research has shown that when multiple behaviors are assessed in one question, neither those doing the assessment nor those being assessed can be sure which components are being measured. The authors of the form find that 10 ratings per subject are needed to provide a reliable assessment. The ABIM has granted permission for use of its core competencies by A/I training programs. The professional associate rating form is an example of assessment via global rating. The ACGME, in its draft Toolbox of Assessment Methods, provides some information about the properties and uses of global rating for assessment .
		TOP

Identification of professionalism problems
Physicianship Evaluation Form		Faculty at the UCSF School of Medicine have created a designed to: “help ‘turn around’ students found to have behaved unprofessionally, to demonstrate the priority placed by the school on the attainment of professional behavior, and to give the school ‘muscle’ to deal with issues of professionalism.”* Development and use of the tool are described in: *Papadakis, M. A., Osborn, E. H. S., Cooke, M., & Healy, K. (1999). A strategy for the detection and evaluation of unprofessional behavior in medical students. Academic Medicine, 74(9), 980-990. The Physicianship Evaluation Form was chosen for the AAAAI Toolbox for a number of reasons. First, it focuses on four important elements of professionalism: professional responsibility, self improvement and adaptability, and relationships with both patients and families and also with health care team members. While this assessment tool was originally developed for use at the undergraduate medical education level, the components of professionalism assessed by the tool are applicable at all levels of training. The form was designed to be used as a way to gather more specific information when a student’s interpersonal skills in a clerkship have been rated as less than satisfactory. This documentation of specific information is needed both to provide targeted feedback designed to improve performance and also, in rare cases, to document a continuing lack of professional behavior. The Physicianship Evaluation Form is an example of a secondary assessment instrument designed to follow-up a problem identified from one or more primary instruments.
		TOP

		Return to Table of Contents
		Return to Assessment Overview
		Return to Professionalism Overview
		Move forward to Professionalism Resources

Home