CBME Visual Summary
Last updated
Last updated
Updated in ME 1.22! An additional views was introduced (Faculty Evaluation Dashboard) and residents are now able to access their own dashboard.
The CBME visual summary is optional and is controlled by a database setting (cbme_enable_visual_summary). To enable this for your organization, please speak to a developer.
IMPORTANT PREREQUISITE: Assessment Plan Builder
In order to leverage the visual summary, your program must have assessment plans entered in Elentra. Please visit the Assessment Plan Builder lesson for more information.
To access the visual summary:
Log in to Elentra as an administrator (e.g. program administrator, course director).
At the top right, click on the "Assessment & Evaluation" task list icon.
Click on the "My Learners" tab
You will land on the Program Dashboard.
From the tab menu below the filter option, click Visual Summary.
You will be directed to the Visual Summary dashboard.
Toggle between the different dashboards, and/or programs as applicable.
The Normative Assessment Dashboard shows the performance of all residents in a program relative to each other and their training phases and is meant to be only viewed by Competency Committee members.
The normative dashboard presents summarized data metrics of all the residents in a program relative to each other. The data is presented as both a visual representation (left) and a tabular representation. Users are provided with an option to either view all the residents in a program or selectively view the metrics of residents in one training stage by using the dropdown at the top of the dashboard. This can be useful during Competency Committee meetings when residents are judged on their performance relative to others in their training group.
By default, the normative dashboard filters out residents without any completed EPAs. However, this behavior can be turned off by toggling the checkbox at the top of the dashboard.
The bar chart visualizes the following four metrics individually and to switch between the metric being visualized, users can select the corresponding radio buttons above the chart. Each bar represents one resident and users can hover their mouse over a bar in the chart to see the name of the resident it represents, and the value of the current metric being shown. Clicking on a bar in the bar chart switches the user to the resident dashboard to view a detailed breakdown of all assessments of that resident.
Total EPAs - This is a count of the total number of EPAs filled out by a resident.
Currently the total EPAs only considers EPAs that have been collected on valid assessment forms. However, In a future release this count will be updated to also include EPAs completed on archived, expired or deleted assessment forms.
Achievement Rate - This is the total number of EPAs a resident has achieved divided by the total number of EPAs completed by that resident. An achieved EPA is one where the EPA meets certain requirements such as acquiring a rating of 4 or above on a 5-point scale, or satisfying specific contextual variable requirements in an EPA, or meeting diverse assessor role requirements.
Progress Rate - This is the number of EPAs a resident has achieved divided by the total number of EPAs they are required to complete in all the valid EPA forms in a program across the different training phases.
While the achievement rate is a personal indicator for each resident to see what number of EPAs they attempt are achieved, the progress rate is a global indicator that shows where they are in their residency training program.
Total EPAs vs Achieved EPAs - This chart visualizes two bars for each resident showing their total EPAs and achieved EPAs next to each other. While this metric is similar to achievement rate, it can offer a better picture of a resident’s overall performance as high achievements rates can be occasionally misleading for residents with a very low number of completed EPAs all of which are achieved.
Finally, there is a tabular representation of the same metrics with the ability to sort the residents in the table by clicking on the column header. By default, the residents are sorted in the table by name in an ascending order. This can be changed to a descending order by simply clicking on the column header “Name”. Similarly clicking on each column header sorts the table in an ascending order based on that particular metric and clicking the same column header again changes the order to descending.
The normative dashboard is linked to the resident dashboard so to view a particular residents’ assessments in detail users can simply click on a bar corresponding to a resident in the bar chart or their corresponding row in the table. This will automatically switch the user to the resident dashboard with that particular resident preselected.
The Resident Metrics Dashboard focuses on individual residents and is designed to be used by Residents and Competency Committee members**.**
The resident dashboard has a wealth of information that is grouped into different categories for easier comprehension. First if you arrived at the resident dashboard by selecting a resident on the normative dashboard their data is automatically fetched for you. However, if you manually switched over to the resident dashboard by clicking on the navigation tabs above, you will need to select a resident from the dropdown in the filter panel situated at the top of the dashboard. The dropdown contains the list of all the residents in the program with their name and their
corresponding progress rate. The names of residents are further grouped by their training stage and then sorted alphabetically for easier access.
The dropdown is also an editable text box and so you can type a resident’s name partially to automatically filter the available options in the dropdown. This makes it easier to search for a particular resident in a program with many residents.
After selecting a resident, users can then click on the “GET RECORDS’’ button to visualize their assessment data. You might notice the small button with the calendar icon on it. This is used to highlight assessment data gained by the resident in a particular time period. For now, ignore it but we will learn more about further down. The resident dashboard consists of 4 main sub sections. Let us look at each one individually.
This section provides the following summarized metrics of the resident:
Total EPAs observed - Count of all EPAs filled by a resident.
This number may vary from the total EPAs count for the same resident on the normative dashboard as this number also includes assessments filled on expired/archived assessment forms and not just currently valid assessment forms.
Progress Rate - This is the number of EPAs a resident has achieved divided by the total number of EPAs they are required to complete in all the valid EPA forms in a program across the different training phases.
Achievement Rate - This is the total number of EPAs a resident has achieved divided by the total number of EPAs completed by that resident. An achieved EPA is one where the EPA meets certain requirements such as acquiring a rating of 4 or above on a 5 point scale, or satisfying specific contextual variable requirements in an EPA, or meeting diverse assessor role requirements.
To the right of the acquisition metrics is a line chart that shows the weekly EPA acquisition rate for the last six months by the resident. This is meant to give a high level overview at a quick glance of the residents assessment gathering in the recent past.
This section is meant to quickly lookup a residents’ recent performance with the option to view records in the following ranges: last 10 days, last 25 days, last month and last 3 months. The chart does not visually distinguish the different EPA types (i.e., EPA-F1 vs EPA-C2), instead, it provides this and other additional information in a pop-up menu that can be invoked by hovering the mouse over a point.
The line chart provides a simple representation of the last “N’’ assessments filled by the resident where every EPA is represented as a point with the oldest record starting on the left. The points are arranged vertically using the O-Score Entrustability scale with 5 being the highest (I did not need to be there) and 1 being the lowest (I had to do). The better a resident performs in an EPA, the higher is the vertical position of the point in the chart. We use background lines to show the 5 levels, instead of labelling the points, to reduce visual clutter as the levels are easy to understand without providing additional context.
The final section provides a detailed overview of every single EPA completed by the resident. The entire list of EPAs that residents are required to complete are broken down into four groups based on the training phase during which a resident is supposed to complete them and are numbered accordingly.
Each training phase is presented as a block with the title of the training phase and a label indicating whether the resident has completed the training phase. If a training phase is in progress a completion rate is shown to indicate the number of assessments the resident has achieved in that training phase relative to the total number of required assessments for every EPA in that phase. Each training phase block acts as an accordion and can be expanded or collapsed to view the list of all EPAs in that block.
Although residents generally complete the EPAs of their current training phase before they pick up EPAs of later phases, there are exceptions. Due to various external factors such as their rotation schedules and the nature of medical cases of the patients they attend to, residents can occasionally end up completing EPAs which are not in their current training phase. This means residents can have a non-zero completion rate for training phases that they have not yet started officially. When a training block is open, all the EPAs in that block are arranged sequentially based on the numbering order in a 3-column layout as shown in the following figure.
First Column: EPA ID and textual description of the corresponding medical scenario that the EPA targets.
Second Column: The residents acquisition metrics for each EPA are provided as four numbers along with two bullets charts that visualize how far along the resident is in completing that EPA. The first bullet chart (blue) visualizes the observed EPA count relative to the required count while the second bullet chart visualizes the achieved EPA count relative to the required count. If an EPA is complete (the required numbers of EPAs are achieved), the “TO GO’’ metric changes into a completed check mark icon.
Third Column: This is a visualization of all assessment filled by the resident for that particular EPA. The information is visualized similar to the recent EPA chart discussed above. Assessments are arranged chronologically on the X axis with the oldest to the left and are arranged vertically based on the EPA rating (5-point O-Score Entrustability scale) with 5 being the highest (resident managed the situation independently) and 1 being the lowest (Assessor had to completely take over the situation). Each point in this chart can be hovered upon to view additional information about that EPA such as narrative feedback in an onscreen popup window.
Finally, two buttons are provided as seen in the bottom left corner of the chart. The first one (book icon) can be clicked to see all the records in a table that can be sorted and filtered by columns. To filter the table start typing in the input box at the top of each column in the table. This will dynamically filter the table as you type. To sort the table by a column simply click on the column header.
The second button (sliders icon) brings up dropdown filter lists that can be used to visually identify a particular record based on patient demographics or other contextual variables such as “Case Complexity’’ or “Clinical Presentation’’. For example, if a user wanted to see which of the records were for senior patients, they could select the ‘Senior’ option from the drop-down list and the corresponding points (observation scores) would turn light red.
This is a common feature across the dashboard that highlights all assessments that were filled in particular time period. To enable this, head over the filter panel at the top of the dashboard and click on the small button with the calendar icon on it. This will open a panel where you can set the start date and end date for the time period. You can either type in directly into the input box or use the date selector on the calendar below.
Once the start date and end date are set all assessments that fall in that particular time period are converted into diamonds across the dashboard. This provides a way to visually distinguish these EPAs while still viewing them relative to other EPAs filled outside of the selected time period. This feature can be particularly useful during competence committee meetings which happen once every three months such that the time period can be set to highlight only the EPAs filled by the resident since the last meeting.
Another way to set the time period on the dashboard is by simply clicking on rotation block in the rotation schedule. This will automatically set the start date and end date of the time period to the dates of the rotation block and all assessments filled in that rotation block are automatically highlighted.
The checkbox provided in the date filter panel when enabled hides all EPA levels which do not have any assessments filled in the selected training period. If an entire training phase does not have any EPAs filled in the training period, then the whole training phase block as a whole is also hidden. This can be useful to reduce the visual clutter on the dashboard and only focus on a small subset of EPAs.
From this dashboard, program directors and coordinators can see a breakdown of all assessments completed in a program by a faculty member for informing faculty development.
