Add a Framework

1. Log in as a MedTech or Admin user.

2. Open the Admin drop-down on the top menu and click Manage Curriculum.

3. Click Curriculum Framework Builder in the left-side menu.

4. Click the Add Framework button.

5. In the Adding New Framework box, input the Title of the framework and click the Add Framework button.

6. The new framework will appear in the Curriculum Builder table.

Create a Tag Set

1. Click on the framework title in the Curriculum Builder table that you want to add tag sets.

2. On the Tag Sets page, click the Create Tag Set button.

3. On the Adding Tag Set page, under the Details section, input the Code, Name, Shortname. These fields are required.

4. Scroll down to select which course(s) the tag set will be associated with.

5. Scroll down to select from the options under Framework Specific Options, if applicable. See Curriculum Framework Tag Set Options for more details.

6. Scroll down to select from the options under Form Options, if applicable. See Curriculum Framework Tag Set Options for more details.

7. Scroll down to select from the options under Advanced Options, if applicable. See Curriculum Framework Tag Set Options for more details.

8. Click the Save button to add the tag set.

Note: With CBME, there may be multiple tag sets to add using:

- STG - Stages

- EPA – Entrustable Professional Activity

- KC – Key Competencies

- EC – Enabling Competencies

- MS – Milestones

Add an Existing Tag Set

1. Click on the framework title in the Curriculum Builder table that you want to add tag sets.

2. On the Tag Sets page, click the Use Existing Tag Set button.

3. On the Use an Existing Objective Set box, use the Existing Objective Set drop-down to select a tag set.

4. Click the Import button.

5. On the Edit Tag Set page, under the Details section, ensure the correct Code, Name, Shortname appear.

6. Scroll down to ensure the proper course(s) are associated with the tag set.

7. Scroll down to ensure the necessary tag set options are selected, if applicable:

   • Framework Specific Options

   • Form Options

   • Advanced Options

8. Click the Save button to add the tag set.

Note: With CBME, there may be multiple tag sets to add using:

- STG – Stages

- EPA – Entrustable Professional Activity

- KC – Key Competencies

- EC – Enabling Competencies

- MS – Milestones

Remove a Tag Set

1. On the Curriculum Builder page, select the tag set on the Tag Set table.

2. Click the Remove Tag Set button.

3. On the Remove Objective Sets box, review the instructions:

4. Click the Remove button to delete.

Manage Tag Set Relationships

1. Click on the framework title in the Curriculum Builder table that you want to manage the tag set relationships for.

2. Click on the Relationships tab.

3. Click on the Manage Relationships buttons for the tag set you want to add a relationship.

4. In the Adding Relationships to [insert tag set title] box, use the Relationship Type and Tag Set drop-downs to add the proper relationship.

5. Repeat the above steps for all necessary tag sets. This will create the framework for your CBE Course (program).

Create an Organization Tree

1. Click on the framework title in the Curriculum Builder table where you want to create, or edit an organization tree.

2. Click the Organization Tree tab.

3. On the Create New Organization Tree page, input the Title name.

4. Click the Save and Proceed to Uploader button.

Note: You can also set up an Organization Tree under the CBME tab in Manage Courses, but this workflow currently has the most usable user interface.

Delete an Organization Tree

1. On the Organization Tree tab, click the trash icon associated with the tree you want to delete.

2. On the Delete an Organization Tree page, click the Delete this tree button.

3. You will receive a notification that the tree was deleted, and the page will return to the Organization Trees table.

Setup a CBME Framework in CBE

The following steps are for manual setup and not migrated data.

1. Create the following tag sets with their corresponding tag set options and select Stages as the Primary tag set:

2. Select the following options for Stages:

3. Select the following options for Entrustable Professional Activities:

4. Do not select options for Key Competencies or Enabling Competencies:

5. Select the following options for Milestones:

CBE in Elentra allows institutions to create their own Competency Based Curriculum structure. This new module expands on the functionality developed for Competency-Based Medical Education (CBME), Enhancements have been made so that institutions can add their own competency nomenclature and structure. Your institution may use CBE differently than others, but all will be able to:

• Create a Curriculum Framework

• Upload Objectives for Curriculum Tag Set

• Upload or add CV (Contextual Variable) responses

• Set up an Assessment Plan

• Create Form Templates

• Review Learner Progression

• Set up a CBME program in CBE

• CBME to CBE Post-Migration Review

Add an Assessment Plan

1. In Manage Courses, click the Course Title you want to configure CBME for.

2. Click the Assessment Plans tab.

3. Select the Version (required) using the drop-down menu.

4. Input the Title (required) of the Assessment Plan.

5. Click the Save Plan button. The page will refresh to display the Plan Objectives.

CBME Migration

Pre-requisite: A developer has already migrated CBME data to CBE.

1. Course’s CBME tab in the course setup under Manage Courses.

   • Configure CBME tab

   • CV Responses tab

   • Assessment Plans tab

2. Curriculum Framework in the Curriculum Framework Builder under Manage Curriculum.

3. Form Template in Assessment & Evaluation.

Notes

• Two additional views were introduced in 1.24:

  • Program Evaluation

  • Program Oversight

• The CBME visual summary dashboard suite is optional and is controlled by a database setting (cbme_enable_visual_summary). To enable this for your organization, please speak to a developer. There is also a corresponding cron job that needs to be run nightly to update the data shown in the visual summary dashboards. More information about this can be found in the upgrading to ME 1.24 guide.

Pre-requisite

Assessment Plan Builder

To leverage the visual summary, your program must have assessment plans entered in Elentra. Please visit the Assessment Plan Builder lesson for more information.

Access Visual Summary

Step 1

Log in to Elentra as an administrator (e.g. program administrator, course director).

Step 2

At the top right, click on the "Assessment & Evaluation" task list icon.

Step 3

Click on the "My Learners" tab

Step 4

You will land on the Program Dashboard.

Step 5

From the tab menu below the filter option, click Visual Summary.

Step 6

You will be directed to the Visual Summary dashboard.

Step 7

Toggle between the different dashboards, and/or programs as applicable.

Note

Residents have access to the visual summary page only though their Learner Dashboard where they can click on “visual summary” from the tab menu.

Visual Summary Permissions

The permission system within the visual summary page has been set up in a specific way such that most users are only shown a small subset of the dashboards based on their user role and the rest of the dashboards are hidden completely. Only a small set of elevated user types have access to all the dashboards.

A breakdown of the visibility by user type of the 5 dashboards within the visual summary module:

Residents: Resident Dashboard (only their data) Competency Committee Members/Chairs: Resident Dashboard, and Normative Assessment Course Director: Resident Dashboard, Normative Assessment, Faculty Development, and Program Evaluation Program Coordinator: Resident Dashboard, Normative Assessment, Faculty Development and Program Evaluation Course Director/Program Coordinators, with access to multiple programs: Resident Dashboard, Normative Assessment, Faculty Development, Program Evaluation and Program Oversight (limited to the programs they have access to) Medtech Admin: Resident Dashboard, Normative Assessment, Faculty Development, Program Evaluation and Program Oversight (Access to all programs)

When creating an Objective Tree allowing for Objectives to be uploaded, Objectives may (not mandatory) be uploaded individually for each of the Curriculum Framework’s containing Curriculum Tags.

Prerequisite:

The titles of the column headings in the uploaded spreadsheets must be the following:

Two Upload Options:

Default Upload

• The “Parent(s)” field must be specific as to where the objective is to be located in the Curriculum Framework structure:

  • Example: for objective PTYPE1, Parent(s) field may be: “DOCC1-EPA1-POE1, DOCC1-EPA1-POE2” - This will attach objective PTYPE1 to Curriculum Tag PEO1 under EPA1, which is under DOCC1, as well as PEO2, under EPA1, which is under DOCC1.

• Parent(s) field can be empty until used in another upload. For example, objectives at the highest level would be uploaded with the “Parent(s)” field empty.

Sample .csv format:

Sample .csv format:

Upload Referencing a Single Parent Code

• Upload objectives referencing their immediate parent.

  • Example: when uploading objective PTYPE1, the Parent(s) field may only specify “POE1, POE2”. This attaches PTYPE1 to wherever Curriculum Tags POE1 and POE2 appear, no matter their parents or where they appear in the hierarchy.

Sample .csv format:

Sample .csv format:

Uploading Objectives Example

Demo Session

The Core Team recorded a demo session with a complete walk-through of the creation and versioning of Objective Trees:

• Demo

• Samples

Overview

Given Curriculum Framework with structure:

DOCC - Domains of Clinical Care

EPA - Entrustable Professional Activity

POE - Phase of Encounter

PTYPE - Procedure Type

P - Procedure

SD/FMR - Skill Dimensions & FM CanMEDs Role

PT - Priority Topic

P&P - Progress & Promotion

When selecting to create a new Objective Tree Version and choosing to Upload, a page will be displayed to allow the user to upload objectives for each of the above Curriculum Tag Sets in the hierarchy, in order of hierarchy:

1. Domains of Clinical Care

2. Progress & Promotion

3. Entrustable Professional Activity

4. Phase of Encounter

5. Skill Dimensions & FM CanMEDs Role

6. Priority Topic

7. Procedure Type

8. Procedure

Note: For each of the above, the user can upload a CSV of objectives and select either Default upload or Upload referencing a single parent code.

Level 1 - Root Level

For Curriculum Tag Sets with no parent in the Curriculum Framework structure, the selected method does not matter, since no Parent is specified for the Objectives.

For example, the following can be used to set objectives for Curriculum Tag Domains of Clinical Care, no matter the upload method since no Parent is specified.

Root Level Objectives

Level 2

In addition, for Curriculum Tag Sets that appear in the second level (below the top) in the Curriculum Framework, the upload method will also no matter since only a single Parent is specified:

Root Level Objectives

In the above example, objective EPA2 has two parents, DOCC2 and DOCC3.

For Curriculum Tag Sets that appear in lower levels than the top in the Curriculum Framework, the selected upload method will matter.

Level 3

Default Upload Method Example

The following is an example of how POE objectives can be uploaded using the Default Method, under EPAs:

The above will set objective POE1 under DOC1-EPA1 in the hierarchy for example. Also note the difference between POE4 and POE5.

POE4 is assigned to EPA2, but only EPA2 under DOCC2 in the hierarchy, where POE5 is assigned to EPA2 under DOCC3:

The above will set objective POE1 under DOC1-EPA1 in the hierarchy for example. Also note the difference between POE4 and POE5.

POE4 is assigned to EPA2, but only EPA2 under DOCC2 in the hierarchy, where POE5 is assigned to EPA2 under DOCC3:

Single Parent Code Upload Example

The following is an example of how POE objectives can be uploaded using the Single Parent Code method, under EPAs:

In the above, EPA2, no matter where it is in the hierarchy, is assigned the set objectives POE2, POE3, POE4 and PEO5. Therefore, both DOCC2-EPA2, and DOCC3-EPA2 are assigned those objectives.

Subsequent levels would be assigned objectives in a similar way.

Upload Objectives for a Curriculum Tag Set

You can upload objectives within a course under Manage Courses. The following steps are formatted under the lens of CBME.

Step 1

Within a course, select the CBME tab and then the Configure CBME tab to begin Step 1: Configuration. Select Use a curriculum framework as a base and click Next Step.

Step 2

Select a framework in the drop-down and click Next Step.

Step 3

Ensure the objective .csv files are formatted as indicated in the highlighted instructions.

Step 4

For the Stages upload, ensure the Default upload option is selected and click Choose a file or drag and drop to add a .csv file.

Step 5

Once the .csv file is selected, the upload area will display the file name. Click the Save and upload Royal College Stages button.

Step 6

A confirmation highlighted in green will display when the file is uploaded successfully.

Step 7

For the Enstrustable Professional Activities upload, ensure the Upload referencing a single parent code option is selected and click Choose a file or drag and drop to add a .csv file.

Once the .csv file is selected, the upload area will display the file name. Click the Save and upload Royal College Stages button. A confirmation highlighted in green will display when the file is uploaded successfully.

Step 8

For the Milestones upload, ensure the Upload referencing a single parent code option is selected and click Choose a file or drag and drop to add a .csv file.

Once the .csv file is selected, the upload area will display the file name. Click the Save and upload Royal College Stages button. A confirmation highlighted in green will display when the file is uploaded successfully.

Scroll down to click the Next Step button.

Notes:

• If you forget to select the correct Upload radio button (ex. Upload referencing a single parent code), you must reset the upload by going back to Step 1: Configuration in the CBME tab of the course or in the Organization Trees section of the Curriculum Framework Builder.

• Depending on your framework, Enabling Competencies and Key Competencies objective files do not need to be uploaded. Under the lens of a CBME framework, they are not required and the Default upload option would remain selected.

Step 9

You can review the new proposed tree structure with the uploaded objectives added. Click the Publish button to save.

Note: if there are errors in the upload, then the objective uploads have not been published. The publish button also does not appear when there is an error. You will need to reset the objective upload. See Notes section in Step 8 of this documentation.

Step 10

Once a CRON job interval has been completed, you can refresh the Configure CBME tab screen to review and search within the new tree mappings.

Test Uploading Objectives

Using the CBME framework, you can test uploading objectives for a curriculum tag set in your staging environment.

Sample Stages File

Sample EPAs File

Sample Milestones File

Upload CV Responses

1. In Manage Courses, click the Course Title you want to configure CBME for.

2. Click the CV Responses tab.

3. Click the Upload CV Responses button.

4. You can click Choose a File or drag and drop a file.

5. Click the Save and Upload Contextual Variable Responses button to complete.

Sample CV Response File

Add CV Responses

1. In Manage Courses, click the course title you want to configure CBME for.

2. Click the CV Responses tab.

3. Click a Contextual Variable in the list to expand.

4. Click Add Response to expand to add the response(s).

5. Click the Save icon to save response(s).

Framework Specific Options

Show in Dashboard

This tag set will display on the learner dashboard.

When this option is selected, a dashboard scale can also be selected. The dashboard scale will display to progress reviewers (with permission) to set a status for the objective (when clicking the status circle). If you don’t have permission to set the status for an objective, you will only see the status.

Display Under Parent on Dashboard

This tag set option will display the objectives under their parents as categories on the dashboard.

"EPA's" are displayed under their parent, "Stages", on the Learner Dashboard

Allow tag set to be used as a filter on the dashboard

This option will add the tag set as a filter on the learner dashboard.

Allow tag set to be triggered

This option should be selected when triggering an on-demand assessment.

Allow tag set to be mapped to Clinical Experiences

This option allows the tag set to be mapped to clinical experiences (e.g., rotations).

Allow tag set to be used with Assessment Plans

This option should be selected if the tag set is to be used in the assessment plan builder.

Allow tag set to be displayed on the Program Dashboard

This option will allow the tag set to be displayed on the program dashboard.

Group Program Dashboard progress by this tag

This option will group children of this tag set on the program dashboard.

Allow tag set to be used as a filter on the Program Dashboard

This option will use the tag set as a filter on the program dashboard.

Allow editing of objective text

This option will enable editing of the tag set once it has been uploaded.

Form options

Allow tag set to be shown on a smart tag form

This option will add this tag set as a smart tag item on the smart tag form templates. Smart tag items will display the mapping of the triggered branch after the form has been generated.

Allow tag set to be added to form templates (used with CBME/Royal College Frameworks)

This option will allow this tag set to be used on other CBME form templates (Supervisor Form, Procedure Form, Field Note Form).

Programs map to this objective set (used with CBME/Royal College Frameworks)

This option will add this tag set as a mapped objective on supervisor forms. Example: Milestones scale can be selected on a Supervisor Form template

Advanced Options

This objective set is a primary competency

• This is meant to show an EPA; where we expect to show EPA related information, this option allows the objective set to be used in its place.

• This is also important to select as the child of a primary objective (example: Parent - Stages, Child - EPAs).

  • You will want to select this to ensure all children and grandchildren of this tag set/objective set will be displayed in the objective tree mapping for your CBE course.

• Select this to ensure uploading objective files is successful.

Allow tag set to be used as a filter in the triggering interface

This option sets the tag set as a filter for on-demand triggering.

If the objective is a primary competency, this filter will display the set of objectives the learner is currently in-progress (e.g., Display EPAs by the current stage). When this option is selected, name the filter.

This objective set is a container of its related children (used with CBME/Royal College Frameworks)

This option may be used if your curriculum uses stages. For example, your learners must complete a foundations stage before they move into a core stage.

Apply ordering at this objective set's hierarchical depth

This option determines that ordered objectives must start at the depth of this tag set. This is relevant when multiple types appear on the same depth. The tree in the example below is the Framework and not actual branches of objectives. If the order for B is set as a value of 2, and the order for C is set to 1 and D is left unset, then, the relevant interfaces (currently, only Smart Tag Triggering) would render as such: C, B, D

Use Case

When two objectives live on the same level, both are triggerable and you would like to identify which order they should be displayed in. Relevant to smart tag form triggering.

This objective set utilizes objective ordinality

Objective code list (order of objectives in this set)

• Clicking this option allows you to enter the order of the objective codes on each tag in this tag set, in the order that the learner will complete it. In the case of the Stages tag set, enter D,F,C,P. This indicates that the learner would complete the D, F, C, P objective codes in this tag set, in this order.

• This applies to curriculum versioning.

CBME Program

Pre-requisite: A Course is already created.

1. Create a framework in Curriculum Framework Builder.

2. Create a course in Manage Courses.

3. Add the course name to tag set in the framework under Curriculum Framework Builder.

4. Follow the tab flow under the Configure CBME tab in the course under Manage Courses.

5. Create an A&E form template.

The Resident Metrics Dashboard focuses on individual residents and is designed to be used by Residents and Competency Committee members.

The resident dashboard has a wealth of information that is grouped into different categories for easier comprehension. First if you arrived at the resident dashboard by selecting a resident on the normative dashboard their data is automatically fetched for you. However, if you manually switched over to the resident dashboard by clicking on the navigation tabs above, you will need to select a resident from the drop-down in the filter panel situated at the top of the dashboard. The drop-down contains the list of all the residents in the program with their name and their corresponding progress rate. The names of residents are further grouped by their training stage and then sorted alphabetically for easier access.

The drop-down is also an editable text box and so you can type a resident’s name partially to automatically filter the available options in the drop-down. This makes it easier to search for a particular resident in a program with many residents.

After selecting a resident, users can then click on the “GET RECORDS’’ button to visualize their assessment data. You might notice the small button with the calendar icon on it. This is used to highlight assessment data gained by the resident in a particular time period. For now, ignore it but we will learn more about it further down. The resident dashboard consists of several main sub sections. Let us look at each one individually.

Acquisition Metrics

This section provides the following summarized metrics of the resident:

• Total EPAs observed - This is a count of the total number of EPAs filled out by a resident.

• Progress Rate - This is the number of EPAs a resident has achieved divided by the total number of EPAs they are required to achieve for all the valid EPA forms in a program across the different training phases.

• Achievement Rate - This is the total number of EPAs a resident has achieved divided by the total number of EPAs completed by that resident. An achieved EPA is one where the EPA meets certain requirements set in the assessment plan, such as acquiring a rating of 4 or above on a 5-point scale, or satisfying specific contextual variable requirements, or meeting diverse assessor role requirements.

To the right of the acquisition metrics is a line chart that shows the weekly EPA acquisition rate for the last six months by the resident. This is meant to give a high level overview at a quick glance of the residents assessment gathering in the recent past.

Recent EPAs

This section is meant to quickly lookup a residents’ recent performance with the option to view records in the following ranges: last 10 days, last 25 days, last month and last 3 months. The chart does not visually distinguish the different EPA types (i.e., EPA-F1 vs EPA-C2), instead, it provides this and other additional information in a pop-up menu that can be invoked by hovering the mouse over a point.

The line chart provides a simple representation of the last “N’’ assessments filled by the resident where every EPA is represented as a point with the oldest record starting on the left. The points are arranged vertically using the O-Score Entrustability scale with 5 being the highest (I did not need to be there) and 1 being the lowest (I had to do). The better a resident performs in an EPA, the higher is the vertical position of the point in the chart.

If a resident has assessments that were filled on EPA forms across several different rating scales then each rating scale is provided with its own recent EPA chart as shown in the above image.

Individual EPA Assessments

The final section provides a detailed overview of every single EPA completed by the resident. The entire list of EPAs that residents are required to complete are broken down into four groups based on the training phase during which a resident is supposed to complete them and are numbered accordingly. With the addition of support for dynamic CBE the number of training phases can be higher or lower than 4. Some programs such as surgical foundation for example only have two training phases.

Each training phase is presented as a block with the title of the training phase and a label indicating whether the resident has completed the training phase or not. If a training phase is in progress a completion rate is shown to indicate the number of assessments the resident has achieved in that training phase relative to the total number of required assessments for every EPA in that phase. Each training phase block acts as an accordion and can be expanded or collapsed to view the list of all EPAs in that block.

Although residents generally complete the EPAs of their current training phase before they pick up EPAs of later phases, there are exceptions. Due to various external factors such as their rotation schedules and the nature of medical cases of the patients they attend to, residents can occasionally end up completing EPAs which are not in their current training phase. This means residents can have a non-zero completion rate for training phases that they have not yet started officially.

When a training block is open, all the EPAs in that block are arranged sequentially based on the numbering order in a 3-column layout as shown above.

First Column

EPA ID and a textual description of the corresponding medical scenario that the EPA targets.

Second Column

The residents acquisition metrics for each EPA are provided as 3 numbers along with two bullet charts that visualize how far along the resident is in completing that EPA. If an assessment plan is not available for an EPA the required and achieved numbers default to “N/A” (not available). The first bullet chart (blue) visualizes the observed EPA count relative to the required count while the second bullet chart visualizes the achieved EPA count relative to the required count. A green check mark icon indicates the completed status of the EPA. It can show up either because the resident has achieved the required number of EPAs or if the competence committee has marked the EPA as complete (even if the achieved count is not met). In the scenario shown in the image above the latter is true. However, if an EPA has not been marked complete and the resident has not met the required achieved EPA count then a “TO GO” metric is shown in place of the check mark icon as shown above.

Third Column

This is a visualization of all assessments filled by the resident for that EPA. The information is visualized like the recent EPA chart discussed above. Assessments are first grouped by the EPA form type and version number. This ensures that assessments of a similar variety are visualized together. Within each chart, assessments are arranged chronologically on the X axis with the oldest to the left and are arranged vertically based on the EPA rating (5-point O-Score Entrustability scale) with 5 being the highest (resident managed the situation independently) and 1 being the lowest (Assessor had to completely take over the situation). However, the rating scale is not always a standard 5-point scale and can change depending on the type of scale used in the assessment plan for a given EPA. For example, in the image shown below the EPA form has a 2-point (yes/no) rating scale.

Further each point in this chart can be hovered upon to view additional information about that assessment such as narrative feedback, situational context, assessor name and their role in an onscreen popup window as shown above.

Finally, three buttons are provided as seen in the bottom left corner of each chart. The first button (sliders icon) brings up a set of drop-down filter lists that can be used to visually identify a particular record based on patient demographics or other contextual variables such as “Case Complexity’’ or “Clinical Presentation’’. For example, if a user wanted to see which of the records were for “respiratory distress”, they could select that option from the Clinical presentation drop-down list and the corresponding points (observation scores) would turn light red.

The second button (book icon) can be clicked to see all the records in a table, which can be sorted and filtered through. To filter the table start typing in the input box at the top of each column in the table. This will dynamically filter the table as you type. To sort the table by a column simply click on the column header. For example, in the image below the table is being sorted in an ascending order by the first column (date).

The third button brings up a popup screen that shows the achievement criteria breakdown for the EPA. This feature has been duplicated from the main CBME dashboard where hovering over the “i” icon in an EPA gives a breakdown of the assessment criteria as shown above.

Note

If a school has enabled the ability to track expired assessments, an optional section is visible at the end of the dashboard which shows a tabular breakdown of all the expired assessments filled against a selected resident.

Date Filtering

This is a common feature across all the sections of the resident dashboard that highlights all assessments that were filled in a particular period. To enable this, head over to the filter panel at the top of the dashboard and click on the small button with the calendar icon on it. This will open a panel where you can set the start date and end date for the period. You can either type in directly into the input box or use the date selector on the calendar above.

Once the start date and end date are set, all assessments that fall in that period are converted into diamonds across the dashboard. This provides a way to visually distinguish these EPAs while still viewing them relative to other EPAs filled outside of the selected period. This feature can be particularly useful during competence committee meetings which happen once every three months such that the period can be set to highlight only the EPAs filled by the resident since the last meeting.

The checkbox provided in the date filter panel when enabled hides all EPA levels which do not have any assessments filled in the selected training period. If an entire training phase does not have any EPAs filled in the training period, then the whole training phase block is also hidden. This can be useful to reduce the visual clutter on the dashboard and only focus on a small subset of EPAs.

The Normative Assessment Dashboard presents summarized data metrics of all the residents in a program relative to each other. It is meant to be viewed by Competency Committee members to assess a resident against their peers.

The data is presented as both a visual representation(left) and a tabular representation(right). Users are provided with an option to either view all the residents in a program or selectively view the metrics of residents in a particular training stage by using the drop-down at the top of the dashboard. This can be useful during Competency Committee meetings when residents are judged on their performance relative to others in their training group.

By default, the normative dashboard filters out residents without any completed EPAs. However, this behaviour can be turned off by toggling the checkbox at the top of the dashboard.

The bar chart visualizes the following four metrics individually and to switch between the metric being visualized, users can select the corresponding radio buttons above the chart. Each bar represents one resident and users can hover their mouse over a bar in the chart to see the name of the resident it represents, and the value of the current metric being shown. Clicking on a bar in the bar chart switches the user to the resident dashboard to view a detailed breakdown of all assessments of that resident.

• Total EPAs- This is a count of the total number of EPAs filled out by a resident.

• Achievement Rate- This is the total number of EPAs a resident has achieved divided by the total number of EPAs completed by that resident. An achieved EPA is one where the EPA meets certain requirements set in the assessment plan, such as acquiring a rating of 4 or above on a 5-point scale, or satisfying specific contextual variable requirements, or meeting diverse assessor role requirements.

• Progress Rate- This is the number of EPAs a resident has achieved divided by the total number of EPAs they are required to achieve for all the valid EPA forms in a program across the different training phases.

• Total EPAs vs Achieved EPAs- This chart visualizes two bars for each resident showing their total EPAs and achieved EPAs next to each other. While this metric is like achievement rate, it can offer a better picture of a resident’s overall performance as high achievement rates alone can be occasionally misleading for residents with a very low number of completed EPAs when all of them are achieved.

Finally, there is a tabular representation of the same metrics with the ability to sort the residents in the table by clicking on the column header. By default, the residents are sorted in the table by name in an ascending order. This can be changed to a descending order by simply clicking on the column header “Name”. Similarly clicking on each column header sorts the table in an ascending order based on that metric and clicking the same column header again changes the order to descending.

The normative dashboard is linked to the resident dashboard so to view a particular residents’ assessments in detail, users can simply click on a bar corresponding to a resident in the bar chart or their corresponding row in the table. This will automatically switch the user to the resident dashboard with that resident pre-selected.

Learner Dashboard

1. Select the desired course in the drop-down.

2. The course disciplines will be displayed in a list.

3. Click the drop-down icon to expand a specific discipline to review the learner’s progress.

Start an Assessment/Evaluation on the Learner Dashboard

1. Click the Start Assessment/Evaluation button.

2. Select an assessor from the drop-down.

3. Select the date of the encounter.

4. Select the Assessment Method:

• Email blank form The assessor will receive an email notification to complete an assessment based on the selected tool.

• Complete and confirm via email Complete an assessment using the selected tool. Upon completion, the assessor will receive an email notification asking them to review/edit and confirm the assessment.

• Complete and confirm via PIN An assessor must have a PIN set-up to use this method. The learner will initiate the assessment. The assessor will complete the form and enter their PIN to indicate they have completed the assessment.

• Self Assessment, then email blank form Complete an assessment based on the selected tool. Upon completion, the assessor will receive a blank assessment task with the same assessment tool.

4. Select the desired course in the drop-down menu.

5. Select the desired EPA in the drop-down menu.

6. Begin or Send the Assessment, depending on your choice.

This dashboard organizes all the EPAs that have been completed by learners in the program by the year they were completed in with the goal of informing program evaluation. There are two main sections in this dashboard. The first section of the dashboard visualizes key metrics across several academic years as shown below.

EPAs Acquired and Expired Per Learner

This graph displays the number of EPAs that have been completed and expired per learner in each year in the program. Users can mouse over the bars to see the number of active residents with assessments in each year.

EPA Rating Distribution

This stack chart displays the proportion of EPAs in each year that have been rated at each level of entrustment ('I had to do' to 'I didn't need to be there'). Users can mouse-over each row for additional details. This chart only considers assessments that have been completed on "Supervisor Forms" that have a standard 5 point "O score" scale.

Average Words per Comment

This graph displays the average number of words contained within the completed EPAs of each year. The length of the feedback has been found to correlate with the quality of feedback and so a higher word count is preferable.

Monthly Distribution

This graph visualizes the number of EPA observations submitted per month over multiple years. It is intended to identify increases and decreases in EPA assessments over seasons and years. The X axis of this chart spans across a given academic year from July to June.

The second section of the dashboard shows EPA metrics that contextualize the EPAs that have been completed within a program over a selected academic year.

The summarized metrics are followed by three pie charts that group the assessors by their type, group, and role. Assessor role and group distributions are only available for "internal" assessors. These charts let program evaluators get a better picture of which user group gives them a higher share of EPA assessments. For example, in programs like Surgical foundations where senior residents often assess junior residents. The role “Trainee” might have a higher share compared to “Lecturers” or “Directors”.

Export

The “export program data” button available at the top of the dashboard lets users download a detailed CSV export of all assessments completed or expired in the program. This can be used for additional downstream analysis of metrics that are not shown in the dashboard.

This dashboard organizes all of the EPAs that have been completed in a program by the assessor that completed them with the goal of informing faculty development. To get started, users need to first select the academic year in which the assessments were collected and click “GET RECORDS”.

Filters

There are three filters available in this dashboard. The first filter lets users select an assessor group to only look at assessments completed by users in that group. This filter can be used to remove or include student assessors. The second filter lets users select a department to only look at assessments completed by assessors from that department. This can be used to remove assessors from external programs. The final filter lets users select a specific assessor for their metrics to be highlighted. Alternatively, users can click on any of the bars in the charts below to select and highlight that assessor.

Acquisition Metrics

There are two sections each showing summarized metrics related to faculty assessments. The first section shows the amalgamated metrics for EPAs completed by all assessors in a given academic year. Users can mouse-over the EPA Rating visual to see the proportion of EPAs rated at each level of entrustment (EPAs completed on Supervisor Forms with 5-point O score scales are only considered for this metric). Users can also mouse-over the Training Stage visual to see the proportion of EPAs completed in each stage of training. The second section shows the metrics for EPAs completed by the selected Assessor. Like the section above, users can mouse-over the EPA Rating visual to see the proportion of EPAs rated at each level of entrustment and also mouse-over the Training Stage visual to see the proportion of EPAs completed in each stage of training.

The acquisition metrics panel is followed by four charts each visualizing a specific metric of an assessor relative to others in the program. The red highlighted bar represents the current selector assessor/faculty.

Total EPAs Observed

This chart displays the number of EPAs observed by each assessor. Users can mouse-over for each assessor's name and click to highlight that assessor's data. If an assessor is selected, their EPA count is shown in the chart title in red.

EPA Expiry Rate

This chart displays the percentage of EPAs sent to each assessor that expired before completion. If your school doesn't track expiry metrics of assessments, then the values in this chart would default to zero for all faculties.

Average Entrustment Score

This chart displays the average entrustment score of EPAs completed by each assessor. If an assessor is selected, their average entrustment score is shown in the chart title in red. This chart only considers assessments that have been completed on "Supervisor Forms" that have a standard 5 point "O score" scale.

Average Words per Comment

This chart displays the average number of words per comment included with the EPAs completed by each assessor. If an assessor is selected, their average words per comment metric is shown in the chart title in red.

The charts are followed by two tables that are only visible when a faculty is selected. The first table displays all the EPAs completed by the selected assessor. It is searchable (click the white box) and sortable (click the column header). The second table displays Expired EPAs that were not completed by the selected assessor. It is also searchable (click the white box) and sortable (click the column header). Both the tables can be exported as a CSV file.

Exports

The dashboard has two different types of exports that can be triggered by using the two buttons at the top of the page.

Download Assessor Report

This provides a CSV export with a list of all faculties and their related metrics. This can be used for downstream analysis that is beyond the scope of the dashboard. If a faculty is selected, the report contains their data alone. If a user wants the data of all the faculties in the export then they must select “All” in the “Assessor” filter.

Print Assessor Dashboard

This button is only visible when a faculty has been selected and it lets the user export the entire dashboard as a PDF file that can be shared with other users who don't have access to the dashboard. It prompts a “Save as PDF” popup as shown below.

Add a Supervisor Form Template

1. Open the Admin drop-down on the top menu and click Assessment & Evaluation.

2. Click the Form Templates tab.

3. Click the Add Form Template button.

4. Input the following:

   • Form Title (required)

   • Form Type: Supervisor Form

   • Program (required): [select course name]

5. Click the Add Form button to create the form.

6. The creation of the form template will show you the Edit Form Template page.

7. Scroll down to Template Components.

8. Choose the Course Objective(s) from the list you want to include in the assessment.

9. If you want to remove a Course Objective(s), click the drop-down above the list to deselect.

10. Click the Save and Next button to continue to the Contextual Variables.

11. To associate Contextual Variables with Course Objectives:

    • Deselect a Course Objective(s) to create a new list below.

    • Select the corresponding Contextual Variables beside each list.

12. Click the Save and Next button to continue to the Milestone Scale Selector.

13. Click the Select Scale drop-down on the Milestone Scale Selector to select the desired scale.

14. You can click the Comment drop-down to enable or disable comments.

15. You can select the desired responses from the Responses list.

16. Click the Select Scale drop-down on the Global Rating Scale Selector to select the desired scale.

17. You can click the Comment drop-down to enable or disable comments.

18. You can select the desired responses from the Responses list.

19. Click the Publish button to complete the form template.

Add a Smart Tag Form Template

1. Open the Admin drop-down on the top menu and click Assessment & Evaluation.

2. Click the Form Templates tab.

3. Click the Add Form Template button.

4. Input the following:

   • Form Title (required)

   • Form Type: Smart Tag Form

   • Program (required): [select course name]

5. Click the Add Form button to create the form.

6. The creation of the form template will show you the Edit Form Template page.

7. Scroll down to Template Components.

8. In the Template Components, under the Associated Objectives to display on form drop-down, select the desired tag set objective.

Note: The tag set options of Allow tag set to be shown on a smart tag form and Allow tag set to be added to form templates must be selected in the Curriculum Framework Builder.

8. Click the Save and Next button to continue.

9. Select the desired Contextual Variables in the list of Contextual Variables.

10. Click the Save and Next button to continue.

11. Complete the remaining sections of the form as outlined in Add a Supervisor Form Template.

This dashboard lets you compare metrics among different programs for the purpose of program oversight. The dashboard is mainly intended for users that oversee several programs or other higher-level users like Medical Administrators and Deans.

Users need to first select an academic year. This updates the Program dropdown and shows an alphabetically sorted list of programs with the “EPA count” next to the name of each program. The list only contains programs that have assessments that were completed in the selected year. The user can then multi-select all the programs they need to compare and then click “GET RECORDS”.

EPAs Acquired and Expired

This graph displays the number of EPAs that have been completed and expired in each year by program. If your school doesn't track expired assessments the values default to zero for all programs.

EPAs Acquired and Expired per Resident

This graph displays the number of EPAs that have been completed and expired per learner in each year by program. The dotted line represents the average completed EPA count per learner for all the selected programs.

EPA Rating Distribution

This stack chart displays the proportion of EPAs in each year that have been rated at each level of entrustment ('I had to do' to 'I didn't need to be there'). Users can mouse-over each row for additional details. This chart only considers assessments that have been completed on "Supervisor Forms" that have a standard 5 point "O score" scale.

Mean Words per Comment

This graph displays the average number of words contained within the completed EPAs of each year. The dotted line represents the mean value of all the selected programs. The length of the feedback has been found to correlate with the quality of feedback and so a higher word count is preferable.

Monthly Distribution

These are a collection of graphs that visualize the completed and expired count by month through the academic year (spanning from July to June) for each program individually.

Export Program Data

This button lets you download the metrics shown in this dashboard as a CSV file for other types of downstream analysis beyond the scope of this dashboard.