1 of 62

Competency-Based Medical Education

If your institution is new to competency-based medical education (CBME), especially in the context of Canadian PG programs, please read this additional learning module to prepare to launch CBME with Elentra: http://bit.ly/CBMELaunchReadiness

The Competency-Based Medical Education module allows institutions to map Entrustable Professional Activities (EPAs) and link them to Key and Enabling Competencies and Milestones. In turn, this allows the creation of assessments and the collection of learner performance data, all tied to EPAs.

Different institutions will be using Elentra differently to help them transition to CBME. There are some CBME features that rely on the use of other parts of Elentra, however, using CBME without any additional features like rotation scheduling, you will be able to:

map EPAs and milestones,
use supervisor, procedure and field note form templates to generate triggerable assessment forms,
create periodic performance assessment (PPA) and rubric forms that are linked to EPAs (and also triggerable),
allow faculty and learners to trigger assessment forms on demand as needed,
store and report on learner performance data based using the CBME dashboard,
assign faculty as competence committee members and academic advisors,
log meeting information for each resident,
promote learners through stages,
collect feedback from learners on completed assessment forms, and,
create EPA-based assessment plans for each course/program.

If you wish to create additional forms for assessment or evaluation purposes, use the clinical rotation scheduler, or schedule individual learning events, resources for those actions are also available on this site at the following links:

Assessment and Evaluation (used to create and distribute additional forms like course or faculty evaluations) Clinical Rotation Schedule (used to schedule learners into clinical rotations - note this will allow use of some additional learner tools in the CBME module) Scheduling Learning Events (used to schedule events like academic half days)

Setting Up CBME: Overview

This is an overview only. For more detailed information on each step of setting up CBME, please go to the sub-topics.

To ensure CBME works correctly, each organisation that has CBME enabled must have a developer or technical administrator set the default_stage_objectivesetting value in the elentra_me.settingstable to the global_lu_objectives.objective_idof the first stage of competence (ex: Transition to Discipline).

Using Elentra with learners and faculty requires some general basic setup including the creation of curriculum layouts and periods, loading of user information, and creation of courses. Those tasks are not CBME specific and are covered elsewhere in our documentation.

Beginning to specifically use Elentra for CBME can be broken down into several tasks:

mapping your EPAs,
building form templates,
setting up learners and faculty to access the forms, and,
reviewing and reporting on learner progress.

Setting up CBME Curriculum Map Version and Curriculum Tags

Before institutions can map program EPAs and build form templates, a Curriculum Map Version needs to be created. This is done by navigating to Admin > Manage Curriculum > Curriculum Map Versions > Add Version. The purpose of the Curriculum Map Version is the ability to easily change Curriculum Layouts. The Curriculum Map Versions can have a status of draft or published. The draft status is useful for making changes that learners and faculty will not see until it is published. When creating a new Curriculum Map Version, there is also the ability to import a curriculum map from a previous version to quickly update curriculum layouts in the new map version.

To begin using CBME, specific curriculum tag sets must be in place. To build the necessary tag sets and tags:

Create a Course by going to Admin > Manage Courses > Add New Course. Once the information in the Course Setup tab has been completed, click on the CBME tab where the auto-setup curriculum tag wizard will appear. Click on 'Yes, enable CBME features.'

The auto-setup wizard is only required once for an organisation and populates three tag sets: CanMEDS Roles, Royal College Stages, and Contextual Variables. You will receive a confirmation once the wizard has set up the tag sets.

In Admin > Manage Curriculum > Curriculum Tags, populate the CanMEDS Key and Enabling Competencies at the organization level. This happens once for an organisation. Even if your organization will be using program specific objectives, you are still required to upload a standard set of key and enabling competencies.
Import EPAs, Key and Enabling Competencies, Milestones and Contextual Variable Responses (e.g. options for case complexity or clinical setting) through a course's CBME tab by navigating to Admin > Manage Courses > CBME > Import CBME Data. This happens for each course/program using the CBME module.
- Upon importing the first set of EPAs, Key and Enabling Competencies, Milestones and Contextual Variable Responses, you will be brought to Step 1: Flag EPAs, click 'Next' as there is no data in this section yet as this step is used for versioning.
- Step 2: Set the Status of All Objectives doesn't require any action as there is no data and is used for versioning. Click 'Next'.
- Step 3 called Import CBME Data is where you will upload the files for EPAs, Key and Enabling Competencies, Milestones and Contextual Variable Responses.
Define and upload procedure attributes to make use of the procedure form template. Procedure attributes become assessment criteria for procedures. Each program using CBME will need to upload procedure attributes but this can be completed after the other imports. It may be preferable to complete this step when programs are ready to build their assessment tools.

Building Form Templates

Once the required curriculum tags sets are in place and populated, an institution can begin to use assessment features linked to CBME. A series of form templates can be completed at the program level to facilitate assessment of learners. The existing form templates are:

Supervisor Form Template
Procedure Form Template
Field Note Form Template

Using these form templates allows Elentra to rely on a program's EPAs, competencies, milestones and contextual variables to generate and publish multiple forms based on the parameters established on each form template.

Additional assessment forms can also be used for CBME. These include:

The Periodic Performance Assessment Form
The Rubric Assessment Form

As long as items added to these forms are linked to an EPA the form will be triggerable once it is published. Note that using these forms allows you to add customized questions or prompts. You can build those items either in advance or during form creation using the Assessment and Evaluation module.

If you want to be able to create additional customized forms and distribute them to learners and/or faculty based on rotation schedules, specific learning events, etc. you must be using additional features of Elentra. For additional information about creating forms, please see information for the Assessment and Evaluation module here.

Managing Faculty and Learners

Through the Groups tab of a course/program page, set up optional Academic Advisor Groups to link faculty members to groups of learners. Academic Advisors (listed as tutors in the course groups) will be able to view the CBME dashboards for all of their associated learners. The labels next to each tutor of Teacher, Tutor, Teacher's Assistant and Auditor are only used to differentiate between the different tutors in the same group and provide additional information. They do not change any of the user's permissions.

Through the Course Contacts section of a Course Setup page, add faculty to the Competence Committee Members list to allow them to review learner performance and make decisions regarding learner progress.

Reviewing and Reporting on Learner Progress

The CBME Dashboard is one place to review learner progress and the different forms completed on a learner. An Admin can access the CBME Dashboard of their learners by clicking on My Assessment & Evaluation (located in the top right hand corner between your profile and the Logout button), then by navigating to My Learners. From the CBME Dashboard Stages tab, users can quickly see learner progress in different EPAs and stages and see an overview of completed forms and performance trends on the global assessment ratings on those forms. An aggregated report of learner performance in an EPA is available directly from the Stages tab of the CBME dashboard.

From the CBME dashboard you can also access an Assessments tab which allows you apply a variety of filters, search in forms, and view completed, in progress, pending and deleted forms where the learner is the target of the form.

As learners and faculty review and discuss forms, a variety of additional tools are available including the ability to pin comments, items, and forms, a way to log meeting notes, and the ability for learners to give a "thumbs up" to a completed form.

Additional tabs accessible from the CBME dashboard include Assessment Items (a view of individual items completed on the learner), Trends (visualizations of learner performance), Comments (access to all comments in one place) and Pins (view of anything residents or faculty have pinned for review).

In addition to the CBME dashboard, learner progress can be reviewed and reported from the learner's Assessments page by clicking on Assessments, in the My Learners tab of My Assessment & Evaluation.

This displays the tasks where the learner is the target or the assessor/evaluator. Reports are accessible from this page as well, however many of these reports only apply to forms managed via distributions, not CBME forms triggered by faculty or learners.

Mapping Curriculum

Setting up CBME Curriculum Map Version and Curriculum Tags

Mapping Entrustable Professional Activities, Key and Enabling Competencies, Milestones, Contextual Variable Responses and Procedure Attributes for each program is what allows administrators to use the CBME form templates to quickly create assessment tools. There are two stages in mapping curriculum; in the first stage a central administrator (e.g. PGME office) needs to configure Elentra, then, in the second stage, the curriculum information specific to each program needs to be uploaded.

Steps taken once, at the organisational level, include:

Auto-setup CBME features,
Import standard list of Key and Enabling Competencies,
Import custom list of contextual variables as needed, and
Build assessment rating scales as needed.

Then individual programs can upload Entrustable Professional Activities, Key and Enabling Competencies, Milestones, Contextual Variable Responses and Procedure Attributes.

Auto-Setup Curriculum Tag Sets

To use the CBME Module, nine required curriculum tag sets must be in place. An auto-setup tool will help users create these. Only certain users have permission to complete this step.

Note: If you are a UGME organization using CBME, some of these may be changed; please speak to a developer or the Elentra Core Team as needed.

Navigate to Admin > Manage Course > create a new program (course), and click the CBME tab.

If the required curriculum tag sets aren't yet configured, the program will prompt users to auto-setup the required curriculum tag sets and will populate the CanMEDS Roles, Royal College Stages, and Contextual Variables tags.

Users without permission to use the auto-setup feature will be directed to contact a system administrator with the message below.

After an administrative user has run the auto-setup tool, several tag sets will be created and three of them populated (users can view these via Admin > Manage Curriculum). Unless you are working with a developer, you cannot change the tags in these tags sets or after you do, you will be prompted to run the auto-setup feature again.

Populate CanMEDS Competencies

Although Elentra allows programs to upload specific Key and Enabling Competencies, organizations are still required to upload a standard list of Key and Enabling Competencies at the organization level. For Canadian schools using CBME, we recommend making the CanMEDS Key and Enabling Competencies your standard list. These are available online or by contacting Elentra's Training and Development Coordinator.

To download the required templates, complete them, and upload standard Key and Enabling Competencies information:

Navigate to Admin > Manage Curriculum.
Click 'Curriculum Tags' from the Manage Curriculum card on the left sidebar.
Click on a curriculum tag set name (e.g. CanMEDS Key Competencies) and then click 'Import from CSV' in the top right to access and download the template to complete.

After you complete each required template, save it in a .csv format and upload it to the corresponding tag set page.

Accessing CanMEDS Key and Enabling Competencies

You can access the CanMEDS Key and Enabling Competencies at http://canmeds.royalcollege.ca/guide. The Key Competencies are preceded by one number like 1 or 2, and are stored under CanMEDS roles. The Enabling Competencies are preceded by a number like 1.1 or 2.2 and are also stored under CanMEDS roles.

Sample Key Competency: 1 Practise medicine within their defined scope of practice and expertise

Sample Enabling Competency: 1.1 Demonstrate a commitment to high-quality care of their patients

Both of these are from the Medical Expert role so would be coded with the prefix ME. Alphanumeric codes are provided in the templates, you will need to input the key and enabling competency text.

You can export the text from the CanMEDS guide.

CanMEDS Key Competencies

Download the template through Manage Curriculum > Curriculum Tags > CanMEDS Key Competencies.

Note that the first column, objective_code, is completed for you with alphanumeric codes based on the CanMEDS roles.

Complete the second column, objective_name, by providing the key competency text. You can access the key competencies at http://canmeds.royalcollege.ca/guide or request them from another institution.

Save your file as a CSV.

CanMEDS Enabling Competencies

Download the template through Manage Curriculum > Curriculum Tags > CanMEDS Enabling Competencies.

Note that the first column, objective_code, is completed for you with alphanumeric codes based on the CanMEDS roles.

Complete the second column, objective_name, by providing the enabling competency text. You can access the enabling competencies at http://canmeds.royalcollege.ca/guide or request them from another institution.

Save your file as a CSV.

Uploading Templates and Setting as Standardized Curriculum Tag Set

To upload completed templates:

Navigate to Admin > Manage Curriculum.
Click 'Curriculum Tags' from the Manage Curriculum card on the left sidebar.
Click on a curriculum tag set name (e.g. CanMEDS Key Competencies) and then click 'Import from CSV' in the top right.
Drag and drop or select the appropriate file from your computer and click 'Upload'.
You will get a success message and the curriculum tags you've added will appear on the screen.

Accessing Templates for Importing CBME Data

After the required, standardized curriculum tag sets have been populated, all programs must also complete and upload multiple templates through the CBME tab for a program. Note that program directors and program coordinators will have access to the CBME tab for their affiliated programs. The templates to complete are:

The EPA Template
The Program Specific Key Competency Template (optional)
The Program Specific Enabling Competency Template (optional)
The Milestones Template (or the Enabling Competency Map Template)
The Contextual Variables Template

To access these templates:

Navigate to Admin > Manage Courses.
Search for the required program (you may only have access to one depending on your role in your organization).
Using the gear icon beside the program name, click 'CBME'.
Click 'Import CBME Data' from the tab menu below the Competency-Based Medical Education heading.

With the introduction of EPA Versioning, you will now be prompted to indicate which EPAs you are changing; if you are uploading a program's CBME data for the first time, this does not apply and you can click 'Next' on Steps 1 and 2 to move forward.

After skipping Steps 1 and 2, you'll arrive at a screen showing Steps 1 - 6 for importing CBME data.
On the right you will see a dropdown menu titled 'Download Example CSV Templates' for your use.

Click on the required template and it will download to your computer. Look for it in your downloads file if it doesn't immediately open.

Why are there two enabling competency templates?

There are two similarly named templates because of two options available: the option to import program specific enabling competencies, and the option to map EPAs to enabling competencies instead of milestones. Most programs will map EPAs to milestones and will never use the 'Enabling Competency Map Template.' However, if you have a program piloting CBME before its national launch they may choose to map to enabling competencies if milestones aren't finalized yet and we have left that option available. Use the Enabling Competency Template to import program specific enabling competencies. Use the Enabling Competency Map Template only if a program chooses to map EPAs to enabling competencies instead of milestones.

Import EPAs

The EPA Template needs to be completed by all programs. This template organizes the information about EPAs decided by each national specialty committee and allows the information to be uploaded to Elentra.

Navigate to Admin > Manage Course and select a course. Click CBME and Import CBME Data to access and download the required template. (See for more information on downloading templates if needed.)

Template Columns to Complete

Each program should complete an EPA template with the following information:

Code: This is the EPA code. It should be recorded in uppercase letters and include the learner stage of training and the EPA number. Use the learner stage letters outlined below to ensure that the EPAs correctly map to other curriculum tags. D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice Sample EPA Codes include D1, D2, D3, F1, F2, F3, C1, C2, etc. Note that there is no space between the letter and number in the EPA Code.
Title: This should be the EPA text. This can be copied from the EPAs as decided by each national specialty committee.
Detailed Description: In some cases, these may be the same as the EPA title; in other cases, an EPA has an additional description. Not every program will use the ‘Detailed Description’ column. Note that this information is displayed to users in the EPA Encyclopedia.
Entrustment required (optional): This field is optional but will populate the EPA Encyclopedia which is visible to all users.

Remember to use these codes for the learner stage on all templates:

D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice

(The exception to this is if you have a unique CBME setup, e.g. for a UG program.)

Save as a .csv and upload

After saving your file as a .csv, scroll to Step 1 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

Choose Competency and Curriculum Tag Options

After you upload an EPA Template, you will be prompted to select how you'd like to continue mapping your EPAs. Each program has two decisions to make: 1. Competency Options: Decide whether or not to upload course/program specific key and enabling competencies. If you select to use the standard key and enabling competencies (e.g., CanMEDS key and enabling competencies) the system will copy the items previously uploaded via Manage Curriculum and make them the program's default key and enabling competencies. 2. Curriculum Tag Options: Decide whether to map EPAs to Milestones or Enabling Competencies. Most programs will map their EPAs to Milestones.

The ability to choose whether to use course/program specific competencies vs. the standard list is configurable in the Elentra settings table. This means that with a developer's help it is possible to prevent users from having the Competency Options choice. If you get to an option that ONLY asks about Curriculum Tag Options, consult with your CBME implementation lead as to whether your organization should enable the option to provide program specific KCs and ECs.

Import Key Competencies (Program Specific)

Note that you'll only have this step available if you decided to map program specific key and enabling competencies in import Step 2.

Use the Key Competencies Template if a program wants to import program specific key competencies. This template organizes the information about Key Competencies decided by each national specialty committee and allows the information to be uploaded to the system.

Navigate to Admin>Manage Course and select a course. Click 'CBME' and 'Import CBME Data' to access and download the required template. (See here for more information on downloading templates if needed.)

Template Columns to Complete

Note that the first column, Key Competency Code, is completed for you with alphanumeric codes based on the CanMEDS roles. Adjust this list as needed to match the information from the national specialty committee.

When coding the Key Competencies remember these required codes for the CanMEDS roles: Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME (The exception to this is if you have a unique CBME set up, e.g., different using different roles.)

Complete the second column, Title, by providing the key competency text.

The third column, Description, is optional. This will store information but will not be visible to users.

Save your file as a CSV.

Uploading Your File

After saving your file as a .csv, scroll to step 3 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

Import Enabling Competencies (Program Specific)

Note that you'll only have this step available if you decided to map program specific key and enabling competencies in import Step 2.

Use the Enabling Competencies Template if a program wants to import program specific enabling competencies. This template organizes the information about Enabling Competencies decided by each national specialty committee and allows the information to be uploaded to the system.

Template Columns to Complete

Note that the first column, Enabling Competency Code, is completed for you with alphanumeric codes based on the CanMEDS roles. Adjust this list as needed to match the information from the national specialty committee.

When coding the Enabling Competencies remember these required codes for the CanMEDS roles: Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME

(The exception to this is if you have a unique CBME setup, e.g., different roles.)

Complete the second column, Title, by providing the enabling competency text.

The third column, Description, is optional. This will store information but will not be visible to users.

Save your file as a CSV.

Uploading Your File

After saving your file as a .csv, scroll to step 4 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

Import Milestones

A program needs to complete this template OR the Enabling Competencies Template, not both. Most programs will complete the Milestones Template. Users define which option they will use in Step 2 of the import CBME process.

Navigate to Admin>Manage course and select a course. Click 'CBME' and 'Import CBME Data' to access and download the template. (See here for more information on downloading templates if needed.)

Three Columns to Complete

2) Milestone code: Codes should be recorded in uppercase letters. The format for the milestone code is: Leaner Stage letter, followed by a space, CanMEDS Role letters, Key Competency number, followed by a period, Enabling Competency number, followed by a period, Milestone number.

Note that there should be no space between the CanMEDS Role letter and the Key Competency number.

CanMEDS Role refers to the CanMEDS roles and should be coded as follows:

Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME

(The exception to this is if you have a unique CBME setup, e.g., different roles.)

Creating and confirming your Milestones codes takes patience. You'll likely notice that in some Royal College (RC) documents there is only a two digit milestone code. For the purposes of mapping your curriculum in Entrada, you must have three digit milestone codes. We recommend you add the third digit in the order the milestones appear in the RC documents you're using. Make sure you check for duplication as you go so that unique milestones have their own codes but different codes aren't applied to repeating milestones (within one stage). Using the data organization tools to reorder the milestone code columns and title columns can help you identify unneeded duplication.

You may also notice that the RC allows programs to use milestones coded from one stage in another stage (so you may see an F milestone in a C stage EPA). How programs and organizations handle this is ultimately up to them but we recommend that you align the milestone code with the stage that it is actually being assessed in/mapped to. For example, if it’s a “D” milestone being mapped to an F EPA, rename it as an “F” milestone. This is because you likely have a different expectation of performance on the milestone in a different stage, even if it’s the same task.

3) Title: The text of the milestone should be copied verbatim from the national specialty committee EPAs.

Mapping multiple EPAs to one milestone

The system is designed to allow multiple EPAs to map to one milestone. To do so, simply insert a comma between the EPA code entries in column one. e.g. D1,D2,F1

Save as a .csv and upload

Import Enabling Competencies Mapping Template

A program needs to complete this template OR the Milestones Template, not both. Most programs will map milestones and there is no need to complete this template. Users define which option they will use in in Step 2 of the import CBME process.

Navigate to Admin>Manage course and select a course. Click 'CBME' and 'Import CBME Data' to access and download the template. (See here for more information on downloading templates if needed.)

Columns to Complete

EPA code: Codes should be recorded in uppercase letters. Codes should refer to the stage of training and the EPA number. e.g. D1, D2, D3, F1, C2. Using consistent codes across your program and school is important.The system is setup to accept the following codes for learner stage: D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice
Enabling Competency Code: Note that the template already includes a list of Enabling Competency Codes using the CanMEDS role prefixes. In the first column, simply add the EPAs codes you wish to map to each enabling competency.
If there are unmapped Enabling Competency Codes left over you must delete those rows before you upload your .csv file.

Mapping multiple EPAs to one enabling competency

The system is designed to allow multiple EPAs to map to one milestone. To do so, simply insert a comma between the EPA code entries in the first column.

Save as a .csv and upload

Import Contextual Variable Responses

Contextual Variable Responses are used to describe the context in which a learner completes something. Examples of contextual variables include diagnosis, clinical presentation, clinical setting, case complexity, patient demographics, etc. For each variable, programs must define a list of response options. For example, under clinical presentation, a program might include cough, dyspnea, hemoptysis, etc.

Elentra will autosetup a specific list of contextual variables in the Manage Curriculum interface. Institutions can also add contextual variables (e.g., assessor's role) to this list before uploading contextual variable responses (e.g., nurse, physician, senior resident) via each program's CBME tab.

Do not delete any contextual variables from the existing list within Manage Curriculum or the CBME auto-setup feature will prompt you to re-add them the next time you use CBME. If you wish to add to the contextual variables list you may do so through Manage Curriculum and then import your responses through a program's CBME tab. By default Elentra is set up to provide access to the same Contextual Variables for all programs. You don't necessarily have to provide responses for all CVs in all programs but by default each program will see the same list of CVs.

Adding Additional Contextual Variables

Institutions can add additional contextual variables to make available to their programs through Admin > Manage Curriculum. You'll need to be a medtech:admin or staff:admin user to do this.

Navigate to Admin > Manage Curriculum.
Click on 'Curriculum Tags' from the Manage Curriculum Card on the left sidebar.
Click on the Contextual Variables tag set.
Click 'Add Tag' or 'Import from CSV' to add additional tags (more detail regarding managing curriculum tags is available here).
Any new tags you add to the Contextual Variable tag set (e.g. observer role, consult type, domain) will be visible and useable to all programs within the organization.
When you add new tags to the tag set, you'll be required to provide a code and title. It is recommended that you make the code and title the same, but separate the words in the code with underscores. For example: Title: Observer Role Code: observer_role

Please note that there is by default a 24 character limit for curriculum tag codes and a 240 character limit for curriculum tag titles. If you enter more characters than the limit, the system will automatically cut off your entry at the maximum characters.
After you have added Contextual Variables to the existing tag set, programs will be able to add the new response codes (and responses) to the contextual variable response template and successfully import them.

Importing Contextual Variable Responses

Once you have built out the list of all contextual variables you want included in the system you can import specific contextual variable responses for each program.

Navigate to Admin>Manage course and select a course. Click 'CBME' and 'Import CBME Data' to access and download the template. (See here for more information on downloading templates if needed.)

Template Columns to Complete

Response Code: This column is pre-populated by fields including assessor’s role, basis of assessment, case complexity, case type, etc. You can add to this list if you have built additional contextual variables through Manage Curriculum (see above). Make sure to use the exact response code from the newly created contextual variable tag in the response_code column.
Response: In this column, you can add the response variables required (e.g. Case complexity response variables: low, medium, high). To add more than one response variable per response code, simply insert another line and fill in both the response code and response columns.
If there are items in the response code column that you don’t use, you must delete them before you upload the file.
Description (optional): This field is not seen by users at this point and simply serves as metadata.

Save as a .csv and upload

Reordering Contextual Variable Responses

New in ME 1.14! Users can now reorder contextual variable responses within a program.

Reordering Contextual Variable Responses

Navigate to Admin > Manage Courses/Programs.
Search for the appropriate course as needed.
From the cog menu on the right, select 'CBME'.
Click on 'CV Responses' from the tab menu below the Competency-Based Medical Education heading.
Click on a contextual variable to show its existing responses.
Click on the double arrow in the Order column to drag and drop a response to a new location.
You'll get a green success message that your change has been saved.

Reordering Contextual Variables List

If users want to reorder the overall list of contextual variables they can do so from Admin > Manage Curriculum. To change a tag's display order you need to edit the tag by clicking on the pencil icon beside a tag name. Then you need to adjust the display order for each individual tag.

Defining and Uploading Procedure Attributes

When importing data to create EPA maps, programs are required to upload a contextual variable template to provide contextual variable response options (e.g. case complexity: low, medium, high). In addition to this information, programs will need to add specific details about any procedures included in their contextual variable response options. This additional information, called procedure attributes, plays an important part in building procedure assessment forms. Think of the procedure attributes as assessment criteria (e.g. obtained informed consent, appropriately directed assistants, etc.) for each procedure .

Required Information

For each procedure in your program, you must define headings, rubrics and list items that may be used to assess a learner completing the procedure. Headings appear at the top of a form, rubric becomes the subtitle of a rubric, and list items are the things that will actually be assessed on the rubric.

All this information must be stored and uploaded in a CSV spreadsheet that uses two columns: Type and Title.

Within ‘Type’ there are three things you can include: H to represent heading (e.g. Participate in Surgical Procedures, NG Insertion) R to represent rubric (e.g. Procedure, Pre-Procedure) L to represent list items (e.g. Apply knowledge of anatomy)

The three procedure criteria form a hierarchy. A heading can have multiple rubrics, and a rubric can have multiple list items. Arrange the criteria in your spreadsheet to reflect their nesting hierarchies.

In the 'Title' column enter the information to be uploaded.

When complete, save the file in a CSV format.

Sample Procedure Form On this sample form, the following information was uploaded as procedure attributes: H: Participate in Surgical Procedures R: Procedure L: (ME1.3.2) Apply knowledge... (ME3.4.2) Perform procedural tasks... Use common surgical instruments...

Tip: You can use the same file for multiple procedures if they share the same heading, rubric, and list information.

Uploading Procedure Attributes

Ensure you are in a Program Coordinator or Program Director role.
Navigate to Admin > Manage Courses/Programs.
Scroll down or type in text and click on the relevant program name to select it.
Click on ‘CBME’ on the tab menu.
Click on ‘Contextual Variable Responses’ on the second tab menu.
From the list of contextual variables, click on ‘Procedure’.
Beside each procedure you will see a small black up arrow beside a green disk icon.
Click on the up arrow to upload a .csv file of information for each procedure.
Either choose your file name from a list or drag and drop a file into place.
Use the dropdown menu to select the EPA to apply your procedure attributes to.
Click ‘Save and upload criteria’.

Repeat this process for each procedure and each EPA. Remember, a program can use the same procedure attributes for multiple procedures and EPAs if appropriate.

Managing Existing Procedure Attributes

You can view what attributes were already uploaded to a procedure by clicking on the green check mark in the criteria column. You can review the criteria posted for a specific EPA and opt to replace the criteria for that specific EPA.

Trouble Shooting EPA Content

Uploading Templates

All templates you upload to the system must be in .csv format.

If you are working in Excel or another spreadsheet manager use the “Save as” function to select and save your file as a .csv

To upload your files, either search for or drag and drop the file into place. You'll often see a green checkmark when your file is successfully uploaded.

If you receive an error message make sure that you have:

Deleted any unused lines. This is especially relevant in templates with pre-populated columns including the contextual variables template, and enabling competencies mapping template.
Completed all required columns. If a column is empty for a specific line the file may fail to upload.

Viewing EPA Maps

If you've imported all CBME data for a program but you are not able to see EPA maps in the EPA Encyclopedia or in the Map Curriculum Tags tab of CBME, double check that you've uploaded your files without spaces between the EPA Code letter and number, nor between the CanMEDS Role and Key Competency number. Having spaces in the incorrect places will prevent the system from retrieving the required information to produce maps.

Mistakes When Mapping Relationships

While you can correct minor typos and add additional mapping info. through the user interface you can not delete, or undo mapping between, EPAs and milestones or enabling competencies.

The option to reset all CBME data for a program was removed in a previous Elentra version.

Modifying Uploaded Templates

If you need to add new information to your EPAs, enabling competencies, or milestones templates, simply add new information to the existing template and upload it again in the appropriate spot. The system will identify any NEWLY ADDED EPAs/milestones/enabling competencies and append them to the existing EPA information.

Because of this feature, it can be valuable to keep a copy of the most recent file you’ve uploaded at all times.

If you need to correct a typo or minor error in an EPA, milestone, etc. you can now do so in through the user interface. Navigate to the Import CBME Data screen and follow the instructions on the screen to edit existing EPAs, milestones, etc.

Modifying Contextual Variable Responses

After a program has uploaded an initial set of contextual variable responses, they can be managed through the CBME tab.

Navigate to Admin>Manage course and select a course. Click CBME and then click the Manage Contextual Variable Responses tab below.

If contextual variable response information has been uploaded for the program, you will see a list of contextual response variables.

To modify the contextual variables within a category, click the title of the contextual variable or click 'Show' to the right of the category you want to edit.

To edit an existing response, make the required change and click the green disk icon.

To delete an existing response, click the red minus button beside the response.

To add a new contextual variable response, click the black plus button at the bottom of the category window. This will open a blank response space at the end of the list. Fill in the required content and save.

Troubleshooting

If you get a yellow bar across the screen when you try to modify contextual variables it means none have been uploaded for the program you are working in. Return to the Import CBME tab and complete Step 4.

Manually Mapping EPAs

If you want to manually map a connection between an EPA and a milestone, you can do so through the Map Curriculum Tags tab. This allows you to map additional milestones to EPAs after you have uploaded your templates.

Navigate to admin, manage course and select a course.
Click CBME and then map curriculum tags.
Scroll down to the Map Curriculum Tags area.
Either type in and select the tags required to sort the displayed results OR click through the displayed options to create the mapping structure you require.
- You can display the results for more than one role at a time by clicking on multiple roles. Note that any already mapped milestones will still show up in the milestone list.
Click to highlight the appropriate tags and then click ‘Save Curriculum Map’.

Editing Existing EPAs

To modify existing EPAs and milestones navigate to the CBME tab of a program page. In the appropriate step (e.g. Import EPAs or Import Milestones), look for "Click here if you wish to make changes to existing EPAs" and click on the blue text.

This will open a window where you can edit the text, description, and entrustment of EPAs. Note that this should only be used to make small modifications to this information and should not change the meaning of the text.

If you edit an EPA, it will be updated automatically across the system including on all completed forms. If you edit any milestones, you will have to republish forms on which they are used. Edits made to milestones will not be made on already published forms.

Viewing the EPA Encyclopedia and EPA Maps

After sufficient information has been added to the system, a user can generate a map showing the connections between EPAs, roles, key competencies, enabling competencies, and milestones. This information can be access through the EPA Encyclopedia or a program's CBME page.

EPA Encyclopedia

From the dashboard, access the EPA Encyclopedia from the left sidebar on the Helpful Links card.

There are several filter options when viewing the EPA Encyclopedia.

Search for a specific learner. After you select a learner the page will refresh to show that learner's program(s).
Switch between multiple programs as needed.
Search for a specific EP by typing in an EPA code or key word. The tool will search the EPA title, detailed description and entrustment.
Switch between CBME Curriculum versions as needed.

After clicking on an EPA, click on Detailed Description, Entrustment or Program Map to view details about the EPA.

Viewing Maps from the CBME page

This option is only available to users with Admin > Manage Courses access (e.g. program direcotrs, program coordinators).

Navigate to Admin > Manage course/program and select a course.
Click CBME and the Map Curriculum tab.
Scroll down to click on an EPA or type the beginning of the EPA code and title into the dialogue box and click on the relevant EPA when it appears below.

Hover the mouse over a point on the map to view the complete text for the mapped item.

Setting Priority EPAs and Likelihood

Elentra allows you to map EPAs to specific rotations and indicate for each rotation whether an EPA is a priority and how likely a learner is to encounter the EPA. For this information to be useful for learners, you must be using the Elentra Rotation Scheduler.

From the CBME dashboard, EPAs can display their priority and the likelihood of being encountered within a learner's rotation. Additionally, EPAs relevant to the learner's current rotation are outlined.

When learners and faculty trigger assessments they will also have the option to apply pre-set filters (e.g. priority EPA, current stage EPAs). All of these tools can help learners quickly identify EPAs to focus on in their clinical work.

Adding Priority and Likelihood Levels

Navigate to Admin > Clinical Experience.
Click on the Rotation Schedule tab.
Create a draft schedule or open a published schedule. Note that if you are creating a new rotation schedule, you will also need to add rotations to the schedule.
Once rotations exist within a schedule a small Objectives badge will appear beside the rotation name.
Click on the CBME Objectives badge to open a list of EPAs tagged to the rotation.

Click the plus sign beside any EPA you'd like to label with priority or likelihood.
Indicate the likelihood by clicking on the appropriate button in the Likelihood.
Set an EPA as a priority by clicking the exclamation mark in the priority column.
Repeat as necessary.

Return to the rotations list using the Back to Rotations button in the top right.

EPA Versions - Overview

New in ME 1.14! Elentra now supports the ability to have versions of EPAs and milestones for a program.

Which learners will get the new version of the curriculum?

The new curriculum version will be applied to each learner at the stage level. This means that if a learner is currently in Foundations, they will retain the previous version of Transition to Discipline and Foundations but will be given the new version of Core and Transition to Practice. Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. Remember: a green checkmark on the stage indicates that the learner has completed that stage. A learner is currently in the stage that comes after the last green stage checkmark.

As soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.

What is considered a new EPA?

Think in terms of the EPA or Milestone code. A new version of an EPA is when:

The EPA title/description has changed sufficiently to change the context of the EPA, AND/OR
You are changing, adding, or retiring any milestones within the EPA, AND/OR
You are changing the EPA code association (e.g., you have retired an EPA in the middle of your numbering, such as D2, and now need to shift D3, D4, D5, into D2, D3, D4).

A new version of a Milestone is when:

The Milestone title has changed sufficiently to change the context of the Milestone.

What do the objective statuses mean?

Replaced

Select this option if you wish to replace the EPA entirely, including all milestones. This is the easiest option to work through the wizard. This option will require you to have a complete set of CSV files, rather than just the changes. Use this option if you have many changes to implement across EPAs and milestones.

Not changing

The objectives will remain unchanged. This exact version of the objective will be carried-forward into the new curriculum version. Note that selecting this option will stop any objectives with the same code from being uploaded. Any EPAs marked as not changing will have all existing tools carried over.

Changing

The objective will be changing. A new version of this EPA will be created, which may include objectives within it that you mark as changing.

For a milestone (or competency), if you select “changing” on one EPA, the system will update all other instances of that milestone to “changing”. This will ensure that you only have one version of that milestone.

Retire

The EPA (remember, think in terms of EPA Code, e.g. "C1") is being retired from the next version. This means that there will be no new version of that EPA Code in the next curriculum version. If you include an EPA with the retired code in the spreadsheet, it will not get uploaded. You are unlikely to use this status for EPAs unless there is an EPA at the end of your numbering that is being removed. For example, you have D1, D2, D3, D4 currently. You are retiring D4 from the next version and no new D4 will be uploaded.

For milestones, a “retired” status will remove that milestone from that EPA in the new version. This means that there will be no new version of that milestone for that EPA. It will not retire it from other EPAs unless you explicitly tell it to.

What do I need to do to prepare?

Please ensure that you have documented any changes that will need to be made, in addition to creating (or modifying) new CSV templates for upload. From the user interface, you will be able to indicate:

Whether an EPA is being replaced, changing, not changing, or being retired
Within each EPA that has been marked as changing, indicate if any KC, EC, or Milestones will be changing, not changing, or retired
If you are adding milestones to an EPA, mark it as changing so that the uploader will add the new milestones
Entirely new EPAs will be detected by the uploader without you needing to indicate anything. For example, if you currently have D1, D2, and D3, but will be adding a D4 and D5, the system will detect these in your spreadsheet and add them to the new curriculum.
The new curriculum version will be applied to each learner at the stage level. This means that if a learner is currently in Foundations, they will retain the previous version of Transition to Discipline and Foundations but will be given the new version of Core and Transition to Practice. Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. Remember: a green checkmark on the stage indicates that the learner has completed that stage.

Which CSV files do I need to have ready?

EPA Template
Milestone Template
If applicable, any new program-specific Key and Enabling Competencies

SPREADSHEET TIP:

To ensure that you do not have any duplicated milestone text or duplicated milestone codes, use the "Duplicate Values" feature in Microsoft Excel. This will highlight any duplicated cells for you to ensure that you have only entered each milestone once. Caveat: this method is not perfect and will not catch the same milestone text if one instance of it has an additional space or missing period.

How will my assessors know which EPA version to pick for each learner?

Great news: When triggering an assessment, only the EPAs and associated forms relevant to that learner will be displayed. You will not have to choose between multiple versions for a given learner.

Do I need to build new assessment tools?

You only need to build new assessment tools for new EPAs or EPAs that have changed. Assessment tools will be carried over for all EPAs that were marked as "not changing." As noted above, as soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.

Using the Versioning Wizard

Navigate to the CBME Sub-Module

Log in to Elentra.
Click "Admin" at top right
Click "Manage Programs"
Select your program.
Click "CBME"
Click "Import CBME Data".

Add New Version

Click "Actions"
Click "Add New Version"

Versioning Wizard Step 1

In this step, simply indicate which EPAs will be:

Replaced
Changing
Not Changing
Retired

Note: Each program can only have one version "in progress" at a time. You can continue editing that new version, or reset the progress.

Replaced

Select this option if you wish to replace the EPA entirely, including all milestones. This option will require you to have a complete set of CSV files for the objectives that are being replaced (all EPAs, all Milestones), rather than just the changes. Use this option if you have many changes to implement across EPAs and milestones.

Retired

Select this option to retire an EPA (remember, think in terms of EPA Code, e.g. "C11") from the next version. This means that there will be no new version of that EPA Code (C11) in the next curriculum version. If you include an EPA with the retired code in the spreadsheet, it will not get uploaded. You are unlikely to use this status for EPAs unless there is an EPA at the end of your numbering that is being removed. For example, you have D1, D2, D3, D4 currently. You are retiring D4 from the next version and no new D4 will be uploaded.

Changing

The objective will be changing. This will allow you to upload new EPA and Milestone text. For a milestone (or competency), if you select “changing” for that milestone under one EPA, the system will update all other instances of that milestone to “changing”. This will ensure that you only have one version of that milestone.

Not changing

Versioning Wizard Step 2

In this step, indicate which KCs, ECs, and Milestones will be: Changing, Not changing, or Retired.

If you marked all of your EPAs as "Replaced," you can click next without needing to go through each EPA.

PLEASE NOTE: If there are any milestones in a Changing EPA that you want removed, you MUST mark them as RETIRED. This is with respect to the Milestone code - if "C PR4.3.2" will no longer be in this EPA, mark it as retired.

If you would like to add a new Milestone to an EPA only, then just set the EPA to changing and nothing else.

Note that "retiring" a milestone will only retire it from that EPA and no others unless otherwise indicated.

Import CBME Data

Upload CSV file with new and/or changing EPAs

You will notice that the yellow box at the top will list all objectives that have been set to changing. You can still upload new EPAs and Milestones that were not already in the system.

Choose the file and click "Save and upload EPAs"

IF APPLICABLE: Upload Key and Enabling Competencies

Moving forward, we are recommending that programs upload the specialty-specific EC and KC files. This is especially important for programs that have added or removed KCs or ECs (e.g., Otolaryngology).

Upload CSV file with new and/or changing Milestones

Choose the file and click "Save and upload Milestones".

Some programs at Queen's only mapped to the Enabling competency level. If you will continue mapping to enabling competencies, and are not converting to milestones, skip the milestone upload and use Step 6 to upload your EC map. If you are converting to milestones, use the milestone upload.

When the upload is complete, click "Next".

Versioning Wizard Step 3

Confirm the Version

This overview lists all of the EPAs and their version status. When you are ready to publish the new version of the curriculum, click the blue "Publish" button.

REMINDER: Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. The green checkmark on the stage indicates that the learner has completed that stage. A learner is currently in the stage that comes after the last green stage checkmark. Failing to ensure that learners are in the correct stages will result in those learners receiving (or not receiving) new stages when they shouldn't have. If you do this by accident, there is a developer tool to reset the learners stages - please contact the support team if you need help with this.

As soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.

The new curriculum version will be available the next day. For most installations, new versions are published nightly.

Build and Publish Assessment Tools for the New Version

Now that you have finished creating your new curriculum version, please ensure that you create new assessment tools for any changed/replaced/new EPAs. After the published version is live, learners will not be able to be assessed on any new/changed EPAs until the new tools have been built. Remember that in most cases this will have minimal impact, since the learners only get the new version for the stages beyond the one they are currently in.

Please do not delete forms or form templates from previous versions unless you are certain that there are no longer any learners using that version.

Form Building with Versioning

Building Form Templates for New EPA Versions

Please note that versioning support for Procedure Form Templates is awaiting review and is not currently included in Elentra.

Navigate to Admin > Assessment and Evaluation.
Click ‘Form Templates’ on the tab menu.
Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear.
Type in a form name and select the form type from the dropdown menu. Select ‘Supervisor Form’. If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.

Now that you have two (or more) curriculum versions in the system, the Form Template Builder will default to loading the most recent version. In the "EPA Version" tab, simply select the appropriate version. Click Save.

If you want to build new forms for learners using Version 1, change the EPA Version to Version 1, click Save, and it will load the appropriate EPAs.

EPA Selection

Select which EPAs can be assessed using forms generated from this template.

To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

Click the grey badge beside an EPA to select or remove specific milestones for forms built from this template.

Click Save.

Contextual Variables

If you want all EPAs to have the contextual variables, leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable.

You may only select between 1 and 6 contextual variables per EPA per supervisor form.

Click Save.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

Indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The default response feature allows you to pre-populate a form with the selected response.

Disabled - Comments are disabled at the milestone level.

Optional - An optional comment box will appear for each milestone. Comments are not required for a form to be submitted.

Mandatory - A mandatory comment box will appear for each milestone. Comments are required for all milestones before a form can be submitted.

Prompted - A mandatory comment box will appear only for the prompted responses indicated. This is a popular option for responses at the lower end of a scale.

Global Rating Scale Selector

From the first dropdown menu, select a Global Rating Scale. This will populate the Item Text and the Responses sections.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

Publish Form Template

Click 'Publish' to make your template available for use. The forms will be available within the hour.

Once a form template has been published, you can rearrange the template components for each form; however, you cannot makes changes to the scales or contextual variables. To make these changes, copy the form template and create a new version.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed.

Rubrics and PPAs

Within a given form, you can only tag curricular objectives (e.g., EPAs or milestones) from the same curriculum version. To ensure that you do not accidentally add an EPA from a different version, you must create the form first and then "Create & Attach" new items to the form.

Build a PPA or Rubric/Flex Form

Click Admin > Assessment & Evaluation.
Click on Forms from the subtab menu
Click Add Form.
Provide a form name and select PPA Form or Rubric/Flex Form from the Form Type dropdown menu; then click Add Form.

Now that you have two (or more) curriculum versions in the system, the Form Editor will default to loading the most recent version. Under "EPA Version", simply select the appropriate version. Click Save.

If you want to build new forms for learners using Version 1, simply change the EPA Version to Version 1 and it will load the appropriate EPAs.

Note: In order to use the "Programs" filter in the form bank, you need to add Program-level permissions to each form.

Click "Individual", change to "Program"
Begin typing in your program name in the "Permissions" box.
Click on your program to add it.

If applicable, select the relevant contextual variables. Click the grey badge beside a variable to select or remove specific responses.

Feedback and Concerns items will be added when the form is published.

Add Items

Click "Add Item(s)" to add an item, or click the down arrow for more options. "Add Free Text" will add an instruction box. Do not use the "Add Curriculum Tag Set" at this time.

Create & Attach a New Item

Click "Create & Attach a New Item" to add an item.

Create New Item

Click "Create & Attach a New Item" to add an item. Select the Item Type and add any item responses, if applicable.

Tag Curriculum Objectives

Because you are using a form that is mapped to "Version 2", the curriculum tag sets will be locked to "Version 2". This will ensure that you do not accidentally tag an EPA from a different version.

Add Additional Items & Publish

Continue adding items as desired. When you have finished creating and attaching new items, click "Publish" to publish your new form.

Assessment Plans

The assessment plan builder is undergoing significant rebuilding during the winter of 2019. We do not recommend using the assessment plan builder at this time and if your institution is operating Elentra ME 1.14 the assessment plan builder will appear broken. You will likely see something like below:

Planned enhancements for a future release include:

1) Adding version scoping to the AP builder

2) Adding ability to group contextual variable responses for use in the AP builder (will not be reflected in forms as they will still honour the CV Response ordering)

3) Adding ability to indicate if the AP should look at specific CV responses (same as original version of AP builder whereby you indicate how many of each response is required) , a group of CV responses (i.e. 3 different responses from a given CV response group), or a general number of CV responses (i.e. 3 different responses).

_______________________________________________________________________________________________________

Introduced in Elentra ME 1.12, the assessment plan builder allows you to create assessment plans for a program. Currently these simply house information about the requirements for each program, but eventually reporting tools will be built to reflect the assessment plan information.

The assessment plan allows you to set minimum completion requirements for the different assessment tools built for a program. These minimums can apply to the number of assessments, the number of assessors, and specific contextual variable responses.

Please note that currently the assessment plan does not apply to procedure forms.

You need to have EPAs mapped and forms built before you can create an assessment plan.

Managing Faculty and Residents

Recommendations for CBME User Roles

Ideally your users will be loaded into Elentra from a central, authoritative source like a student information system. Manually adding users through Admin>Manage Users is a possibility and more detailed instructions for using this tool are available on the Elentra help website.

If you are manually setting up CBME users, please note the following:

Resident learners should be assigned group: student, role: predicted graduating year.
Program Administrators should be assigned group: staff, role: Pcoordinator. Further, they must be assigned to a specific program in order to access the relevant CBME and Assessment and Evaluation features. To give specific staff more ability to act within Elentra set them up as Staff>Admin. Note that Staff>Admin have access to almost all features of Elentra and can access System Settings.
Faculty should be assigned group: faculty, role: faculty, lecturer, or director depending on their role. Someone with faculty director permissions will still have to be added to a specific program in order to access the relevant CBME and A and E features. Faculty directors will have access to all faculty evaluations within their department (except their own).
Any faculty, regardless of their role, can be added to Competence Committees through a course/program setup page or as Academic Advisors through the program Groups tab. (Additional instructions for each are in additional lessons.)

Learner Levels and Stages

Please note that there is currently no user interface to manage learner levels (e.g., PGY1, PGY2). The learner level will display on the learner card when you view a list of My Learners. In the example below the learner is a PGY4.

The D in front of the learner level indicated the learner CBME stage. In this case the learner is in stage D or Transition to Discipline. There is a user interface for promoting a learner through different stages. Competency Committee members can mark a stage as complete from the learner's CBME dashboard.

For learner level and stage to show on learner cards, developers or technical staff will need to configure tables in the database to display this information.

Note that when people trigger assessments and hover over a name, they will also see a summary of information about the user. In this case they will see the learner group and role displayed. In the example below group is Student and the role is 2015.

Cohorts/Course Lists

Residents should be stored in course lists in order to easily add them as the audience of a course. Course lists are managed through Admin>Manage Cohorts. We recommend that you create a course list for each academic year and populate the course list with all learners, CBME and non-CBME, who will participate in that program for the specified time.

If you are syncing your users with another source, your course lists may be automatically populated. It will depend on the script written by developers to connect Elentra to your other source of user information.

When you create courses, you can set the Course List as the audience for the course. It is important that learners be enrolled in a current curriculum period for them to be able to trigger forms and have forms triggered on and to them.

Setting up Academic Advisor Groups

Note that before using the groups function a course needs to have enrolled learners with a designated curriculum period.

You need to be logged in as a program coordinator or program director to create or modify academic advisor groups.

Navigate to Admin>Manage course and select a course. Under the program title, click on 'Groups.' After clicking on Groups, any existing groups will be listed below. Ensure that you are working in the appropriate Period by reviewing the dropdown menu in the top right. Click the green button on the right to add a new group. Select a prefix for the group name and type it into the Group Name Prefix area. This prefix will be applied to all group names.

Select your group type: you can either define a number of empty groups and manually populate them with learners OR have the system automatically populate groups.

To create empty groups, enter the Group Name Prefix, and a designated number of groups. Click the blue add button on the bottom right.

Add academic advisors to a group

Within the current setup, the term tutor refers to the academic advisor role. To add tutors/academic advisors to the group, click on a group name, find the Group Details section, and use the tutor search bar to pull up names.

Click on the required name and repeat as necessary. When names have been added they will appear in a list below the tutor search bar.

Adding tutors/academic advisors to a group allows those faculty to view the assessment data for any learners in the group.

Add learners to a group

To add learner members to a group, click on the group name, scroll down to Add Members and use the dropdown menu to tick off members. Note that you can add multiple learners at once by ticking off multiple checkboxes beside names.

After you have added the required names, click the blue Proceed button on the bottom right. This will save your work.

Renaming a group

From the Groups tab, click on the group you want to rename. In the Group Details section look for the Group Name box and manually change the name of the group as needed.

Scroll to the bottom of the page and click the blue proceed button on the right. This will save your work.

Deleting group members

To delete tutors, go to the Group Details section, and click on the red minus sign beside a tutor name.

To delete learners from a group, go to the View Members section, and tick off the box beside any learner names you want to delete, and then click the red Delete Members button on the right side.

Managing Competency Committees

You need to be logged in as a program coordinator or program director to create or modify competence committees.

Navigate to Admin>Manage course and select a course.

Under the program title, click on Setup.

Scroll down to the Course Contacts section and find the 'Competency Committee Members' section.

Begin to type a name to retrieve a list of potential committee members.

Click on the name you want to add and repeat as necessary.

When a name has been added it will appear below the search box.

After you have added the required names, scroll down to the bottom of the page and click ‘Save’.

Note that all Competency Committee Members for a program will be able to access all resident profiles in that program and will have the ability to promote them.

Promoting Learners Through Stages

Members of Competency Committees can promote learners through stages from the learner CBME dashboard. For more information please see the Reviewing Learner Progress>Promoting Learners Through Stages lesson.

Forms/Tools: Introduction

Building assessment forms for learners and faculty to access is an important step in using the CBME tools.

Two options for building forms exist:

Building forms with a template (e.g. supervisor, procedure, and field note forms)
Building forms without a template (e.g. generic, periodic performance assessment (PPA) or rubric form)

Using a form template allows Elentra to quickly generate forms pulling in the relevant milestones, contextual variables and rating scales defined by administrative staff. After form templates are published, they can be triggered by faculty and learners at any point. Form templates have defined items included and administrators can not add additional items to a form template. Examples of form templates are the supervisor form, procedure form, and field note form.

You can also build forms without a template which gives you increased flexibility to add items of your own design. Examples include the generic form, rubric form, and periodic performance assessment form. The rubric and PPA have some required elements but you can also add additional items to these forms. As long as one item added to a rubric or PPA form is linked to an EPA the form will be triggerable by faculty and learners.

If a form is not something that a faculty or learner will trigger themselves, program administrators can create forms and send them out to be completed via distributions. A distribution defines who or what is being assessed/evaluated, who is competing the form, and when it is to be completed. Building items and forms to use with distributions is not a requirement if you only want to use triggerable forms. For additional information on building and distributing forms that are not triggerable, please see the Assessment and Evaluation help resources .

For Staff and Faculty Groups to have access to the CBME features in the Distribution wizard, a developer will need to add an additional resource_type called CBME into the elentra_auth.acl_permissions table to include the organisation that has enabled CBME (if your instance of Elentra has more than one organisation) and the two groups of staff and faculty with an app_id of 1 and only a read permission.

An example of this entry for an instance where CBME is enabled in organisation_id 1 and 3:

Creating and Managing Forms

Form Basics

Form templates and forms behave fairly similarly in terms of how you create and manage them. Specific details for each form type are included in other lessons but here is some general information.

Form Permissions: These permissions dictate who has access to and can edit a form or form template while it is still a draft. To quickly make a form available to multiple users consider adding a program permission to the form (that will give anyone affiliated with that program (e.g. PA, PD) and with access to the administration of forms access to that form).
Form Instructions: To include instructions on a form template, look in the Form Template Information section and click the box beside 'Include Instructions.' This will open a rich text editor where you can type. What you type here will be included on all forms generated from this template. You could include specifics about the use of a form, the number of assessments a learner must complete, or even the key features of an EPA. Remember that each form produced will have the same instructions so if you include key features make sure they only relate to the applicable EPAs for the template.

Publish a Form: To make a form template generate forms to be available for learners or faculty to trigger you must publish the form template. Most installations of Elentra run a behind-the-scenes action to publish form templates every hour or so.
Editing a Form: Once a form template is published you cannot change the content of the resulting forms (you can change permissions or rearranged item order on published forms). If there is an error on a form you will need to copy the form, correct the error, and publish your new form. You can delete old forms but note that any already completed assessments will remain on the learner dashboard.

Rating Scales

To use supervisor and procedure forms and certain item types, an organization needs to have rating scales in place. Elentra includes several default rating scales but organizations can also add their own.

There are three types of rating scales: default, Global Assessment and MS/EC which stands for Milestone Scale /Enabling Competency. There is no user interface to modify the scale types and both Global Assessment and MS/EC are used on form templates including supervisor and procedure form templates.

There is a user interface to manage rating scales but only System Administrators and users with Medtech>Admin group and role can modify the available scales. Note that scales are applied to an entire organization and can be accessed by multiple programs.Managing rating scales is part of the Assessment and Evaluation module; please see additional help resources here.

Note that as of ME 1.14, when a new rating scale is added by an organization there is some developer work that must be completed to make the rating scale available on form templates. If you are a developer, please see the resource below.

Auto-populated Item Text in Global Rating Scale Selector

Elentra is designed to automatically populate the item text when you select a global rating scale. Using different scales will result in different questions populating the form. See examples below.

Supervisor Form Template

A supervisor form is used to give a learner feedback on a specific EPA and can be triggered by a learner or supervisor. Once an EPA is selected, the form displays the relevant milestones to be assessed. A supervisor can indicate a learner’s progress for each milestone that was observed and can provide a global entrustment rating. Comments can be made optional, prompted or mandatory in each section of the form.

When you create a supervisor form template and publish it, the system automatically looks at the EPAs, milestones, and contextual variables selected and generates the appropriate number of forms. If you kept 3 EPAs on the supervisor form template, the system will generate 3 unique forms (one per EPA) that are available to be triggered by a user.

Accessing Form Templates

You need to be logged in as a Program Coordinator, Program Director or staff:admin role to access Admin > Assessment & Evaluation.
Navigate to Admin > Assessment & Evaluation.
Click ‘Form Templates’ on the tab menu.
Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear.
Type in a form name and select the appropriate form type from the dropdown menu. Select ‘Supervisor Form’.
If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.
Click 'Add Form'.

Form Template Information

Add additional form template information as required:

Template Title: Edit the form template title/name if needed. Description: The form description can be used to store information for administrative purposes, but it is not seen by users completing the form. Form Type: The form type was set when you created the form and cannot be changed here. Course: Program coordinators and faculty directors may not have access to multiple courses, while staff:admin users are likely to. If you have access to multiple courses, make sure you've selected the correct course to affiliate the form with. EPA Version: With the introduction of EPA versioning in Elentra, you can now affiliate a form with a specific EPA version if the selected course has multiple curriculum versions. The Version Details link is not currently hooked up to anything. Permissions: Anyone added under permissions will have access to edit and copy the form. You may wish to include a program in the permissions field so that you can filter by this form type later on. To add individual, program, or organisation permissions, click the dropdown selector to select a category, and then begin to type in the appropriate name, clicking on it to add to the list. You can add multiple individuals, programs, and organisations to the permissions list as needed. Include Instructions: Add additional text at the beginning of the form by clicking the small tick box beside ‘Include Instructions.’ This will open a rich text editor where you can enter text, images, hyperlinks, etc. This information will display to users when they complete forms published from this blueprint.

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms generated from this template.

All EPAs assigned to a course are included on the template by default. To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

Clicking the grey badge beside an EPA will display a list of all the milestones mapped to that EPA and by default, all are checked off. To remove specific milestones from this template, deselect them to remove the blue checkmark and then click the blue ‘Save and Close’ button. Deleting unnecessary milestones is one way to reduce the length of the form and reduce the time required to complete it.

Modify the milestones for each EPA as needed. The system does not enforce a maximum number of selected milestones but the Royal College recommends 8 milestones per EPA.

After you have modified the EPAs and milestones as required, click the small save button in the top right section of the EPA Selection box.

Contextual Variables

If you want all EPAs to have the same available contextual variables leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes. You may only select between 1 and 6 contextual variables per EPA per supervisor form.

By default, all of the options within a contextual variable are included on any forms made from the template. You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable. This will open a list of contextual variable responses that by default are all checked off. To remove specific responses from this template, deselect them to remove the blue checkmark. For convenience, you can also use ‘Check All’ and ‘Uncheck All’. When you have made the required changes, click the blue ‘Save and Close’ button.

When you modify which contextual variable response options will be available on a template, the number in the grey badge will show how many responses have been included out of the total possible responses.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Note that if you all contextual variables will display on this list, even if a program doesn't have contextual variable responses set for that variable. If you attempt to select a contextual variable for which there are no responses set, you will get a red error message that reads "No objectives found to display." Click the X on the red message to remove it and then select a different contextual variable to use.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

If you don't have scales configured you can access them through the Assessment and Evaluation tab.

The default response feature allows you to prepopulate a form with the selected response.This can be useful if there will be a significant number of milestones that will be included on a form.

The Responses fields will be automatically populated depending on the scale selected in the first dropdown menu.

After you have modified the Enabling Competency/Milestone Scale Selector as required, click the small save button in the top right section of the Enabling Competency/Milestone Scale Selector box.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections.FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed except by a developer.

Publish

Click 'Publish' to make forms generated by this template available for use. Remember that the number of forms that will be created from a template depends on the number of EPAs assigned to the template. Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited.

Please note that a behind the scenes task (a cron job) needs to run before your forms will be published. At some schools this may take up to an hour so expect a slight delay between when you publish a form and when it is available to be triggered by users.

Procedure Form Template

A procedure form is an assessment tool that can be used to provide feedback on a learner’s completion of a specific procedural skill. Once a procedure is selected, specific criteria will be displayed. A procedure form can be initiated by a learner or faculty.

When you create a procedure template and publish it, the system looks at the number of EPAs and procedure contextual variable responses selected and generates the appropriate number of forms. If you keep 3 EPAs and indicate 10 procedures on the form template, the system will publish 30 forms that are available to be triggered by a user (one form per EPA per procedure).

To effectively build procedure form templates in a program, the program must have contextual variables responses defined. In addition to uploading contextual variable response options, every procedure needs specific information added. Contextual variable responses and procedural attributes (which function as assessment criteria) can be managed from Admin>Manage Program>select program name>CBME>Manage Contextual Variables.

Accessing Form Templates

Ensure you are logged in as a staff:admin user, or as a Program Coordinator or Program Director affiliated with a program.
Navigate to Admin > Assessment & Evaluation.
Click ‘Form Templates’ on the tab menu.
Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear.
Type in a form template name and select the form type from the dropdown menu. Select ‘Procedure Form.’
If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.
Click 'Add Form'.
You will be taken to the procedure form template build page.

Building a Procedure Form Template

Provide additional information as needed in the Form Template Information section:

Template Title: This is the title of the form and will be seen by users. Description: The form description can be used to store information for administrative purposes, but is not seen by users completing the form. Form Type: This was set in the previous step and cannot be edited here. Course: Program coordinators and faculty directors may not have access to multiple courses, while staff:admin users are likely to. If you have access to multiple courses, make sure you've selected the correct course to affiliate the form with. EPA Version: With the introduction of EPA versioning in Elentra, you can now affiliate a form with a specific EPA version if the selected course has multiple curriculum versions. The Version Details link is not currently hooked up to anything. Permissions: Anyone added under permissions will have access to edit the form before it is in use and use the form if they are setting up a distribution. You may wish to include a program in the permissions field so that you can filter by this form type later on. To add individual, program, or organisation permissions, click the dropdown selector to select a category, and then begin to type in the appropriate name, clicking on it to add to the list. You can add multiple individuals, programs, and organisations to the permissions list as needed.

Include Instructions: Add additional text at the beginning of the form by clicking the small tick box beside ‘Include Instructions.’ This will open a rich text editor where you can enter text, images, hyperlinks, etc. This information will display to users when they complete forms published from this blueprint.

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms made from this template.

Note: You do not specify milestones for use on a Procedure Form.

Contextual Variables

By default, ‘Procedure’ will be selected as a contextual variable.This will require some additional information to be added to the system if the program you are working in hasn’t input procedure response options. (See above.)

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes.

In addition to ‘Procedure’, you must select at least one other contextual variable and you may select up to a total of 6 for each EPA.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

If you don't have scales configured you can access them through the Assessment and Evaluation tab.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted.By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The default response feature allows you to prepopulate a form with the selected response.This can be useful if there will be a significant number of milestones that will be included on a form.

The Responses fields will be automatically populated depending on the scale selected in the first dropdown menu.

After you have modified the Enabling Competency/Milestone Scale Selector as required, click the small save button in the top right section of the Enabling Competency/Milestone Scale Selector box.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections. FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted.By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

Publish

When the form is complete, a green bar will tell you the form can be published.

Click 'Publish' to make your template available for use.

Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited. The number of forms that can be created from a template depends on the number of EPAs assigned to the template.

A Note on Logging Procedures and Encounters Without Assessing Them

In some programs residents may be required to log multiple procedures or encounters and only have a subset of those logged entries be assessed. Elentra does support a logbook outside the CBME module and some programs have opted to have residents use both tools to capture the full picture of residents' progress. For more detail on Elentra's logbook, please see here.

Field Note Form Template

A field note form template is used to give a learners narrative feedback about their performance.

When you create a field note form template and publish it, the system automatically looks at the EPAs and contextual variables selected and generates the appropriate number of forms.

Accessing Form Templates

Ensure you are logged in as a staff:admin user, or as a Program Coordinator or Program Director affiliated with a program.
Navigate to Admin > Assessment & Evaluation.
Click ‘Form Templates’ on the tab menu.
Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear.
Type in a form template name and select the form type from the dropdown menu. Select ‘Field Note Form.’
If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.
Click 'Add Form'.
You will be taken to the field note form template build page.

Building a Field Note Form Template

Form Template Information Section

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms generated from this template.

After you have modified the EPAs as required, click the small save button in the top right section of the EPA Selection box.

Contextual Variables

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Free Text Feedback to Resident

All field note form templates include a Continue and Consider section in which faculty can record comments to provide feedback to learners. These sections cannot be edited in the Field Note Form Template.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections. FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

Publish

Click 'Publish' to make your template available for use.

Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited. The number of forms that will be created from a template depends on the number of EPAs assigned to the template.

Rubric Forms

Rubrics are assessment tools that describe levels of performance in terms of increasing complexity with behaviourally anchored scales. In effect, performance standards are embedded in the assessment form to support assessors in interpreting their observations of learner performance.

If you create a rubric form and at least one item on the form is linked to an EPA the form will be triggerable by faculty and learners once published. Results of a completed rubric form are included on a learner's CBME dashboard information.

Creating a Rubric Form

You need to be a staff:admin, staff:prcoordinator, or faculty:director to access Admin > Assessment & Evaluation to create a form.
From the Admin menu, select Assessment and Evaluation.
Click on Forms from the subtab menu and then click Add Form on the far right.
Provide a form name and select Rubric Form from the Form Type dropdown menu; then click Add Form.
Provide additional form information as needed: Form Description: The form description can be used to store information for administrative purposes, but it is not seen by users completing the form. Course: Program coordinators and faculty directors may not have access to multiple courses, while staff:admin users are likely to. If you have access to multiple courses, make sure you've selected the correct course to affiliate the form with. EPA Version: With the introduction of EPA versioning in Elentra, you can now affiliate a form with a specific EPA version if the selected course has multiple curriculum versions. The Version Details link is not currently hooked up to anything. Permissions: Anyone added under permissions will have access to edit the form before it is in use and use the form if they are setting up a distribution. You may wish to include a program in the permissions field so that you can filter by this form type later on. To add individual, program, or organisation permissions, click the dropdown selector to select a category, and then begin to type in the appropriate name, clicking on it to add to the list. You can add multiple individuals, programs, and organisations to the permissions list as needed.
Select the relevant contextual variables for this form by clicking on the checkbox. Adjust which contextual variable responses should be included by clicking on the gray badge. This allows you to deselect unneeded contextual variable responses which can make the form faster to complete for assessors.
If you want to include an Entrustment Rating on the form, click the checkbox. Select an entrustment rating scaled from the dropdown menu. Note that the responses will be configured based on the scale you select. It is also possible that the Item Text will be autopopulated based on the scale you select. (More information about how to configure the item text is available .)
For the optional Entrustment Rating, set requirements for comments noting that if you select Prompted comments you should also check off which responses are prompted in the Prompt column. If you use this option and any person completing the form selects one of the checked responses, s/he will be required to enter a comment. Additionally, if the form is part of a distribution you'll be able to define how prompted responses should be addressed (e.g. send email to program coordinator whenever anyone chooses one of those response options).
Note that the default Feedback and Concerns items will be added when the form is published.
Add form items by clicking Add Items. You can either add existing items or create new items and add them to the form. Remember that every item created can be mapped to an EPA and milestones. The mapped EPA information about each item will be used to provide access to this form when a learner or faculty member triggers an assessment.
If you need to add a Grouped Item or Free Text to your form, click the down arrow beside the Add Item(s) button to access additional options. If you add free text, remember to click Save in the top right of the free text entry area. Any free text entered will display to people using the form.
After you have added items to your form you may download a PDF, and preview or copy the form as needed.
Save your form to return to it later, or if the form is complete, click Publish. You will see a blue message confirming that a form is published. Unlike form templates which require a behind the scenes task to be published, a rubric form will be available immediately.

Rubric forms can also be scheduled for distribution through the Assessment and Evaluation module.

Note that when a rubric form is completed its results will be included in the learner dashboard.

Periodic Performance Assessment (PPA) Forms

A Periodic Performance Assessment (PPA) Form is designed to capture longitudinal, holistic performance trends.

At least one item on a PPA form must be linked to an EPA for the form to be triggerable by faculty and learners.

Creating a PPA Form

You must be in a staff:admin role, or a Program Coordinator or Program Director to access Admin > Assessment & Evaluation.
Navigate to Admin > Assessment & Evaluation.
Click on 'Forms' from the subtab menu and click 'Add Form'.
Provide a form name and select PPA Form from the Form Type dropdown selector; then click 'Add Form'.
You will be taken to the Editing Form page.

Building a PPA Form

Form Information Section

Contextual Variables

Select the relevant contextual variables for this form. Adjust which contextual variable responses should be included by clicking on the gray badge. This allows you to deselect unneeded contextual variable responses which can make the form faster to complete for faculty.

Entrustment Rating

Select an entrustment rating (if required) by entering item text, and selecting a scale. Note that the response options will be configured based on the scale you select.
Set requirements for comments noting that if you select Prompted comments you should also flag which response options will result in prompted comments.

Form Items

Add form items by clicking 'Add Items' or the down arrow to the right. You can either add existing items or create new items and add them to the form.
Remember that every item created can be mapped to an EPA and milestones. The mapped EPA information about each item will be used to provide access to this form when a learner or faculty member triggers an assessment. At least one item added to the PPA form must be linked to an EPA in order for the form to be triggerable.
For more information about how to create Assessment and Evaulation items, please see here.
Preview and save your form to return to it later. When your form is complete click 'Publish' to make it available to be triggered from the CBME dashboard.

Reordering Items on Published Forms

After creating forms using a form template, you can reorder form items on individual forms. This applies to published Supervisor Forms, Field Notes, and Procedure Forms.

Open the Form Template that you used to build the form of interest.
Click on the title of the form on which you wish to re-order items.
Click and drag on the “cross” icon to re-order the assessment items.
The changes will be saved automatically. This must be done for each individual form that you wish to re-order.

Reviewing Form Feedback

To review feedback provided about the assessment tools themselves, go to Admin>Assessment and Evaluation, and click Reports from the second tab menu (look under the Assessment & Evaluation header). Under Assessments, click on Assessment Tools Feedback Report. CBME form feedback will show up there. You can filter by course and form to refine the list of feedback. If you are logged in as a PA you will only have access to forms associated with your program.

Tracking Completion of Tasks and Forms

Learner and Faculty Triggered Assessment Forms Viewed from CBME Dashboard

Learners and their affiliated faculty and program administrators can track assessment form completion by navigating to the learner's CBME dashboard.

Click on the Assessment and Evaluation badge at the top of the page. Click the My Learners tab and click on CBME Dashboard below the relevant learner name.

From the learner's CBME dashboard click on Assessments. Scroll down past the Filter Options until you see a set of tabs including Completed, In Progress, Pending, and Deleted. Choose the appropriate tab to review assessment form completion.

Tracking Form Completion From the Learner's Assessments Page

Click on the Assessment and Evaluation badge at the top of the page. Click the My Learners tab and click on Assessments below the relevant learner name.

These pages provide information on tasks triggered by faculty and learner's as well as tasks assigned through distributions and provide access to some reporting. For more information see the Reviewing Progress>Assessments Page help section.

Tracking Form Completion From a Faculty's Assessments Page

PAs can view a faculty's tasks from the Assessment and Evaluation tab. It works almost the same as the learner's assessment page but is accessed from the Faculty tab.

Monitoring Progress of Tasks Created via Distributions

When a PA sets up assessment and evaluation tasks to be completed via distributions, progress can quickly be viewed via the Assessment and Evaluation module.

Methods of Assessment Completion

Learners

CBME learners will be taken to their CBME Dashboard upon logging into the system. From there, a learner can click the green 'Trigger Assessment' to begin. The learner can select an assessor from the dropdown menu, fill in the date, and select the appropriate assessment method. A reminder that if you want to test this feature and use the complete via pin option, you should set up the PIN for the faculty member(s). (See next lesson.) Learners access assessment forms by selecting an EPA. For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

After an EPA is selected, the available assessment tools will be displayed. Learners can click 'Begin Assessment' to start.

Faculty

From the Dashboard, faculty have access to a green 'Trigger Assessment' button on the right side of the screen. Users can select a resident and then an EPA. For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

After an EPA is selected, the available assessment tools will be displayed. Faculty can click 'Begin Assessment' to start.

Triggering and Completing Forms

Setting User PINs

If you are building and testing supervisor forms, you'll want to set up the faculty PIN for some sample users so that your test forms can be completed and you can view progress on the CBME dashboard. When CBME is live for your organization, users will need to set their own pins in order to easily complete forms at the point of encounter. Faculty and learners can set PINs. This allows residents to complete peer assessments. Learners will only be able to trigger assessments on themselves and have a peer complete the form via PIN; residents cannot trigger forms on other learners.

Setting the PIN

To set a PIN, log in as the necessary user. Access the user's profile by clicking on the user's name at the top of the screen (to the right of the logout button). Once in the user's profile, click on the Account Information tab. Beside PIN enter a 4 to 6 digit PIN. Click 'Save Profile'. Done!

Trigger Assessments: Faculty

From the Dashboard, faculty have access to a green 'Trigger Assessment' button on the right side of the screen.

First faculty select a resident. They can begin to type a name to limit the list of options. When they mouse over a resident name, faculty can see the resident card including their name, photo (if uploaded) and program.

If the learner is enrolled in two programs the faculty will have to specify a program.

A date of encounter will also have to be set.

Next faculty select an EPA.

For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia (see above).

Users can easily navigate the list of EPAs by applying preset filters including Current Stage EPAs, Current Rotation EPAs, and Priority EPAs. Click on a filter title to apply it. In this example the Priority EPAs filter is being applied.

After an EPA is selected, the available assessment tools will be displayed.

Note the small clock in the top right of each tool. This is an estimate of how long the form will take to complete based on the experience of other users.

Faculty can click 'Begin Assessment' on any tool to begin.

Trigger Assessments: Learners

From the Dashboard, learners have access to a green 'Trigger Assessment' button on the right side of the screen.

First learners select a faculty. They can begin to type a name to limit the list of options. When they mouse over a faculty name, learners can see the faculty card including their name, photo (if uploaded) and program.

Next, learners select an assessment method. Details about each assessment method are provided inline.

Note that faculty must have a PIN setup for learners to select the first option.

A date of encounter will also have to be set.

Next learners select an EPA.

For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

After an EPA is selected, the available assessment tools will be displayed.

Note the small clock in the top right of each tool. This is an estimate of how long the form will take to complete based on the experience of other users.

Learners can click 'Begin Assessment' on any tool to begin.

Depending on the method of assessment learners will either be directed to the assessment form to start it, or the form will be sent to the assessor.

Sample: Submit via PIN

In the first assessment method, learners will be able to save their form as draft or submit it to the assessor to confirm by PIN.

Different assessment methods will provide different options.

Resident Tool: Assessor Cues

This feature allows residents to cognitively situate their assessors by adding an optional note to any assessment that will be emailed (complete and confirm via email; email blank form; self-assessment, then email blank form). It can be used to remind the assessor about specific case details, provide a focus for assessment, or anything else that the resident feels the assessor should know before completing the assessment. This note will stay attached to the assessment and will be visible from the CBME dashboard for reference.

This cue will stay attached to the assessment and will be visible from the CBME dashboard for reference.The cue is optional and if provided it will appear at the top of the assessment for the assessor. The cue can also be seen on the completed assessment for both the resident and the faculty member.

For “Email blank form,” the cue modal will pop up from the Trigger Assessment page after “Send Assessment” has been clicked.
For “Complete and Confirm via Email” and “Self-assessment, then email blank form”, the cue modal will pop up after clicking “Submit and notify attending by email” at the bottom of the form.

PAs Entering Completed Assessments

PAs can enter complete assessment forms on behalf of assessors.

Navigate to Admin>Assessment & Evaluation. Click on the green 'Record Assessment' button below the orange Assessment Tasks heading. Select a resident and assessor from the dropdown menus and provide a date of encounter. Select an EPA as you would to trigger a form. Search for the appropriate form and click Select. Complete the form and click Submit. The PA will be submitting the assessment on behalf of the selected assessor. There is a reminder of the selected assessor displayed at the top of the form in a yellow information bar.

Reopening and Editing Completed Assessments

There is a tool to allow admin users to reopen a completed assessment and edit it as needed. This is a feature that can be turned on or off depending on how your organization wants to use it.

If enabled, staff>admin can access this feature by navigating to their Assessment & Evaluation badge (beside their name in the top left), clicking on My Learners and then the Assessments tab for a specific learner. From the list of Tasks Completed on Learner click on a task to view it. An orange "Reopen Task" button will be above the form. This will set the task to in-progress and the staff>admin or faculty will be able to adjust it. Note that once a task is reopened you can clear all existing form data and forward the task as needed to send an email to faculty alerting them to the new task status.

Use with caution!

This tool should be used judiciously to ensure that residents are not ”gaming” the system and bullying anyone into changing their assessments to be more favourable.

Overview of Outstanding Tasks

Program Administrators and Faculty Directors have the ability to get an overview of pending and in progress tasks from Admin>Assessment and Evaluation. When users access this page they can now view 'Outstanding' tasks as well as 'Upcoming' and 'Deleted' tasks.

Viewing Outstanding Tasks

1. Users can filter for distributed forms or triggered forms.

2. Users can search for specific forms, or owners.

3. The 'Owner' is the person who is responsible for completing the form.

4. Hover over Targets to view the targets of the form.

5. Check the appropriate boxes and then send reminders as needed.

6. Delete tasks as needed.

Click on any form to complete it on behalf of another user.

(Please note, not all users will currently see the Trigger Assessment button included in this image.)

The list of users included in the Outstanding task list are those faculty listed as "Associated Faculty" on the program setup tab, faculty who have been included in a distribution in the course, and learners in the audience of a program enrolment (also configured on the program setup tab).

CBME Dashboard

Faculty, program administrators and residents can easily review a learner's progress from the CBME dashboard.

Note that the CBME dashboard includes several tabs: Stages, Assessments, Assessments Items, Trends, Comments, and Pins. (Reports can be generated from the Stages page.)

There is another Assessments Dashboard that pulls in additional form information if your organization is using the Assessment and Evaluation module for other forms (e.g. program and faculty evaluations). This page discusses just the CBME Dashboard.

Accessing CBME Dashboards

When logged in as a faculty member or program coordinator, click the Assessment & Evaluation badge that appears between the user's name and the logout button in the top right. Click on 'My Learners' from the tab menu. Search for a learner as needed, and if you can't find someone, ensure that you are in the correct curriculum period using the dropdown menu on the right. After finding the correct learner, click on CBME Dashboard to view the learner's progress. Residents automatically land on their CBME Dashboard when they log into Elentra.

Reviewing Learner Progress

From a learner's CBME Dashboard, click through the different tabs to view a range of assessment information. On most tabs, you can apply filters in order to refine the list of assessments visible to others. To use this option, select all the appropriate filters (e.g. date, curriculum tag, contextual variable, assessment tool, etc.) and then click 'Apply Filters'. Note that the filters a PA or PD applies on one learner will be maintained as you move through different pages.

Stages

From the stages tab you can see a summary of a learner's progress across EPAs and stages.

EPAs are displayed in order of learner stage and completed stages can be collapsed. If you are using the rotation scheduler, EPAs specific to a learner's current rotation are outlined in the stage colour and the priority and likelihood of an EPA in the learner's specific rotation is shown through the exclamation mark and bar chart (unlikely, likely, very likely). Whether or not the rotation scheduler is in use, green checkmarks indicate that a stage or EPA is complete (this is set by the Competency Committee).

Click the down arrow on the right side of an EPA card to see a list of completed assessments for that EPA. Depending on the form there may be a count of the learner's global assessment rating, which you can click on to access an aggregated report of the learner's performance on this form.

When in a faculty or PA role you can also access a Milestone Report from the Stages tab. More details about the Milestone Report are in another lesson.

Stages Tab: Reporting Options

Users can produce an aggregated report for any form displaying on this page. Click the grey down arrow on the right side of a EPA card, and click on a form title. A new tab will open to display the aggregated data with a trend chart at the bottom. On the trend chart at the bottom of the report hover over any individual assessment to click 'View Assessment' and see the specific assessment form. Note that you can download a PDF of this form as needed (green button at top right). If you want to look at only the forms completed with a certain global rating click on a blue number on the right hand side.

Assessments

See a list of all completed assessments and filter as desired. Toggle between completed, in progress, pending and deleted assessments below the filter settings. On each individual assessment card, note that the form type and relevant EPA are displayed. You can also click the down arrow on the right to show some form information (e.g., global rating, comments, assessor and assessment method), and click 'View Details' on the left to see the assessment form. Users can mark an assessment as read by clicking on the eye icon. A greyed out eye means an assessment is unread. There is an option to mark all assessments as read on the right side above the assessment cards. You can also quickly view all unread assessments by clicking the small blue number beside 'Assessments' in the tab menu. Marking assessments as read or unread is user specific so learners and faculty will see their own results. Users can pin an assessment from this screen and learners can give an assessment a "thumbs up" to provide feedback to an assessor about their feedback.

Assessments Items

Quickly see a list of all completed assessment items and filter as desired. Click the down arrow on the right to see comments, response descriptor and the name of the assessor (as applicable). Click View Details at the bottom left of each card to access the completed assessment form.

Users can pin an assessment item from this screen.

Trends

View trends in learner performance across global assessment ratings. Note the overall tally of ratings in the top right corner of each card. Hover over on a node on the chart to view information about the form name, date and rating; you can also click through to access the actual assessment form.

Comments

Quickly access a list of all narrative comments provided to the learner on any complete CBME assessment tool. The tool type and relevant EPA will be displayed below the form name on each comment card.

Users can pin comments from this tab.

Pins

Quickly view all assessments, items, or comments that have been pinned. Apply filters as desired, or just scroll down to find the toggle to switch between assessments, items, and comments.

Reviewing Resident Progress

Promoting Learners Through Stages

To promote a learner to a new stage, log in as a Competence Committee member, and click on the Assessment and Evaluation badge at the top of the page (beside the green logout button). Select the 'My Learners' tab and then click on the CBME Dashboard tab of the learner you want to promote. Beside each item you'll see a small grey circle; click on the circle to open a dialogue box where you can enter text and mark an EPA or stage as complete. You can also click 'View History' in the dialogue box and see any previous comments or decisions regarding learner promotion (or demotion).

Updating Learner Stages

Introduction

When publishing a new curriculum version within the CBME module, learners will automatically be updated to have the new curriculum version (i.e., EPAs marked as “replaced” or “changing”) for all of their upcoming stages of training. There may be a time where you would like certain learners to have stages from a specific version. This guide will instruct you on how to properly update a learner or series of learners to have stages from a specific version.

What you will need

In order to update the learner(s) objectives in a timely manner there are a few things to gather before you begin the process:

1. Assemble a list of proxy_ids for all of the learners that you wish to update. This process is done on a per course basis so make sure that all of the learner proxy_ids that you compile belong to the same course.

2. Make note of the stages that you wish to be updated. The script requires the stage letter in order to know which stages to update, so make a list of the stage letters. For example, if you are updating a learner to have an old version of Transition to Discipline (D) then you will need to note D as the stage you are updating.

3. The final thing you will need is the cbme_objective_tree_version_id for the version that you wish to update to. So if the course you are updating is course_id 123 and you would like to update a learner to version 2 for a stage then you need to look up the cbme_objective_tree_version_id for course 123 version 2 in the cbme_objective_tree_versions table. You will also need the cbme_objective_tree_version_id for the version that the learner is already a part of. The script requires that you set the version for every stage available to the learner which is why we need the current versions.

4. Access to the database

5. SSH access to your production environment

Updating the learner stages

Updating the learner stages requires a developer tool to be run from developers/tools/cbme_migration. You must have SSH access to your production server in order to complete these steps. Please Note: It’s recommended that you go through the following steps in a test/staging/development environment first so that you can ensure that the script updated the learners properly.

Steps:

1. Open up your database client and open the cbme_objective_tree_aggregates table.

2. For every learner proxy_id that you compiled ahead of time we will be deleting their cbme_objective_tree_aggregate records for the course that we are updating. Select all of the rows where tree_viewer_value is the proxy_id that we are dealing with, the tree_viewer_type is “proxy_id” and the course_id matches the course that we are using. Once you have all of that data, we are going to DELETE it from the table. Repeat this until we have deleted the aggregates for every learner in the list.

3. Now that the aggregates are deleted, we can update the learners’ stages using the script. SSH into your server and navigate to the following directory: /var/www/vhosts/your installation name here/developers/tools/cbme_migration

4. Once in that directory we are going to be executing the reset-learner-tree-version.php script. Tip: if you run php reset-learner-tree-version.php --usage it will bring up the help dialogue to describe all of the options that are available. Once you have read through all of the available options you will notice that there are multiple modes that this script can be run in. For this scenario we will be using “specify” mode since we want to specify which version of stages the learners will be receiving. We must specify all stages in the --stages parameter so that the script updates them to the correct version.

As an example, if your data is this:

organisation_id = 1
course_id = 123
proxy_ids = 1111,2222,3333,4444
stages to update = C,P
current version id = 10
new version id = 20

The command will look like this:

php reset-learner-tree-version.php --mode=specify 
--course_id=123 --organisation_id=1 
--proxy_ids=1111,2222,3333,4444 --stages=D,F,C,P 
--versions=10,10,20,20

We do not need to provide the --exclude parameter in this case

You will notice with the command above that we have listed all 4 stages in the --stages parameter even though we are only updating C and P. The reason for this is because the script requires that all stages be specified in order to update them to the correct version. In this case we are not changing D and F so we set them as the original version (10) in the script parameters. Each stage corresponds with a version in the --versions parameter, so in this case D will get version 10, F will get version 10, C will get version 20 and so on.

5. Once that script runs the last thing to do is to clear the cbme_objective_tree cache for the course that we are dealing with. The easiest way to do this is through the interface:

Login to your install as an administrator who has admin access to the course that we are updating
Navigate to Admin > Manage Courses (Programs) > Select the course that you are using > CBME tab > Import CBME Data
Click on the Actions button on the left side above the EPA search bar and select the Edit EPAs option.
Whenever one of these EPAs are updated, the cache is cleared for the course. So all that is required is to just click save on the first EPA that is listed and the cache will be cleared. You do not need to change any of the text in the EPA that you just saved. Simply saving what is already there will sufficiently clear the cache.

Conclusion

Once you have cleared the cache for the course then the learners should see the updates on their dashboard. As mentioned before, it's recommended that you do this process in a test environment first so that you can verify the data is the way you would like it before updating production. If you do run into the scenario where you updated a learner to the wrong version then you can always repeat this process and update them to the correct version.

The easiest way to verify that the learners are in the correct version would be to login as some of the learners that were updated and compare their dashboards to the version they were set to. Usually there is a difference between one version to the next whether it be the EPA titles or the number of EPAs.

Milestone Report

Accessible by PDs and PAs from the CBME Dashboard Stages tab, the Milestone Report allows faculty and PAs to generate a milestone report for a specific learner. This report is a breakdown of completed assessments that have been tagged to milestones and EPAs. Assessments are tallied based on the milestone and EPA that they are tagged to and the tool generates one report per unique scale. The number of unique scales is determined by what tools were used to complete the included assessment tools. The reports are generated as CSVs and are zipped into a folder.

From a learner's CBME Dashboard Stages tab, click Milestone Report in the top right (beside Log Meeting).

1. Select a date range for the report. 2. Click “Get Tools" 3. Select the tools that you wish to view the aggregated milestone report for, or click “Select All Tools" 4. Click “Generate Report”. This will open a download modal for you to select where to save the zip file. 5. Unzip the file 6. If multiple rating scales were used to assess the milestones, there will be one CSV file for each rating scale 7. Open a CSV file 8. Each row represents a milestone, and each column represents the rating scale options. These rating scale options are grouped by EPA (e.g., for the Queen’s Three-Point Scale, you will see 4 columns for each EPA: Not Observed, Needs Attention, Developing, and Achieving). 9. Each cell displays the total number of times that milestone was assessed for that EPA, including how many times it was rated at that level on the rating scale (e.g., 3 of 6 completed assessments were rated as “Achieving”.

NOTE: Even though you may have selected only one of many Supervisor Tools (or other tools), if you used the same scale on all tools, the report will display all data for all tools that used that rating scale. We will be enhancing this in future iterations to only report on tool(s) selected.

Resident Tool: Thumbs Up for Helpful Feedback

Residents can highlight assessments that they found helpful to their learning. From the “Assessments” tab of the CBME Dashboard, a learner can click on the “thumbs up” icon to indicate to an assessor that their feedback was helpful to the resident's learning. Leaners can also include a comment on why they found it helpful. This feedback is important to help assessors identify the types of feedback that residents find beneficial for their learning.

Pinning Forms, Items and Comments

Learners or faculty can pin assessments or individual comments to keep them easily accessible for review during meetings. This can help to keep an important piece of feedback or other information front and centre until it has been discussed.

How to pin something

To pin an assessment, simply click on the pushpin icon that appears to the right of the assessment title and details. You'll get a success message that you've pinned the item. In the example to the left, the second and third assessments have been pinned.

To pin a comment, click on the Comments tab from the CBME Dashboard and then click the pushpin beside the desired comment. In the example to the left, the second comment has been pinned.

Viewing Pinned Items

From the CBME Dashboard, click on Pins at the end of the tabs menu. This will open a screen showing all pinned items. To unpin something, just click the pushpin. You'll see a success message that you've unpinned the item.

Log Meeting Notes

Faculty and program administrators can log meetings to maintain a record of conversations about learner progress. Either user needs to navigate to a learner's CBME dashboard to access the Log Meeting button.

Learners have quick access to the their own meeting logs from the CBME dashboard My Meetings button.

Logged meetings display on a meeting log accessible to faculty and learners. Note that only faculty and PAs can add and edit meeting information. Learners have read only access to their meeting log.

Posts can have files uploaded to them, be edited or be deleted using the tools in the Actions column. A program admin., academic advisor or competence committee member can create or add to any entry they have made (but not those made by other users).

Meeting Logs are accessible to competence committees and may be referenced when committees meet to discuss learner progress.

Assessments Page

In addition to the CBME Dashboard, users can access a learner's Assessments page to view tasks completed on the learner, as well as tasks assigned to the learner. These pages reflect forms completed via distributions and CBME forms triggered by faculty and learners, however the reporting tool accessible from Tasks Completed on Learner applies only to forms managed via distributions (for reporting on triggered CBME forms please see the CBME Dashboard page).

Accessing the Assessments Page

When logged in as a faculty member or program administrator, click the Assessment & Evaluation badge that appears between the user's name and the logout button in the top right. Click on 'My Learners' from the tab menu. Search for a learner as needed, and if you can't find someone, ensure that you are in the correct curriculum period using the dropdown menu on the right. After finding the correct learner, click on Assessments on the resident card to view the learner's assessments page.

Note that program administrators and program directors can use these same steps to access the assessments page for faculty.

Assessments Page Contents and Filters

Multiple tabs are available on the Assessments page. Each tab displays a collection of assessment tasks. CBME specific forms are included on this page and so are any forms created through the Assessment and Evaluation module and sent to users through a distribution.

On each assessment card you will see a range of information which may include the form title, form type, delivery and completion dates, progress (e.g., 3 of 14 complete), who triggered the form , the target, and the assessor/evaluator (including their group and role). Not all of this information is included on every task card.

On each tab you can search for tasks, set a date range and apply filters. Click the down arrow beside the search tasks bar to select a filter type and drill down to the necessary filter. Select a filter and click the green Apply Filters button. Your applied filters will display on an Active Filters list.

Tasks Completed On Learner

These are tasks completed by assessors on the learner.

Each task can be downloaded or viewed. To download tasks, click the appropriate box at the bottom of the task card and then click Download PDF(s) at the top of the page. Note that there is a Select All tool available. To view a task click View Task at the bottom of the task card.

More tasks can be displayed by clicking Load More Tasks at the bottom of the page.

Pending Tasks on Learner

From this tab you can view all pending tasks on the learner this includes tasks that have been distributed but are not yet started. To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Upcoming Tasks on Learner

This tab displays scheduled distributions where the learner is the target.

To remove a task from this list click Remove Task and provide a reason for the removal. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Current Tasks

This tab displays active distributions where the learner is an assessor/evaluator.

To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Upcoming Tasks

This tab shows tasks scheduled through a distribution where the learner is an assessor/evaluator. Tasks can be removed from this list by clicking Remove Task and providing a reason for the removal.

Reports for this Learner

From the Tasks Completed on Learner tab, users can access Reports for this Learner. This gives you access to aggregated reports on a learner for each form used. Note that these reports are available for forms managed via distributions, not those triggered by learners and faculty.

Set a report date range if desired.

From the Options dropdown menu you can: 1. Choose to generate the report with or without comments (both can then be downloaded as a PDF); 2. Click View This Form to open a new tab that shows the form; and 3. Click View Assessments to open a tab that lists all completed assessments for this form and provides links to view each one.

Clicking the 'Group by distribution' checkbox will reorganize the forms to reflect completion based on distribution and add a column to the display to show distribution information.