If your institution is new to CBME please view this additional learning module to prepare to launch CBME with Elentra: http://bit.ly/CBMELaunchReadiness

The Competency-Based Medical Education module allows institutions to map entrustable professional activities (EPAs) and link them to key and enabling competencies and milestones. In turn this allows the creation of assessments and the collection of learner performance data, all tied to EPAs.

Different institutions will be using Elentra differently to help them transition to CBME. It is important to note that there are some features relevant to CBME that rely on the use of the Assessment and Evaluation module. Using CBME without any additional features like rotation scheduling, you will be able to map EPAs and milestones, use supervisor, procedure and field note form templates to generate 'triggerable' assessment forms, create Periodic Performance Assessment and Rubric forms that are linked to EPAs (and also triggerable), allow faculty and learners to trigger assessment forms, store and report on learner performance data based on the previously mentioned forms, assign faculty as competence committee members and academic advisors, log meetings, and use the CBME dashboard (including promoting learners through stages).

Beginning to use Elentra for CBME can be broken down into several tasks: mapping your EPAs, building forms, setting up learners and faculty to access the forms, and reviewing and reporting on learner progress.

Setting up CBME Curriculum Tags

Before institutions can map their EPAs or use the CBME features within the assessment and evaluation module, specific curriculum tag sets must be in place. These are the required steps to build the necessary tag sets:

1. Auto-setup the required curriculum tag sets through the CBME tab. This happens once for an organization and populates roles, stages, and contextual variables.

2. Populate the CanMEDS Key and Enabling Competencies at the organization level. This is completed through Manage Curriculum and happens once for an organization. Even if your organization will be using program specific objectives, you are still required to upload a standard set of key and enabling competencies.

3. Import EPAs, key and enabling competencies, milestones and contextual variable responses (e.g. options for case complexity or clinical setting) through a program's CBME tab. This happens for each program using CBME.

4. Define and upload procedure attributes to make use of the procedure form template. Procedure attributes become assessment criteria for procedures. Each program using CBME will need to upload procedure attributes but this can be completed later than the other imports. It may be preferable to complete this step when programs are ready to build their assessment and evaluation tools.

After curriculum tags are setup: Form template creation

Once the required curriculum tags sets are in place and populated, an institution can begin to use assessment and evaluation features linked to CBME. A series of form templates can be completed at the program level to facilitate assessment of learners.

1. Supervisor Form Template

2. Procedure Form Template

3. Field Note Form Template

Using these form templates allows Elentra to rely on the mapped EPAs, competencies, milestones and contextual variables to generate and publish multiple forms based on the parameters established on each form template.

Additional assessment forms can also be used for CBME. These include:

1. The Periodic Performance Assessment Form

2. The Rubric Assessment Form

As long as items added to these forms are linked to an EPA the form will be triggerable once it is published. Note that using these forms allows you to add customized questions or prompts and that you can build those items either in advance or during form creation using the Assessment and Evaluation module.

If you want to be able to create additional customized forms and distribute them to learners and/or faculty based on rotation schedules, specific learning events, etc. you must be using additional features of Elentra.

Managing Faculty and Learners

Through the Groups function of a course page, set up optional Academic Advisor Groups to link faculty members to groups of learners. Academic Advisors will be able to view the CBME dashboard for all of their associated learners.

Through the Course Contacts section of a Course Setup page, set up Competence Committees to allow groups of faculty to review learner performance and make decisions regarding learner progress through the required stages.

Once they are set up, learners and faculty can trigger and complete CBME assessment forms. Completed forms are stored on the learner's CBME dashboard.

Reviewing and Reporting on Learner Progress

The CBME dashboard is one place to review learner progress and the different forms completed on a learner. From the CBME dashboard Stages tab, users can quickly see learner progress in different EPAs and stages and see an overview of completed forms and performance trends on the global assessment ratings on those forms. An aggregated report of learner performance in an EPA is available directly from the Stages tab of the CBME dashboard.

From the CBME dashboard you can also access an Assessments tab which allows you apply a variety of filters, search in forms, and view completed, in progress, pending and deleted forms where the learner is the target of the form.

As learners and faculty review and discuss forms a variety of additional tools are available including the ability to pin comments, items, and forms, a way to log meeting notes, and the ability for learners to give a "thumbs up" to a completed form. This information becomes accessible to the learner and faculty and in some cases dedicated tabs give easy access to it (see below).

Additional tabs accessible from the CBME dashboard include Assessment Items (a view of individual items completed on the learner), Trends (visualizations of learner performance), Comments (access to all comments in one place) and Pins (view of anything residents or faculty have pinned for review).

In addition to the CBME dashboard, learner progress can be reviewed and reported from the learner's Assessments page where tasks completed on the learner, as well as a variety of forms where the learner is the target or the evaluator can be viewed. Reports are accessible from this page as well however those reports only apply to forms managed via distributions, not CBME forms triggered by faculty or learners.

Although there is the ability to upload program specific key and enabling competencies in Elentra ME 1.12, organizations are still required to upload a standard list of key and enabling competencies at the organization level. For Canadian schools using CBME we recommend making the CanMEDS Key and Enabling Competencies as your standard list. These are available online or from another institution.

To download Elentra templates, complete them, and upload key and enabling competencies information, navigate to Admin>Manage Curriculum>Curriculum Tags.

Click on a curriculum tag set name and then click 'Import from CSV' in the top right to access and download the template to complete.

After you complete each required template, save it in a .csv format and upload it from the same screen you were on.

Accessing CanMEDS Key and Enabling Competencies

You can access the CanMEDS Key and Enabling Competencies at http://canmeds.royalcollege.ca/guide. The Key Competencies are preceded by one number like 1 or 2, and are stored under CanMEDS roles. The Enabling Competencies are preceded by a number like 1.1 or 2.2 and are also stored under CanMEDS roles.

Sample Key Competency: 1 Practise medicine within their defined scope of practice and expertise

Sample Enabling Competency: 1.1 Demonstrate a commitment to high-quality care of their patients

Both of these are from the Medical Expert role so would be coded with the prefix ME. Alphanumeric codes are provided in the templates, you will need to input the key and enabling competency text.

You can export the text from the CanMEDS guide.

CanMEDS Key Competencies

Download the template through Manage Curriculum>Curriculum Tags>Key Competencies.

Note that the first column, objective_code, is completed for you with alphanumeric codes based on the CanMEDS roles.

Complete the second column, objective_name, by providing the key competency text. You can access the key competencies at http://canmeds.royalcollege.ca/guide or request them from another institution.

Save your file as a CSV.

CanMEDS Enabling Competencies

Download the template through Manage Curriculum>Curriculum Tags>Enabling Competencies.

Note that the first column, objective_code, is completed for you with alphanumeric codes based on the CanMEDS roles.

Complete the second column, objective_name, by providing the enabling competency text. You can access the enabling competencies at http://canmeds.royalcollege.ca/guide or request them from another institution.

Save your file as a CSV.

Uploading Templates and Setting as Standardized Curriculum Tag Set

To upload completed templates, navigate to Manage Curriculum>Curriculum Tags, and select Enabling Competencies or Key Competencies. Use the 'Import from CSV' button in the top right to upload your templates. Search for or drag and drop a file in place, then click the green 'Import' button.

A program needs to complete this template OR the Milestones Template, not both. Most programs will map milestones and there is no need to complete this template. Users define which option they will use in in Step 2 of the import CBME process.

Navigate to Admin>Manage course and select a course. Click CBME and Import CBME Data to access and download the template.

Columns to Complete

• EPA code: Codes should be recorded in uppercase letters. Codes should refer to the stage of training and the EPA number. e.g. D1, D2, D3, F1, C2. Using consistent codes across your program and school is important.The system is setup to accept the following codes for learner stage: D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice

• Enabling Competency Code: Note that the template already includes a list of Enabling Competency Codes using the CanMEDS role prefixes. In the first column, simply add the EPAs codes you wish to map to each enabling competency.

  If there are unmapped Enabling Competency Codes left over you must delete those rows before you upload your .csv file.

Mapping multiple EPAs to one enabling competency

The system is designed to allow multiple EPAs to map to one milestone. To do so, simply insert a comma between the EPA code entries in the first column.

Save as a .csv and upload

After saving your file as a .csv, scroll to step 3 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

If you want to manually map a connection between an EPA and a milestone, you can do so through the Map Curriculum Tags tab. This allows you to map additional milestones to EPAs after you have uploaded your templates.

• Navigate to admin, manage course and select a course.

• Click CBME and then map curriculum tags.

• Scroll down to the Map Curriculum Tags area.

• Either type in and select the tags required to sort the displayed results OR click through the displayed options to create the mapping structure you require.

  • You can display the results for more than one role at a time by clicking on multiple roles. Note that any already mapped milestones will still show up in the milestone list.

• Click to highlight the appropriate tags and then click ‘Save Curriculum Map’.

Ideally your users will be loaded into Elentra from a central, authoritative source like a student information system. Manually adding users through Admin>Manage Users is a possibility and more detailed instructions for using this tool are available on the Elentra help website.

If you are manually setting up CBME users, please note the following:

• Resident learners should be assigned group: student, role: predicted graduating year.

• Program Administrators should be assigned group: staff, role: Pcoordinator. Further, they must be assigned to a specific program in order to access the relevant CBME and Assessment and Evaluation features. To give specific staff more ability to act within Elentra set them up as Staff>Admin. Note that Staff>Admin have access to almost all features of Elentra and can access System Settings.

• Faculty should be assigned group: faculty, role: faculty, lecturer, or director depending on their role. Someone with faculty director permissions will still have to be added to a specific program in order to access the relevant CBME and A and E features. Faculty directors will have access to all faculty evaluations within their department (except their own).

• Any faculty, regardless of their role, can be added to Competence Committees through a course/program setup page or as Academic Advisors through the program Groups tab. (Additional instructions for each are in additional lessons.)

Improve learner identification of priority EPAs

From the CBME dashboard, EPAs can show their priority and the likelihood of being encountered within a learner's rotation. For this feature to be active you must also be using the rotation scheduler in Elentra.

Adding Priority and Likelihood Levels

Navigate to Admin>Rotation Schedule Create a draft schedule or open a published schedule. If you are create a new rotation schedule, you will also need to add rotations to the schedule. Once rotations exist within a schedule a small Objectives badge will appear beside the rotation name. Click on the Objectives badge to open a list of EPAs tagged to the rotation.

Click the plus sign beside any EPA you'd like to label with priority or likelihood. Indicate the likelihood by clicking on the appropriate button. If you'd like to set an EPA as a priority, click the exclamation mark in the priority column. Repeat as necessary. Return to the rotations list using the Back to Rotations button.

Note that you'll only have this step available if you decided to map program specific key and enabling competencies in import Step 2.

Use the Enabling Competencies Template if a program wants to import program specific enabling competencies. This template organizes the information about Enabling Competencies decided by each national specialty committee and allows the information to be uploaded to the system.

Navigate to Admin>Manage Course and select a course. Click CBME and Import CBME Data to access and download the required template.

Template Columns to Complete

Note that the first column, Enabling Competency, is completed for you with alphanumeric codes based on the CanMEDS roles. Adjust this list as needed to match the information from the national specialty committee.

When coding the Enabling Competencies remember these required codes for the CanMEDS roles: Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME

Complete the second column, Title, by providing the enabling competency text.

The third column, Description, is optional. This will store information but will not be visible to users.

Save your file as a CSV.

Uploading Your File

After saving your file as a .csv, scroll to step 4 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

Please note that there is currently no user interface to manage learner levels (e.g., PGY1, PGY2). The learner level will display on the learner card when you view a list of My Learners. In the example below the learner is a PGY4.

The D in front of the learner level indicated the learner CBME stage. In this case the learner is in stage D or Transition to Discipline. There is a user interface for promoting a learner through different stages. Competency Committee members can mark a stage as complete from the learner's CBME dashboard.

For learner level and stage to show on learner cards, developers or technical staff will need to configure tables in the database to display this information.

Note that when people trigger assessments and hover over a name, they will also see a summary of information about the user. In this case they will see the learner group and role displayed. In the example below group is Student and the role is 2015.

After the required, standardized curriculum tags have been populated, all programs must also complete and upload multiple templates through the CBME tab for a program. Note that program directors and program coordinators will have access to the CBME tab for their affiliated programs. The templates to complete are:

• The EPA Template

• The Program Specific Key Competency Template (optional)

• The Program Specific Enabling Competency Template (optional)

• The Milestones Template

• The Contextual Variables Template

To access these templates, navigate to Admin>Manage Courses, and search for the required program. Using the gear icon beside the program name, click 'CBME' and then click 'Import CBME Data' from the tabs.

On the right you will see a dropdown menu titled 'Download Example CSV Templates' for your use.

Click on the required template and it will download to your computer. Look for it in your downloads file if it doesn't immediately open.

Why are there two enabling competency templates?

There are two similarly named templates because of two options available: the option to import program specific enabling competencies, and the option to map EPAs to enabling competencies instead of milestones. Most programs will map EPAs to milestones and will never use the 'Enabling Competency Map Template.' However, if you have a program piloting CBME before its national launch they may choose to map to enabling competencies if milestones aren't finalized yet and we have left that option available. Use the Enabling Competency Template to import program specific enabling competencies. Use the Enabling Competency Map Template only if a program chooses to map EPAs to enabling competencies instead of milestones.

The EPA Template needs to be completed by all programs. This template organizes the information about EPAs decided by each national specialty committee and allows the information to be uploaded to the system.

Navigate to Admin>Manage Course and select a course. Click CBME and Import CBME Data to access and download the required template.

Template Columns to Complete

Each program should complete an EPA template with the following information:

• Code: This is the EPA code. It should be recorded in uppercase letters and include the learner stage of training and the EPA number. Use the learner stage letters outlined below to ensure that the EPAs correctly map to other curriculum tags. D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice Sample EPA Codes include D1, D2, D3, F1, F2, F3, C1, C2, etc. Note that there is no space between the letter and number in the EPA Code.

• Title: This should be the EPA text. This can be copied from the EPAs as decided by each national specialty committee.

• Detailed Description: In some cases, these may be the same as the EPA title; in other cases, an EPA has an additional description. Not every program will use the ‘Detailed Description’ column. Note that this information is displayed to users in the EPA Encyclopedia.

• Entrustment required (optional): This field is optional but will populate the EPA Encyclopedia which is visible to all users.

Remember to use these codes for the learner stage on all templates:

D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice

Save as a .csv and upload

After saving your file as a .csv, scroll to Step 1 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

Note that you'll only have this step available if you decided to map program specific key and enabling competencies in import Step 2.

Use the Key Competencies Template if a program wants to import program specific key competencies. This template organizes the information about Key Competencies decided by each national specialty committee and allows the information to be uploaded to the system.

Navigate to Admin>Manage Course and select a course. Click CBME and Import CBME Data to access and download the required template.

Template Columns to Complete

Note that the first column, Key Competency, is completed for you with alphanumeric codes based on the CanMEDS roles. Adjust this list as needed to match the information from the national specialty committee.

When coding the Key Competencies remember these required codes for the CanMEDS roles: Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME

Complete the second column, Title, by providing the key competency text.

The third column, Description, is optional. This will store information but will not be visible to users.

Save your file as a CSV.

Uploading Your File

After saving your file as a .csv, scroll to step 3 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

To use the CBME Module, nine required curriculum tag sets must be in place. An auto-setup tool will help users create these.

Navigate to Admin>Manage Course>select a program, and click CBME.

If the required curriculum tag sets aren't yet configured the program will prompt users to auto-setup the required curriculum tag sets and will populate the roles, stages, and contextual variables tags.

Users without permission to use the auto-setup feature will be directed to contact a system administrator with the message below.

Note that before using the groups function a course needs to have enrolled learners with a designated curriculum period.

You need to be logged in as a program coordinator or program director to create or modify academic advisor groups.

Navigate to Admin>Manage course and select a course. Under the program title, click on 'Groups.' After clicking on Groups, any existing groups will be listed below. Ensure that you are working in the appropriate Period by reviewing the dropdown menu in the top right. Click the green button on the right to add a new group. Select a prefix for the group name and type it into the Group Name Prefix area. This prefix will be applied to all group names.

Select your group type: you can either define a number of empty groups and manually populate them with learners OR have the system automatically populate groups.

To create empty groups, enter the Group Name Prefix, and a designated number of groups. Click the blue add button on the bottom right.

Add academic advisors to a group

Within the current setup, the term tutor refers to the academic advisor role. To add tutors/academic advisors to the group, click on a group name, find the Group Details section, and use the tutor search bar to pull up names.

Click on the required name and repeat as necessary. When names have been added they will appear in a list below the tutor search bar.

Adding tutors/academic advisors to a group allows those faculty to view the assessment data for any learners in the group.

Add learners to a group

To add learner members to a group, click on the group name, scroll down to Add Members and use the dropdown menu to tick off members. Note that you can add multiple learners at once by ticking off multiple checkboxes beside names.

After you have added the required names, click the blue Proceed button on the bottom right. This will save your work.

Renaming a group

From the Groups tab, click on the group you want to rename. In the Group Details section look for the Group Name box and manually change the name of the group as needed.

Scroll to the bottom of the page and click the blue proceed button on the right. This will save your work.

Deleting group members

To delete tutors, go to the Group Details section, and click on the red minus sign beside a tutor name.

To delete learners from a group, go to the View Members section, and tick off the box beside any learner names you want to delete, and then click the red Delete Members button on the right side.

Mapping Entrustable Professional Activities, key and enabling competencies, milestones, contextual variable responses and procedure attributes for each program is what allows users to quickly and efficiently create assessment tools and report of resident progress. There are two stages in mapping curriculum; in the first stage a central administrator (e.g. PGME office) needs to configure Elentra, then, in the second stage, the curriculum information specific to each program needs to be uploaded.

Steps taken once, at the organisational level, include:

• auto-setup CBME features

• Import central key and enabling competencies,

• Import custom list of contextual variables as needed

• Build assessment scales as needed

Then individual programs can upload Entrustable Professional Activities, key and enabling competencies, milestones, contextual variable responses and procedure attributes.

You need to be logged in as a program coordinator or program director to create or modify competence committees.

Navigate to Admin>Manage course and select a course.

Under the program title, click on Setup.

Scroll down to the Course Contacts section and find the 'Competency Committee Members' section.

Begin to type a name to retrieve a list of potential committee members.

Click on the name you want to add and repeat as necessary.

When a name has been added it will appear below the search box.

After you have added the required names, scroll down to the bottom of the page and click ‘Save’.

Note that all Competency Committee Members for a program will be able to access all resident profiles in that program and will have the ability to promote them.

Promoting Learners Through Stages

Members of Competency Committees can promote learners through stages from the learner CBME dashboard. For more information please see the Reviewing Learner Progress>Promoting Learners Through Stages lesson.

When importing data to create EPA maps, programs are required to upload a contextual variable template to provide contextual variable response options (e.g. case complexity: low, medium, high). In addition to this information, programs will need to add specific details about any procedures included in their contextual variable response options. This additional information, called procedure attributes, plays an important part in building procedure assessment forms. Think of the procedure attributes as assessment criteria (e.g. obtained informed consent, appropriately directed assistants, etc.) for each procedure .

Required Information

For each procedure in your program, you must define headings, rubrics and list items that may be used to assess a learner completing the procedure. Headings appear at the top of a form, rubric becomes the subtitle of a rubric, and list items are the things that will actually be assessed on the rubric.

All this information must be stored and uploaded in a CSV spreadsheet that uses two columns: Type and Title.

Within ‘Type’ there are three things you can include: H to represent heading (e.g. Participate in Surgical Procedures, NG Insertion) R to represent rubric (e.g. Procedure, Pre-Procedure) L to represent list items (e.g. Apply knowledge of anatomy)

The three procedure criteria form a hierarchy. A heading can have multiple rubrics, and a rubric can have multiple list items. Arrange the criteria in your spreadsheet to reflect their nesting hierarchies.

In the 'Title' column enter the information to be uploaded.

When complete, save the file in a CSV format.

Sample Procedure Form On this sample form, the following information was uploaded as procedure attributes: H: Participate in Surgical Procedures R: Procedure L: (ME1.3.2) Apply knowledge... (ME3.4.2) Perform procedural tasks... Use common surgical instruments...

Tip: You can use the same file for multiple procedures if they share the same heading, rubric, and list information.

Uploading Procedure Attributes

• Ensure you are in a Program Coordinator or Program Director role.

• Navigate to Admin>Manage Courses/Programs.

• Scroll down or type in text and click on the relevant program name to select it.

• Click on ‘CBME’ on the tab menu.

• Click on ‘Contextual Variable Responses’ on the second tab menu.

• From the list of contextual variables, click on ‘Procedure’.

• Beside each procedure you will see a small black up arrow beside a green disk icon.

• Click on the up arrow to upload a .csv file of information for each procedure.

• Either choose your file name from a list or drag and drop a file into place.

• Use the dropdown menu to select the EPA to apply your procedure attributes to.

• Click ‘Save and upload criteria’.

Repeat this process for each procedure and each EPA. Remember, a program can use the same procedure attributes for multiple procedures and EPAs if appropriate.

Managing Existing Procedure Attributes

You can view what attributes were already uploaded to a procedure by clicking on the green check mark in the criteria column. You can review the criteria posted for a specific EPA and opt to replace the criteria for that specific EPA.

Uploading Templates

All templates you upload to the system must be in .csv format.

If you are working in Excel or another spreadsheet manager use the “Save as” function to select and save your file as a .csv

To upload your files, either search for or drag and drop the file into place. You'll often see a green checkmark when your file is successfully uploaded.

If you receive an error message make sure that you have:

• Deleted any unused lines. This is especially relevant in templates with pre-populated columns including the contextual variables template, and enabling competencies mapping template.

• Completed all required columns. If a column is empty for a specific line the file may fail to upload.

Viewing EPA Maps

If you've imported all CBME data for a program but you are not able to see EPA maps in the EPA Encyclopedia or in the Map Curriculum Tags tab of CBME, double check that you've uploaded your files without spaces between the EPA Code letter and number, nor between the CanMEDS Role and Key Competency number. Having spaces in the incorrect places will prevent the system from retrieving the required information to produce maps.

Mistakes When Mapping Relationships

While you can correct minor typos and add additional mapping info. through the user interface you can not delete, or undo mapping between, EPAs and milestones or enabling competencies. There is an option to reset all CBME data for a program. This will remove all the uploaded information and delete all existing forms (complete forms will remain on the learner's CBME dashboard but you'll have to rebuild forms to allow them to be triggered moving forward). The Reset CBME Data option is only available to users with administrator role permissions. Find it on a program's Import CBME Data tab.

Contextual Variable Responses are used to describe the context in which a learner completes something. Examples of contextual variables include diagnosis, clinical presentation, clinical setting, case complexity, patient demographics, etc. For each variable, programs must define a list of response options. For example, under clinical presentation, a program might include cough, dyspnea, hemoptysis, etc.

Elentra will autosetup a specific list of contextual variables in the Manage Curriculum interface. Institutions can also add contextual variables (e.g., assessor's role) to this list before uploading contextual variable responses (e.g., nurse, physician, senior resident) via each program's CBME tab.

Do not delete any contextual variables from the existing list within Manage Curriculum or the CBME auto-setup feature will prompt you to re-add them the next time you use CBME. If you wish to add to the contextual variables list you may do so manually through Manage Curriculum and then import your responses through the CBME tab. By default Elentra is set up to provide access to the same Contextual Variables for all programs. You don't necessarily have to provide responses for all CVs in all programs but by default each program will see the same list of CVs.

Adding Additional Contextual Variables

Elentra's auto-setup feature will pre-populate a specific set of contextual variables. Users will be able to upload response options for that list through the CBME tab. However, institutions may wish to add additional contextual variables to make available to their programs. You can do this by following two steps.

1) Through Admin>Manage Curriculum>Curriculum Tags>Contextual Variables you can add new contextual variable tags (e.g. observer role, consult type, domain) that will become visible and useable to all programs within the organization. When you add these new tags to the tag set, you'll be required to provide a code and title. It is recommended that you make the code and title the same, but separate the words in the code with underscores. Example: Title: Observer Role Code: observer_role Please note that there is by default a 24 character limit for curriculum tag codes and a 240 character limit for curriculum tag titles. If you enter more characters than the limit, the system will automatically cut off your entry at the maximum characters. 2) After you have added Contextual Variables to the existing tag set, you will be able to add the new response codes (and responses) to the contextual variable response template and successfully import them.

Importing Contextual Variable Responses

Once you have built out the list of all contextual variables you want included in the system you can import specific contextual variable responses for each program.

Teamplate Columns to Complete

Navigate to Admin>Manage course and select a course. Click CBME and Import CBME Data to access and download the contextual variable response template. Complete the following columns:

• Response Code: This column is pre-populated by fields including assessor’s role, basis of assessment, case complexity, case type, etc. You can add to this list if you have built additional contextual variables through Manage Curriculum (see above). Make sure to use the exact response code from the newly created contextual variable tag in the response_code column.

• Response: In this column, you can add the response variables required (e.g. Case complexity response variables: low, medium, high). To add more than one response variable per response code, simply insert another line and fill in both the response code and response columns.

• If there are items in the response code column that you don’t use, you must delete them before you upload the file.

• Description (optional): This field is not seen by users at this point and simply serves as metadata.

Save as a .csv and upload

After saving your file as a .csv, scroll to step 4 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

A procedure form is an assessment tool that can be used to provide feedback on a learner’s completion of a specific procedural skill. Once a procedure is selected, specific criteria will be displayed. A procedure form can be initiated by a learner or faculty.

When you create a procedure template and publish it, the system looks at the number of EPAs and procedure contextual variable responses selected and generates the appropriate number of forms. If you keep 3 EPAs and indicate 10 procedures on the form template, the system will publish 30 forms that are available to be triggered by a user (one form per EPA per procedure).

To effectively build procedure form templates in a program, the program must have contextual variables responses defined. In addition to uploading contextual variable response options, every procedure needs specific information added. Contextual variable responses and procedural attributes (which function as assessment criteria) can be managed from Admin>Manage Program>select program name>CBME>Manage Contextual Variables.

Accessing Form Templates

Ensure you are in a Program Coordinator or Program Director role. Navigate to Admin>Assessment and Evaluation. Click ‘Form Templates’ on the tab menu. Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear. Type in a form name and select the form type from the dropdown menu. Select ‘Procedure Form.’ If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.

Form Template Information

Edit the form name if needed, and add an optional description. Descriptions will be displayed at the top of forms published from this template. Often people include assessment guidelines in this section. Set permissions for the form template by selecting a category from the dropdown menu (individual, organization, or program) and then beginning to type in a name. Click on the appropriate name when you see it. Added users will appear in a list below the permissions search field. These permissions dictate who has access to and can edit the form template while it is still a draft.

Add additional text at the beginning of the form by clicking the small tick box beside ‘Include Instructions.’ This will open a rich text editor where you can enter text, images, hyperlinks, etc.

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms made from this template.

All EPAs assigned to a course are included on the template by default. To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

Note: You do not specify milestones for use on a Procedure Form.

Contextual Variables

By default, ‘Procedure’ will be selected as a contextual variable.This will require some additional information to be added to the system if the program you are working in hasn’t input procedure response options. (See above.)

If you want all EPAs to have the same available contextual variables leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes.

In addition to ‘Procedure’, you must select at least one other contextual variable and you may select up to a total of 6 for each EPA.

By default, all of the options within a contextual variable are included on any forms made from the template. You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable. This will open a list of contextual variable responses that by default are all checked off. To remove specific responses from this template, deselect them to remove the blue checkmark. For convenience, you can also use ‘Check All’ and ‘Uncheck All’. When you have made the required changes, click the blue ‘Save and Close’ button.

When you modify which contextual variable response options will be available on a template, the number in the grey badge will show how many responses have been included out of the total possible responses.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

If you don't have scales configured you can access them through the Assessment and Evaluation tab.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted.By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The default response feature allows you to prepopulate a form with the selected response.This can be useful if there will be a significant number of milestones that will be included on a form.

The Responses fields will be automatically populated depending on the scale selected in the first dropdown menu.

After you have modified the Enabling Competency/Milestone Scale Selector as required, click the small save button in the top right section of the Enabling Competency/Milestone Scale Selector box.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections. FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted.By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed except by a developer.

Publish

When the form is complete, a green bar will tell you the form can be published.

Click 'Publish' to make your template available for use.

Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited. The number of forms that can be created from a template depends on the number of EPAs assigned to the template.

A Note on Logging Procedures and Encounters Without Assessing Them

In some programs residents may be required to log multiple procedures or encounters and only have a subset of those logged entries be assessed. Elentra does support a logbook outside the CBME module and some programs have opted to have residents use both tools to capture the full picture of residents' progress. The logbook is being enhanced as of May 2018.

After creating forms using a form template, you can reorder form items on individual forms. This applies to published Supervisor Forms, Field Notes, and Procedure Forms.

1. Open the Form Template that you used to build the form of interest.

2. Click on the title of the form on which you wish to re-order items.

3. Click and drag on the “cross” icon to re-order the assessment items.

4. The changes will be saved automatically. This must be done for each individual form that you wish to re-order.

After you upload an EPA Template, you will be prompted to select how you'd like to continue mapping your EPAs. Each program has two decisions to make: 1. Competency Options: Decide whether to upload program specific key and enabling competencies. If you select to use the standard key and enabling competencies (e.g., CanMEDS key and enabling competencies) the system will copy the items previously uploaded via Manage Curriculum and make them the program's default key and enabling competencies. 2. Curriculum Tag Options: Decide whether to map EPAs to Milestones or Enabling Competencies. Most programs will map their EPAs to Milestones.

It is possible to configure the Elentra database to prevent users from having the choice between using course-specific or standard key and enabling competencies. If you get to an option that ONLY asks whether you want to map to milestones or enabling competencies consult with your CBME implementation lead as to whether your organization should have the option to provide program specific KCs and ECs.

If you need to add new information to your EPAs, enabling competencies, or milestones templates, simply add new information to the existing template and upload it again in the appropriate spot. The system will identify any NEWLY ADDED EPAs/milestones/enabling competencies and append them to the existing EPA information.

Because of this feature, it can be valuable to keep a copy of the most recent file you’ve uploaded at all times.

If you need to correct a typo or minor error in an EPA, milestone, etc. you can now do so in through the user interface. Navigate to the Import CBME Data screen and follow the instructions on the screen to edit existing EPAs, milestones, etc.

To use supervisor and procedure forms, an organization needs to have rating scales in place. Elentra includes several rating scales but organizations can also add their own. There are two types of rating scales: Global Assessment and MS/EC which stands for Milestone Scale /Enabling Competency. There is no user interface to modify the scale type and both scale types are used on form templates including supervisor and procedure form templates.

There is a user interface to manage rating scales but only System Administrators and users with Medtech>Admin group and role can modify the available scales. Note that scales are applied to an entire organization and can be accessed by multiple programs.

Modifying Scales

Navigate to Admin>Assessment and Evaluation. Select the Scales tab from the top tabs list. Any existing rating scales will be displayed. To add a rating scale, click the green add button. Create a scale title, pick a scale type from the dropdown options, and then add descriptors to your response categories. Add or remove responses using the small plus and minus symbols in the top right of the responses categories section. Once scales are created, they will become visible options when creating forms.

If you don't see the Response Categories you want to include in your scale, assessment response descriptors can be added through the Systems Settings menu. This can only be completed by a Medtech:Admin user.

Auto-populated Item Text in Global Rating Scale Selector

Elentra is designed to automatically populate the item text when you select a global rating scale. Using different scales will result in different questions populating the form. See examples below.

There is no user interface to configure the item text used with each scale. This is controlled through the database; if you need to create a scale and will be using it as a global rating scale please speak to your project manager ora developer.

A PPA Form is designed to capture longitudinal, holistic performance trends.

If you create a PPA Form and at least one item on the form is linked to an EPA the form will be triggerable by faculty and learners once published.

Creating a PPA Form

From the Admin menu, select Assessment and Evaluation. Click on Forms from the subtab menu and click Add Form. Provide a form name and select PPA Form from the Form Type dropdown menu; then click Add Form. Provide additional form information noting that anyone added under permissions will have access to edit the form before it is in use. Select the relevant contextual variables for this form. Adjust which contextual variable responses should be included by clicking on the gray badge. This allows you to deselect unneeded contextual variable responses which can make the form faster to complete for faculty. Select an entrustment rating (if required) by entering item text, and selecting a scale. Note that the response options will be configured based on the scale you select. Set requirements for comments noting that if you select Prompted comments you should also flag which response options will result in prompted comments. Add form items by clicking Add Items or the down arrow to the right. You can either add existing items or create new items and add them to the form. Remember that every item created can be mapped to an EPA and milestones. The mapped EPA information about each item will be used to provide access to this form when a learner or faculty member triggers an assessment. Preview and save your form to return to it later. When your form is complete click Publish to make it available to be triggered from the CBME dashboard.

Rubrics are assessment tools that describe levels of performance in terms of increasing complexity with behaviourally anchored scales. In effect, performance standards are embedded in the assessment form to support assessors in interpreting their observations of learner performance.

If you create a rubric form and at least one item on the form is linked to an EPA the form will be triggerable by faculty and learners once published. Results of a completed rubric form are included on a learner's CBME dashboard information.

Creating a Rubric Form

From the Admin menu, select Assessment and Evaluation. Click on Forms from the subtab menu and click Add Form. Provide a form name and select Rubric Form from the Form Type dropdown menu; then click Add Form. Provide additional form information noting that anyone added under permissions will have access to edit the form before it is in use. You may wish to include the program in the permissions field so that you can filter by this form type later on. Select the relevant contextual variables for this form. Adjust which contextual variable responses should be included by clicking on the gray badge. This allows you to deselect unneeded contextual variable responses which can make the form faster to complete for faculty. Select an entrustment rating by entering item text, and selecting a scale. Note that the response options will be configured based on the scale you select. Set requirements for comments noting that if you select Prompted comments you should also flag which response options will result in prompted comments. Add form items by clicking Add Items. You can either add existing items or create new items and add them to the form. Remember that every item created can be mapped to an EPA and milestones. The mapped EPA information about each item will be used to provide access to this form when a learner or faculty member triggers an assessment. Preview and save your form to return to it later. When your form is complete click Publish to make it available to be triggered from the CBME dashboard. Rubric forms can also be scheduled for distribution through the Assessment and Evaluation module.

Note that when a rubric form is completed its results will be included in the learner dashboard.

Sample Rubric Form

Learners

CBME learners will be taken to their CBME Dashboard upon logging into the system. From there, a learner can click the green 'Trigger Assessment' to begin. The learner can select an assessor from the dropdown menu, fill in the date, and select the appropriate assessment method. A reminder that if you want to test this feature and use the complete via pin option, you should set up the PIN for the faculty member(s). (See next lesson.) Learners access assessment forms by selecting an EPA. For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

After an EPA is selected, the available assessment tools will be displayed. Learners can click 'Begin Assessment' to start.

Faculty

From the Dashboard, faculty have access to a green 'Trigger Assessment' button on the right side of the screen. Users can select a resident and then an EPA. For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

After an EPA is selected, the available assessment tools will be displayed. Faculty can click 'Begin Assessment' to start.

After a program has uploaded an initial set of contextual variable responses, they can be managed through the CBME tab.

Navigate to Admin>Manage course and select a course. Click CBME and then click the Manage Contextual Variable Responses tab below.

If contextual variable response information has been uploaded for the program, you will see a list of contextual response variables.

To modify the contextual variables within a category, click the title of the contextual variable or click 'Show' to the right of the category you want to edit.

To edit an existing response, make the required change and click the green disk icon.

To delete an existing response, click the red minus button beside the response.

To add a new contextual variable response, click the black plus button at the bottom of the category window. This will open a blank response space at the end of the list. Fill in the required content and save.

Troubleshooting

If you get a yellow bar across the screen when you try to modify contextual variables it means none have been uploaded for the program you are working in. Return to the Import CBME tab and complete Step 4.

Form templates and forms behave fairly similarly in terms of how you create and manage them. Specific details for each form type are included in other lessons but here is some general information.

• Form Permissions: These permissions dictate who has access to and can edit a form or form template while it is still a draft. To quickly make a form available to multiple users consider adding a program permission to the form (that will give anyone affiliated with that program (e.g. PA, PD) and with access to the administration of forms access to that form).

• Form Instructions: To include instructions on a form template, look in the Form Template Information section and click the box beside 'Include Instructions.' This will open a rich text editor where you can type. What you type here will be included on all forms generated from this template. You could include specifics about the use of a form, the number of assessments a learner must complete, or even the key features of an EPA. Remember that each form produced will have the same instructions so if you include key features make sure they only relate to the applicable EPAs for the template.

• Publish a Form: To make a form template generate forms to be available for learners or faculty to trigger you must publish the form template. Most installations of Elentra run a behind-the-scenes action to publish form templates every hour or so.

• Editing a Form: Once a form template is published you cannot change the content of the resulting forms (you can change permissions or rearranged item order on published forms). If there is an error on a form you will need to copy the form, correct the error, and publish your new form. You can delete old forms but note that any already completed assessments will remain on the learner dashboard.

If you are building and testing supervisor forms, you'll want to set up the faculty PIN for some sample users so that your test forms can be completed and you can view progress on the CBME dashboard. When CBME is live for your organization, users will need to set their own pins in order to easily complete forms at the point of encounter. Faculty and learners can set PINs. This allows residents to complete peer assessments. Learners will only be able to trigger assessments on themselves and have a peer complete the form via PIN; residents cannot trigger forms on other learners.

Setting the PIN

To set a PIN, log in as the necessary user. Access the user's profile by clicking on the user's name at the top of the screen (to the right of the logout button). Once in the user's profile, click on the Account Information tab. Beside PIN enter a 4 to 6 digit PIN. Click 'Save Profile'. Done!

To modify existing EPAs and milestones navigate to the CBME tab of a program page. In the appropriate step (e.g. Import EPAs or Import Milestones), look for "Click here if you wish to make changes to existing EPAs" and click on the blue text.

This will open a window where you can edit the text, description, and entrustment of EPAs. Note that this should only be used to make small modifications to this information and should not change the meaning of the text.

If you edit an EPA, it will be updated automatically across the system including on all completed forms. If you edit any milestones, you will have to republish forms on which they are used. Edits made to milestones will not be made on already published forms.

After sufficient information has been added to the system, a user can generate a map showing the connections between EPAs, roles, key competencies, enabling competencies, and milestones. This information can be access through the EPA Encyclopedia or a program's CBME page.

EPA Encyclopedia

From the dashboard, access the EPA Encyclopedia from the left sidebar.

If you have access to multiple programs, set the program using the dropdown menu in the top left.

Search for an EPA by typing in its code, or scroll down to find an EPA.

After clicking on an EPA, click on Detailed Description, Entrustment or Program Map to view details about the EPA.

Viewing Maps from the CBME page

Navigate to Admin>Manage course and select a course. Click CBME and the Map Curriculum tab.

Scroll down to click on an EPA or type the beginning of the EPA code and title into the dialogue box and click on the relevant EPA when it appears below.

From the Dashboard, faculty have access to a green 'Trigger Assessment' button on the right side of the screen.

First faculty select a resident. They can begin to type a name to limit the list of options. When they mouse over a resident name, faculty can see the resident card including their name, photo (if uploaded) and program.

If the learner is enrolled in two programs the faculty will have to specify a program.

A date of encounter will also have to be set.

Next faculty select an EPA.

For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia (see above).

Users can easily navigate the list of EPAs by applying preset filters including Current Stage EPAs, Current Rotation EPAs, and Priority EPAs. Click on a filter title to apply it. In this example the Priority EPAs filter is being applied.

After an EPA is selected, the available assessment tools will be displayed.

Note the small clock in the top right of each tool. This is an estimate of how long the form will take to complete based on the experience of other users.

Faculty can click 'Begin Assessment' on any tool to begin.

This feature allows residents to cognitively situate their assessors by adding an optional note to any assessment that will be emailed (complete and confirm via email; email blank form; self-assessment, then email blank form). It can be used to remind the assessor about specific case details, provide a focus for assessment, or anything else that the resident feels the assessor should know before completing the assessment. This note will stay attached to the assessment and will be visible from the CBME dashboard for reference.

This cue will stay attached to the assessment and will be visible from the CBME dashboard for reference.The cue is optional and if provided it will appear at the top of the assessment for the assessor. The cue can also be seen on the completed assessment for both the resident and the faculty member.

• For “Email blank form,” the cue modal will pop up from the Trigger Assessment page after “Send Assessment” has been clicked.

• For “Complete and Confirm via Email” and “Self-assessment, then email blank form”, the cue modal will pop up after clicking “Submit and notify attending by email” at the bottom of the form.

Staff>Admin. users have the ability to completely reset a program's CBME data. This is a dramatic step and should be used judiciously. Resetting CBME data will delete all EPAs, competencies and milestones for a program and they will have to be reloaded. Additionally, all existing forms built for the program will have to be rebuilt.

Resident Data

If you reset CBME data for a program, all existing, completed forms remain in existence. They will display on a learner's Assessment and Evaluation tab but will not display on the CBME dashboard Stages tab as the dashboard Stages tab will reflect data that relates to the newly loaded EPAs.

A major change for CBME in Entrada ME 1.12 is the ability to create assessment plans for a program. Currently these simply house information about the requirements for each program, but eventually reporting tools will be built to reflect the assessment plan information.

The assessment plan allows you to set minimum completion requirements for the different assessment tools built for a program. These minimums can apply to the number of assessments, the number of assessors, and specific contextual variable responses.

Please note that currently the assessment plan does not apply to procedure forms.

You need to have EPAs mapped and forms built before you can create an assessment plan.

There is a tool to allow admin users to reopen a completed assessment and edit it as needed. This is a feature that can be turned on or off depending on how your organization wants to use it.

If enabled, staff>admin can access this feature by navigating to their Assessment & Evaluation badge (beside their name in the top left), clicking on My Learners and then the Assessments tab for a specific learner. From the list of Tasks Completed on Learner click on a task to view it. An orange "Reopen Task" button will be above the form. This will set the task to in-progress and the staff>admin or faculty will be able to adjust it. Note that once a task is reopened you can clear all existing form data and forward the task as needed to send an email to faculty alerting them to the new task status.

Use with caution!

This tool should be used judiciously to ensure that residents are not ”gaming” the system and bullying anyone into changing their assessments to be more favourable.

A supervisor form is used to give a learner feedback on a specific EPA and can be triggered by a learner or supervisor. Once an EPA is selected, the form displays the relevant milestones to be assessed. A supervisor can indicate a learner’s progress for each milestone that was observed and can provide a global entrustment rating. Comments can be made optional, prompted or mandatory in each section of the form.

When you create a supervisor form template and publish it, the system automatically looks at the EPAs, milestones, and contextual variables selected and generates the appropriate number of forms. If you kept 3 EPAs on the supervisor form template, the system will generate 3 unique forms (one per EPA) that are available to be triggered by a user.

Accessing Form Templates

Ensure you are in a Program Coordinator or Program Director role. Navigate to Admin>Assessment and Evaluation. Click ‘Form Templates’ on the tab menu. Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear. Type in a form name and select the form type from the dropdown menu. Select ‘Supervisor Form’. If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.

Form Template Information

Edit the form name if needed, and add an optional description. Descriptions will be displayed at the top of forms published from this template. Often people include assessment guidelines in this section. Set permissions for the form template by selecting a category from the dropdown menu (individual, organization, or program) and then beginning to type in a name. Click on the appropriate name when you see it. Added users will appear in a list below the permissions search field. These permissions dictate who has access to and can edit the form template while it is still a draft.

Add additional text at the beginning of the form by clicking the small tick box beside ‘Include Instructions.’ This will open a rich text editor where you can enter text, images, hyperlinks, etc.

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms generated from this template.

All EPAs assigned to a course are included on the template by default. To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

Clicking the grey badge beside an EPA will display a list of all the milestones mapped to that EPA and by default, all are checked off. To remove specific milestones from this template, deselect them to remove the blue checkmark and then click the blue ‘Save and Close’ button. Deleting unnecessary milestones is one way to reduce the length of the form and reduce the time required to complete it.

Modify the milestones for each EPA as needed. The system does not enforce a maximum number of selected milestones but the Royal College recommends 8 milestones per EPA.

After you have modified the EPAs and milestones as required, click the small save button in the top right section of the EPA Selection box.

Contextual Variables

If you want all EPAs to have the same available contextual variables leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes. You may only select between 1 and 6 contextual variables per EPA per supervisor form.

By default, all of the options within a contextual variable are included on any forms made from the template. You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable. This will open a list of contextual variable responses that by default are all checked off. To remove specific responses from this template, deselect them to remove the blue checkmark. For convenience, you can also use ‘Check All’ and ‘Uncheck All’. When you have made the required changes, click the blue ‘Save and Close’ button.

When you modify which contextual variable response options will be available on a template, the number in the grey badge will show how many responses have been included out of the total possible responses.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

If you don't have scales configured you can access them through the Assessment and Evaluation tab.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The default response feature allows you to prepopulate a form with the selected response.This can be useful if there will be a significant number of milestones that will be included on a form.

The Responses fields will be automatically populated depending on the scale selected in the first dropdown menu.

After you have modified the Enabling Competency/Milestone Scale Selector as required, click the small save button in the top right section of the Enabling Competency/Milestone Scale Selector box.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections.FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed except by a developer.

Publish

Click 'Publish' to make your template available for use.

Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited. The number of forms that can be created from a template depends on the number of EPAs assigned to the template.

Residents should be stored in course lists in order to easily add them as the audience of a course. Course lists are managed through Admin>Manage Cohorts. We recommend that you create a course list for each academic year and populate the course list with all learners, CBME and non-CBME, who will participate in that program for the specified time.

If you are syncing your users with another source, your course lists may be automatically populated. It will depend on the script written by developers to connect Elentra to your other source of user information.

When you create courses, you can set the Course List as the audience for the course. It is important that learners be enrolled in a current curriculum period for them to be able to trigger forms and have forms triggered on and to them.

To promote a learner to a new stage, log in as a Competence Committee member, and click on the Assessment and Evaluation badge at the top of the page (beside the green logout button). Select the 'My Learners' tab and then click on the CBME Dashboard tab of the learner you want to promote. Beside each item you'll see a small grey circle; click on the circle to open a dialogue box where you can enter text and mark an EPA or stage as complete. You can also click 'View History' in the dialogue box and see any previous comments or decisions regarding learner promotion (or demotion).

Accessible by PDs and PAs from the CBME Dashboard Stages tab, the Milestone Report allows faculty and PAs to generate a milestone report for a specific learner. This report is a breakdown of completed assessments that have been tagged to milestones and EPAs. Assessments are tallied based on the milestone and EPA that they are tagged to and the tool generates one report per unique scale. The number of unique scales is determined by what tools were used to complete the included assessment tools. The reports are generated as CSVs and are zipped into a folder.

From a learner's CBME Dashboard Stages tab, click Milestone Report in the top right (beside Log Meeting).

1. Select a date range for the report. 2. Click “Get Tools" 3. Select the tools that you wish to view the aggregated milestone report for, or click “Select All Tools" 4. Click “Generate Report”. This will open a download modal for you to select where to save the zip file. 5. Unzip the file 6. If multiple rating scales were used to assess the milestones, there will be one CSV file for each rating scale 7. Open a CSV file 8. Each row represents a milestone, and each column represents the rating scale options. These rating scale options are grouped by EPA (e.g., for the Queen’s Three-Point Scale, you will see 4 columns for each EPA: Not Observed, Needs Attention, Developing, and Achieving). 9. Each cell displays the total number of times that milestone was assessed for that EPA, including how many times it was rated at that level on the rating scale (e.g., 3 of 6 completed assessments were rated as “Achieving”.

NOTE: Even though you may have selected only one of many Supervisor Tools (or other tools), if you used the same scale on all tools, the report will display all data for all tools that used that rating scale. We will be enhancing this in future iterations to only report on tool(s) selected.

Building and using assessment tools for learners and faculty to access is an important step in using the CBME tools. There are different ways to create and access forms in Elentra.

Two options for building forms exist:

• Building forms with a template (e.g. supervisor, procedure form)

• Building forms without a template (e.g. generic or rubric form)

Using a form template allows Elentra to automatically generate forms and have them available to be triggered by faculty and learners at any point. Form templates have defined items included and you can not add additional items to a form template. Examples of form templates are the supervisor form, procedure form, and field note form.

You can also build forms without a template which gives you more flexibility. Examples include the generic form, rubric form, and periodic performance assessment form. The rubric and PPA have some required elements but you can also add additional items to these forms. As long as one item added to a rubric or PPA form is linked to an EPA the form will be triggerable by faculty and learners.

If a form is not something that a faculty or learner will trigger themselves, program administrators can create forms and send them out to be completed via distributions. A distribution defines who or what is being assessed/evaluated, who is competing the form, and when it is to be completed. Building items and forms to use with distributions is not a requirement if you only want to use triggerable forms.

Residents can highlight assessments that they found helpful to their learning. From the “Assessments” tab of the CBME Dashboard, a learner can click on the “thumbs up” icon to indicate to an assessor that their feedback was helpful to the resident's learning. Leaners can also include a comment on why they found it helpful. This feedback is important to help assessors identify the types of feedback that residents find beneficial for their learning.

Learner and Faculty Triggered Assessment Forms Viewed from CBME Dashboard

Learners and their affiliated faculty and program administrators can track assessment form completion by navigating to the learner's CBME dashboard.

Click on the Assessment and Evaluation badge at the top of the page. Click the My Learners tab and click on CBME Dashboard below the relevant learner name.

From the learner's CBME dashboard click on Assessments. Scroll down past the Filter Options until you see a set of tabs including Completed, In Progress, Pending, and Deleted. Choose the appropriate tab to review assessment form completion.

Tracking Form Completion From the Learner's Assessments Page

Click on the Assessment and Evaluation badge at the top of the page. Click the My Learners tab and click on Assessments below the relevant learner name.

These pages provide information on tasks triggered by faculty and learner's as well as tasks assigned through distributions and provide access to some reporting. For more information see the Reviewing Progress>Assessments Page help section.

Tracking Form Completion From a Faculty's Assessments Page

PAs can view a faculty's tasks from the Assessment and Evaluation tab. It works almost the same as the learner's assessment page but is accessed from the Faculty tab.

Monitoring Progress of Tasks Created via Distributions

When a PA sets up assessment and evaluation tasks to be completed via distributions, progress can quickly be viewed via the Assessment and Evaluation module.

Faculty and program administrators can log meetings to maintain a record of conversations about learner progress. Either user needs to navigate to a learner's CBME dashboard to access the Log Meeting button.

Learners have quick access to the their own meeting logs from the CBME dashboard My Meetings button.

Logged meetings display on a meeting log accessible to faculty and learners. Note that only faculty and PAs can add and edit meeting information. Learners have read only access to their meeting log.

Posts can have files uploaded to them, be edited or be deleted using the tools in the Actions column. A program admin., academic advisor or competence committee member can create or add to any entry they have made (but not those made by other users).

Meeting Logs are accessible to competence committees and may be referenced when committees meet to discuss learner progress.

Learners or faculty can pin assessments or individual comments to keep them easily accessible for review during meetings. This can help to keep an important piece of feedback or other information front and centre until it has been discussed.

How to pin something

To pin an assessment, simply click on the pushpin icon that appears to the right of the assessment title and details. You'll get a success message that you've pinned the item. In the example to the left, the second and third assessments have been pinned.

To pin a comment, click on the Comments tab from the CBME Dashboard and then click the pushpin beside the desired comment. In the example to the left, the second comment has been pinned.

Viewing Pinned Items

From the CBME Dashboard, click on Pins at the end of the tabs menu. This will open a screen showing all pinned items. To unpin something, just click the pushpin. You'll see a success message that you've unpinned the item.

A program needs to complete this template OR the Enabling Competencies Template, not both. Most programs will complete the Milestones Template. Users define which option they will use in in Step 2 of the import CBME process.

Navigate to Admin>Manage course and select a course. Click 'CBME' and 'Import CBME Data' to access and download the template.

Three Columns to Complete

1) EPA code:Codes should be recorded in uppercase letters. Codes should refer to the stage of training and the EPA number, e.g. D1, D2, D3, F1, C2. Using consistent codes across your program and school is important. The system is setup to accept the following codes for learner stage: D: Transition to Discipline F: Foundations of Discipline C: Core Discipline P: Transition to Practice

2) Milestone code: Codes should be recorded in uppercase letters. The format for the milestone code is: Leaner Stage letter, followed by a space, CanMEDS Role letters, Key Competency number, followed by a period, Enabling Competency number, followed by a period, Milestone number

Note that there should be no space between the CanMEDS Role letter and the Key Competency number. CanMEDS Role refers to the CanMEDS roles and should be coded as follows:

Professional PR Communicator CM Collaborator CL Scholar SC Leader LD Advocate HA Medical Expert ME

Creating and confirming your Milestones codes takes patience. You'll likely notice that in some RC documents there is only a two digit milestone code. For the purposes of mapping your curriculum in Entrada, you must have three digit milestone codes. We recommend you add the third digit in the order the milestones appear in the RC documents you're using. Make sure you check for duplication as you go so that unique milestones have their own codes but different codes aren't applied to repeating milestones (within one stage). Using the data organization tools to reorder the milestone code columns and title columns can help you identify unneeded duplication.

You may also notice that the RC allows programs to use milestones coded from one stage in another stage (so you may see an F milestone in a C stage EPA). How programs and organizations handle this is ultimately up to them but we recommend that you align the milestone code with the stage that it is actually being assessed in/mapped to. For example, if it’s a “D” milestone being mapped to an F EPA, rename it as an “F” milestone. This is because you are (presumably) having a different expectation of performance on the milestone in a different stage, even if it’s the same task.

3) Title: The text of the milestone should be copied verbatim from the national specialty committee EPAs.

Mapping multiple EPAs to one milestone

The system is designed to allow multiple EPAs to map to one milestone. To do so, simply insert a comma between the EPA code entries in column one. e.g. D1,D2,F1

Save as a .csv and upload

After saving your file as a .csv, scroll to step 3 on the Import CBME screen and either search for or drag and drop the file into place. You'll see a green checkmark when your file is successfully uploaded.

A field note form template is used to give a learners narrative feedback about their performance.

When you create a field note form template and publish it, the system automatically looks at the EPAs and contextual variables selected and generates the appropriate number of forms.

Accessing Form Templates

Ensure you are in a Program Coordinator or Program Director role. Navigate to Admin>Assessment and Evaluation. Click ‘Form Templates’ on the tab menu. Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear. Type in a form name and select the form type from the dropdown menu. Select ‘Field Note Form’. If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.

Form Template Information

Edit the form name if needed, and add an optional description. Descriptions will be displayed at the top of forms published from this template. Often people include assessment guidelines in this section. Set permissions for the form template by selecting a category from the dropdown menu (individual, organization, or program) and then beginning to type in a name. Click on the appropriate name when you see it. Added users will appear in a list below the permissions search field. These permissions dictate who has access to and can edit the form template while it is still a draft.

Add additional text at the beginning of the form by clicking the small tick box beside ‘Include Instructions.’ This will open a rich text editor where you can enter text, images, hyperlinks, etc.

Template Components Section

EPA Selection

Specify which EPAs can be assessed using forms generated from this template.

All EPAs assigned to a course are included on the template by default. To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

After you have modified the EPAs as required, click the small save button in the top right section of the EPA Selection box.

Contextual Variables

If you want all EPAs to have the same available contextual variables leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

Select which contextual variables you want to include with which EPAs by checking and unchecking the tick boxes.

By default, all of the options within a contextual variable are included on any forms made from the template. You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable. This will open a list of contextual variable responses that by default are all checked off. To remove specific responses from this template, deselect them to remove the blue checkmark. For convenience, you can also use ‘Check All’ and ‘Uncheck All’. When you have made the required changes, click the blue ‘Save and Close’ button.

When you modify which contextual variable response options will be available on a template, the number in the grey badge will show how many responses have been included out of the total possible responses.

After you have modified the contextual variables as required, click the small save button in the top right section of the Contextual Variables box.

Free Text Feedback to Resident

All field note form templates include a Continue and Consider section in which faculty can record comments to provide feedback to learners. These sections cannot be edited in the Field Note Form Template.

Global Rating Scale Selector

The Item Text area will be automatically completed depending on the scale you select.

From the first dropdown menu, select a Global Rating Scale.This will populate the Item Text and the Responses sections. FYI: Rating Scales can be managed from Admin>Assessment & Evaluation, Scales tab. If the Item Text doesn't populate after you have picked a scale, some additional configuration may be required in your database. Please see the Scales help section.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The Responses fields will be auto-populated depending on the scale selected in the first dropdown menu.

After you have made your choices for the global rating scale, click the small save button in the top right section of the Global Rating Scale Selector box.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed except by a developer.

Publish

Click 'Publish' to make your template available for use.

Once a form template has been published, forms created from it will live on the resident dashboard and can no longer be edited. The number of forms that will be created from a template depends on the number of EPAs assigned to the template.

To review feedback provided about the assessment tools themselves, go to Admin>Assessment and Evaluation, and click Reports from the second tab menu (look under the Assessment & Evaluation header). Under Assessments, click on Assessment Tools Feedback Report. CBME form feedback will show up there. You can filter by course and form to refine the list of feedback. If you are logged in as a PA you will only have access to forms associated with your program.

In addition to the CBME Dashboard, users can access a learner's Assessments page to view tasks completed on the learner, as well as tasks assigned to the learner. These pages reflect forms completed via distributions and CBME forms triggered by faculty and learners, however the reporting tool accessible from Tasks Completed on Learner applies only to forms managed via distributions (for reporting on triggered CBME forms please see the CBME Dashboard page).

Accessing the Assessments Page

When logged in as a faculty member or program administrator, click the Assessment & Evaluation badge that appears between the user's name and the logout button in the top right. Click on 'My Learners' from the tab menu. Search for a learner as needed, and if you can't find someone, ensure that you are in the correct curriculum period using the dropdown menu on the right. After finding the correct learner, click on Assessments on the resident card to view the learner's assessments page.

Note that program administrators and program directors can use these same steps to access the assessments page for faculty.

Assessments Page Contents and Filters

Multiple tabs are available on the Assessments page. Each tab displays a collection of assessment tasks. CBME specific forms are included on this page and so are any forms created through the Assessment and Evaluation module and sent to users through a distribution.

On each assessment card you will see a range of information which may include the form title, form type, delivery and completion dates, progress (e.g., 3 of 14 complete), who triggered the form , the target, and the assessor/evaluator (including their group and role). Not all of this information is included on every task card.

On each tab you can search for tasks, set a date range and apply filters. Click the down arrow beside the search tasks bar to select a filter type and drill down to the necessary filter. Select a filter and click the green Apply Filters button. Your applied filters will display on an Active Filters list.

Tasks Completed On Learner

These are tasks completed by assessors on the learner.

Each task can be downloaded or viewed. To download tasks, click the appropriate box at the bottom of the task card and then click Download PDF(s) at the top of the page. Note that there is a Select All tool available. To view a task click View Task at the bottom of the task card.

More tasks can be displayed by clicking Load More Tasks at the bottom of the page.

Pending Tasks on Learner

From this tab you can view all pending tasks on the learner this includes tasks that have been distributed but are not yet started. To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Upcoming Tasks on Learner

This tab displays scheduled distributions where the learner is the target.

To remove a task from this list click Remove Task and provide a reason for the removal. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Current Tasks

This tab displays active distributions where the learner is an assessor/evaluator.

To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Upcoming Tasks

This tab shows tasks scheduled through a distribution where the learner is an assessor/evaluator. Tasks can be removed from this list by clicking Remove Task and providing a reason for the removal.

Reports for this Learner

From the Tasks Completed on Learner tab, users can access Reports for this Learner. This gives you access to aggregated reports on a learner for each form used. Note that these reports are available for forms managed via distributions, not those triggered by learners and faculty.

Set a report date range if desired.

From the Options dropdown menu you can: 1. Choose to generate the report with or without comments (both can then be downloaded as a PDF); 2. Click View This Form to open a new tab that shows the form; and 3. Click View Assessments to open a tab that lists all completed assessments for this form and provides links to view each one.

Clicking the 'Group by distribution' checkbox will reorganize the forms to reflect completion based on distribution and add a column to the display to show distribution information.

Faculty, program administrators and residents can easily review a learner's progress from the CBME dashboard.

Note that the CBME dashboard includes several tabs: Stages, Assessments, Assessments Items, Trends, Comments, and Pins. (Reports can be generated from the Stages page.)

There is another Assessments Dashboard that pulls in additional form information if your organization is using the Assessment and Evaluation module for other forms (e.g. program and faculty evaluations). This page discusses just the CBME Dashboard.

Accessing CBME Dashboards

When logged in as a faculty member or program coordinator, click the Assessment & Evaluation badge that appears between the user's name and the logout button in the top right. Click on 'My Learners' from the tab menu. Search for a learner as needed, and if you can't find someone, ensure that you are in the correct curriculum period using the dropdown menu on the right. After finding the correct learner, click on CBME Dashboard to view the learner's progress. Residents automatically land on their CBME Dashboard when they log into Elentra.

Reviewing Learner Progress

From a learner's CBME Dashboard, click through the different tabs to view a range of assessment information. On most tabs, you can apply filters in order to refine the list of assessments visible to others. To use this option, select all the appropriate filters (e.g. date, curriculum tag, contextual variable, assessment tool, etc.) and then click 'Apply Filters'. Note that the filters a PA or PD applies on one learner will be maintained as you move through different pages.

Stages

From the stages tab you can see a summary of a learner's progress across EPAs and stages.

EPAs are displayed in order of learner stage and completed stages can be collapsed. If you are using the rotation scheduler, EPAs specific to a learner's current rotation are outlined in the stage colour and the priority and likelihood of an EPA in the learner's specific rotation is shown through the exclamation mark and bar chart (unlikely, likely, very likely). Whether or not the rotation scheduler is in use, green checkmarks indicate that a stage or EPA is complete (this is set by the Competency Committee).

Click the down arrow on the right side of an EPA card to see a list of completed assessments for that EPA. Depending on the form there may be a count of the learner's global assessment rating, which you can click on to access an aggregated report of the learner's performance on this form.

When in a faculty or PA role you can also access a Milestone Report from the Stages tab. More details about the Milestone Report are in another lesson.

Stages Tab: Reporting Options

Users can produce an aggregated report for any form displaying on this page. Click the grey down arrow on the right side of a EPA card, and click on a form title. A new tab will open to display the aggregated data with a trend chart at the bottom. On the trend chart at the bottom of the report hover over any individual assessment to click 'View Assessment' and see the specific assessment form. Note that you can download a PDF of this form as needed (green button at top right). If you want to look at only the forms completed with a certain global rating click on a blue number on the right hand side.

Assessments

See a list of all completed assessments and filter as desired. Toggle between completed, in progress, pending and deleted assessments below the filter settings. On each individual assessment card, note that the form type and relevant EPA are displayed. You can also click the down arrow on the right to show some form information (e.g., global rating, comments, assessor and assessment method), and click 'View Details' on the left to see the assessment form. Users can mark an assessment as read by clicking on the eye icon. A greyed out eye means an assessment is unread. There is an option to mark all assessments as read on the right side above the assessment cards. You can also quickly view all unread assessments by clicking the small blue number beside 'Assessments' in the tab menu. Marking assessments as read or unread is user specific so learners and faculty will see their own results. Users can pin an assessment from this screen and learners can give an assessment a "thumbs up" to provide feedback to an assessor about their feedback.

Assessments Items

Quickly see a list of all completed assessment items and filter as desired. Click the down arrow on the right to see comments, response descriptor and the name of the assessor (as applicable). Click View Details at the bottom left of each card to access the completed assessment form.

Users can pin an assessment item from this screen.

Trends

View trends in learner performance across global assessment ratings. Note the overall tally of ratings in the top right corner of each card. Hover over on a node on the chart to view information about the form name, date and rating; you can also click through to access the actual assessment form.

Comments

Quickly access a list of all narrative comments provided to the learner on any complete CBME assessment tool. The tool type and relevant EPA will be displayed below the form name on each comment card.

Users can pin comments from this tab.

Pins

Quickly view all assessments, items, or comments that have been pinned. Apply filters as desired, or just scroll down to find the toggle to switch between assessments, items, and comments.

From the Dashboard, learners have access to a green 'Trigger Assessment' button on the right side of the screen.

First learners select a faculty. They can begin to type a name to limit the list of options. When they mouse over a faculty name, learners can see the faculty card including their name, photo (if uploaded) and program.

Next, learners select an assessment method. Details about each assessment method are provided inline.

Note that faculty must have a PIN setup for learners to select the first option.

A date of encounter will also have to be set.

Next learners select an EPA.

For a reminder on what is included in a specific EPA the black question mark provides a quick link to the EPA Encyclopedia.

Users can easily navigate the list of EPAs by applying preset filters including Current Stage EPAs, Current Rotation EPAs, and Priority EPAs. Click on a filter title to apply it. In this example the Priority EPAs filter is being applied.

After an EPA is selected, the available assessment tools will be displayed.

Note the small clock in the top right of each tool. This is an estimate of how long the form will take to complete based on the experience of other users.

Learners can click 'Begin Assessment' on any tool to begin.

Depending on the method of assessment learners will either be directed to the assessment form to start it, or the form will be sent to the assessor.

Sample: Submit via PIN

In the first assessment method, learners will be able to save their form as draft or submit it to the assessor to confirm by PIN.

Different assessment methods will provide different options.

PAs can enter complete assessment forms on behalf of assessors.

Navigate to Admin>Assessment & Evaluation. Click on the green 'Record Assessment' button below the orange Assessment Tasks heading. Select a resident and assessor from the dropdown menus and provide a date of encounter. Select an EPA as you would to trigger a form. Search for the appropriate form and click Select. Complete the form and click Submit. The PA will be submitting the assessment on behalf of the selected assessor. There is a reminder of the selected assessor displayed at the top of the form in a yellow information bar.

Program Administrators and Faculty Directors have the ability to get an overview of pending and in progress tasks from Admin>Assessment and Evaluation. When users access this page they can now view 'Outstanding' tasks as well as 'Upcoming' and 'Deleted' tasks.

Viewing Outstanding Tasks

1. Users can filter for distributed forms or triggered forms.

2. Users can search for specific forms, or owners.

3. The 'Owner' is the person who is responsible for completing the form.

4. Hover over Targets to view the targets of the form.

5. Check the appropriate boxes and then send reminders as needed.

6. Delete tasks as needed.

Click on any form to complete it on behalf of another user.

(Please note, not all users will currently see the Trigger Assessment button included in this image.)

The list of users included in the Outstanding task list are those faculty listed as "Associated Faculty" on the program setup tab, faculty who have been included in a distribution in the course, and learners in the audience of a program enrolment (also configured on the program setup tab).