Building Form Templates for New EPA Versions.

1. Navigate to Admin > Assessment and Evaluation.

2. Click ‘Form Templates’ on the tab menu.

3. Click the green ‘Add Form Template’ button in the top right and a pop-up window will appear.

4. Type in a form name and select the form type from the dropdown menu. Select ‘Supervisor Form’. If you have permission to access multiple programs, use the dropdown menu to select the appropriate program for this form. This option will only show up if you have access to multiple programs.

Now that you have two (or more) curriculum versions in the system, the Form Template Builder will default to loading the most recent version. In the "EPA Version" tab, simply select the appropriate version. Click Save.

If you want to build new forms for learners using Version 1, change the EPA Version to Version 1, click Save, and it will load the appropriate EPAs.

EPA Selection

Select which EPAs can be assessed using forms generated from this template.

To remove EPAs, click on the small 'x' to the left of the EPA code. You can add back any required EPAs by clicking on the dropdown menu and checking off the tick box for a desired EPA.

Click the grey badge beside an EPA to select or remove specific milestones for forms built from this template.

Click Save.

Contextual Variables

If you want all EPAs to have the contextual variables, leave all EPAs checked off. If you’d rather specify which contextual variables apply to which EPAs simply uncheck an EPA and it will appear below with its own customizable list of contextual variables.

You can remove specific contextual variable responses by clicking on the grey button beside a contextual variable.

You may only select between 1 and 6 contextual variables per EPA per supervisor form.

Click Save.

Enabling Competency/Milestone Scale Selector

Use the first dropdown menu to select the scale you want to use to assess enabling competencies or milestones.

Indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

The default response feature allows you to pre-populate a form with the selected response.

Disabled - Comments are disabled at the milestone level.

Optional - An optional comment box will appear for each milestone. Comments are not required for a form to be submitted.

Mandatory - A mandatory comment box will appear for each milestone. Comments are required for all milestones before a form can be submitted.

Prompted - A mandatory comment box will appear only for the prompted responses indicated. This is a popular option for responses at the lower end of a scale.

Global Rating Scale Selector

From the first dropdown menu, select a Global Rating Scale. This will populate the Item Text and the Responses sections.

From the second dropdown menu, indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’ from the Comments dropdown menu, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

Publish Form Template

Click 'Publish' to make your template available for use. The forms will be available within the hour.

Once a form template has been published, you can rearrange the template components for each form; however, you cannot makes changes to the scales or contextual variables. To make these changes, copy the form template and create a new version.

Default Content

On each form template you create you’ll notice a greyed out area at the bottom including Next Steps, Concerns, and a place for feedback. This is default form content that cannot be changed.

Rubrics and PPAs

Within a given form, you can only tag curricular objectives (e.g., EPAs or milestones) from the same curriculum version. To ensure that you do not accidentally add an EPA from a different version, you must create the form first and then "Create & Attach" new items to the form.

Build a PPA or Rubric/Flex Form

1. Click Admin > Assessment & Evaluation.

2. Click on Forms from the subtab menu

3. Click Add Form.

4. Provide a form name and select PPA Form or Rubric/Flex Form from the Form Type dropdown menu; then click Add Form.

Now that you have two (or more) curriculum versions in the system, the Form Editor will default to loading the most recent version. Under "EPA Version", simply select the appropriate version. Click Save.

If you want to build new forms for learners using Version 1, simply change the EPA Version to Version 1 and it will load the appropriate EPAs.

Note: In order to use the "Programs" filter in the form bank, you need to add Program-level permissions to each form.

1. Click "Individual", change to "Program"

2. Begin typing in your program name in the "Permissions" box.

3. Click on your program to add it.

If applicable, select the relevant contextual variables. Click the grey badge beside a variable to select or remove specific responses.

Indicate whether comments are disabled, optional, mandatory, or prompted. By selecting ‘Prompted’, you can set the system to prompt and require a comment when any flagged response is selected by an assessor.

Feedback and Concerns items will be added when the form is published.

Add Items

Click "Add Item(s)" to add an item, or click the down arrow for more options. "Add Free Text" will add an instruction box. Do not use the "Add Curriculum Tag Set" at this time.

Create & Attach a New Item

Click "Create & Attach a New Item" to add an item.

Create New Item

Click "Create & Attach a New Item" to add an item. Select the Item Type and add any item responses, if applicable.

Tag Curriculum Objectives

Because you are using a form that is mapped to "Version 2", the curriculum tag sets will be locked to "Version 2". This will ensure that you do not accidentally tag an EPA from a different version.

Add Additional Items & Publish

Continue adding items as desired. When you have finished creating and attaching new items, click "Publish" to publish your new form.

Navigate to the CBME Sub-Module

1. Log in to Elentra.

2. Click "Admin" at top right

3. Click "Manage Programs"

4. Select your program.

5. Click "CBME"

6. Click "Import CBME Data".

Add New Version

1. Click "Actions"

2. Click "Add New Version"

Versioning Wizard Step 1

In this step, simply indicate which EPAs will be:

1. Replaced

2. Changing

3. Not Changing

4. Retired

Note: Each program can only have one version "in progress" at a time. You can continue editing that new version, or reset the progress.

Replaced

Select this option if you wish to replace the EPA entirely, including all milestones. This option will require you to have a complete set of CSV files for the objectives that are being replaced (all EPAs, all Milestones), rather than just the changes. Use this option if you have many changes to implement across EPAs and milestones.

Retired

Select this option to retire an EPA (remember, think in terms of EPA Code, e.g. "C11") from the next version. This means that there will be no new version of that EPA Code (C11) in the next curriculum version. If you include an EPA with the retired code in the spreadsheet, it will not get uploaded. You are unlikely to use this status for EPAs unless there is an EPA at the end of your numbering that is being removed. For example, you have D1, D2, D3, D4 currently. You are retiring D4 from the next version and no new D4 will be uploaded.

For milestones, a “retired” status will remove that milestone from that EPA in the new version. This means that there will be no new version of that milestone for that EPA. It will not retire it from other EPAs unless you explicitly tell it to.

Changing

The objective will be changing. This will allow you to upload new EPA and Milestone text. For a milestone (or competency), if you select “changing” for that milestone under one EPA, the system will update all other instances of that milestone to “changing”. This will ensure that you only have one version of that milestone.

Not changing

The objectives will remain unchanged. This exact version of the objective will be carried-forward into the new curriculum version. Note that selecting this option will stop any objectives with the same code from being uploaded.

Versioning Wizard Step 2

In this step, indicate which KCs, ECs, and Milestones will be: Changing, Not changing, or Retired.

If you marked all of your EPAs as "Replaced," you can click next without needing to go through each EPA.

PLEASE NOTE: If there are any milestones in a Changing EPA that you want removed, you MUST mark them as RETIRED. This is with respect to the Milestone code - if "C PR4.3.2" will no longer be in this EPA, mark it as retired.

If you would like to add a new Milestone to an EPA only, then just set the EPA to changing and nothing else.

Note that "retiring" a milestone will only retire it from that EPA and no others unless otherwise indicated.

Import CBME Data

Upload CSV file with new and/or changing EPAs

You will notice that the yellow box at the top will list all objectives that have been set to changing. You can still upload new EPAs and Milestones that were not already in the system.

Choose the file and click "Save and upload EPAs"

IF APPLICABLE: Upload Key and Enabling Competencies

Moving forward, we are recommending that programs upload the specialty-specific EC and KC files. This is especially important for programs that have added or removed KCs or ECs (e.g., Otolaryngology).

Upload CSV file with new and/or changing Milestones

Choose the file and click "Save and upload Milestones".

Some programs at Queen's only mapped to the Enabling competency level. If you will continue mapping to enabling competencies, and are not converting to milestones, skip the milestone upload and use Step 6 to upload your EC map. If you are converting to milestones, use the milestone upload.

When the upload is complete, click "Next".

Versioning Wizard Step 3

Confirm the Version

This overview lists all of the EPAs and their version status. When you are ready to publish the new version of the curriculum, click the blue "Publish" button.

REMINDER: Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. The green checkmark on the stage indicates that the learner has completed that stage. A learner is currently in the stage that comes after the last green stage checkmark. Failing to ensure that learners are in the correct stages will result in those learners receiving (or not receiving) new stages when they shouldn't have. If you do this by accident, there is a developer tool to reset the learners stages - please contact the support team if you need help with this.

As soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.

The new curriculum version will be available the next day. For most installations, new versions are published nightly.

Build and Publish Assessment Tools for the New Version

Now that you have finished creating your new curriculum version, please ensure that you create new assessment tools for any changed/replaced/new EPAs. After the published version is live, learners will not be able to be assessed on any new/changed EPAs until the new tools have been built. Remember that in most cases this will have minimal impact, since the learners only get the new version for the stages beyond the one they are currently in.

Please do not delete forms or form templates from previous versions unless you are certain that there are no longer any learners using that version.

Which learners will get the new version of the curriculum?

The new curriculum version will be applied to each learner at the stage level. This means that if a learner is currently in Foundations, they will retain the previous version of Transition to Discipline and Foundations but will be given the new version of Core and Transition to Practice. Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. Remember: a green checkmark on the stage indicates that the learner has completed that stage. A learner is currently in the stage that comes after the last green stage checkmark.

As soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.

What is considered a new EPA?

Think in terms of the EPA or Milestone code. A new version of an EPA is when:

1. The EPA title/description has changed sufficiently to change the context of the EPA, AND/OR

2. You are changing, adding, or retiring any milestones within the EPA, AND/OR

3. You are changing the EPA code association (e.g., you have retired an EPA in the middle of your numbering, such as D2, and now need to shift D3, D4, D5, into D2, D3, D4).

A new version of a Milestone is when:

1. The Milestone title has changed sufficiently to change the context of the Milestone.

What do the objective statuses mean?

Replaced

Select this option if you wish to replace the EPA entirely, including all milestones. This is the easiest option to work through the wizard. This option will require you to have a complete set of CSV files, rather than just the changes. Use this option if you have many changes to implement across EPAs and milestones.

Not changing

The objectives will remain unchanged. This exact version of the objective will be carried-forward into the new curriculum version. Note that selecting this option will stop any objectives with the same code from being uploaded. Any EPAs marked as not changing will have all existing tools carried over.

Changing

The objective will be changing. A new version of this EPA will be created, which may include objectives within it that you mark as changing.

For a milestone (or competency), if you select “changing” on one EPA, the system will update all other instances of that milestone to “changing”. This will ensure that you only have one version of that milestone.

Retire

The EPA (remember, think in terms of EPA Code, e.g. "C1") is being retired from the next version. This means that there will be no new version of that EPA Code in the next curriculum version. If you include an EPA with the retired code in the spreadsheet, it will not get uploaded. You are unlikely to use this status for EPAs unless there is an EPA at the end of your numbering that is being removed. For example, you have D1, D2, D3, D4 currently. You are retiring D4 from the next version and no new D4 will be uploaded.

For milestones, a “retired” status will remove that milestone from that EPA in the new version. This means that there will be no new version of that milestone for that EPA. It will not retire it from other EPAs unless you explicitly tell it to.

What do I need to do to prepare?

Please ensure that you have documented any changes that will need to be made, in addition to creating (or modifying) new CSV templates for upload. From the user interface, you will be able to indicate:

• Whether an EPA is being replaced, changing, not changing, or being retired

• Within each EPA that has been marked as changing, indicate if any KC, EC, or Milestones will be changing, not changing, or retired

• If you are adding milestones to an EPA, mark it as changing so that the uploader will add the new milestones

• Entirely new EPAs will be detected by the uploader without you needing to indicate anything. For example, if you currently have D1, D2, and D3, but will be adding a D4 and D5, the system will detect these in your spreadsheet and add them to the new curriculum.

• The new curriculum version will be applied to each learner at the stage level. This means that if a learner is currently in Foundations, they will retain the previous version of Transition to Discipline and Foundations but will be given the new version of Core and Transition to Practice. Please ensure that all of your learners are in the correct stage (on their CBME Dashboards) prior to publishing the new curriculum version. Remember: a green checkmark on the stage indicates that the learner has completed that stage.

Which CSV files do I need to have ready?

1. EPA Template

2. Milestone Template

3. If applicable, any new program-specific Key and Enabling Competencies

How will my assessors know which EPA version to pick for each learner?

Great news: When triggering an assessment, only the EPAs and associated forms relevant to that learner will be displayed. You will not have to choose between multiple versions for a given learner.

Do I need to build new assessment tools?

You only need to build new assessment tools for new EPAs or EPAs that have changed. Assessment tools will be carried over for all EPAs that were marked as "not changing." As noted above, as soon as you have published your new version, you will need to build tools for all of the new or changing EPAs before residents can be assessed on them.