To effectively use the various assessment tools in Elentra ME some initial setup is required.

For the Assessment & Evaluation module specifically, a staff:admin or medtech:admin will need to configure assessment response categories in System Settings. For information about configuring assessment response categories (required to build scales), please see the System Setup help section. A medtech:admin will have to configure scales through Admin > Assessment & Evaluation (see next section).

Note that assessment characteristics (in system settings), are used in the the Gradebook module to define assessment types like written exam, test, quiz, project, etc. You do not need to build a list of assessment characteristics to use the Assessment & Evaluation module.

The assessment and evaluation module provides a way for learners to be assessed, especially in clinical learning environments, and a way for all users to evaluate courses/programs, faculty, learning events, etc. Any forms used for assessment and evaluation require form items (e.g., questions, prompts, etc.).

How to create single items for use on assessment and evaluation forms

• Note that you can copy existing items which may save time. To copy an existing item, click on the item and click 'Copy Item' which is beside the Save button.

• Navigate to Admin>Assessment & Evaluation.

• Click 'Items'.

• Click 'Add A New Item'.

• Complete the required information, noting that different item types will require different information. Common item details are below: Item Type: This list shows the item types supported by Elentra. Item codes will display when you use a list view of items and A complete list of item types is provided below. Item Text: This is what will show up on a form this item is added to. When you view items in the detail view you'll also see the item text. Item Code: This is an optional field. Item codes do display when you view items in a list and they are searchable. Some organizations apply their own coding system to their items, but another use case might be if you are importing items from another tool or vendor and they have a coding system you want to match. Rating Scale: Rating scales can be configured through the Scales tab within Assessment & Evaluation. First select a scale type and then select the specific scale. Selecting a rating scale will prepopulate the response categories for this item. In some item types you will also be required to add response text (e.g. multiple choice items) and that text will show up on the actual form. In other question types you may rely on just the response categories. Mandatory: Click this checkbox if this item should be mandatory on any forms it is added to. Allow comments: Click this checkbox to enable comments to be added when a user responds to this item. If enabled, you have several options to control commenting.

  • Comments are optional will allow optional commenting for any response given on this item.

  • Require comments for any response will require a comment for any response given on this item.

  • Require comments for a prompted response means that for any response where you check off the box in the Prompt column, a user will be required to comment if they select that response.

  Allow for a default value: If you check this box you will be able to select a default response that will prepopulate a response when this item is used on any form. Set a default response by clicking on the appropriate response line in the Default column.

• Depending on the question type, add or remove answer response options using the plus and minus icons.

• Depending on the question type, reorder the answer response options by clicking on the crossed arrows and dragging the answer response option into the desired order.

• Add curriculum tags to this item as needed.

• If you have access to multiple courses/programs, first use the course/program selector to choose the appropriate course/program which will limited the available curriculum tags to those assigned to the course/program. Click the down arrow beside the course selector and search for the course by beginning to type the course name. Click the circle beside the course name.

• Click through the hierarchy of tags as needed until you can select the one(s) appropriate for the item.

• As you add curriculum tags, what you select will be listed under the Associated Curriculum Tags section.

• Scroll back up and click 'Save'.

Item types

• Horizontal Multiple Choice (single response): Answer options will display horizontally on the form and the user can select one answer. Response text required; response category optional. Response descriptors provide another data point so you can potentially report on them in the future. They are metadata in MC questions whereas in a rubric they are displayed. Horizontal MC will let you assign the same response descriptors to multiple responses.

• Vertical Multiple Choice (single response): Answer options will display in a vertical list on the form and the user can select one answer. Response text required; response category optional.

• Drop Down (single response): answer options will display in a dropdown menu. Response text required; response category optional.

• Horizontal Multiple Choice (multiple responses): Answer options will display horizontally on the form and the user can select two or more answers. Response text required; response category optional.

• Vertical Multiple Choice (multiple responses): Answer options will display in a vertical list on the form and the user can select two or more answers. Response text required; response category optional.

• Drop Down (multiple responses): Answer options will display in a dropdown list that remains open and allows users to select multiple responses using the control or command and enter/return keys.

• Free Text Comments: Use this item type to ask an open ended question requiring a written response. (In ME 1.11 and lower you can not map a free text comment to a curriculum tag set.)

• Date Selector: Use this item type to ask a question to which the response is a specific date (e.g. What was the date of this encounter?)

• Numeric Field: Use this item type to ask a question to which the response is a numeric value (e.g. How tall are you?)

• Rubric Attribute (single response): Use this to create an item that relies on response categories as answer options. If you enter text in the response text area it will not show up to the user unless you create a grouped item. If you create a grouped item remember you need to use the same scale across all items to be grouped together. If you want a rubric item to display response text, create a grouped item with just one item included.

• Scale Item (single response): Use this to create an item that relies on response categories as answer options. If you enter text in the response text area it will not show up to the user unless you create a grouped item. If you create a grouped item remember you need to use the same scale across all items to be grouped together.

• Field Note: This item type was developed specifically for post-graduate family medicine prior to the current work on they dynamic competency based education tools.

  • To begin you select a curriculum tag set.

  • For a tag you want to be available to be assessed on a form, you add 4 response descriptors (e.g., Excellent, Needs Improvement) and corresponding level of competency descriptions.

  • If you have a hierarchical curriculum tag set you can develop items for the bottom/most granular tags in the set.

  • When you create an assessment form, you can add the relevant curriculum tag set to the form. When individuals use the form they can navigate through the tag set and select the appropriate tag to assess.

• Autocomplete (multiple responses): Use this item to allow users to search the option list in an autocomplete fashion and then make multiple selections. The response options will display in the order they were added to the item and when a user begins to type a response, the list of options will be filtered. The user can select more than one response as needed.

How to create grouped items

Creating a grouped item allows you to group items and guarantee that they appear together on forms. If you use the rubric attribute or scale item item types, creating a grouped item will create a rubric with common response categories (e.g. developing, achieved) and specific response text for each field (e.g. performed a physical exam with 1-2 prompts from supervisor, independently performed a physical exam). There is also the ability to Copy a Grouped Item which is next to the Create & Attach a New Item and Attach Existing Item(s) buttons. You can choose to create a new item linkage to keep all items as grouped or to create new individual items from the original grouped item.

• Navigate to Admin>Assessment & Evaluation.

• Click on the Items tab.

• Click on the Grouped Items sub-tab.

• Click 'Add A New Grouped Item'.

• Provide a grouped item name and select a rating scale type and then a rating scale. All items in the group will have the same response categories assigned to them, as configured through the rating scale. (Rating scales can be set up through the Scales tab in Admin>Assessment and Evaluation.)

• Click 'Add Grouped Item'.

• Complete the required information, noting the following: Title: This will display when you view a list of grouped items. Description: This field is optional; note that the grouped item description will display below the grouped item title on forms produced for users (see image below). Permissions: Adding a group, course, or individual here will give those users access the the grouped item.

• To add items to a grouped item you can either create and attach a new item or add existing items (click the appropriate button).

• If attaching existing items, use the search bar and filters to find items. You will only be shown items that match the rating scale parameters you've selected. Click the checkbox beside a question (in list view) or beside the pencil icon (in detail view) and click 'Attach Selected'. Because an existing item may already be in use on another form, in some cases you will not be able to modify the response descriptors for that item.

• If creating and attaching items, follow the instructions above for creating items. The rating scale for your new items will be set to match the rating scale of the grouped item. After creating one item, you can repeat the steps to create and attach as many items as needed.

• Click 'Save'.

• To edit an item click on the pencil icon. Bear in mind that an existing item may already be in use on another form.

• To delete an item from a grouped item, click on the trashcan icon.

• To reorder the items in the grouped item, click on the crossed arrows and drag the item into the appropriate location.

• When you have added all required items to the grouped item, click 'Save'.

• Click 'Grouped Items' at the top of the screen to return to the list of grouped items.

How to add permissions to existing items

When users create items, they automatically have permission to access and use those items. Users can optionally give permission for other individuals and coursess to access items. As much as possible, we recommend permissioning items to courses to reduce future workload to reassign items to individuals as you experience staff changes.

• Navigate to Admin>Assessment and Evaluation.

• Click 'Items'.

• From list view, click on any item to open it. From grid view, click the pencil icon to edit an item.

• Give permission to an individual or course by first selecting the appropriate title from the dropdown menu, and then beginning to type in the search bar. Click on the desired name from the list that appears below the search bar.

  • If you give permission to a course, anyone listed on as a course contact on the setup page AND with access to the Assessment and Evaluation module will have access to the item.

• Currently, permissioning an item to an organization only allows medtech:admin users to access it. For that reason, permissioning items to courses is likely more useful.

• After you've added all permissions, you can return to the list of all items by clicking 'Items'.

Tips for viewing existing items

• Toggle between list view and detail view using the icons beside the search bar.

• In detail view, see the details of an existing item by clicking on the eye icon.

• In detail view, edit an existing question by clicking on the pencil.

• To delete items, check off the tick box beside a question (list view) or beside the pencil icon (detail view) and click 'Delete Items'.

• From an Edit Item page you can click on a link to view the forms that use an item or the grouped item an item is included in.

• When viewing items in list view, the third column shows the number of answer options the item has. Clicking on it takes you to the item, and by clicking again you can see all the forms that use this item.

How to search for existing items

• From the Items tab type into the search box to begin to find questions.

• You can apply a variety of filters to refine your search for existing items.

• To select a filter, click on the down arrow beside the search box. Select the filter type you want to use, click on it, and then begin to type what you want to find or continue clicking to drill down and find the required filter field. Filter options will pop up based on your search terms or what you’ve clicked through and you can check off the filters you want to apply. Apply multiple filters to further refine your search.

• If you're working with a filter with multiple hierarchies, use the breadcrumbs in the left corner of the filter list to go back and add additional filters.

• When you’ve added sufficient filters, scroll down and click Apply Filters to see your results.

• To remove individual filters from your search, click on the down arrow beside the search field, click a filter type and click on the small x beside each filter you want to remove. Scroll down and click ‘Apply Filters’ to apply your revised selections.

• To remove all filters from your search, click on the down arrow beside the search field, click a filter type, scroll down, and click on ‘Clear All’ at the bottom of the filter search window.

Retiring Items

Retiring items allows a user to manage archived and future or currently active items effectively. When users create forms they will not see retire items, however retired items will still be included in all relevant reports. Retired items will continue show in the item bank, but will have their item type highlighted red.

• Click the tick box on an item card.

• Click the orange Retire Items button.

• Confirm your action by clicking 'Retire'.

• The recently retired item will have the item type bar highlighted in red.

Deleting Items

Deleting an item will remove it from the list of Assessment and Evaluation Items.

• Click the tick box on an item card.

• Click the red Delete Items button.

• Confirm your action by clicking 'Delete'.

• The deleted item will no longer show on the list of items.

Please note that there is no way through the user interface to recover an accidentally deleted item. If you have deleted something by mistake you will need help from a developer to correct the mistake.

Deleted items will remain on forms that already include that item.

There are multiple tools available to facilitate assessment and evaluation through Elentra.

• To create items and forms to assess learner performance on tasks (e.g., clinical skills and workplace based assessment), use the Assessment and Evaluation Module.

• To create items and forms to have learners evaluate courses, faculty, themselves, and other activities, use the Assessment and Evaluation Module.

• To create assessment items to use in small, low-stakes assessments use the Exams Module. (Elentra does include a Quizzes Module but we recommend you use the much newer Exams Module instead.)

• To create assessment items to use for tests and exams use the Exams Module.

• To store learner grades and facilitate faculty grading of assignments use the Gradebook feature in each course. (To allow inline faculty grading using a rubric form or similar, please note that you will build the form in Assessment and Evaluation, then attach it to a Gradebook Assessment. Only specific item types are supported on forms attached to gradebook assessments.)

Elentra ME also includes a portfolio, which some organizations may choose to use for student assessment purposes. The portfolio can be connected to a course gradebook. For more detail about the portfolio please see the Portfolio help section.

Assessment & Evaluation Module

The Assessment & Evaluation Module in Elentra is predominantly used to assess learner performance in a clinical environment, and provide multiple user groups with the ability to evaluate faculty, courses/programs, and other activities within the context of your organization. Since the Assessment and Evaluation Module essentially allows you to create forms for people to complete you could even use it to do a pre-course survey, or as a way to collect information from a group about their work plans in a collaborative activity.

To use the Assessment & Evaluation Module users must create items (e.g., questions, prompts, etc.), and create forms (a collection of items). Organizations can optionally allow users to initiate specific forms based on a form workflow (e.g. to allow learners to initiate an assessment on themselves by a faculty member), or organizations can create tasks for individuals to complete using a distribution. A distribution defines who will complete a form, when the form will be completed, and who or what the form is about.

The Assessment and Evaluation Module users also have a quick way to view their assigned tasks and administrators can monitor the completion of tasks assigned to others.

Reporting exists to view and in some cases export the results of various assessments and evaluations.

Rotation Based Distributions allow you to set up a distribution based on a rotation schedule. This means you can easily send a form to all enrolled learners to be delivered when they are actively in the rotation. Note that you must have rotations built using the Clinical Experience Rotation Scheduler to use this distribution method.

Add New Distribution

• Navigate to Admin>Assessment and Evaluation.

• Click 'Distributions' above the Assessment and Evaluation heading.

• Click 'Add New Distribution'.

Step 1: Form

• Distribution Title: Provide a title. This will display on the list of Distributions that curriculum coordinators and program coordinators can view.

• Distribution Description: Description is optional.

• Task Type: Hover over the question mark for more detail about how Elentra qualifies assessments versus evaluations. If a distribution is to assess learners, it's an assessment. If it is to evaluate courses, faculty, learning events, etc. it is an evaluation. Notice that the language on Step 4 will change if you switch your task type.

• Disable Target Release: If you do not want assessors/evaluators to be able to release the form to the target upon completion, check this box. If left unchecked, assessors/evaluators will be able to release the form to the target upon completion.

• Assessment/Evaluation Mandatory: This will be checked off be default.

• Disable Reminders: Check this box to exclude this distribution from reminder notification emails.

• Select Form: The form you want to distribute must already exist; pick the appropriate form.

• Select a Curriculum Period: The curriculum period you select will impact the list of available learners and associated faculty.

• Select a Course: The course you select will impact the list of available learners and associated faculty.

• Click 'Next Step'

Step 2: Method

• Distribution Method: Select 'Rotation Distribution' from the dropdown menu.

• Rotation Schedule: Select the appropriate rotation schedule from the dropdown menu.

• Specific Sites: Select a site if you wish to include only targets from a specific site on this distribution.

• Release Date: This tells the system how far back on the calendar to go when creating tasks. Hover over the question mark for more detail.

• Delivery Type:

  • Basic delivery (single): allows you to send a single evaluation task repeatedly, once per block, or once per rotation.

  • Dynamic delivery rules (multiple): allows you to deliver multiple evaluation tasks at specific intervals during a rotation. For example, if you have a 5-month rotation, and you’d like to deliver an interim evaluation at the 2-month mark and the 4-month mark, you can set a rule for the distribution to do so.

• If you select Basic delivery (single), then select your Delivery Period:

  • Choose between delivering tasks repeatedly, once per block, or once per rotation.

    • Repeatedly means you set how often during the rotation the task sends (e.g. every 5 days)

    • Once per block means that for each booking the learner has in a rotation they will be sent a task

    • Once per rotation means that consecutive bookings (i.e., two or more back-to-back blocks) will be treated as one unit of time

  • For each delivery period, additional customization allows you to control the timing of the distribution (e.g. 1 day after the start of the block, or 3 days before the end of the rotation).

    • Options are before/after the start, before/after the middle, and before/after the end.

In this sample schedule, a 'Once per block' delivery for the IM rotation would send learner 012022 three tasks - one each for Blocks 1, 2, and 3. A 'Once per rotation' delivery for the IM rotation would send learner 012022 two tasks - one for Blocks 1+2, and one for Block 4.

Note that if learners have consecutive bookings at different sites they will not be treated as one rotation. For example a 'Once per rotation' delivery for the IM rotation for Tibor would result in two assessment tasks because he is at Hospital 1 and then Hospital 2.

• If you select Dynamic deliver rules (multiple), then set your delivery rules.

  • Identify the specific delivery rules depending on the length of a rotation (you identify the length of the rotation in months, weeks, or days).

  • If you enter an integer larger than 1 for the number of tasks, you will be able to define when each task will be delivered (percent of the way through the rotation) and a visualization will be supplied by the Task bar below.

  • Clicking on Add new delivery will allow you to add additional delivery schedules that depend on the length of the rotation, which allows you to set different parameters for shorter rotations and longer rotations.

  • As noted in the interface, “larger” rules take precedence over smaller ones. So if you had a 0-1 month rule and a 0-2 month rule, the 0-2 month would supersede the other.

  • Remember: if a learner's schedule has two or more of the same rotation scheduled in a row, the system will treat them as a single rotation.

  • You can have as many rules as you want, but they must be contiguous (e.g., 0-2 months, 2-4 months, 4-6 months) to avoid leaving learners without an assessment.

Examples of Dynamic Delivery Scenarios:

• If you would like two interim assessment tasks delivered for any rotation 5 to 8 months in length, you could identify that for a 5 – 8-month rotation, task one should be delivered at 40% of the way through the rotation and task 2 should be delivered at 80% of the way through the rotation. Then, for a learner on a 5-month rotation, the interim task would be delivered at 2 months into the rotation and again at 4 months into the rotation, for example.

• If you would like three tasks delivered for any rotation 3 – 6 months in length, you could set task one at 25% through the rotation, task 2 at 50% through the rotation, and task 3 at 75% through the rotation, which would represent delivery of the task at ¼ of the way through the rotation, 1/2 the way through the rotation, and ¾ of the way through each 3-month, 4- month, 5-month, or 6-month rotation that a learner might be scheduled on.

• Task Expiry: If you check this box, the tasks generated by the distribution will automatically expire (i.e., disappear from the assessor's task list and no longer be available to complete). You can customize when the task will expire in terms of days and hours after the delivery.

  • Warning Notification: If you choose to use the Task Expiry option, you'll also be able to turn on a warning notification if desired. This can be set up to send an email a specific number of days and hours before the task expires.

Step 3: Targets

The target is who or what the form is about (e.g., learners, faculty members, a rotation). Note that you'll only see the option to set the rotation or faculty member as a target if you are creating an evaluation.

• Assessments will be delivered for: Use this area to specify the target of the form.

  • Targets are the assessors (self assessment): use this for self-assessments. Learners will be delivered a task where they are both the target and assessor.

  • Targets are learners

    • Learner Options:

      • All learners in this rotation

        • Learners from My Program/Outside of my Program: This refers to organizations that use on and off-service rotation slots (usually PGME programs).

          • Checking My Program will target leaners who are scheduled in the rotation AND enrolled in the course.

          • Checking Outside of my program will target learners who are scheduled in the rotation but enrolled in a different course than that in which the rotation exists.

        • Learner Levels: This refers to learner levels usually used in CBME enabled organizations. Checking a learner level will restrict the distribution to target only learners at that learner level. Leaving all boxes unchecked will target all learners, regardless of their learner level.

        • Additional Learners

          • Click Browse Additional Learners. Search for a learner and check the box beside the learner name. To delete a learner, click the x beside their name.

      • Specific learners in this rotation: Use the drop down selector to add the required learners. (Hover over a learner name to see their profile information.)

      • CBME Options: This will display whether you select All Learners or Specific Learners. This option applies only to schools using Elentra for CBME. Ignore it and leave it set to non CBME learners if you are not using CBME. If you are a CBME school, this allows you to apply the distribution to all learners, non CBME learners, or only CBME learners as required.

  • Targets are peers

    • This option allows for targets who have completed the rotation to assess all of their peers in the rotation block.

    • CBME Options: This option applies only to schools using Elentra for CBME. Ignore it and leave it set to non CBME learners if you are not using CBME. If you are a CBME school, this allows you to apply the distribution to all learners, non CBME learners, or only CBME learners as required.

  • Targets for this Distribution will be determined by the Assessor

    • With this option, the assessor/evaluator selects the target based on their role and distribution assessor options when they are completing the task.

    • In the Distribution Progress Report, administrators will see the target listed as "Assessor will choose before completing task."

• Target Attempt Options: Specify how many times an assessor can assess each target, OR whether the assessor can select which targets to assess and complete a specific number (e.g. assessor will be sent a list of 20 targets, they have to complete at least 10 and no more than 15 assessments but can select which targets they assess).

  • If you select the latter, you can define whether the assessor can assess the same target multiple times. Check off the box if they can.

• Click 'Next Step'

Step 4: Assessors

The assessors are the people who will complete the form.

• Assessor Options:

  • Assessors are learners

    • Learner Options: All Learners of Specific Learners

      • All Learners: Select all learners in the rotation or specific learners in the rotation

        • All learners: Select from My Program and/or Outside of my program, use the drop down selector to add additional learners

        • Specific learners in the rotation: Use the drop down selector to add required learners

      • Additional Learners: Check this option to add additional learners from outside the program.

  • Assessors are faculty members

    • Browse faculty and click on the required names to add them as assessors

    • Select Associated Faculty: This tool will pull the names of faculty listed on the course setup page as associated faculty

  • Assessors are preceptors associated with a rotation schedule

    • This allows you to associated this distribution with all preceptors affiliated with the rotation based on the slot bookings made for learners. The distribution will dynamically update based on changes made to the rotation schedule.

  • Select individuals regardless of role (Individuals or external assessors)

    • This allows you to add external assessors to a distribution

    • Begin to type an email, if the user already exists you'll see them displayed in the dropdown menu. To create a new external assessor, scroll to the bottom of the list and click 'Add External Assessor'

    • Provide first and last name, and email address for the external assessor and click 'Add Assessor'

  • Exclude Self Assessments

  • Feedback Options: This will only display when the assessors are faculty. Checking the box will add a default item to the distribution asking if the faculty member met with the trainee to discuss their assessment.

  • Give access to the results of previous assessments: This relates to Elentra's ability to provide a summary assessment task to users. If enabled, tasks generated by this distribution will link to tasks completed in the listed distributions. When users complete the summary assessment task they will be able to view tasks completed in the other distributions. For any items that are used on both forms, results from previously completed tasks will be aggregated for the assessor to view.

    • Click to add the relevant distribution(s).

    • Currently for rotation-based distributions there is no option to configure minimum tasks completed on linked distributions nor a fallback date for the summary assessment task.

• Click 'Next Step'

Step 5: Results

You can immediately save your distribution at this point and it will generate the required tasks, but there is additional setup you can configure if desired.

• Authorship: This allows you to add individual authors, or set the distribution to be accessible to everyone with A+E access in a course or organization. (This may be useful if you have multiple users who manage distributions or frequent staffing changes.)

• Target Release: These options allow you to specify whether the targets of the distribution can see the results of completed forms.

  • Task List Release:

    • "Targets can view tasks completed on them after meeting the following criteria" can be useful to promote completion of tasks and is often used in the context of peer assessments. Targets will only see tasks completed on them after they have completed the minimum percentage of their tasks set by you.

  • Target Self-Reporting Release: This controls whether targets can run reports for this distribution (i.e. to generate an aggregated report of all responses). When users access their own A+E they will see a My Reports button. This will allow them to access any reports available to them.

• Target Self-Reporting Options: This allows you to specify whether or not comments included in reports are anonymous or identifiable. (This will only be applied if you have set reports to be accessible to the targets.)

• Reviewers: This allows you to set up a reviewer to view completed tasks before they are released to the target (e.g. a staff person might review peer feedback before it is shared with the learner).

  • Check off the box to enable a reviewer.

  • Click Browse Reviewers and select a name from the list. Note that this list will be generated based on the course contacts (e.g. director, curriculum coordinator) stored on the course setup page.

• Notifications and Reminders:

  • Prompted Responses: This allows you to define whom to send an email to whenever a prompted response is selected on a form used in the distribution. For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response, any time "I had to do it" is picked as an answer an email notification will be sent.

    • You can optionally select to email Program Coordinators, Program/Course Directors, Academic Advisors, Curricular Coordinators, or you can add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

  • Sender Details: Define the email address that notifications and reminders will be sent from for a distribution.

    • Options are the distribution author, an existing user, an existing generic email, or a new generic email.

    • To create a new generic email provide a name and email address. This will be stored in the system and available to other users to use as needed.

• Click 'Save Distribution'.

Additional Notes on "The targets for this Distribution will be determined by the Assessor"

After the distribution is created, the distribution progress page will show targets as "Assessor will choose before completing task."

When an assessor/evaluator accesses the task they will see that the target is to be defined by them.

When completing the task, the assessor/evaluator will define their target.

When you create assessment and evaluation items you will have the option of applying rating scales to certain item types; creating rating scales promotes consistency across items and can be a time saver for the administrative staff creating items and forms.

You must be a medtech:admin user to manage rating scales. Scales can have permissions configured if needed (see end of page).

For additional detail about rating scales in CBME, please see here.

How to create new rating scales

• Navigate to Admin>Assessment and Evaluation.

• Click 'Scales' from the A&E tabs list. Any existing rating scales will be displayed.

• Click the green 'Add Rating Scale' button.

• Complete the required information, noting the following: Title: Title is required and is what users will see when they build items and add scales so make it clear. Description: This is optional and is not often seen though the platform. Rating Scale Type: This defines the type of rating scale you are creating. Later, if you add rating scales to items, or add standard scales to form templates, you will first have to select a scale type. There is no user interface to configure rating scale types. Sample scale types include generic, global rating scale, milestone scale, etc. In a default Elentra installation you'll likely just see a default scale type. (In installations with CBME enabled you'll see global rating and milestone/enabling competency scales.)

• Add or remove response categories by clicking the plus and minus icons.

• For each response category, select a descriptor (these are configured through the assessment response categories). Note that you can search for descriptors by beginning to type the descriptor in the search box.

• To edit an existing rating scale click on the scale title, make changes as needed, and click 'Save'.

• To delete a rating scale click the checkbox beside the rating scale and click 'Delete Rating Scale'.

Once scales are created, they will become visible options when creating items and using some form templates.

How to set permissions for a scale

Setting permissions for a scale dictates which users will be able to access a scale when they create assessment and evaluation items. For example, if you set a scale's permissions to Undergraduate Medicine, all users with access to Admin > A & E in the undergraduate organisation will be able to use the scale when creating items. If you set a scale's permissions to several individual users, only those users will be able to access the scale when creating items.

You must create a scale before you can edit the permissions for it. After a scale is created you will automatically be redirected to the edit page.

• In the Rating Scale Information section, look for the Scale Permissions heading.

• Select Individual, Organisation, or Course from the dropdown options.

• Type in the search bar to find the appropriate entity.

• Click on the entity name to add it to the permission list.

• Add as many permissions as required.

• Scroll down and click 'Save'.

The pdf below provides a sample view of all item types that can be included on an assessment and evaluation form (please note this does not include some items specific to forms built from CBME templates). Some items are designed to be used online (e.g. autocomplete multiselect). To view sample items in use in an online environment, check out your Elentra demo site.

In this context, we use the term off-service to mean learners who are completing rotations outside their home program (e.g. a family medicine resident on a pediatrics rotation).

Clinical Experiences > Rotation Scheduler includes support for off-service rotations or slots. When an administrator builds rotations she can optionally create off-service slots and make them available to all other programs or specific programs. The following information assumes you are an organization using off-service slots and refers to rotation-based distributions. (This page does not discuss automated rotation evaluations nor rotation evaluations completed on demand via workflows.)

As the home program you can:

• Create assessment tasks targeting your learners while they are on an off-service rotation.

• Report on assessment tasks targeting your learners while they were on an off-service rotation.

  • For distributions you created, you can view results via the Learner Assessments report or the Learner Reports (Aggregated).

  • For distributions created by host programs, but targeting your learners, you can only view results via Learners Reports (Aggregated).

• Create evaluation tasks asking your learners to evaluate their experience on an off-service rotation.

• View individual responses and download a Weighted CSV of your learners' responses to an evaluation of their off-service experience (do this via the Completed Distribution Report); up to and including ME 1.17 a home program cannot use the Rotation Evaluation (Aggregated) Report to see an overview of their learners' experience on an off-service rotation (whether the distribution was set up by the home or host program).

As the host program you can:

• Create assessment tasks targeting visiting, off-service learners while they are on your rotation.

• Report on assessment tasks targeting visiting, off-service learners while they were on your rotation.

• Create evaluation tasks asking visiting, off-service learners to evaluate their experience on your rotation.

• Report on evaluation tasks completed by visiting learners to your rotation, as long as you created the distribution. If the home program created an evaluation asking their learners to evaluate your rotation or faculty, you will not be able to access the results.

If you are in an organisation that is not using CBME, the form templates tab is unlikely to be something you will ever use. Form templates allow users to make forms more consistent and in the case of the Competency-Based Medical Education module, allow users to configure the requirements for multiple forms through one user interface. Administrative staff specify EPAs and milestones, contextual variables, and rating scales to be used on assessment forms for clinical environments, and these forms get created after a form template is published. If you are using CBME please see here to learn more about using form templates.

Form templates are built off of something called a form blueprint and there is currently no way through the user interface to configure a form blueprint. If you want different form templates from those in a default installation of Elentra, you'll need a developer's help.

A distribution defines who will complete a form, who/what the form is about, and when the form will be completed. Usually, administrative staff manage distributions of forms to cohorts of learners, faculty, event participants, etc. Generally, the following language is used in the A&E module:

Assessor/Evaluator: The person completing a form/task Target: The person, course, experience or other that the form/task is about Distribution method: How the form/task is assigned to people (e.g. based on a date range, rotation schedule, event schedule, etc.)

The available distribution methods and some of their use cases are listed below:

Rotation Schedule: Use a rotation schedule distribution to send forms based on a rotation schedule. This could include assessments of learners participating in a rotation, evaluation by learners of a specific rotation site or experience, etc.

Delegation: Use a delegation to schedule a distribution to be sent to an intermediate person who will later forward the tasks to the appropriate assessor/evaluator. For example, if you want to set up distributions in September, but it is unknown who learners will work with in January because a clinic schedule hasn't been set yet, you could send the distribution to a delegator to forward once the clinic schedule is known.

Learning Event Schedule: Use a learning event schedule distribution when you want to have participants evaluate an event, when you want participants to evaluate the faculty teaching particular types of events, or when you want faculty to assess the learners participating in certain types of events.

Date Range: Use a date range to create the most generic type. You could use this for faculty or course evaluations, or for assessment of learners at any point.

A distribution wizard walks users through creating any distribution. Note that a distribution is only saved once all five steps of the wizard are complete. You can navigate forward and backward in a distribution using the Previous Step and Next Step buttons. You can cancel a distribution at any time by clicking 'Cancel'.

How to create a distribution

• Navigate to Admin>Assessment & Evaluation.

• Click 'Distributions'.

• Click 'Add New Distribution'.

• This will open a five-step wizard that walks you through creating a distribution. On the following pages you can read about the specifics of each distribution method.

How to copy a distribution

• Navigate to Admin>Assessment & Evaluation.

• Click 'Distributions'.

• Use the search tool to look for the distribution you want to copy. Click the down arrow beside the search bar to apply filters to refine your search results.

• Click on the cog icon to the right of the distribution you want to copy. Select 'Copy Distribution'.

• Edit the information in each step of the distribution wizard as needed and save your work.

How to delete a distribution

• Navigate to Admin>Assessment & Evaluation.

• Click 'Distributions'.

• Use the search bar to look for the distribution you want to delete. Click the down arrow beside the search bar to apply filters to refine your search results.

• Click the checkbox beside the name of the distributions you want to delete.

• Click the red Delete Distributions button.

How to manage distributions (monitor completion of tasks, remind people to complete tasks, etc.)

• Navigate to Admin>Assessment & Evaluation.

• Click 'Distributions'.

• Use the search bar to look for the distribution you want to manage. Click the down arrow beside the search bar to apply filters to refine your search results. You will only see distributions to which you have access.

• Click on the cog icon to the right of the distribution you want to copy. Select 'View Distribution Report'.

• Review progress. Click on any category (Pending, In Progress, Completed) to view more details about specific targets and assessors.

• Pending tasks have not yet been started. In Progess tasks have been started but not complete. Completed tasks are done and have been submitted.

• To delete tasks tick off the box below the garbage icon on each task card and then click the red Delete Task(s) button.

• To send reminders to those with incomplete forms, tick off the box below the bell icon on each task card and then click the blue Manage Distribution button and select Send Reminders from the dropdown list. To select all tasks for reminders click on the bell icon.

• Review your choices and, if correct, click Confirm Reminders. You will get a green success message.

• To add a task to a distribution, click the blue Manage Distribution button and select 'Add a Task' from the dropdown list. Complete the required information and click 'Confirm Task'.

Viewing progress results for learning event-based distributions: Program coordinators who set up such distributions should view progress from their My Assessments page.

How to manage tasks at the individual level

• Many of the functions described above can also be completed at an individual task level when logged into the system as a program coordinator or faculty director.

• Click the Assessment and Evaluation badge at the top right (beside the logout button).

• Click the My Learners tab.

• Click on the appropriate tab for a learner (e.g. CBME, Assessments, Logbook).

• When looking at a user's Assessment & Evaluation dashboard, some users will be able to send reminders, remove tasks, and download PDFs of selected tasks from their assigned learners or faculty.

Retiring a distribution

If you want to change a distribution partway through its use but you want to leave all existing tasks active, you can copy the distribution, alter it as needed, and then retire the original distribution on a defined date. When you retire a distribution, you preserve any existing data and pending tasks but prevent future tasks from generating.

• To retire one distribution, click the menu cog to the right of the distribution name and select 'Retire Distribution'.

• To retire multiple distributions at once, click the checkbox beside distribution names and click the 'Retire Distribution' button.

• Set a Retirement Date for the distribution(s). As of this date, no more tasks will generate for this distribution.

• Click 'Retire'

• You will get a success message and can click to return to the Distribution index.

• Retired distributions will still display on the list of Distributions, but they will have a small orange clock beside the name to show they have been retired.

• An informational notice will be included on distributions that are set to retire soon. The notice will be seen when you access the Progress Report for a distribution.

Deleting a Distribution

If you delete a distribution you will no longer see it in the list of Distributions and it can't be recovered through the user interface.

• Click the checkbox beside a distribution name.

• Click 'Delete Distribution'

• Confirm your decision.

Send Reminders from the Admin. A & E Dashboard

• Navigate to Admin > Assessment & Evaluation.

• Filter the task list as necessary to find the task you want.

• Check the box under the bell icon on the right side of the task table.

• Click 'Send Reminders'.

• You will see a summary of the reminders that will be sent (e.g. assessor name and number of notifications).

• Click 'Confirm Reminders'.

• You will get a success message. Close the success message to return to the A & E Dashboard.

Send Reminders from a Distribution

• Navigate to Admin > Assessment & Evaluation.

• Click the Distributions tab.

• Find the appropriate distribution and click on it or use the cog icon and select 'View Distribution Report'.

• Click on a tab (e.g., Available Assessments) to view tasks.

• Check the box under the trash can icon on the right side of the task table.

• Click 'Manage Distribution' and select 'Send Reminders'.

• You will get a confirmation message and see a summary of the reminders to be send (e.g. assessor name).

• Click 'Confirm Reminders'.

• You will get a success message. Close the success message to return to the Distribution Report.

Forward Tasks from the Admin. A & E Dashboard

Administrators can forward tasks on behalf of assessors/evaluators.

• Navigate to Admin > Assessment & Evaluation.

• Filter the task list as necessary to find the task you want.

• Open the task.

• Click 'Forward Task'.

• Choose a user to forward the task to.

  • Note that emails will be sent to the old and new assessor/evaluator.

• Click 'Forward Task'.

• You will get a success message. Close the success message to return to the A & E Dashboard.

Delete Tasks from the Admin. A & E Dashboard

• Navigate to Admin > Assessment & Evaluation.

• Filter the task list as necessary to find the task you want.

• Check the box under the trash can icon on the right side of the task table.

• Click 'Delete Task(s)'.

• You are required to provide a reason for deleting the task and can optionally provide additional notes.

  • The reason entered will be displayed when tasks are viewed on the Deleted tab.

• Click 'Remove Task(s)'.

• You will get a success message. Close the success message to return to the A & E Dashboard.

Delete Tasks from a Distribution

• Navigate to Admin > Assessment & Evaluation.

• Click the Distributions tab.

• Find the appropriate distribution and click on it or use the cog icon and select 'View Distribution Report'.

• Click on a tab (e.g., Not Started) to view tasks.

• Check the box under the trash can icon on the right side of the task table.

• Click 'Delete Task(s)'.

• You are required to provide a reason for deleting the task and can optionally provide additional notes.

  • The reason entered will be displayed when tasks are viewed on the Deleted tab.

• Click 'Remove Task(s)'.

• You will get a success message.

Reopen Deleted Tasks from the Admin. A & E Dashboard

• Navigate to Admin > Assessment & Evaluation.

• Toggle to the Assessment or Evaluations view as needed.

• Click the Deleted tab.

• Filter the task list as necessary to find the task you want.

• Click on a task to view it.

• Click 'Reopen Task'.

• Provide a reason for reopening the task.

• Click 'Reopen Task'.

• The task will now display under the Outstanding tab and the admin dashboard and will be set to in-progress for the task owner. New notifications will not automatically be sent.

Recover Deleted Tasks in Distributions

• Navigate to Admin > Assessment & Evaluation.

• Click the Distributions tab.

• Find the appropriate distribution and click on it or use the cog icon and select 'View Distribution Report'.

• Click on a tab (e.g., Not Started) to view tasks.

• Check the box under the trash can icon on the right side of the task table.

• Click 'Recover Task(s)'.

• You will get a success message and the task will appear as active on the task list.

Users with access to Admin > Assessment and Evaluation will be able to view A&E reports. Generally this will include medtech:admins, staff:admins, and staff:pcoordinator and faculty:director assigned to a specific course or program.

To view A&E Reports:

• Click Admin > Assessment & Evaluation.

• From the tab menu, click 'Reports'.

Reports are categorized under evaluations, assessments, leave and distributions. Please read report descriptions carefully as not all reports necessarily include data from both distributed forms and user-initiated on-demand forms.

Additional reporting tools for user-initiated on-demand forms used with the CBME tools can be viewed form the CBME dashboard.

A distribution based on a learning event schedule allows you to schedule forms to be sent out where the targets of the distribution are the attendees of the event OR faculty who taught the event OR a specific event within the selected event type.

Add New Distribution

• Navigate to Admin>Assessment and Evaluation.

• Click 'Distributions' above the Assessment and Evaluation heading.

• Click 'Add New Distribution'.

Form

• Distribution Title: Provide a title. This will display on the list of Distributions that curriculum coordinators and program coordinators can view.

• Distribution Description: Description is optional.

• Task Type: Hover over the question mark for more detail about how Elentra qualifies assessments versus evaluations. If a distribution is to assess learners, it's an assessment. If it is to evaluate courses, faculty, learning events, etc. it is an evaluation. Notice that the language on Step 4 will change if you switch your task type. The task type will also dictate other fields available throughout the wizard.

• Disable Target Release: This will only display for Evaluations. If you do not want evaluators to be able to release the form to the target upon completion, check this box. If left unchecked, evaluators will be able to release the form to the target upon completion.

  Assessment/Evaluation Mandatory: This will be checked off be default.

  Disable Reminders: Check this box to exclude this distribution from reminder notification emails.

• Select Form: The form you want to distribute must already exist; pick the appropriate form.

• Select a Curriculum Period: The curriculum period you select will impact the list of available learners and associated faculty.

• Select a Course: The course you select will impact the list of available learners and associated faculty.

• Click 'Next Step'

Method

• Distribution Method: Select 'Learning Event Schedule' from the dropdown menu.

• Select Event Type: Select the appropriate event type(s) from the dropdown menu. You can select multiple event types as needed.

• Release Date: This tells the system how far back on the calendar to go when creating tasks. Hover over the question mark for more detail.

• Task Expiry: If you check this box, the tasks generated by the distribution will automatically expire (i.e. disappear from the assessors task list and no longer be available to complete). You can customize when the task will expire in terms of days and hours after the delivery.

  • Warning Notification: If you choose to use the Task Expiry option, you'll also be able to turn on a warning notification if desired. This can be set up to send an email a specific number of days and hours before the task expires.

• Delay Task Creation: This option relates specifically to creating a summary assessment task distribution. This allows you to create a distribution that will pull in and display completed tasks from other distributions. In effect, the assessor of the distribution you create will be able to see items from previously completed tasks and take them into account when completing their own task.

  • Select a distribution (note that you will only see distributions to which you have access).

  • Define how many tasks must be completed in the linked distribution(s) before the new distribution will take effect.

  • Enter a fallback date. This date represents when your distribution will create tasks whether or not the minimum number of tasks have been completed in the linked distribution.

    • You can adjust the fallback date on the fly and the distribution will adjust accordingly to only generate tasks as appropriate (previously created tasks will still remain).

    • If you set a fallback date that falls before the release date, the fallback date will be ignored.

Targets

The target is who or what the form is about.

If you are creating an Evaluation:

• Evaluations delivered for:

  • Faculty who taught events with the selected event types (this will generate an evaluation for any faculty member who taught the event type specified in the Method section, e.g. lecture, lab, etc.)

  • Events with the selected event types (this will generate an evaluation for any event of the event type specified in the Method section)

• Target Attempt Options: Specify how many times an evaluator can evaluate each target, OR whether the evaluator can select which targets to assess and complete a specific number (e.g. evaluator will be sent a list of 20 targets, they have to complete at least 10 and no more than 15 evaluations but can select which targets they evaluate).

  • If you select the latter, you can define whether the evaluator can evaluate the same target multiple times. Check off the box if they can.

  • Click 'Next Step'

If you are creating an Assessment:

• Assessments delivered for: Currently, the only option here is attendees who are enrolled in events with the specified event type.

• Target Attempt Options: Specify how many times an assessor can assess each target, OR whether the assessor can select which targets to assess and complete a specific number (e.g. assessor will be sent a list of 20 targets, they have to complete at least 10 and no more than 15 assessments but can select which targets they assess).

  • If you select the latter, you can define whether the assessor can assess the same target multiple times. Check off the box if they can.

• Click 'Next Step'

Assessors/Evaluators

The assessors/evaluators are the people who will complete the form.

• Assessor Options

  • Assessors/evaluators are attendees enrolled in the event

    • Attendee Options (note that you must be using Elentra's attendance module within events to use this feature)

      • Send to all enrolled audience attendees that attended the event

      • Sent to all enrolled attendees, even if they did not attend

      • Send this assessment to a percentage of enrolled attendees that attended the event

        • You supply the percentage; this option will randomly select the appropriate number of attendees and send them tasks. You'll be able to see which users have been sent the task after the distribution is saved.

  • Assessors/evaluators are faculty members associated with the event

    • This will send the distribution to the faculty listed on the event (e.g. teacher, tutor, etc.)

  • Assessors/evaluators are external to the installation of Elentra

    • This allows you to add external assessors/evaluators to a distribution

    • Begin to type an name or email, if the user already exists you'll see them displayed in the dropdown menu. To create a new external user, scroll to the bottom of the list and click 'Add External Assessor/Evaluator'

    • Provide first and last name, and email address for the external assessor and click 'Add Assessor/Evaluator'

• Give access to the results of previous assessments: This relates to Elentra's ability to provide a summary assessment task to users. If enabled, tasks generated by this distribution will link to tasks completed in the listed distributions. When users complete the summary assessment task they will be able to view tasks completed in the other distributions. For any items that are used on both forms, results from previously completed tasks will be aggregated for the assessor to view.

  • Click to add the relevant distribution(s).

  • Currently for learning event based distributions there is no option to configure minimum tasks completed on linked distributions nor a fallback date for the summary assessment task.

• Click 'Next Step'

Results

You can immediately save your distribution at this point and it will generate the required tasks, but there is additional setup you can configure if desired. (Not all options will display depending on the other parameters of the distribution.)

• Authorship: This allows you to add individual authors, or set the distribution to be accessible to everyone with A+E access in a course or organization. (This may be useful if you have multiple users who manage distributions or frequent staffing changes.)

• Target Release: These options allow you to specify whether the targets of the distribution can see the results of completed forms.

  • Task List Release:

    • "Targets can view tasks completed on them after meeting the following criteria" can be useful to promote completion of tasks and is often used in the context of peer assessments. Targets will only see tasks completed on them after they have completed the minimum percentage of their tasks set by you.

  • Target Self-Reporting Release: This controls whether targets can run reports for this distribution (i.e. to generate an aggregated report of all responses). When users access their own A+E they will see a My Reports button. This will allow them to access any reports available to them.

• Target Self-Reporting Options: This allows you to specify whether or not comments included in reports are anonymous or identifiable. (This will only be applied if you have set reports to be accessible to the targets.)

• Reviewers: This allows you to set up a reviewer to view completed tasks before they are released to the target (e.g. a staff person might review peer feedback before it is shared with the learner).

  • Check off the box to enable a reviewer.

  • Click Browse Reviewers and select a name from the list. Note that this list will be generated based on the course contacts (e.g. director, curriculum coordinator) stored on the course setup page.

• Notifications and Reminders:

  • Prompted Responses: This allows you to define whom to send an email to whenever a prompted response is selected on a form used in the distribution. For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response, any time "I had to do it" is picked as an answer an email notification will be sent.

    • You can optionally select to email Program Coordinators, Program/Course Directors, Academic Advisors, Curricular Coordinators, or you can add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

  • Sender Details: Define the email address that notifications and reminders will be sent from for a distribution.

    • Options are the distribution author, an existing user, an existing generic email, or a new generic email.

    • To create a new generic email provide a name and email address. This will be stored in the system and available to other users to use as needed.

• Click 'Save Distribution'.

A form is a collection of items used to assess a learner, or evaluate a faculty member, event, rotation, course or anything else in your organisation. Forms can be created for specific courses, or for use across an entire organization. When building a form administrators can optionally indicate a form workflow which allows Elentra users to initiate a form on-demand (e.g. for the purposes of clinical assessment).

This help section is about creating forms without CBME templates. See here for information about creating CBME-specific forms.

Form Types

Elentra supports several types of forms.

• Generic Form

  • Users can add any items to this form and it will be immediately available for us.

• Standard Rotation Evaluation Form

  • You will require developer assistance to add this form to your organization.

  • Only use this form type if you are also using the Clinical Experience rotation scheduler and you use an identical form to evaluate all rotations within a course.

  • This form can have preset items that appear on the form each time one is created. There is no user interface to build the templated items and a developer needs to do so.

  • When a new form is made, the templated items (if any exist) will be automatically added to it. Users can then optionally add additional items. After adding items, the form must be published before it can be used.

  • Forms must be permissioned to a course to be used, and each course can only have one active form at a time. If a new form is created it will overwrite the existing form.

  • This form can be automatically distributed based on a rotation schedule (set by administrators when building rotations) OR can be made available to be accessed by learners on demand (use the on-demand workflow for this option).

• Standard Faculty Evaluation Form

  • You will require developer assistance to add this form to your organization.

  • Only use this form type if you use an identical form to evaluate all faculty within a course.

  • This form can have preset items that appear on the form each time one is created. There is no user interface to build the templated items and a developer needs to do so.

  • When a new form is made, the templated items (if any exist) will be automatically added to it. Users can then optionally add additional items. After adding items, the form must be published before it can be used.

  • Forms must be permissioned to a course to be used, and each course can only have one active form at a time. If a new form is created it will overwrite the existing form.

  • This form can be made available for learners to initiate on demand using on-demand workflows.

Form Permissions

Any individual given permission to a form will be able to edit it until it is used in a distribution.

You can also grant permission on a form to a course. If you do this, any user who is a course contact for that course and who also has permission to access Admin > Assessment & Evaluation, will be able to access the form and include it in a distribution if needed. If you are using on-demand workflows and/or Standard Rotation Evaluation or Standard Faculty Evaluation forms you must permission them to a course for them to work properly.

Currently, permissioning a form to an organization only allows medtech:admin users to access it. For that reason, permissioning forms to courses is more useful and is our recommendation.

How to create forms

• Navigate to Admin>Assessment & Evaluation.

• Click 'Forms'.

• Click 'Add Form'.

• Provide a form name and select a type (if applicable). See list of form types above.

• Depending on the form type selected, you may be required to identify a course/program.

• Click 'Add Form'.

• Provide a form description if desired. This will display to users when the form is accessed.

• Indicate the form workflow if your organization uses workflows (i.e. allows users to initiate specific forms on-demand). Set to None if you do not want users to access this form on-demand.

• Set form permissions to give other users access to this form. Anyone given permission to the form will be able to edit it until it is used in a distribution.

  • Please note that forms using a workflow must be permissioned to the appropriate course.

  • Additionally, we strongly recommend permissioning forms to at least a course so that staffing changes are simplified. (If Jane Doe is the only person with access to a form and she retires, you'll need to manually reassign all her forms to a new user. If the form is permissioned to a course, any course contact with access to Assessment & Evaluation will be able to access the form.)

  • Currently, permissioning a form to an organization only allows medtech:admin users to access it. As such we recommend relying mostly on course permissions.

• Click 'Add Item(s)' to add existing items.

  • Note that you can also add grouped items, free text (e.g., to provide instructions), or a curriculum tag set to your form. To add any of these, click on the down arrow beside 'Add Items'. If you choose to add Free Text or a Curriculum Tag Set, please note that you must save your choices within the item box using the small 'Save Required' button.

    • Note: Adding a Curriculum Tag Set is a very specific tool that supports field notes for use in family medicine. Most users should ignore this option.

• Search for existing items and tick off the check boxes, then click 'Attach Selected' to apply your choices.

• To create new items while creating your form, click Add Items and then click Create & Attach a New Item. When you complete your new item and save it, you will be returned to the form you in the process of building.

• Save the form when you have added all the relevant items.

• To preview your form, by clicking on the eye icon/Preview Form button.

• To download a copy of the form, use the Download PDF button.

How to manage forms (delete items, rearrange items, etc.)

• To delete items on a form, tick off the box on the item card and then click the red Delete button on the left.

• To rearrange items on a form, click the crossed arrow icon on the item card and drag the item to where you want it to be.

• To edit an item, click on the pencil icon on the item card. Note that an item already in use on a form that has been distributed will not be able to be edited. Instead you must copy and attach a new version of the item to edit and use it.

• To quickly view the details of an item, click on the eye icon on the item card.

How to copy an existing form

• Navigate to Admin>Assessment & Evaluation.

• Click 'Forms'.

• Use the search bar to look for the form you want to copy. Click the down arrow beside the search bar to apply filters to refine your search results.

• Click on the name of the form you want to copy.

• Click 'Copy Form' and provide a new name for the copied form.

• Click 'Copy Form'.

• Edit the form as needed (e.g., add additional items, change permission, etc.).

  • If you edit an item on a form and that item is in use on other forms, you will affect all of the associated forms. You can optionally view all forms that include the item.

    • For grouped items you can optionally copy and attach the grouped item to the form allowing you to change it as needed.

      • Create new item linkage

      • Create new items

    • For single items you can optionally copy the item to edit it. This will create a brand new item with no connection/link to the item it is copied from.

• Click 'Save'.

How to retire an existing form

Retiring a form means it will remain available in existing distributions, and reports, but will not be available for any new distribution.

• Navigate to Admin>Assessment & Evaluation.

• Click 'Forms'.

• Use the search bar to look for the form you want to retire. Click the down arrow beside the search bar to apply filters to refine your search results.

• Tick off the box beside the form name (you can select multiple forms to retire at once), and then click the orange Retire Form button.

• You will be prompted to confirm your action. Click 'Retire'.

• Retired forms will display with a red highlight around them.

How to delete an existing form

Deleting a form means that all pending and in-progress tasks that used that form will not have a form associated with them and will display an error message stating that the form has been deleted when an assessor/evaluator tries to access the form.

• Navigate to Admin>Assessment & Evaluation.

• Click 'Forms'.

• Use the search bar to look for the form you want to delete. Click the down arrow beside the search bar to apply filters to refine your search results.

• Tick off the box beside the form name (you can select multiple forms to delete at once), and then click the red Delete Form button.

You can use a delegation distribution to send tasks to a delegator who can then forward the tasks to the appropriate assessor/evaluator. The use case for this might be setting up a distribution at the beginning of the year and not knowing exactly which preceptors will be working in a specific environment. You can send the distribution to a staff member who can forward the tasks once a clinic schedule is set or send the distribution to a user who is actually the target of the task and can then delegate the task to the appropriate assessor/evaluator.

Add New Distribution

• Navigate to Admin > Assessment and Evaluation.

• Click 'Distributions' above the Assessment and Evaluation heading.

• Click 'Add New Distribution'.

Form

• Distribution Title: Provide a title. This will display on the list of Distributions that curriculum coordinators and program coordinators can view.

• Distribution Description: Description is optional.

• Task Type: Hover over the question mark for more detail about how Elentra qualifies assessments versus evaluations. If a distribution is to assess learners, it's an assessment. If it is to evaluate courses, faculty, learning events, etc. it is an evaluation. Notice that the language on Step 4 will change if you switch your task type.

• Disable Target Release: This will only display for Evaluations. If you do not want evaluators to be able to release the form to the target upon completion, check this box. If left unchecked, evaluators will be able to release the form to the target upon completion.

• Assessment/Evaluation Mandatory: This will be checked off be default.

• Disable Reminders: Check this box to exclude this distribution from reminder notification emails.

• Select Form: The form you want to distribute must already exist; pick the appropriate form.

• Select a Curriculum Period: The curriculum period you select will impact the list of available learners and associated faculty.

• Select a Course: The course you select will impact the list of available learners and associated faculty.

• Click 'Next Step'

Method

• Distribution Method: Select 'Delegation' from the dropdown menu.

• Delegator: Click Browse Delegators and select the appropriate name from the dropdown menu. You can only select one delegator.

• Delegation Options:

  • Delegation based on date range

  • Delegation based on rotation schedule

• Delegator Type:

  • The delegator is an individual

  • The delegator is the target

• From this point forward the distribution wizard will be configured based on whether you selected a date range or rotation-based distribution. Please refer the the help pages for those two methods for additional details.

Assessors

You can optionally add Assessors for a delegation distribution. The users you add to this list will be made available to the delegator to select from a quick pick list. The delegator will optionally be able to search for and add additional assessors/evaluators as needed.

If you don't want to define any assessors, check "The assessors for this Delegation Distribution are unknown."

Completing a Delegation

Please see here for more information about how to complete a delegation task.

A date range distribution allows you to send a form to the appropriate assessors/evaluators in a specific date range.

Add New Distribution

• Navigate to Admin>Assessment and Evaluation.

• Click 'Distributions' above the Assessment and Evaluation heading.

• Click 'Add New Distribution'.

Form

• Distribution Title: Provide a title. This will display on the list of Distributions that curriculum coordinators and program coordinators can view.

• Distribution Description: Description is optional.

• Task Type: Hover over the question mark for more detail about how Elentra qualifies assessments versus evaluations. If a distribution is to assess learners, it's an assessment. If it is to evaluate courses, faculty, learning events, etc. it is an evaluation. Notice that the language on Step 4 will change if you switch your task type, as will other steps of the wizard.

• Disable Target Release: This will only display for Evaluations. If you do not want evaluators to be able to release the form to the target upon completion, check this box. If left unchecked, evaluators will be able to release the form to the target upon completion.

• Assessment/Evaluation Mandatory: This will be checked off be default. Mandatory tasks can't be deleted by assessors/evaluators.

• Disable Reminders: Check this box to exclude tasks from this distribution from reminder notification emails. This option will only display if specific database settings are enabled. If you check this box, assessors/evaluators will still be sent an initial task creation notification, but weekly reminder summary emails will be disabled.

• Select Form: The form you want to distribute must already exist and you must have permission to access it; pick the appropriate form.

• Select a Curriculum Period: The curriculum period you select will impact the list of available learners and associated faculty.

• Select a Course: The course you select will impact the list of available learners and associated faculty.

• Click 'Next Step'

Method

• Distribution Method: Select 'Date Range Distribution' from the dropdown menu.

• Start Date: This is the beginning of the period the form is meant to reflect.

• End Date: This is the end of the period the form is meant to reflect.

• Delivery Date: This is the date the task will be generated and delivered to the assessors/evaluators. (The delivery date will default to be the same as the start date.)

• Task Expiry Date: Optional. Set the date on which the tasks generated by the distribution will automatically expire (i.e. disappear from the assessor's task list and no longer be available to complete). Tasks will expire at 12:00 AM on the day that you select, so it is best to select the day after your intended expiry date.

  • Warning Notification: If you choose to use the Task Expiry option, you'll also be able set an automatic warning notification if desired. This will send an email to assessors a specific number of days and hours before the task expires.

• Delay Task Creation: This option relates specifically to creating a summary assessment task distribution. This allows you to create a distribution that will pull in and display completed tasks from other distributions. In effect, the assessor of the distribution you create will be able to see items from previously completed tasks and take them into account when completing their own task.

  • Select a distribution (note that you will only see distributions to which you have access).

  • Define how many tasks must be completed in the linked distribution(s) before the new distribution will take effect.

  • Enter a fallback date. This date represents when your distribution will create tasks whether or not the minimum number of tasks have been completed in the linked distribution.

    • You can adjust the fallback date on the fly and the distribution will adjust accordingly to only generate tasks as appropriate (previously created tasks will still remain).

    • If you set a fallback date that falls before the release date, the fallback date will be ignored.

Targets

The target is who or what the form is about.

• Assessments/Evaluations delivered for: Use this area to specify the targets of the form.

  • For Assessments your options include: targets are the assessors (self-assessment), and learners (

  • For Evaluations your options include: faculty members, courses, course units (if in use), individuals regardless of role, external targets, and rotation schedules.

• Select Targets: Here you can specify your targets using the dropdown selector.

  • CBME Options: This option applies only to schools using Elentra for CBME. Ignore it and leave it set to non CBME learners if you are not using CBME. If you are a CBME school, this allows you to apply the distribution to all learners, non CBME learners, or only CBME learners as required.

• Target Attempt Options: Specify how many times an assessor can assess each target, OR whether the assessor can select which targets to assess and complete a specific number (e.g. assessor will be sent a list of 20 targets, they have to complete at least 10 and no more than 15 assessments but can select which targets they assess).

  • If you select the latter, you can define whether the assessor can assess the same target multiple times. Check off the box if they can.

• Click 'Next Step'

Assessors

The assessors are the people who will complete the form.

• There are three options:

  • Select faculty members

    • Browse faculty and click on the required names to add them as assessors

    • Select Associated Faculty will add the names of all faculty listed as course contacts on the course setup page

    • Exclude self-assessments: If checked, this will prevent the assessor from completing a self-assessment

    • Feedback Options: This will add a default item to the distribution asking if the faculty member met with the trainee to discuss their assessment.

  • Select learners

    • Click Browse Assessors and click on the appropriate option to add learners as assessors

    • Exclude self-assessments: If checked, this will prevent the assessor from completing a self-assessment

  • Select individuals external to the installation of Elentra

    • This allows you to add external assessors to a distribution

    • Begin to type an email, if the user already exists you'll see them displayed in the dropdown menu. To create a new external assessor, scroll to the bottom of the list and click 'Add External Assessor'

    • Provide first and last name, and email address for the external assessor and click 'Add Assessor'

    • Feedback Options: This will add a default item to the distribution asking if the faculty member met with the trainee to discuss their assessment.

• Exclude Self Assessments: Check this to stop learners from completing a self-assessment

• Feedback Options: Check this to add an item to the form requiring the target and assessor to confirm they spoke. More detail here.

• Give access to the results of previous assessments: This relates to Elentra's ability to provide a summary assessment task to users. If enabled, tasks generated by this distribution will link to tasks completed in the listed distributions. When users complete the summary assessment task they will be able to view tasks completed in the other distributions. For any items that are used on both forms, results from previously completed tasks will be aggregated for the assessor to view.

  • Click to add the relevant distribution(s).

• Click 'Next Step'

Results

You can immediately save your distribution at this point and it will generate the required tasks, but there is additional setup you can configure if desired.

• Authorship: This allows you to add individual authors, or set a course or organization as the author. This may be useful if you have multiple users who manage distributions or frequent staffing changes. Adding someone as an author will allow them to more quickly access the distribution from their distribution list.

  • Distributions are automatically accessible to all users with staff:admin group and role permissions.

  • Adding a course permission will make the distribution show, without filters applied, to program coordinators and faculty directors associated with the course.

  • Adding an org. permission will make the distribution accessible to anyone with administrative access to Assessment & Evaluation. (Note that most users will need to apply filters to access the distribution.)

• Target Release: These options allow you to specify whether the targets of the distribution can see the results of completed forms.

  • Task List Release:

    • "Targets can view tasks completed on them after meeting the following criteria" can be useful to promote completion of tasks and is often used in the context of peer assessments. Targets will only see tasks completed on them after they have completed the minimum percentage of their tasks set by you.

  • Target Self-Reporting Release: This controls whether targets can run reports for this distribution (i.e. to generate an aggregated report of all responses). When users access their own A+E they will see a My Reports button. This will allow them to access any reports available to them.

• Target Self-Reporting Options: This allows you to specify whether or not comments included in reports are anonymous or identifiable. (This will only be applied if you have set reports to be accessible to the targets.)

• Reviewers: This allows you to set up a reviewer to view completed tasks before they are released to the target (e.g. a staff person might review peer feedback before it is shared with the learner).

  • Check off the box to enable a reviewer.

  • Click Browse Reviewers and select a name from the list. Note that this list will be generated based on the course contacts (e.g. director, curriculum coordinator) stored on the course setup page.

• Notifications and Reminders:

  • Prompted Responses: This allows you to define whom to send an email to whenever a prompted response is selected on a form used in the distribution. For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response, any time "I had to do it" is picked as an answer an email notification will be sent.

    • You can optionally select to email Program Coordinators, Program/Course Directors, Academic Advisors, Curricular Coordinators, or you can add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

  • Sender Details: Define the email address that notifications and reminders will be sent from for a distribution.

    • Options are the distribution author, an existing user, an existing generic email, or a new generic email.

    • To create a new generic email provide a name and email address. This will be stored in the system and available to other users to use as needed.

• Click 'Save Distribution'.

The Reciprocal Distribution method allows an administrator to set up a distribution that delivers tasks based on the status of tasks delivered by another distribution. Reciprocal tasks can be delivered either when the corresponding distribution tasks are delivered (i.e., the corresponding distribution tasks don't have to be completed in order for corresponding tasks be be delivered by the reciprocal distribution) or when the corresponding distribution tasks are completed (i.e., the corresponding distribution tasks must be completed in order for the reciprocal distribution tasks to be delivered).

An example of a reciprocal distribution might be that you set up a Learner Assessment distribution that delivers tasks to an assessor to assess learner performance; when those are delivered, you'd also like the learners being assessed by the assessors identified in the Learner Assessment distribution to automatically receive an Evaluation task to complete on their assessor.

Summary of Steps Required to Facilitate a Reciprocal Distribution

1. Build the distribution off of which you would like the reciprocal distribution to run; note that this distribution must use either a date range or rotation-based distribution method.

   1. E.g., an admin wants a faculty evaluation task sent to the target of an assessment so that the learner can evaluate to the assessor, first set up the assessment distribution.

2. Build a second distribution that will look at the first distribution in order to deliver tasks

   1. E.g., in the example given above, the next step would be to set up the evaluation distribution.

3. When building the second distribution, select the distribution method of Reciprocal Distribution on Step 2: Method

   1. When Reciprocal Distribution is chosen, the admin can then select the corresponding distribution they want the current distribution to be based on.

   2. The next step allows the admin to select either "I want this distribution to deliver a task each time the corresponding distribution delivers a task" or "I want this distribution to deliver a task each time a task from corresponding distribution is completed".

   3. Step 3: Targets: allows the admin to select "The targets of this distribution will be determined by the corresponding distribution."

   4. Step 4: Assessors/Evaluators: allows the admin to select "The assessors (or evaluators) of this distribution will be determined by the corresponding distribution."

   5. Step 5: the selections available on Step 5 of the distribution will be determined by the type of distribution (assessment or evaluation) that was selected in Step 1.

Create a Reciprocal Distribution

After you have created your first distribution, based on which you would like tasks delivered by a reciprocal distribution, create a corresponding reciprocal distribution.

In this particular example, a faculty evaluation distribution is being created to deliver tasks based on a previous learner assessment distribution; i.e., the first distribution will ask faculty to assess a learner; based on that distribution, this reciprocal distribution will ask the learner to complete an evaluation of the faculty who assessed them.

Step 1: Form

• Enter a Distribution Title and make it as descriptive as possible.

• The optional Distribution Description allows you to clarify the intention of the task and will appear on the task card on the assessor/evaluator’s My Assessments page.

• Select Assessment or Evaluation from the Task Type dropdown (depending on the type of distribution you to create). This selection is important; it will determine available settings in the Distribution Wizard going forward.

  • In this case, Evaluation is selected.

• Leave the Disable Target Release checkbox unticked if you would like the evaluator to be able to choose to release their completed evaluation to the target.

• Leave the Assessment/Evaluation Mandatory checkbox ticked if the assessment/evaluation is considered mandatory.

• Leave the Disable Reminders checkbox unticked if you would like the tasks in this distribution to be included in the weekly reminder summary email notification. If you place a tick in this checkbox, initial task creation email notifications will be sent, but weekly reminder summary emails will be disabled.

• Select a form using the Browse Forms dropdown.

• Select a curriculum period from the Browse Curriculum Periods dropdown.

• Select a course/program from the Browse Programs dropdown.

• Click Next Step to move on to Step 2: Method.

Step 2: Method

Note: the ‘Previous Step’ button allows you to navigate back to Step 1. You can cancel out of the distribution on any page; it will only be saved once you complete all 5 steps.

• Select a Distribution Method from the dropdown: choose Reciprocal.

• Select the Corresponding Distribution: from the list of distributions that you have access to, select the distribution whose tasks you want your reciprocal distribution's task delivery to be based on.

• Specify when the new task should be delivered: identify when you would like the tasks generated by your reciprocal distribution to be delivered based on the corresponding distribution.

  • On previous task delivery: tasks will be delivered by the reciprocal distribution when the corresponding tasks are delivered, regardless of whether or not they are actually completed.

  • On previous task completion: tasks will be delivered by the reciprocal distribution only when the associated tasks on the corresponding distribution are completed.

• Task Expiry: If you check this box, the tasks generated by the distribution will automatically expire; i.e., they will disappear from the evaluator’s task list and no longer be available to complete. You can customize when the task will expire in terms of days and hours after the delivery. When in doubt, leave this field blank.

• Warning Notification: If you choose to use the Task Expiry option, you'll also be able to turn on a warning notification if desired. This can be set up to send an email a specific number of days and hours before the task expires.

• Click Next Step to move on to Step 3: Targets.

Step 3: Targets

Targets are the object of the assessment: those learners whose academic performance is being assessed.

• Target option: select the target will be the assessor/evaluator of the previous task.

• Target attempt options: (always select the first choice, the second choice is for peer assessments). Leave the default setting of: Assessors can assess each target a minimum of (1) times and a maximum of (1) times.

• Click Next Step to move on to Step 4: Assessors/Evaluators.

Step 4: Assessors/Evaluators

• Select an Assessor/Evaluator Option: select the assessor/evaluator will be the target of the previous task.

• Click Next Step to move on to Step 5: Results.

Step 5: Results

The Results settings are optional. If you do not wish to edit them, simply click the Save Distribution button. Otherwise, click on a heading to toggle a section open.

• Authorship:This allows you to add individual authors or set the distribution to be accessible and editable by everyone with A&E access in a program/course or organization. This may be useful if you have multiple users who manage distributions or frequent staffing changes.

  • Choose an Author Type from the dropdown: Individual, Organization or Course (Program)

  • Individual Author names can be added using the Distributions Authors search box.

• Target Release: This option was created for Undergraduate Medicine and can be skipped over by Postgraduate Medicine programs. This option allows you to specify whether the targets of the distribution can see the aggregated results of completed forms. Note: completed evaluation tasks are automatically hidden from target view by the system.

• Task List Release: “Targets can view tasks completed on them after meeting the following criteria” can be useful to promote completion of tasks and is often used in the context of peer assessments. Targets will only see tasks completed on them after they have completed the minimum percentage of their own tasks set by you.

• Target Self-Reporting Release: This controls whether targets can run reports for this distribution; i.e., to generate an aggregated report of all responses. When users access their own A&E badge, they will see a ‘My Reports’ button. This will allow them to access any reports available to them.

• Target Self-Reporting Options: This allows you to specify whether or not comments included in reports are anonymous or identifiable. This will only be applied if you have set reports to be accessible to the targets.

• Reviewers: This allows you to set up a reviewer to view completed tasks before they are released to the target; e.g., a staff person might review peer feedback before it is shared with the learner. They can decide to forward the form to the target or to hide the form from the target.

  • Check off the box to enable a reviewer.

  • Click Browse Reviewers and select a name from the list. Note: This list will be generated based on the course/program’s contacts (e.g., director, curriculum coordinator) stored on the course Setup page.

• Prompted Response Notifications can be enabled to send selected individuals an email message when evaluators choose a response that has been flagged on the form as a ‘prompted response’. E.g., if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response, a notification can be delivered to the individual(s) you identify in this step. You can optionally select to email the Evaluation Reviewers, Program Coordinators, Program/Course Directors, or Distribution Authors.

• Click Save Distribution. The distribution will be ‘live’ as of the next day. It will be triggered when a learner is on the rotation and forms will be sent out on the date defined by the rules you set up.

When tasks are created, if the criteria for delivering the task has not yet been met, you'll see that detail in the distribution progress report.

An ad hoc distribution allows an administrator to set up a form that is available to a specific audience, during a specific time frame, and can be initiated on demand by learners or faculty. A potential use case is a clinical environment where it is unknown which learners will work with which assessors. Using an ad hoc distribution, administrative staff can set up the distribution then allow learners or faculty to trigger the assessment and provide details about who the target or assessor in the situation was.

One advantage of using an ad hoc distribution over a workflow is that you will have a distribution report to review and monitor task completion for the specific distribution.

Please be aware of the following when using ad hoc distributions:

• Users can only select 'complete and confirm via PIN' or 'send blank form' as form completion methods (you can't currently begin a form and send it via email or use the self-assess and send blank form options).

• When setting the targets of a distribution you can only use cohort and individuals (you can't currently set the distribution up with a course audience or course group as the target).

• The available assessors list will be based on course contacts from the course setup page.

• Use the ad hoc distribution for assessments only.

Creating an Ad Hoc Distribution

Permissions: Anyone with access to Admin > Assessment & Evaluation will be able to create an ad hoc distribution. For default installations this will include Medtech: Admin, Staff:Admin, Faculty:Admin, Staff:PCoordinator and Faculty:Directors.

Add New Distribution

• Navigate to Admin > Assessment & Evaluation.

• Click 'Distributions' above the Assessment and Evaluation heading.

• Click 'Add New Distribution'.

Form

• Distribution Title: Provide a title. This will display on the list of Distributions that curriculum coordinators and program coordinators can view. In addition, this distribution title will be used by learners and faculty to access the form. For this reason, a clear title including a course or use is recommended (e.g. Clinical Skills Week 1 History).

• Distribution Description: Description is optional.

• Task Type: Hover over the question mark for more detail about how Elentra qualifies assessments versus evaluations. If a distribution is to assess learners, it's an assessment. If it is to evaluate courses, faculty, learning events, etc. it is an evaluation. Notice that the language on Step 4 will change if you switch your task type as will other steps of the wizard.

• Assessment Mandatory: This will be checked off be default. Currently this information is recorded but does not impact how a task displays to a user.

• Select Form: The form you want to distribute must already exist and you must have permission to access the form; pick the appropriate form from the dropdown menu.

• Select a Curriculum Period: The curriculum period you select will impact the list of available learners and associated faculty.

• Select a Course: The course you select will impact the list of available learners and associated faculty.

• Click 'Next Step'.

Method

• Distribution Method: Select 'Adhoc' from the dropdown menu.

• Start Date: This is the beginning of the period the form is meant to reflect.

• End Date: This is the end of the period the form is meant to reflect.

• Task Expiry: Optional. Set the date on which the tasks generated by the distribution will automatically expire (i.e. disappear from the assessor's task list and no longer be available to complete). Tasks will expire at 12:00 AM on the day that you select, so it is best to select the day after your intended expiry date.

• Delay Task Creation: This option relates specifically to creating a summary assessment task distribution. This allows you to create a distribution that will pull in and display completed tasks from other distributions. In effect, the assessor of the distribution you create will be able to see items from previously completed tasks and take them into account when completing their own task.

  • Select a distribution (note that you will only see distributions to which you have access).

  • Define how many tasks must be completed in the linked distribution(s) before the new distribution will take effect.

  • Enter a fallback date. This date represents when your distribution will create tasks whether or not the minimum number of tasks have been completed in the linked distribution.

    • You can adjust the fallback date on the fly and the distribution will adjust accordingly to only generate tasks as appropriate (previously created tasks will still remain).

    • If you set a fallback date that falls before the release date, the fallback date will be ignored.

• Click 'Next Step'.

Targets

The target is who or what the form is about. You will only set specific targets if you are creating an assessment; if you are creating an evaluation you will not set a target list.

• Select Targets: Use this area to specify the targets of the form.

  • When setting the targets of an ad hoc distribution for an assessment you can only use cohort and individuals (you can't currently set the distribution up with a course audience or course group as the target).

• Target Attempt Options: Specify how many times an assessor can assess each target, OR whether the assessor can select which targets to assess and complete a specific number (e.g. assessor will be sent a list of 20 targets, they have to complete at least 10 and no more than 15 assessments but can select which targets they assess).

  • If you select the latter, you can define whether the assessor can assess the same target multiple times. Check off the box if they can.

• Click 'Next Step'.

Assessors

The assessors are the people who will complete the form. Currently, ad hoc distributions only support making faculty who are course contacts assessors. That means the faculty must be listed on the Course Setup tab as Associated Faculty or as Course Director.

• There are three options:

  • Select faculty members

    • Browse faculty and click on the required names to add them as assessors

    • Select Associated Faculty to add the names of all faculty listed on the course setup page

  • Select learners

    • Click Browse Assessors and click on the appropriate option to add learners as assessors

  • Select individuals external to the installation of Elentra

    • This allows you to add external assessors to a distribution

    • Begin to type an email, if the user already exists you'll see them displayed in the dropdown menu. To create a new external assessor, scroll to the bottom of the list and click 'Add External Assessor'

    • Provide first and last name, and email address for the external assessor and click 'Add Assessor'

• Exclude Self Assessments: Check this to stop learners from completing a self-assessment

• Give access to the results of previous assessments: This relates to Elentra's ability to provide a summary assessment task to users. If enabled, tasks generated by this distribution will link to tasks completed in the listed distributions. When users complete the summary assessment task they will be able to view tasks completed in the other distributions. For any items that are used on both forms, results from previously completed tasks will be aggregated for the assessor to view.

  • Click to add the relevant distribution(s).

• Click 'Next Step'

Results

You can immediately save your distribution at this point and it will generate the required tasks, but there is additional setup you can configure if desired.

• Authorship: This allows you to add individual authors, or set the distribution to be accessible to everyone with A&E access in a course or organization. (This may be useful if you have multiple users who manage distributions or frequent staffing changes.)

• Target Release: These options allow you to specify whether the targets of the distribution can see the results of completed forms.

  • Task List Release:

    • "Targets can view tasks completed on them after meeting the following criteria" can be useful to promote completion of tasks and is often used in the context of peer assessments. Targets will only see tasks completed on them after they have completed the minimum percentage of their tasks set by you.

  • Target Self-Reporting Release: This controls whether targets can run reports for this distribution (i.e. to generate an aggregated report of all responses). When users access their own A+E they will see a My Reports button. This will allow them to access any reports available to them.

• Target Self-Reporting Options: This allows you to specify whether or not comments included in reports are anonymous or identifiable. (This will only be applied if you have set reports to be accessible to the targets.)

• Reviewers: This allows you to set up a reviewer to view completed tasks before they are released to the target (e.g. a staff person might review peer feedback before it is shared with the learner).

  • Check off the box to enable a reviewer.

  • Click Browse Reviewers and select a name from the list. Note that this list will be generated based on the course contacts (e.g. director, curriculum coordinator) stored on the course setup page.

• Notifications and Reminders:

  • Prompted Responses: This allows you to define whom to send an email to whenever a prompted response is selected on a form used in the distribution. For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response, any time "I had to do it" is picked as an answer an email notification will be sent.

    • You can optionally select to email Program Coordinators, Program/Course Directors, Academic Advisors, Curricular Coordinators, or you can add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

  • Sender Details: Define the email address that notifications and reminders will be sent from for a distribution.

    • Options are the distribution author, an existing user, an existing generic email, or a new generic email.

    • To create a new generic email provide a name and email address. This will be stored in the system and available to other users to use as needed.

• Click 'Save Distribution'.

Managing An Existing Ad Hoc Distribution

• Once an ad hoc distribution has been created you can monitor its progress from Admin > Assessment & Evaluation. Click on Distributions, and then the name of the distribution.

• The Show Progress screen you:

  • Not Started (forms that have been triggered but not started)

  • In Progress (forms that have been triggered, started, and saved in draft mode)

  • Completed (forms that have been triggered and completed).

• Click on any of the labels to view the names of the targets and assessors and delivery dates.

• Administrative staff can also send reminders, and add and delete tasks from here.

Learner and Faculty Experience

After an ad hoc distribution has been set up (remember it may take up to a day for your distribution to become active depending on when behind the scenes cron jobs happen), learners and faculty can initiate a task from their main dashboard.

For Learners

• From the dashboard, click 'Start Assessment/Evaluation'.

• Click the Adhoc Distributions tab to move to the correct menu to initiate a task.

• Select a distribution (begin to type a distribution name to quickly filter the list).

• Select an assessor. Hover over an assessor name to see their photo (if uploaded) and details about them including email and course affiliation.

• Select the assessment method. (The methods of completion available to users can be configured in the database for an organization. Speak to a developer if you want to change which assessment methods are available to users.)

  • Email blank form will send a copy of the form to the selected assessor.

  • • If the assessor has a PIN set the system will automatically default to Complete and confirm via PIN as the assessment method.

• Select a date of encounter.

• Click 'Submit'.

• If the learner selected 'Email blank form', the assessor will receive an email alerting them to the task; additionally, they will see the task added to their Assessment Tasks list accessible from their A&E badge in the dashboard header.

• If the learner selected Complete and confirm via PIN, the form will display on the screen. The learner can begin to complete the form, then pass the device to the assessor. When the form is complete, the assessor can enter their PIN to confirm and complete the form.

  • Note that if a form triggered using Complete and confirm via PIN is saved as a draft, the learner will need to reopen the form when in the company of the assessor to have the form completed.

• After a form is completed, the learner's ability to view it will depend on the distribution settings. If the distribution allows the learners to view the tasks, they will be able to view them from their Tasks Completed on Me list accessed from their A&E badge in the dashboard header.

For Faculty

• From the dashboard, click 'Start Assessment/Evaluation'.

• Click the Adhoc Distributions tab to move to the correct menu to initiate a task.

• Select a distribution (begin to type a distribution name to quickly filter the list).

• Select a target. Hover over a target name to see their photo (if uploaded) and details about them including email, group and role (e.g. student, 2022) and enrolled courses.

• Select a date of encounter.

• Click 'Begin Assessment'.

• The form will display on the screen.

• The faculty member can complete the form and Save as Draft or Submit. Note that if the user saves the form as draft, they will have to reopen and complete it at a later date.

  • The faculty member can delete the task if they triggered it in error.

  • The faculty member can forward the task to another faculty member if needed.

• Faculty can view their draft forms, or forms that leaners have triggered to them from their Assessment & Evaluation badge located in the dashboard header. The tasks to be completed will display in the Assessment Tasks list.

The Evaluation Reports section will allow you to generate reports based on evaluations completed via distributions. Evaluation reports will not include commenter names, even if you check off the commenter name option when setting the report options.

Rotation Evaluations (Aggregated)

This report is relevant only if your organization uses the Clinical Experiences rotation schedule. If you distribute rotation evaluations through a rotation based distribution, you can use this report to view results. Obviously the exact format of the report will depend on the form it is reporting on.

• Select a course, date range, rotation, curriculum period and form.

• Set the report parameters regarding displaying comments and averages.

• Click 'Generate Report'.

This report will not include learner names, even if you check off the commenter name option when setting the report options.

Learning Event Evaluations (Aggregated)

For use with distributions completed by event type.

• Set a date range.

• Select Individual Events: Check this off if you want the ability to select individual events (otherwise you will have to report on all events).

• Select the event type, distribution by a curriculum period, learning event and form.

This report will not include learner names, even if you check off the commenter name option when setting the report options.

Learning Event Feedback Forms (Aggregated)

Use this to report on feedback provided by participants when a feedback form is attached to an event as a resource.

• Select a course, date range, event type, form and learning event (optional).

You will notice some extra report options for this type of report.

• Separate Report for Each Event: This will provide separate files for each report if multiple events are selected to include.

• Include Event Info Subheader: This will provide a bit of detail about the event being evaluated (title, date, and teacher).

This report can include an average and an aggregate positive/negative score. This report will not include learner names, even if you check off the commenter name option when setting the report options.

Faculty Reports (Aggregated)

For use in viewing a summary report of learner evaluation of an instructor.

• Select a course and set a date range.

• Select a faculty from the dropdown menu by clicking on his/her name. Note that only faculty associated with the selected course in the given time period will show up on the list. Additionally, they must have been assigned as an assessor in another distribution in the organization. Please see additional information below.

• Select a form and distribution (optional).

• Set the report parameters regarding displaying comments and averages.

• Click 'Download PDF(s)'.

This report will not include an average, even if you check of Include Average when setting the report options.

The way faculty names become available to select for this report is when the faculty member is also an assessor on a distribution in the organization. This is designed in part to protect the confidential nature of faculty evaluations and prevent staff from being able to generate reports on any faculty at any time. If your organization does not use assessments or you require reports on faculty whose names aren't available, please reach out to us and we can help you put a work around in place.

You may also be able to report on faculty evaluations by accessing an aggregated report from a specific distribution. Please see more detail in the Weighted CSV Report section in Distribution.

Course Reports (Aggregated)

For use in viewing a summary report of learner evaluation of a course.

• Select a course and set a date range.

• Select a form and distribution (optional).

• Set the report parameters regarding displaying comments and averages.

• Click 'Download PDF(s)'.

This report will not include learner names, even if you check off the commenter name option when setting the report options. This report can include an average and an aggregate positive/negative score.

Individual Learning Event Evaluations

• Select a Course

• Set a date range for the report.

• Select the event types.

• Select the appropriate distributions.

• Select the relevant learning event (optional).

• Optionally select to download as one file.

Automatic Rotations Evaluations (Aggregated)

Although this report is included in Elentra ME 1.17, it will not be populated unless you are using automated rotation evaluations enabled via the Clinical Experiences Rotation Schedule.

• Select a course/program

• Set the date range

• Select a form (if you had multiple published forms you might be able to pick them)

• Select a rotation schedule (the options available will be based on the set date range)

  • If you select multiple rotations, results will be aggregated (this can function a bit like a program evaluation if desired)

• Choose whether to view results as a CSV or PDF

This report will not include commenter names because it is an evaluation.

Note that in a rotation based distribution you are able to toggle between a block view and date range view of the distribution report. When using Block view you can view completion per block and switch blocks as needed using the forward and back arrows.

Additionally, if the target option for a distribution was that the assessor/evaluator would define their target when completing a task, pending tasks will display the target as to be defined.

In ME 1.14 Elentra introduced a way for learner's to optionally release completed evaluation tasks to the targets of the task. In effect, a learner can complete an evaluation of a faculty member and choose to immediately release the completed task to said faculty member. The faculty can see who the feedback came from (in contrast, if you allow faculty to run reports on a distribution of which they are a target, they do not see the personal details of the evaluators).

While advantageous in some circumstances, allowing learners to optionally release completed evaluation tasks to facutly is not desirable to all organisations. There are database settings that can be used to turn this feature on or off at the course and organisation level. To use these you need a developers help. However, you can also control the learner's ability to release completed evaluation tasks at the distribution level on Step 1 of the distribution wizard.

The Disable Target Release option will only be visible on Evaluation tasks and it does not impact the faculty member's ability to create a report on a distribution (that is still controlled by the distribution in Step 5 of the distribution wizard).

To see what the learner experiences when allowed to optionally release completed evaluation tasks, please see here.

After a distribution is created you can view a distribution report. Reports include an overview of the tasks generated and not started, the tasks in progress and completed tasks. Click on any of the status cards to display the individual tasks in each status category.

From the Distribution Report you can also complete a variety of actions including: deleting a task, recovering a deleted task, sending a reminder, adding a task, downloading completed tasks, and completing a task on someone's behalf. For some distribution types you can also generate a weighted csv report from the Distribution Report.

More details about these actions are available in the .

• Elentra's Summary Assessment Task option allows you to create a distribution that allows the assessor for said distribution to access the results of previously completed tasks (on a per distribution basis).

• Please note that this feature is currently for distributed assessment tasks only. Additionally, it can't be used with form templates and tasks triggered on-demand (i.e., CBME tools).

• The assessor of the summary assessment task can review data multiple ways:

1. View aggregate data (but not average scores) for any items that were included on tasks in associated distributions and their current task (note, the assessment item ids on each form must match for this feature to work)

2. View comments from tasks from associated distributions and see the date, time, assessor and distribution

3. Search for and view completed tasks from associated distributions, even if not all items are included on the summary assessment task

• While reviewing previous tasks, easily navigate between distributions, tasks, or learners as needed.

• When administrators configure a distribution to create summary assessment tasks, they can:

  • Define which previous distributions to link to (associated distributions)

  • For date based distributions and adhoc distributions only:

    • Optionally define a minimum number of tasks to be completed in the associated distribution(s) before summary assessment tasks are generated

    • Optionally define a failsafe date on which to send the summary assessment task, even if minimum numbers tasks in associated distributions have not been achieved

Summary Assessment Tasks FAQ

1. If a summary assessment task has been completed and then more tasks are completed in the associated/evidence distributions, will the summary assessment task reopen to be edited?

   Not automatically. If an admin user wants to manually reopen the summary assessment task so that the assessor can edit their responses, they can do so. Note that if the task was already made visible to learners, editing it may cause confusion.

2. What happens if you set the delivery date for the distribution before the fallback date? The fallback date will be ignored.

3. How soon after the tasks in the associated distributions are completed will the summary assessment task be created? The summary assessment task will be eligible to be created as soon as the tasks in the associated distributions are completed. When the tasks actually get delivered to the assessor will depend on the schedule of some behind the scenes tasks (cron jobs) that your installation of Elentra performs. At many schools, the tasks will be delivered the next day.

When you create a distribution, one of the options available via the Results tab is Target Release and Target Self-Reporting Options.

A further option from there is: Targets can view tasks completed on them after meeting the following criteria.

This option allows you to require that users complete a specific percent of tasks in a distribution before they can review their own results from the same distribution. This is typically used for peer assessments so that learners are required to provide feedback to others before they can see their own feedback.

Please note that these settings are specific to a single distribution only. This tool does not require users to have completed a percentage of tasks across all distributions.

When completing a distribution you will notice a "Feedback Options" section in Step 4 if the assessor is set to faculty members (it will not appear when the assessor/evaluator is set to learners). If you check this off it will add an item to the form for this distribution asking both the assessor and the target whether or not they met to discuss the target's performance (see sample text below).

You might choose to use this option and add the item if you want to collect data on how often preceptors are meeting with learners.

In effect, this tool adds two items to a form. The first item will appear as above for the assessor to complete. Once the form is completed it will be stored in the learner's "Tasks Completed On Me" tab. The learner can access the form from there, and the learner will also receive an email notification that they can take action on the form.

The form is available for the learner to complete and answer the same question from their perspective. The comment box is available for learners to record any additional details.

As of Elentra ME 1.17, there is no reporting tool to compare learner and assessor responses, although they can be viewed on any completed forms. There is also no visual cue to an admin user that the response from the learner is pending; if there is no learner response displaying it means the learner hasn't answered the question.

Setting up on-demand workflows allows you to configure forms so that Elentra users can initiate a form when needed without administrative staff creating a distribution. You can use workflows to:

• allow users to initiate assessment tasks (EPA-based and other), and

• allow learners to initiate rotation and faculty evaluations.

If there are forms you do not want users to be able to initiate on-demand, simply set the workflow to None.

You will need to decide as an organization how you want to use workflows and provide the relevant information to your developers for them to appropriately set up the database.

Sample uses include:

• Allow learners or faculty to initiate CBME assessments (e.g. Supervisor form) on demand

• Allow learners or faculty to initiate any generic assessment form on demand

• Allow learners to initiate faculty evaluations on demand

• Allow learners to initiate rotation evaluations on demand

There are some things to note about using on-demand workflows:

• Forms will only show on the Start Assessment screen if they have an affiliated workflow. If a form's workflow is set to None it will not be available on demand and should be used only in distributions.

• To use on-demand workflows, all forms associated with a workflow must be permissioned to a course.

• Only learners should be set up to initiate faculty and rotation evaluations on demand. This is because Elentra uses learners' schedules to determine which forms they should be able to access. There is no user interface to support faculty completing standard course and faculty evaluation forms.

Types of Workflows

1. EPA - This is designed specifically for organizations using CBME. Use for forms tagged to EPAs that you want learners to be able to trigger. Forms using the EPA workflow contribute to CBME dashboards. Please note that forms generated from CBME form templates (e.g., supervisor form) will automatically have an EPA workflow added to them.

2. Other Assessment Form - Use for forms that you want learners to be able to trigger and complete on demand without tagging to EPAs, or forms that you don't want to appear in the EPA list when triggering. Can be applied to both forms tagged with EPAs/milestones and forms that are not tagged with EPAs/milestones. Only forms with EPAs/milestones tagged will contribute to CBME Learner and Program dashboards.

3. Faculty Evaluation - Use for faculty evaluations. Learners can initiate on demand. A date range is included when triggered. Optionally use with standard faculty evaluation form types.

4. Rotation Evaluation - Use only for rotation evaluations and when scheduling learners in rotations using the Clinical Experience Rotation Scheduler. Learners can initiate on demand. Optionally use with standard faculty evaluation form types.

   • Note: Learners can only evaluate rotations that they have completed or are currently in. They cannot initiate an evaluation for a rotation scheduled in the future.

5. None - Use for forms that you do not want users to be able to initiate on demand. You can still attach these forms to distributions. You can still tag EPAs or milestones as needed.

Optional Automatic Expiry Date for On Demand Tasks

If an organization wants to optionally enable automatic expiry dates for workflows they can do so using a database setting (cbme_ondemand_expiry). Developers must additionally define which workflows will have the automatic expiry dates applied to them (cbme_ondemand_expiry_workflow_shortnames). By default, Elentra will apply a expiry date 7 days after the task is generated. Organizations can modify this via a database setting (cbme_ondemand_expiry_offset).

This tool also includes support for administration users to reset expiry dates manually from the assessments interface. Program coordinators and course directors can view expired assessments which have a visual indicator that they have expired. A Reset button can optionally be used to reset the expiry date for the task.

User Experience

Users click the green Start Assessment/Evaluation button on their dashboard to access a form using a workflow. They may see two tab menus: one for Start Assessment/Evaluation and one for Adhoc Distributions. To use a workflow they should click 'Start Assessment/Evaluation'.

• Select a workflow (options available will depend on how you have configured workflows in your database).

• Remaining steps will depend on the workflow selected.

Developer Information for Workflows

To allow the use of form workflows in an organization, some setup is required. You need to associate form types with workflows, and, if necessary, restrict the user groups who can access specific workflows.

You can review the available form types in cbl_assessments_lu_form_types If your organization wants to use standard rotation evaluation or standard faculty evaluation forms, you will need to add those to the database and associate them with the correct audience.

Specific workflows are listed in cbl_assessments_lu_form_workflows

cbl_assessments_lu_form_type_workflow_link acts as a junction table to associate form workflows with form types.

The migration that runs will assign a workflow of None to all form types. How you configure things depends on how your organization will use workflows. For example, in a CBME setup you could associate Supervisor Forms with the EPA workflow. You could associate Generic Forms with Other Assessment Form workflow. How you set this up depends on how you want to allow users to initiate forms on demand.

Finally, cbl_assessments_lu_form_workflow_groups is responsible for restricting access to a specific workflow to specific user groups.

• The migration that runs will populate this table so that by default, only students can initiateFaculty Evaluation or Rotation Evaluation workflows on demand.

• Any form and user group combination added to this table creates exclusive rights for those user groups to access a form workflow.

• If no workflow is associated with a user group, then all user groups should be able to initiate that workflow on demand.

Program or Curriculum Coordinators have two methods to monitor completion of forms and tasks associated with their program, faculty and learners.

Coordinators can use Admin > Assessment & Evaluation to access distributions they are permissioned to, view A & E Reports, and see outstanding, upcoming, deleted, and completed tasks for users associated with their course/program.

Coordinators can access their own tasks (including delegations sent to them) and monitor the progress of individual faculty and learners through the Assessment & Evaluation Badge located at the top right of the screen.

There is a tool to allow staff:admin users to reopen a completed assessment and edit it as needed.

• Navigate to your Assessment & Evaluation badge (beside own name in the top left).

• Click on My Learners and then Assessments under a specific learner's name.

• From the list of Tasks Completed on Learner click on a task to view it.

• Click the "Reopen Task" button just below the form title. This will set the task to in-progress and the staff>admin or faculty will be able to adjust it.

• Provide a reason to reopen the task (e.g. was accidentally deleted, was missing data, other).

• Click 'Reopen Task'.

• Once reopened, a user can complete the task and submit it on behalf of the original assessor, or they can forward the task to the original assessor to update.

If an administrator is looking at a task with an expiry date, she can extend that date if needed. (This option will display when the task was initiated on-demand by a user AND an organization has their database setting to enabled.)

• Click 'Reset Expiry' in the top right

• Check the date indicated as the new expiry date

• Confirm by clicking 'Reset Expiry'

• You will return to the view of the task

If users view the task from their Assessment & Evaluation badge, they'll see the updated expiry date on their task card.

Administrative users have the ability to trigger an assessment to send tasks on an as needed basis. This might be useful if a specific learner does not have enough assessments in a specific area or there is a reason to prompt a specific assessor to complete a task on a specific learner.

• Navigate to Admin > Assessment & Evaluation.

• Click 'Trigger Assessments'.

• Select a learner.

• Select an assessor.

• Define a form type (e.g. CBME or Generic).

• Set a Start and End Date for the Assessment (this defines the period of time the assessment addresses).

• Optionally set the assessment as mandatory. (This means the asessor will not be able to delete the task.)

• Optionally indicate if feedback is required for the assessments. If you check this off it will add an item to the form asking the assessor whether or not they met with the trainee to discuss their performance (see sample text below). The target will also be asked to answer this question.

• Optionally set prompted response notifications. This allows you to decide whom to contact if any answer on the form is designated as a prompted or flagged response option. (For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response.) You can optionally select to email the Program Coordinators, Program/Course Directors, or add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

• Click 'Get Tools'.

• Search for the appropriate tool as needed.

• Click 'Preview This Form' to confirm you have the form you want or click 'Send Assessment' to send the assessment to the selected assessors.

• You will be prompted to optionally provide an Assessment Cue. This can be a short note to explain the context of the assessment. The text entered will display at the top of a form when the assessor is completing it.

• Click 'Send Assessment.'

• A green success message will display at the top of the page and you will be returned to the beginning of the bulk trigger assessment process to repeat if needed.

A staff:admin or program coordinator can enter completed assessment forms on behalf of assessors. This allows faculty to complete a pen and paper version of a form and still have the data entered into Elentra.

To enter a completed form on someone's behalf:

• Navigate to Admin > Assessment & Evaluation.

• Click on the green 'Record Assessment' button below the orange Assessment Tasks heading.

• Select a learner (you will need to know the curriculum period the learner is in) and assessor from the searchable dropdown menus.

• Select an assessor (this should be the person who completed the pen and paper form).

• Select a course/program (this will only be necessary if the learner is enrolled in multiple programs).

• Select a CBME Version if necessary. (This helps limit to tools returned to the appropriate curriculum version).

• Select a Date of Encounter (i.e., the day the form was completed).

• Select an EPA as you would to initiate a form. Filters and search are available.

• Search for the appropriate form. You can preview the form to make sure it is the one you want or click 'Begin Assessment' to start a form.

• You will be submitting the assessment on behalf of the selected assessor. There is a reminder of the selected assessor displayed at the top of the form in a yellow information bar.

• Complete the form and click 'Submit'.

• You will get a green success message and be returned to the assessment entry screen to complete another form if needed.

Curriculum and program coordinators as well as some faculty directors can access Admin > Assessment & Evaluation to monitor the status of assessment and evaluation tasks.

Typically, coordinators and faculty directors will access this page to see task progress for learners and faculty associated with their course. For example, if Jane Doe is set as the program coordinator for Pediatrics, she will see the tasks of all learners enrolled in Pediatrics and all faculty who are associated with Pediatrics because they are an assessor on a distributed form associated with Pediatrics, or they are the target or assessor on a user-initiated on-demand form associated with Pediatrics.

Admin > Assessment and Evaluation is a useful access point to send reminders for specific tasks, record assessments completed outside Elentra, trigger bulk assessments, etc. More details about those actions are included in the following pages.

Filters on the A & E Dashboard

On the A & E dashboard tab, tasks can be filtered to easily view outstanding, upcoming, deleted, and completed tasks, as well as prompted responses.

• Click on the task status you want to review (e.g. Outstanding, Deleted, etc.)

• Enter a search term (e.g. a user name or form title).

• Apply a delivery type filter if desired (e.g. distribution based).

• Set a date range if desired.

• Click on a task to view it in a new tab.

Field Display Options (showing additional labels on tasks)

• Click the Field Display Option button to optionally add additional fields to your dashboard display

  • Start and End Date - represents the start and end date or the relevant task (if defined, e.g., in a distribution or when a task is initiated on-demand)

  • Course - the course a task is tied to

  • Event - the event a task is tied to (if applicable)

• Choosing a field will add the appropriate labels to the tasks in the Tasks column

• When viewing a task a coordinator can optionally complete the task on behalf of another user, forward the task to be completed by someone else, or delete the task.

Assessments and Evaluations Tabs

Click 'Assessments' or 'Evaluations' to toggle between the types of tasks you are reviewing. Remember, in Elentra Assessments are about learners, and Evaluations are about other things (e.g., faculty, courses, rotations, events).

Administrative users have the ability to trigger bulk assessments to send tasks to multiple assessors, for multiple targets, in the combinations required.

• Navigate to Admin > Assessment & Evaluation.

• Click 'Trigger Bulk Assessments'.

• Select a course/program.

• Select an assessor and target(s) as required. Note that to search for targets you will need to filter by curriculum period.

• For each target you can set a start and end date to reflect what the assessment period should be.

• To add additional assessors (and targets) click the 'Add Assessor' button.

• Optionally set the assessments as mandatory.

• Optionally indicate if feedback is required for the assessments. If you check this off it will add an item to the form asking the assessor whether or not they met with the trainee to discuss their performance (see sample text below). The target will also be asked this question.

• Optionally set prompted response notifications. This allows you to decide whom to contact if any answer on the form is designated as a prompted or flagged response option. (For example, if you have an item asking about student completion of a procedure and "I had to do it" was set as a prompted/flagged response.) You can optionally select to email the Program Coordinators, Program/Course Directors, or add a Custom Reviewer. If you select to add a Custom Reviewer you can select their name from a searchable list of users.

• Click 'Tools'. The tools returned will be those that are permissioned to the course/program you selected at the top of the page.

• Click 'Preview This Form' to confirm you have the form you want or click 'Send Assessment' to send the assessment to the selected assessors.

• You will be prompted to optionally provide an Assessment Cue. This can be a short note to explain the context of the assessment. The text entered will display at the top of a form when the assessor is completing it.

• Click 'Send Assessment.'

• A green success message will display at the top of the page and you will be returned to the beginning of the bulk trigger assessment process to repeat if needed.

If your organization uses a standard rotation or faculty evaluation form (i.e., the same form is used to evaluate all rotations in a course, or for all faculty teaching in a course), there are some options available to reduce the administrator work necessary to collection evaluation data.

Please note that Elentra can include a Standard Rotation Evaluation form type and a Standard Faculty Evaluation form type. These form types are not in Elentra by default and will need to be added by a developer to be used. Use of these specific form types is required if you want to either a) provide templated items on these forms to promote consistency across courses, or b) plan to use the automated rotation evaluation option (set when building a rotation). If neither of those situations apply to you, you can still use on-demand workflows to allow for rotation and faculty evaluation.

1)

• If you want to allow learners to initiate evaluations of rotations and faculty on an as needed basis you can use on-demand workflows.

• Faculty Evaluation and Rotation Evaluation workflows can be applied to any type of form permissioned to a course to allow learners to initiate the form on-demand.

  • Not that you can not set up faculty evaluation or rotation evaluation workflows to allow faculty or staff to evaluate faculty or rotations

• Note:

  • If a learner initiates a form using a Rotation Evaluation workflow, Elentra checks the learner's rotation schedule. The learner will be able to complete a rotation evaluation for any past or current rotation they are scheduled into. A learner will not be able to initiate an evaluation for a rotation they are scheduled into in the future.

  • If a learner initiates a form using a Faculty Evaluation workflow, they will be able to access a complete list of all faculty stored in Elentra.

2) Automatic Standard Rotation Evaluation Option

If your organization uses the Clinical Experience rotation scheduler and a Standard Rotation Evaluation form type to evaluate rotations within courses/programs, you could use the automatic standard rotation evaluation option. This feature is enabled when you build rotations, and removes the need to have distributions or use workflows for rotation evaluations. Instead, each course/program can create a standard rotation evaluation form (which can have preset standard items shared across courses), and the form will be auto-distributed to learners based on their rotation schedules. This option requires developer assistance. A developer must add the appropriate form type in the database and build the templated items if desired (there is no user interface to build the templated items).

Please note that the automatic rotation evaluation option is specifically to support rotation evaluations, and that course evaluations should still be sent via distributions.

Each program/course can only have one standard rotation evaluation form active at a time. If you create a second form of this type it will override the first form you made.

To use the automated standard rotation evaluation feature, when you build a rotation make sure to check off the "Automated standard rotation evaluation" button. Set the preferred delivery schedule for these automated tasks.

Implementation Notes

If your organization wants to use a standard faculty or rotation evaluation form a developer will need to add those two form types to the database and associate them with the correct organisation (relevant table: cbl_assessments_form_type_organisation).

Additionally, a developer will need to create the standard rotation evaluation items. These items that will be included on every standard rotation evaluation form (courses/programs will be able to add additional items as needed).

Once the form types and standard items are added, administrative users can build standard rotation evaluation forms and publish them as needed.

Leave by Block Report

For use when you have clinical learning courses with block schedules and want an overview of those learners who had an approved leave during a specific block.

• Select a curriculum period.

• Select a block.

• Click 'Download PDF'.

Rotation Leave Report

For use when you have clinical learning courses with rotation schedules and want an overview of those learners who had an approved leave during a specific rotation.

• Set a date range.

• Select one or more learners.

• Set the report parameters regarding displaying description and comments.

• Click 'Generate Report'.

• Note that once generated, this report is available to download by clicking 'Download PDF'.

Elentra includes a variety of reporting options through the Admin > Assessment & Evaluation tab. These reports reflect data collected through forms managed through distributions. To view reports for triggered forms when using the Competency Based Medical Education module, use the learner CBME dashboard. (More information about CBME reporting is available in the CBME section.) . One exception to this is if you are trying to report on form feedback from form built with form templates in CBME - that feedback is available through the Assessment Tools Feedback Report in the Assessments section.

A&E Reporting is an administrative reporting tool. Note the individual users may have access to their own reports via their Assessment & Evaluation button, but the availability of such reports depends on how distributions were set up.

Reporting options

When you create most reports you will have some additional options after selecting the appropriate course/faculty/learner/form, etc. These options allow you to customize the reports you run for different audiences.

Please note that not all these options will actually display on all reports, despite the fact that you have the option to select them through the user interface. Please see each specific report for additional detail about what will or will not be visible.

Include Comments: Enable this option if you'd like the report to include narrative comments made on the selected form. Unique Commenter ID: If you select to include comments you'll see this option. It allows you to apply a masked id number to each assessor/evaluator. This can be useful to identify patterns in comments (e.g., multiple negative comments that come from one person) while protecting the identity of those who completed the form. Include Commenter Name: If you would like to display the names of commenters click on the checkbox. Include Description: If you click this checkbox you can give a description to the report. The text you enter will be displayed at the top of the report generated. Include Average: Click this checkbox to include a column showing the average score. Include Aggregate Scoring: If you enable the average, you'll have the option to also include a column with aggregate positive and negative scoring in some reports. This gives a dichotomous overview of positive and negative ratings.

Distribution Delivery Schedule

This report can be used by administrative staff to keep an inventory of distributions.

• Select a report type. You can see an overview of distributions or individual tasks.

• Select a course.

• Set a date range.

• Select a task type.

• Select a distribution or an individual task (your option will depend on the first selection you made on the page).

• Click 'Generate Report'.

• From here you can search within the results or click on any distribution to see its progress.

Distribution Reviewer

Use to see an overview of who is set as a reviewer for distributed tasks. (When you create a distribution you can assign a reviewer who serves as a gatekeeper of completed tasks before they are released to be seen by their target. This is completed during the final step of a distribution. For additional information please see the Assessment and Evaluation>Distributions help section.)

• More information coming soon.

Embedded Distribution Reviewer

This report functions like the report above but offers users a view of the report in the interface without requiring them to open a PDF.

Weighted CSV Report

Please see the next page for information on the Weighted CSV Report.

What is the Impact of Retiring vs. Deleting a Form?

Retiring a PPA or Rubric/Flex form that has been TRIGGERED:

• We recommend that if the form was in use at all, simply retire it instead of deleting it so that any Pending or In progress assessments/evaluations can still be completed by their assessor/evaluator.

Deleting a PPA or Rubric/Flex form that has been TRIGGERED:

• All pending and in-progress tasks that used that form will not have a form associated with them and will display an error message stating that the form has been deleted when assessor/evaluator tries to access them (retiring is the better choice!).

• If needed, a developer can restore the old form to allow assessors/evaluators to complete the assessment/evaluation task that uses the form.

• If you delete a form that has been used to trigger an assessment, then any pending assessments can no longer be completed.

Retiring a PPA or Rubric/Flex form used in a DISTRIBUTION:

• All pending and in-progress tasks that used the form will still be accessible to the assessors, but will be not be available for selection on any new distributions.

• You can go ahead and replace the form on the distribution whose form has been retired; all future tasks will use the new form.

Deleting a PPA or Rubric/Flex form used in a DISTRIBUTION:

• All pending and in-progress tasks that used that form will not have a form associated with them and will display an error message stating that the form has been deleted when assessor/evaluator tries to access them (retiring is the better choice!).

• If you replace that deleted form with a new form on the distribution, all future tasks will use the new form.

• If there are pending or in-progress tasks where users need to have the deleted form replaced with the new form, a developer can replace the old form with the new one for pending tasks.

  • EXCEPT: They cannot switch assessment forms for tasks where the assessor/evaluator has already completed the assessment/evaluation form for one or more targets, but not all of them. In this case, they would have to temporarily restore the old forms so that the assessor/evaluator can finish their tasks. They can only switch assessment/evaluation forms for completely pending assessments (meaning they don't have any progress).

Deleting a FORM TEMPLATE:

• Deleting a form template does NOT delete the forms that were previously generated by the deleted template. They remain to allow assessors/evaluators to complete them; however, the old forms will not be available going forward.

What is the Impact of Replacing a Form on an Active Distribution?

You can create your distribution using one form, and then switch the form on the distribution to the new form when it is ready. All future tasks created by the distribution will use the new form.

It is recommended that, if you replace a form that you no longer wish to use, you retire the form so that previously delivered tasks using that form can still be completed.

As long as you do not delete the old form altogether, any pending or in progress tasks that have been delivered using the old form can still be completed by the assessors/evaluators.

If you need to replace the old form with the new form for any pending tasks (pending tasks are tasks that have not yet been started by the assessor/evaluator) you will need to get help from a developer.

 Once all in progress and/or pending tasks using the old form have been completed, you can delete the old form if you wish. You can also choose to delete the old form if there are in progress and/or pending tasks that are using it, but the assessor/evaluator will receive an error message stating that the form no longer exists when they attempt to complete it. In this case, it is better to retire the form, so that these tasks can still be completed using the old form, but the old form will no longer be available to select when creating future distributions.

What is the Impact of Editing Other Aspects of an Active Distribution?

• Editing the Expiry Date: If an expiry date has been applied to the active distribution, you can edit the expiration date to 're-open' tasks to the targets who haven't completed them yet. Those tasks will be visible to the targets the day after the update is made to the expiry date (because the job that runs behind the scenes to deliver tasks runs at night). A developer can run the job immediately if it is time sensitive.

• Changing the Targets: If you add a new targets to a distribution, the system will reopen the task for the assessor/evaluator to allow them to complete the task on the new target.

• Editing the Rotation Schedule of Targets Associated with a Distribution: will cause new tasks to be delivered to the assessor.

• Changing the Reviewer of a Distribution: will cause the distribution to redeliver previously delivered tasks; be sure to apply a Release Date when making this change.

• Changing the Target Release Options in Step 5: Changes the visibility of tasks/reports retroactively for all completed tasks.

The Assessor/Evaluator Completed an Assessment/Evaluation on the Incorrect Target. What do I do?

• Navigate to Admin > Assessment & Evaluation.

• Click on the Distributions tab.

• Edit the distribution that had the incorrect target selected (remove the incorrect target, add the new target).

• Ask a developer to push the distribution through to reflect the changes OR the next day (after the system has a chance to run the changes to the distribution overnight), navigate to the distribution and click on its title to view its Progress page.

• Find the task in the Available assessments tab (the assessment is available because it is new).

• Click on the name of correct target.

• Use the Choose a Target dropdown menu to open the completed assessment, which was completed on the incorrect target.

• Click the Choose a Target dropdown menu and click the circular double arrow to the right of the correct target under Forms Not Started to assign the completed assessment to the correct target; you will see the page refresh with the correct target and the completed form.

• Now the incorrect target will be listed in the Forms Not Started list, and you can delete that target for the assessor/evaluator: click the Choose a Target dropdown menu, click on the incorrect target in the Forms Not Started list, click Delete Task, and complete the required fields to remove the task.

Elentra automatically sends email notifications to users when they have an Assessment and Evaluation task to complete or review.

Tasks scheduled via distribution

Assessors/evaluators will receive one email notification overnight with all new tasks linked in the body of the email.

For tasks scheduled via a distribution, the timing of when specific tasks will trigger an email will depend on how the distribution is configured (e.g., the task delivery date set, the timing of learning events for event-based distributions, dynamic delivery settings for rotation-based distributions).

Note that if a delegation-based distribution has been created, when a delegator forwards a task to a user, a distinct email will immediately be sent for that task.

User-initiated on-demand tasks

For user-initiated, on-demand tasks, email notifications will be created when the initiating user submits the form and the task moves to a new owner (i.e., assessor).

Additional Notifications

Other circumstances that can generate email notifications for a task include:

• A program administrator sends a reminder for a specific task;

• A task is forwarded from one user to another user.

Course directors, program administrators, and users set to receive prompted response notifications for specific distributions will also receive email notifications when they have tasks to review. This could include:

• In a CBME context, when any user completing a form has answered "Yes" to a concern item on a CBME form generated from a form template (i.e. Supervisor Form, Field Note Form, Procedure Form)

• When a user has indicated an answer configured as a prompted response on an item, when the form containing that item is included in a distribution configured to have prompted response notifications.

Database Configurations for email Options

There are some configurable options for email notification which can be controlled through the database (not through the user interface). This includes the ability to adjusts emails for tasks scheduled via distribution to send a nightly summary of all new tasks, and then a weekly summary of tasks that are still incomplete.

Distributions in Elentra can be configured to allow assessors/evaluators to view the results of previously completed tasks when completing their own tasks.

Say you have a learner on a clinical rotation. Every three days they may get assessed by a preceptor using a daily encounter form. At the end of the rotation, a director is expected to complete a final assessment form on the learner. Creating a head form distribution can allow the director to view the previously completed daily encounter tasks and use aggregated data to help them make their ratings on the final assessment form.

For any items that were used on both forms, the head form assessor will be able to see an aggregate count of ratings and comments made on previously completed tasks.

The head form assessor can also view any completed task in full (e.g. see all additional items included on form, any comments given, etc.).

Faculty have quick access to review their assessment and evaluation tasks and those of their affiliated learners through the Assessment and Evaluation badge located at the top right.

• Click the Assessment & Evaluation badge in the top right.

• A series of tabs will open under the Assessment & Evaluation header.

• When viewing results on any of the tabs in A&E, use the search bar to look for a specific task by name and click the down arrow to limit your results by applying search filters like distribution method (date range, delegation, learning event, rotation schedule), curriculum period, course, and task status (pending, in progress, completed).

• Apply start and end dates to limit your results to a specific time frame.

• Remember to click 'Apply Filters' to apply your choices.

• Remember to click 'Remove Filters' to clear filters and view all results. Note that your previous filter settings may still be applied so if you are seeing no results, or fewer results than you expected, try 'Remove Filters'.

Assessment Tasks

This tab shows users all pending tasks for which they are responsible. Mandatory tasks are indicated with a red highlight with a bold red "Mandatory" title on the card.

• Faculty can view and complete a task by clicking on it.

• Faculty can remove a task by clicking 'Remove Task' and providing a reason for its removal.

• Faculty can download individual forms, or download multiple forms by clicking the download checkbox on each card and then clicking 'Download PDF(s)' at the top right. When users do this they will be able to choose whether to download all tasks as one file or as multiple files.

Tasks Completed on Me

This tab displays all tasks completed on the user and which s/he has been given access to view. This can be controlled in the creation of a distribution.

My Completed Tasks

• The user can view all completed tasks and again, download an individual file or download multiple files at once.

My Feedback

This tab shows feedback received from learners who have given a thumbs up and optionally commented on an assessment form completed by the faculty user. It is learner feedback to the faculty member on how comments/information provided on a completed assessment task helped the learner.

My Learners

The My Learners tab will allow faculty to access a variety of information for learners associated with them. This list of available options will vary depending on which modules are in use in Elentra but could include CBME Dashboard, Assessments (refers to tasks managed through Assessment and Evaluation), Learner Explorer (refers to course gradebooks), Schedule (i.e., rotation schedule) and Log Book.

From the My Learners tab, faculty can:

• Download a list of all learners associated with a faculty by clicking 'Download Enrolment'. A pdf title "learners-for-faculty-name" will download and list all learners, including their primary email and learner level.

• Search for an individual by beginning to type his/her name in the search learners area; the learner cards displayed will automatically display to reflect the searched name.

• Refine the list of learners by switching the curriculum period using the dropdown menu on the far right.

• Click on the appropriate tab to view the desired information.

Faculty

Faculty who are course/program directors have access to view the assessment and evaluation pages of faculty associated with their course/program (including external assessors). Faculty appear on this list if they have been the assessor or target in a distribution tied to the course.

• Hide cards of external faculty by clicking Hide Card. Bring hidden cards back into view by clicking 'Show Hidden Faculty'.

• Update external faculty emails by clicking on Update Email and providing revised information.

• Program directors will be able to view the Current Tasks, Completed Tasks, Upcoming Tasks, and Tasks Completed on Faculty tabs for their faculty.

• Send reminders, remove tasks or download tasks. (There is an option to select all and send reminders.)

• From the Tasks Completed on Faculty tab the course/program director can also view Reports for this Faculty.

Prompted Responses

When building items for use on assessment or evaluation forms, administrators have the option to identify response selections as prompted responses; then, they can identify on a distribution who should be notified by email if a prompted response is selected. The Prompted Response Notifications tab allows recipients of prompted response email notifications to review and interact with the form that includes the prompted response.

• When you click on your Assessment & Evaluation badge, you will see a Prompted Responses tab.

• Once you have clicked the Prompted Response Tab, you will see a table displaying a list of your prompted response notifications, the course it belongs to, and the owner, target, and review the status of each task.

• You can sort the table by clicking on any of the headers.

• Search for a specific prompted response by using the search field.

• Clicking on the title of the task will allow you to interact with the form.

Once you are viewing the form:

• Click in the comment box to add in your comments about the assessment, if you have any.

• Comments can only be viewed by other prompted response recipients and administrators associated with the course/program.

• If you do not wish to add comments to a form, you can simply click the Submit Review/Comment to mark it as reviewed.

My Reports

To view information from forms completed on oneself, the user can click My Reports on the right hand side.

• Set the date range.

• Remember that the creator of a distribution can set whether a user can view reports on a distribution, so not every distribution completed on a target is available for the target to view.

When building a distribution a reviewer can be set in Step 5 of the Distribution Wizard. This person will need to review and approve, reopen or hide all tasks before the tasks are completed as required. The use case for this might be providing an intermediary reviewer who checks the appropriateness of narrative comments on course and faculty evaluations or peer assessments.

For more information on setting up a reviewer on a distribution, please see the page for the distribution type you are working with.

Managing Tasks As a Reviewer

If you have been set as the reviewer for a distribution, click your Assessment and Evaluation badge and you'll see tasks with a Reviewer label under your Assessment Tasks tab.

• Click View Task.

• Review the form contents.

• Choose one of three options:

  • Reopen the task: This will send the task back to the assessor/evaluator and it will remain in their pending/in progress task list until they complete it. If you reopen a task you will get a message indicating that the task has successfully been reopened.

  • Approve the task: This will essentially mark the task as finalized and release the task to the target (if allowed by the parameters of the distribution).

  • Hide the task: This will hide the task from being viewed by the target. The person who completed the task gets no feedback. If you hide a form, you will be prompted to enter a reason for why you hid the form. Note that hiding a form does not remove its data from reports generated about this form and distribution.

• Note that currently, there is no user interface to retrieve all hidden forms or to view the comments about why the form was hidden.

One of the distribution methods supported by Elentar is a delegation. This can be used with a date-based or rotation-based distribution to send tasks to a delegator to send to the appropriate assessor/evaluator at a later date. If there are multiple targets for a delegation, users will be able to forward some tasks to one assessor, and other tasks to another assessor.

How to complete a delegation

• If a distribution was created using a delegation, the designated user needs to complete the delegation by assigning assessors.

• Log in as the delegator.

• From the Assessment Tasks tab, click on the relevant assessment.

• You'll see a list of targets in a table.

• Click the checkbox beside a target name.

• At the bottom right click 'Select Assessors'.

• Search for as assessor and click the checkbox beside the assessor name.

• If you need to add an additional internal and/or external assessor you may do so by clicking the 'Add Additional Assessor' button. This will allow you to enter the name of an internal user and/or email address of an existing external assessor. You can even add a new external assessor if necessary.

• After adding the required assessors, click 'Proceed to Confirm Assessments'.

• If you wish to mark the delegation as complete, click the checkbox. This will move the task to the delegator's My Completed Tasks list. If you have targets without assessors/evaluators, you will receive a warning. If you'd rather that the task to stay on the Assessment Tasks tab, do not click the checkbox.

• Confirm your choice by clicking 'Create Assessment Tasks'.

• The list of targets and assessors/evaluators will be updated with the newly entered information.

In the date range based delegation above, the delegator has assigned 2 targets to Alex Adams and 2 to Bennett Adkins. The rest of the tasks will be assigned at a later date.

The Assessments Reports section mostly allows you to generate reports based on assessments completed via distributions. There are some exceptions, however most reports are for distributed forms.

Learner Assessments

This report allows you to compile all assessments completed on a target in one or more courses into one or many files. It does not aggregate results, just compiles multiple forms.

• Select a course, set a date range and select a course group (optional).

• Select a learner.

• Select a form (optional).

• Click 'Download PDF(s)'.

• Choose whether to download as one file (all forms will be stored in one file) or not (you'll download a file for each form).

A file will download to your computer with the appropriate forms included. Each form will include the target and assessor, delivery and completion date, form responses and comments, etc. The file names will be: learnerfirstname-learnerlastname-assessment-datereportrun-#.pdf. For example: earnest-acosta-assessment-20181005-1.pdf

Learner Reports (Aggregated)

Use this report to create an aggregated report on learner performance on a single form that may have been used multiple times and completed by multiple assessors. For this report, the list of learners available will depend on someone's affiliation with a course/program if they are a staff:pcoordinator or faculty:director.

• Set the date range.

• Select a learner.

• Select a form.

• Set the report parameters regarding displaying comments and averages.

• Click 'Download PDF(s)'.

An average and aggregate positive and negative score are available with this report. This report will not include learner names beside comments, even if you check off the commenter name option when setting the report options.

Please note that if in the distribution for a form, you set comments to be identifiable, learners themselves will be able to view commenter names when they self-report the form results as seen below.

Timeliness of Completion Report

For use in reporting on tasks delivered to and completed by faculty. Report columns include the number of tasks delivered and completed as well as the average time to completion from delivery date and average time to completion from the end of the experience (e.g., a block) per user. It also provides an overall average across all users. Available as a PDF or CSV.

• Select a course.

• Set a date range.

• If there were external assessors used you will have the option to include externals in the report or not.

• Select one or more users by clicking the checkbox beside each required name. Please note that if you select all faculty it can take some time for all names to appear. Please be patient! To delete a user from the report click the 'x' beside the user's name.

• Include Average Delivery Date: Enable this if desired.

• Click 'Download PDF(s)' or 'Download CSV(s)'.

• The report should download to your computer.

Assessment Tools Feedback Report

For use in collating responses provided by faculty completing forms produced through form templates in the competency-based medical education module. This report aggregates comments from forms that are sent out using a distribution. When logged in as an admin. you'll see a full list of all form feedback provided thus far display on the screen.

• Set a date range.

• Select a course from the dropdown options.

• Select a tool from the dropdown options.

• Click 'Apply Filters'.

• Results will display on the screen and you can click 'Download PDF(s)' if you need to download a copy.

• To begin a new search be sure to click 'Reset Filters' and select a new course and/or tool as appropriate.

You can click on the page icon to the right of the feedback column to view the form being referenced.

Triggered Assessments Report

An organization must be using the rotation scheduler to use this report. This report displays the number of completed assessments over the number of triggered assessments per block for all students in a course during the selected curriculum period. Data is grouped by block, based on the selected block type. The learner is considered the target of the assessments.

• Select a course (you can only select one).

• Select a curriculum period.

• Select a block template type (e.g. 1 week, 2 week). The block templates available will depend on the setup of the rotation schedule for the course.

• Click 'Download CSV(s)'

Triggered/Completed Assessments by Faculty Report

For use in monitoring the progress of faculty in completing the tasks assigned to them.

• Select a course.

• Set a date range.

• Decide whether or not to include External assessors. (This is only relevant if you allow users to send tasks to external assessors, e.g., users without an Elentra account.)

• Select the relevant faculty (you can select more than one).

• Click 'Download CSV(s)'.

• A csv file will download to your computer.

Assessment Tool Data Extract

Generates one CSV file per tool (you can select multiple tools at the same time) and lists all completed instances of that tool across all learners within a custom date range. The report displays the encounter date, delivery date, and completed date for each form, as well as each form item and its associated response scale in the column header. If comments were entered for a selected response, they are included in a subsequent column. Users can opt to view only completed instances of a form or to view all instances of a form (pending, in progress, completed, and deleted).

Form Responses Report

Provides an overview while also generating a CSV that displays all/selected assessment responses for a specified course, form, assessment status and date range.

Staff have quick access to review their assessment and evaluation tasks and those of their affiliated learners and faculty through the Assessment and Evaluation badge accessed in the top right.

When viewing results on any of the tabs in A&E, you can:

• Use the search bar to look for a specific task by name and click the down arrow to limit your results by applying search filters like distribution method (date range, delegation, learning event, rotation schedule), curriculum period, course, and task status (pending, in progress, completed), and task type (mandatory/optional).

• Sort by Delivery Date or Expiry Date (select one from the drop down selector and then enter the desired start and end dates).

• Remember to click 'Apply Filters' to apply your choices.

• Remember to click 'Remove Filters' to clear filters and view all results.

Assessment Tasks

This tab shows users all pending tasks for which they are responsible. Mandatory tasks are indicated with a red highlight with a bold red "Mandatory" title on the card.

This is particularly important for curriculum and program coordinators because of the delegation type distributions. If a distribution was created and set as a delegation to be sent to a program admin., s/he can access that delegation here and assign the task to the appropriate faculty (or other user).

• View and complete a task by clicking on it.

• Remove a task by clicking 'Remove Task' and providing a reason for its removal.

• Download individual forms, or download multiple forms by clicking the download checkbox on each card and then clicking 'Download PDF(s)' at the top right. When users do this they will be able to choose whether to download all tasks as one file or as multiple files.

Tasks Completed on Me

This tab displays all tasks completed on the user and which s/he has been given access to view. This can be controlled in the creation of a distribution.

My Completed Tasks

• The user can view all completed tasks and again, download an individual file or download multiple files at once.

My Learners

The My Learners tab will allow administrative staff to access a variety of information for learners associated with them. This list of available options will vary depending on which modules are in use in Elentra but could include CBME Dashboard, Assessments (refers to tasks managed through Assessment and Evaluation), Learner Explorer (refers to course gradebooks), Schedule (i.e., rotation schedule) and Log Book.

From the My Learners tab, users can:

• Download a list of all learners associated with them by clicking 'Download Enrolment'. A pdf title "learners-for-name" will download and list all learners, including their primary email and learner level (if learner levels are in use).

• Search for an individual by beginning to type his/her name in the search learners area; the learner cards displayed will automatically display to reflect the searched name.

• Refine the list of learners by switching the curriculum period using the dropdown menu on the far right.

• Click on the appropriate tab to view the desired information.

Faculty

Curriculum/program coordinators associated with a course will be able to access a tab listing faculty. Faculty will appear on this list if they have been the assessor in a distribution or the target or assessor of a triggered form associated with the same course as the program coordinator.

• Hide cards of external faculty by clicking Hide Card. Bring hidden cards back into view by clicking 'Show Hidden Faculty'.

• Update external faculty emails by clicking on Update Email and providing revised information.

• Program coordinators will be able to view the Current Tasks, Completed Tasks, Upcoming Tasks, and Tasks Completed on Faculty tabs for the available faculty.

• Program coordinators can send reminders, remove tasks or download tasks. (From this screen there is an option to select all and send reminders.)

• From the Tasks Completed on Faculty tab the program coordinator can also view Reports for this Faculty.

• This tool allows the program coordinator to view and download a pdf report (with or without comments) that aggregates evaluations of the selected faculty in one report. Program coordinators can also use the Options dropdown menu to quickly view the form and the individual evaluations.

• Responses and comments aggregated in these reports are de-identified.

• Click group by distribution as desired to sort the forms by distribution. If the same form has been used across multiple distributions this will tease apart each distribution and you can report on forms completed in a single distribution.

Prompted Responses

When building items for use on assessment or evaluation forms, administrators have the option to identify response selections as prompted responses; then, they can identify on a distribution who should be notified by email if a prompted response is selected. The Prompted Response Notifications tab allows recipients of prompted response email notifications to review and interact with the form that includes the prompted response.

• When you click on your Assessment & Evaluation badge, you will see a Prompted Responses tab.

• Once you have clicked the Prompted Response Tab, you will see a table displaying a list of your prompted response notifications, the course it belongs to, and the owner, target, and review the status of each task.

• You can sort the table by clicking on any of the headers.

• Search for a specific prompted response by using the search field.

• Clicking on the title of the task will allow you to interact with the form.

Once you are viewing the form:

• Click in the comment box to add in your comments about the assessment, if you have any.

• Comments can only be viewed by other prompted response recipients and administrators associated with the course/program.

• If you do not wish to add comments to a form, you can simply click the Submit Review/Comment to mark it as reviewed.

Prompted Responses on forms can also be configured to send email notifications. If users are the recipients of Prompted Response notifications, they will see the prompted response indicated in the email notification.

My Reports

To view information from forms completed on oneself, the user can click My Reports on the right hand side.

• Set the date range.

• Remember that the creator of a distribution can set whether a user can view reports on a distribution, so not every distribution completed on a target is available for the target to view.

External Assessors are members of a learning community who do not have Elentra accounts. Examples could include social workers, patients, allied health professionals, etc. External assessors can be included in distributions and learners can initiate tasks to external assessors.

When an external assessor is created their name and email are stored but they do not have a complete user record in Elentra. We avoid duplication of external assessor records by having the system check email addresses.

Existing external assessors will display at the end of a list of users when searching for a user when creating a distribution.

If a learner searches for an existing external assessor to initiate a task to, the name will display with other returned results and the assessor will be labelled as external on the user card.

Most reports are currently available as PDFs.

Some screen shots of sample reports are posted below but remember that your reports will contain the items relevant to the forms you've designed and used. In some cases information has been redacted to protect users' privacy.

The Weighted CSV Report provides a csv file that includes the data collected through completed forms. It lists users who completed the form down the side and form items across the top. Inside each cell will be data from the form representing the scale ratings made by those who completed the forms. If your response descriptors include numbers (e.g. 1 - Overall, this instructor is an effective teacher.) note that those numbers will not necessarily be reflected in the csv.

It is important to note that the weighted CSV report was specifically designed to be used in conjunction with items using a rating scale (e.g. grouped items using a rubric), and allows you to create custom weights for scale response descriptors which get reflected in the report. There is currently no way to configure these weights through the user interface and you will need a developers help to assign weights to scale response descriptions in the database. (Developers, you'll need to use the cbl_assessment_rating_scale_responses table.)

If no weights are applied to the scale responses, the report defaults to assign a value of 0, 1, 2, 3, 4 to the responses in a left to right order. In effect, the Weighted CSV Report will work best if the rating scale you apply to the items mimics a 0-4 value (e.g. Not Applicable, Strongly Disagree, Disagree, Agree, Strongly Agree).

Accessing the Weighted CSV Report

You can access the Weighed CSV Report from two places: from the Assessment & Evaluation Reports tab or from an individual distribution.

To access the Weighted CSV Report from Admin>Assessment & Evaluation you must have general access to the Admin>A&E tools. Such access will usually apply to staff:admin, staff:pcoordinator, and faculty:director users when the staff:pcoordiantors and faculty:directors are affiliated with a course/program.

The weighted CSV report is accessible from the Admin>Assessment & Evaluation Reports tab.

• Click Admin>Assessment & Evaluation.

• From the second tab menu, click on 'Reports'.

• Scroll to the bottom of the list and in the Distribution section and click on 'Weighted CSV Report'.

To access the Weighted CSV Report from an individual distribution you must have access to that distribution.

• Click on Admin>Assessment & Evaluation.

• From the first tab menu, click on 'Distributions'.

• Search for or click on the title of the relevant distribution.

• Click on the Completed Assessments card (far right).

• Click on the Weighted CSV button under the Assessments Completed heading.

• A file should download to your computer.

Learners have quick access to review their assessment and evaluation tasks through the Assessment and Evaluation badge.

• Click the Assessment & Evaluation badge in the top right beside the logout button.

• Three tabs will be visible to learners: Assessment Tasks, Tasks Completed on Me, and My Complete Tasks.

When viewing results on any of the tabs in A&E, use the search bar to look for a specific task by name and click the down arrow to limit your results by applying search filters like distribution method (date range, delegation, learning event, rotation schedule), curriculum period, course, and task status (pending, in progress, completed). Apply start and end dates to limit your results to a specific time frame. Remember to click 'Apply Filters' to apply your choices. Remember to click 'Remove Filters' to clear filters and view all results. Note that your previous filter settings may still be applied so if you are seeing no results, or fewer results than you expected, try 'Remove Filters'.

Assessment Tasks: This tab displays tasks the learner is currently responsible for completing (e.g. faculty evaluation, service evaluation, etc.). Mandatory tasks are now indicated with a red highlight with a bold red "Mandatory" title on the card. Task cards may display a rotation or block name depending on how they were scheduled. Learners can view a task, download individual tasks or multiple tasks at once, and can remove tasks. If the learner removes a task, they will have to provide a reason.

Tasks Completed on Me: This tab displays all the tasks that have been completed on the learner. Task cards display the form title and type, task delivery and completion dates, form triggerer name (if applicable), and name and role of the assessor. Learners can view a task and download individual tasks or multiple tasks at once. A Seen by Target icon now appears in the top right hand corner so that the learner, faculty and staff can confirm the learner has seen the completed form.

My Reports: From this button on the right hand side, learners can access reports on forms completed on them (if the distribution allowed for learners to view results). Provide the appropriate dates. Click 'Report'.

My Completed Tasks: This tab displays forms the learner has completed. Task cards display the form title and type, rotation/block name, task delivery and completion dates, and progress. Learners can view a task and download individual tasks or multiple tasks at once.

When you toggle to card view of a list of users (e.g. within a distribution) you'll be able to see their user photo (if they have provided one).

Learner release of complete task to faculty

When administrators create distributions they can optionally decide whether or not to release the tasks and associated reports to the targets of the forms. Choosing to release tasks allows the target to view forms completed on them. In the case of evaluations, administrators only have the option to release task reporting to targets, not release individual tasks. However, a distribution can be set up to allow learners to optionally release completed evaluation tasks to a target. This applies in the case of evaluations only.

If this optional target release feature is used, at the bottom of an assigned evaluation task, learners will see the option to release the evaluation to the selected target. If the learner picks 'Yes' from the dropdown selector, the task will become available for the target to view through their Assessment & Evaluation tab.

The faculty member can now view that completed form.

Note that learners opting to individually release tasks for faculty to view does not change the faculty member's ability to create a report on a distribution (that is still dictated by the distribution).

Learners can initiate assessments on demand from their dashboard. A learner's specific user interface will depend on whether or not they are a part of an organization with CBME enabled and/or workflows enabled, and whether they are responsible for any adhoc distributions.

• From the main dashboard, click 'Start Assessment/Evaluation'.

• Toggle between 'Start Assessment/Evaluation' and 'Adhoc Distributions' as needed.

  • Start Assessment/Evaluation should be used for any CBME or workflow forms.

  • Adhoc Distributions should be used if a distribution was set up that requires the learner to initiate a task to send to faculty.

A Faculty user can delete a task when viewing it as long as the task is not set as mandatory.

• In the top right hand corner, click 'Delete Task'.

• You are required to provide a reason for deleting the task and can optionally provide additional notes.

  • The reason entered will be displayed to administrative users viewing a list of deleted tasks.

If a task is set as mandatory, faculty will see an explanation when they hover over the Delete Task button.

Accessing the Assessments Page

• When logged in as a faculty member or program administrator, click the Assessment & Evaluation badge that appears in the top right.

• Click on 'My Learners' from the tab menu.

• Search for a learner as needed, and if you can't find someone, ensure that you are in the correct curriculum period using the dropdown menu on the right.

• After finding the correct learner, click on Assessments on the learner card to view the learner's assessments page.

Assessments Page Contents and Filters

Multiple tabs are available on the Assessments page. Each tab displays a collection of assessment tasks. CBME specific forms are included on this page and so are any forms created through the Assessment and Evaluation module and sent to users through a distribution.

On each assessment card you will see a range of information which may include the form title, form type, delivery and completion dates, progress (e.g., 3 of 14 complete), who triggered the form , the target, and the assessor/evaluator (including their group and role). Not all of this information is included on every task card.

On each tab you can search for tasks, set a date range and apply filters. Click the down arrow beside the search tasks bar to select a filter type and drill down to the necessary filter. Select a filter and click the green Apply Filters button. Your applied filters will display on an Active Filters list.

Tasks Completed On Learner

These are tasks completed by assessors on the learner.

Each task can be downloaded or viewed. To download tasks, click the appropriate box at the bottom of the task card and then click Download PDF(s) at the top of the page. Note that there is a Select All tool available. To view a task click View Task at the bottom of the task card.

More tasks can be displayed by clicking Load More Tasks at the bottom of the page.

Pending Tasks on Learner

From this tab you can view all pending tasks on the learner this includes tasks that have been distributed but are not yet started and tasks that are in progress.

Tasks that have expired will not display on this tab. To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Upcoming Tasks on Learner

This tab displays scheduled distributions where the learner is the target.

To remove a task from this list click Remove Task and provide a reason for the removal. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Current Tasks

This tab displays active distributions where the learner is an assessor/evaluator.

To remove a task from this list click Remove Task and provide a reason for the removal. To send reminders click the applicable tasks (or select all) and click Send Reminders in the top right. To download a task select the appropriate tasks (or select all) and click Download PDF(s) in the top right. To view a task click View Task at the bottom of the task card.

Learner's Upcoming Tasks

This tab shows tasks scheduled through a distribution where the learner is an assessor/evaluator. Tasks can be removed from this list by clicking Remove Task and providing a reason for the removal.

Reports for this Learner

From the Tasks Completed on Learner tab, users can access Reports for this Learner. This gives you access to aggregated reports on a learner for each form used. Note that these reports are available for forms managed via distributions, not those triggered by learners and faculty.

Set a report date range if desired.

From the Options dropdown menu you can: 1. Choose to generate the report with or without comments (both can then be downloaded as a PDF); 2. Click View This Form to open a new tab that shows the form; and 3. Click View Assessments to open a tab that lists all completed assessments for this form and provides links to view each one.

Clicking the 'Group by distribution' checkbox will reorganize the list of forms to reflect completion based on distribution and add a column to the display to show distribution description information.