Exam Question Management (Beta)

IMPORTANT: This information is for people who are looking at the work completed up to March 2021 on Elentra's new question management tools. To access this work you should have received a link from the developer at your school or from the Elentra Core Team.

These instructions are not for exam question management in any live production environment.

ℹ️ Important Notes on Changes to Question Management

Labels (formally known as Folders)

Labels replace the folders that previously stored questions.

• Question folders that already existed in an installation of Elentra will be migrated to become a Label. (You can also create new Labels as needed.)

• Permissions that were on an existing folder will be applied to the corresponding Label created during migration.

• More than one label can be applied to a question. That means that if you want to associate a question with more than one specialty, more than one body system, or more than one topic, labels lets you do that.

Associating a question with a course.

You can avoid creating Labels for courses by instead just associating a question with a course.

• If a question is assigned to a course, by default, course contacts for that course (e.g. Course Director, Curriculum Coordinator, Program Coordinator, Associated Faculty) will have access to all questions associated with the course.

• You can associate a question to one or more courses and/or one or more labels.

Question Assessment Types (Summative and Formative)

These categories will be applied in future when exams you create must be flagged as Summative or Formative. You'll only be able to apply Formative questions to Formative Exams, and Summative questions to Summative Exams.

• All migrated questions from existing Elentra installations will become Summative.

Question Status (Draft, Published, etc)

Users can designate a status (this allows for the creation of Draft questions which can later be reviewed and published).

• System statuses like Draft/Publish can only be changed by the appropriate action, not on the Status tab. Selecting either Draft/Publish will always create a new version.

• Multilingual schools can utilize statuses like "Translation Requested", "Translation Approved", etc.

• Users can use a "Review Status" such as "Requested", "Approved", etc.

Enhancements to Multiple Choice, Multiple Response

Enhances functionality of multiple choice multiple responses questions to support weighting different response options.

Ability to Copy an Existing Question

Users can now create a duplicate copy of an existing question

Enhancements to Deleting a Question

Users can now retire a question, allowing for better management of long-term question banks.

Default Permissions

Currently only staff:admin can create questions by default. To grant the ability to view, edit or create questions, assign that user to a label or course.

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

The new Question Management feature handles permissions via two methods: course association and label association.

Course Associations Any course contact will be able to access questions that are associated with that course.

• Note that Course Contact access to questions is controlled through the access control list (a database table in Elentra) so that it can be configured for an institution's need. For example, you could adjust things so that Associated Faculty have the ability to save drafts of questions/groups, but not publish them.

Label Associations Labels are built in Elentra and you can associate individuals with one or more labels. For each association you make, you can control whether the user can read or write questions, groups and exams (in this case groups refers to question groups).

Labels Management

Currently only staff:admin can create and manage labels.

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

Question Index

When you click on the Questions tab you will now land on the Question Index page. From this page you can search for questions by typing in a word, and you can apply multiple filters to refine your search.

When viewing results, card view will show you the question text and responses along with an option to see more information at the bottom of each card.

List view will let you see more questions in a smaller field and get a quick overview of their status, associated groups, assigned curriculum tags, application to exams, and more.

Search and Filter

The search and filter options have been expanded to improve functionality and user experience.

Search query can target any visible field. Therefore, if you are showing additional columns via the "Columns" button, search queries will include those additional columns allowing for more granular ability to see only what you want to see.

Filter options greatly expanded and now include Label, Date, Type, Author, Exam Usage (allows you to pick a specific exam and see the questions used), Course, Group, Retired, Status, Curriculum Tags, Assessment Type. Filters allow for more than one selection at a time allowing more granular ability to scope the returned question items to be exactly what you need.

Each time you pick a category from Filter By (e.g. Question Types) you'll then be able to select your specific filters (e.g. Multiple Choice). One exception to this is the Retired option (it will just show you retired questions or all questions).

Currently applied filters are displayed as badges, and can be cleared by selecting the x beside the individual badge, or via the Clear All Filters button.

Enhanced Navigation Experience

The following enhancements have been made to both the table and card view of the index:

• The "More" button allows you to navigate to the various tabs of the Edit Question process
• The "Edit" button will start the Edit Question process
• Paginated screens have controls to display how many can be displayed, and the ability to jump to first and last pages

Table View

In Table View you can control which columns to show for your filtered questions via the "Columns" button.

Default Columns	Additional Columns
ID	Author
Type	Date
Text	Languages
Status	Exam Type
Groups	Description
Tags	Code
Exams

Card View

The Card View will allow you a preview of the question as it might appear to learners, the ability to view more information about specific questions and the ability to show or hide the correct answer.

You can additionally expand or collapse all the information on a card by selecting the "Expand/Collapse" chevron.

Status

Question Status is a new field. You can now create questions in draft mode and request that they be reviewed. This allows for careful oversight of all questions before they are published.

Currently you do not set a specific reviewer per question. Instead you can filter by status to find questions that require attention.

When viewing a question its status will be represented with a blue airplane (published) or a grey page (draft).

Additional icons will show a question's Review status:

• Review Requested (yellow flag)
• Question Approved (green checkmark)
• Question Rejected (red x)

You can Ability to set status on multiple questions by checking the question you'd like to edit and selecting: "Set Status".

Note: Draft questions can't be removed from the Question Index through the bulk "Remove Questions" options. Instead, click into a draft question as if to edit and then choose "Discard Draft".

Data Visibility

The Question Index user interface has been improved to include more data in a more visible state. This includes:

• Associated Groups

  • See groups this question belongs to and click edit to quickly go edit a group

• Associated Exams

  • You'll see the total question count, a count of any existing posts, and take the exam in a preview mode; each of these will link you to another tab.

• Associated Exam Types

• Associated Curriculum Tags

Click on the count for each entity to see more detail about a question.

Batch (aka Bulk) Index actions

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

When viewing the question index you can make changes to multiple questions at once to enhance productivity when using the Question Index. These actions include:

• Group questions

• Set status

• Set course

• Set curriculum tags

  • This process will open the curriculum tag selector and show you the tags already assigned, which question is assign with which tag.

  • Tags added will be applied to every question selected via checkbox

  • You are able to un-assign tags via this method as well

• Reactivate retired questions

Add Question

When you build a question through the user interface, you will now walk through several tabs to add the required information.

You will need to complete the first two tabs ("Setup" and "Details") before access to the other tabs becomes available.

Setup

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

The ability to Publish a Question is available on this tab.

The Setup tab contains mostly new additions to the adding process. It includes the configuring if a question is flagged as Summative or Formative. It also includes the ability to associate a question with a course or a label.

A question must be associated with either a label or a course. It can be associated to both.

For organizations using the migration, all existing questions will be set to summative.

This tab also contains the version history of a question, for new questions this will be blank.

Details

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

The ability to Publish a Question is available on this tab.

The Details tab contains the majority of the interface users of the existing Question Management module will be familiar with. It will allow you to specify the type of question, the question stem, description and options if applicable. Some new features and user experiences have been added as well.

Question Types Question types remain the same, the only change is that Multiple Choice, Multiple Selection are no longer a separate question type. The ability to specify a Multiple Choice question has Multiple Selections are done after a Multiple Choice question is selected. Note that you can also now set the maximum number of responses to limit learners from being able to select all responses.

Overall User Experience Enhancement Hints and helpful messages appear when applicable to help guide you through complicated question types (i.e. Fill in the Blank).

Question Answer Creation User Experience Enhancement Answers now get added after you tab to or select the next question in a list, meaning you no longer need to scroll up or down to reach the "Add Answer" button.

Clearer, More Accessible Reorder Controls You can now select the arrow up or down beside an answer, or drag and drop them into any order you desire.

Clearer Right/Wrong Flagging You now can explicitly see which answers have been marked as right and wrong via a toggle.

Ability to Preview a Question You can now select the "Preview Question" button at the bottom of the screen to see how the question will appear to Learners.

Ability to Save a Draft of a Question You can now save a Question as a draft, meaning users can work on that draft over time without worrying it may be accidentally associated to an exam.

Tags

Changes are saved automatically on this tab after an interaction is performed.

Note that the Curriculum Tag Selector does not currently include an option to contextually link other tags

The Tags tab integrates the Improved Curriculum Tag Selector including all the user experience and functionality upgrades including improved searching and filtering.

Status

Changes are saved automatically on this tab after an interaction is performed.

The Status tab allows you to configure various statuses to indicate different conditions of a question or reflect workflows.

System statuses are displayed on the Question Index.

System Status This is a read-only select box to indicate the current status and other available statuses. To change this use the appropriate actions on Details or Setup or on the Question Index.

Review Status Allows you to flag the question to indicate a review is "Requested", "Approved", or "Rejected".

Translation Status If your institution supports multiple languages this status is made available. It will allow you to indicate a translation is "Requested" or "Completed".

A comment field is available to track the history or provide context for the statuses set.

Groups

The Groups tab displays a list of any Question Groups this question currently belongs to.

You can read the description or jump to the index page for the group as well.

Exams

The Exams tab displays a list of any Exams this question has been associated with.

You can easily see all the other questions attached to that exam, any posts for that exam and preview the exam as a learner.

Copying a Question

From the Setup or Details tab of an existing question you now have the option to create a copy of that question.

When viewing the Question Text on the Details tab, the Question Text will be prefaced with "Copy of." The newly created copy will automatically be given draft status and will maintain the existing details and tags.

Remove a Question from the Bank

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

Retired Questions

Retiring a question is a new type of system status for questions. It allows you to prevent a question from being attached to new exams or being used in new exam posts when attached to existing exams. These are the features of a retired question:

• Not eligible to any new exam

• Excluded from new posts of existing exams

• Has a comment field when retiring as to why something is being retired

• Filtered out from the question index unless you opt to filter to view retired questions

• Can reactivate retired questions and then will become available for use on exams again

  • Filter question bank to show retired questions

  • Check off the questions you want to reactivate

  • Click Reactivate (at bottom right)

Deleted Questions

Deleting a question works the same way it did previously in Question Management. Only a question not attached to any existing exams can be deleted, otherwise we recommend Retiring that question. Deleting a question will remove it from a Question Group.

Question Versions

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

Question versions have been expanded to be clearer and more useful in the new Question Management feature. However, the inclusion of a draft mode and review statuses will hopefully reduce the number of errors, and thus versions over all.

You can view the versions on a question by selecting "Versions" from the "More" button menu.

The More menu, highlighting the Version option.

Versions listed in the modal will let you preview the previous version to be better able to compare what changed. Reverting to a previous version will create a new version based on that previous version, thereby preserving the complete history of a question.

Creating a new version of a question is as easy as making a change on a question and publishing or saving that version as a draft.

Labels

In this updated version of question management, labels replace folders. Like folders, labels can be associated with specific users to give those users access to questions assigned that label. More than one label can be assigned to a question which gives you maximum flexibility in identifying who questions from your bank are used (e.g. associated with a course and a specialty topic and something else as needed).

Label Index

Labels Index

The Label Index allows you to easily search and filter labels attached to exams & questions. You can decide which columns are visible (searchable).

Deleting a label will remove it from any associated questions and remove any permissions associated with that label.

New Labels

Currently only staff:admin can create labels

To create a label

1. Give the Label a name

2. Optionally give the Label a description

3. Specify the users that should be given permissions for anything associated with this label

4. For each user set if they should have Read Only or Read and Write permissions.

Note: Currently the permissions table for each user only affects questions, it does not affect exams or groups.

The use of labels extends any existing ACL rules for a question. If a question has both a course and label applied Elentra first uses the course contacts table, if fails to authorize at course level, it will check label and grant access as appropriate.

If labels have conflicting permissions and one question is tagged with both, the permissions are cumulative (so a user will get their highest permissable access).

Groups Index

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

The Question Groups interface has been re-done to mimic the features and experience of the Question Index. This includes the ability to search and filter with increased granularity (including the ability to filter by assessment type, language, course, exam, updated by author and updated date). You can also choose which columns are visible, thus searchable.

• Select the "Show Tags" button to view all the tags associated with Questions within this group.
• Select the "Show Associated Exams" button to view all the exams associated with Questions within this group.
• Select "Remove Selected Question Groups" to remove the group from the Index. Questions within that group will not be deleted.

Add Question Group

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

Currently only staff:admin can create groups.

Setup

The Setup tab allows you to specify the tile, an optional description, and to set if this Question Group will contain either Summative or Formative questions.

Questions

The Questions tab allows you to easily browse the Question Index, and attach the needed Questions via the same search and filter interface from the Question Index.

You can easily reorder the questions using the drag and drop method, or by selecting the up and down arrow on each individual item.

You can easily remove questions from a group on this tab as well.

Each question has corresponding interactions to show you versions, associated exams, associated curriculum tags, other groups that question is a part of, associated exam types and any statuses applied to a question.

Tags

The Tags tab displays all Curriculum Tags associated with the questions within this group.

Exams

The Exams tab displays any exams associated with questions in this group.

You can also see the other exam questions attached to that exam, posts from that exam and even preview the exam itself.

Add Exam

Exam Setup

The Exam Setup tab has received an interface refresh and now includes the ability to set Exams as Summative or Formative.

You can also specify an Exam Type, this is to support future enhancements of the Exam Management system and does not currently appear anywhere else in the system.

If your school supports multiple languages, you can specify what languages this exam is intended to support, this will assist you by automatically filtering out any questions that shouldn't apply to your exam.

Existing exams will not have an assessment type associated with it.

Add Questions

Known Area of Improvement Currently this functionality does not meet expectations or is suffering some unintended loss of functionality. Please refer to the Epic: https://elentra.atlassian.net/browse/ME-4326 if you are experiencing unexpected behaviour or experiences.

Note: When adding a question, the Question Index will be filtered to only display exams that match what was provided during Exam Setup, specifically, if your Exam was set as a Summative Exam, only Summative questions will display. If your school supports multiple languages, the language setting will be filtered as well.

When adding a question to an exam you will now use the Question Index, it will allow you to search and filter as normal.

To add a question you select all the questions intended and then select "Attach Questions". To add a group, select "Show Groups" on a Question and then select Attach.