1018
Practice and exam questions can be set to display difficulty levels generated by our system. As learners answer or review questions, they’ll be able to see the difficulty of each question to better gauge their performance. The default difficulty levels displayed include Easy, Moderate, Difficult, and Expert. To use difficulty levels, the feature access Question Difficulty will need to be turned on at the tenant and course level. More info on how to enable the feature access can be found here.
Difficulty levels of questions will adjust overtime based on how learners perform on the questions. For instance, as more learners answer a question correctly, that question will become ranked as an easier question. Similarly, as more learners answer a question incorrectly, that question will become ranked as a more difficult question. This ensures that difficulty levels become more accurate overtime and adapt based on learners’ performance.
In more technical terms, difficulty levels are determined in our system by Elo ranking. Elo is a self-correcting formula for comparison and is a key aspect to adaptive learning. Each question in your course will have an Elo rank, which will fall within a difficulty bound. For example, let’s say the difficulty bounds for your course are Easy [0-500], Moderate [500-1000], and Difficult [1000-1500]. If a question’s Elo rank is 1200, then it will be ranked as a Difficult question. As more learners get the question correct or incorrect, its Elo rank will decrease or increase accordingly.
Setting Up Difficulty Levels
There are two options when it comes to setting up difficulty levels for your course. Review both to decide what option fits your needs.
Option 1: Set Up Predefined Difficulty Levels
If you’re setting up a new course and would like to use your predefined difficulty levels for questions, please contact our Support team here and provide the following information:
- Course ID of the course that contains the questions you want to set difficulty levels for
- Spreadsheet that includes the following columns:
- Question ID / External ID
- Assigned level for each question (Easy, Moderate, Difficult, or Expert)
- All of these levels do not need to be used.
- If you have custom names in place for the difficulty levels, please specify which custom names correspond to each of our standard levels.
- Course IDs of 3 comparable courses (if applicable)
- These should be courses that are similar in content and practice question structure to your new course.
- They should also be live courses where enough data has been collected such that the questions’ Elo rankings have been adjusted to reflect user performance. In other words, the courses should have a good amount of users that have already completed the questions so that we can use this data as a basis for setting up the levels in your new course.
Once your request is received, BenchPrep will complete setting up the difficulty levels for your course. This will typically take about 5 business days. After the difficulty levels are set up, please note that they will still adjust in the future based on learners’ performance. If you’d prefer for your difficulty levels to not adjust once setup is complete, read our section below on Freezing Difficulty Levels.
Option 2: Automatically Setup Difficulty Levels
Difficulty levels can also be automatically set up for your course. As more and more learners answer questions within your course, the difficulty level of each question will update based on their performance. In this case, additional setup is not required by BenchPrep.
If you’d like to automatically set difficulty levels, we recommend disabling the Question Difficulty feature access for the first month after the course’s launch, as this will allow time for the platform to collect difficulty data based on real learners’ question attempts. After a month, enough data should exist to assign each question an accurate difficulty rating based on learners’ performance.
Custom Names for Difficulty Levels
If you’d like to customize the names of your difficulty levels to something other than our standard ones (Easy, Moderate, Difficult, and Expert), please reach out to your CSM to discuss further.
Freezing Difficulty Levels
Whether you set up your own predefined difficulty levels or automatically set them up, they will continue to adjust overtime as more learners answer the questions. However, if you’d prefer for the difficulty levels not to update, they can be set to freeze at their current ranks. This behavior is determined by two feature accesses at the tenant-level that can only be toggled by BenchPrep Admins.
Update question Elo data daily based on answers recorded - Default: ON
Freeze user level Elo updating on a course for all enrollments - Default: OFF
If you’re interested in freezing your difficulty levels, please reach out to your CSM to discuss further.
Replicating Difficulty Levels
If your course already has difficulty levels setup, and you’d like to create another course with similar content, questions can be replicated over and still retain their difficulty ranks. However, this can only be done during a full course replication and not when individual questions are replicated over. Instructions on how to replicate a course can be found here.
When replicating a full course, check the box Include Question Ranking to include the difficulty levels. If this box is not checked before replication, difficulty levels will NOT replicate over to your new course.
If you’re replicating a bank course, there are a few additional options you can select, which will affect how data is shared between the bank and cloned course:
- Send Question Ranking (Cloned Course Will Send ELO Data To The Bank) - As more learners answer questions in the cloned course, this data will be shared with the bank course and adjust its Elo rankings.
- Feed Question Ranking (Bank Will Send Aggregated ELO Data To Cloned Course) - As more learners answer questions in the bank course, this data will be fed down to the cloned course and adjust its Elo rankings.
If neither of these options are selected, then the difficulty levels in the bank and cloned course will update independently of each other after replication.
Features That Use Difficulty Levels
Other than showing the difficulty of your questions within your course, difficulty levels also impact other features that can be enabled. These include Custom Practice, Strengths & Weaknesses, and the Adaptive Study Plan.
Custom Practice
If Custom Practice is enabled for your course, learners will have the option to choose questions based on their difficulty levels. Before starting a new session, they can check the option Choose the type, difficulty, and number of questions. Then, they can either choose all questions within a difficulty level or choose a specific number of questions within a difficulty level and question category. To learn more about how to use Custom Practice, check out our article here.
Strengths & Weaknesses
If the Strengths & Weaknesses feature is enabled for your course, learners will be able to see how proficient they are in each question category. Their proficiency levels will be determined by how difficult the questions are, how well they answer the questions, and how all other users within the course also answer them. More information about Strengths & Weakness can be found in our article here.
Adaptive Study Plan
If the Adaptive Study Plan is enabled for your course, learners will be able to choose this study plan and determine how proficient they are in each adaptive category. The questions that are chosen in the Adaptive Assessment and the order in which Knowledge Sets are presented to learners both take into account the difficulty levels of questions. To find out more about how the Adaptive Study Plan works, take a look at our articles here.
