How difficult is a question?

Every instructional designer creating questions for adaptive learning exercises or tests faces the same uncertainty: will this question feel easy or difficult to learners? It's surprisingly hard to tell. To a subject matter expert who knows the material inside out, almost every question feels easy.

As a result, difficulty is often misjudged in both directions. A question can look simple to its author and still trip up most learners. And a question meant to be challenging can give away its answer through its wording or answer options.

One rule, three levels of difficulty

For example, take a hand hygiene protocol every nurse learns: when and how must hands be washed with soap and water? Here are three ways to ask about it.

• Too easy: Ask what to do and offer four options: wash them with soap and water, wipe them on a uniform, wait until the end of the shift, or leave them as they are. Anyone can pick the right answer without training.
• Well balanced: Offer soap and water, alcohol hand rub or an antiseptic wipe instead. All three sound reasonable, so learners need to know the rule.
• Challenging: Remove the options and let learners type the answer. They have to recall it from memory instead of recognizing it in a list.

All three test the same knowledge. The only thing that changes is how the question is written.

Why the right difficulty matters

A question, scenario, or dilemma that is too easy doesn't tell you much about what someone masters. A question that is hard for the wrong reasons, such as confusing wording, tests reading skills rather than competencies. Either way, the results no longer show what learners really master.

This matters even more in Drillster, because our adaptive technology uses every answer to decide what each learner should practice next, and when. Well-calibrated questions give a reliable picture of who already masters the material and who still needs practice.

Yet until learners have answered a question, situation or problem, the only guide to its difficulty is the author's judgment. The Difficulty Calibration Service fills that gap.

How Difficulty Calibration works

Difficulty Calibration predicts how difficult a question is likely to feel before any learner has answered it. It looks at each question from six angles:

• Thinking required: Does the learner need to memorize a fact, understand it, or apply it in a realistic situation?
• Believable wrong answers: How hard is it to rule out the incorrect options?
• Chance of guessing: How likely is a lucky guess? A four-step sequence seems hard to guess, but much less so when common sense gives away the first step.
• Information to keep in mind: How many details does the learner need to combine?
• Clues in the wording: Does the question give away its answer, for example because the correct option is much longer than the others?
• Fine distinctions: How precisely does the learner need to tell similar rules or meanings apart?

The Difficulty Calibration Service combines checks on the question's structure, such as its format and number of options, with AI that assesses the wording and meaning in the context of the learning element and its source material.

From intuition to informed question design

For each question, situation, dilemma or problem, you get a prediction of how difficult it will feel and can see which factors drive it. Common writing flaws are flagged separately, such as asking two things at once or allowing more than one defensible answer. All information will be presented to you in an understandable report.

That makes it easy to spot questions that are too easy, too difficult or difficult for the wrong reasons. You can then fix them in a targeted way: make the wrong answers more believable, remove a clue or simplify the wording. This lets you make a question easier or harder without changing what it tests.

AI doesn't replace the expertise of instructional designers or subject matter experts. It gives them a second pair of eyes before any learner data exists, and alongside that data once it does.

Interested in Difficulty Calibration?

Difficulty Calibration is available to Drillster customers who want more insight into the difficulty and quality of their question sets. Want to know where your questions could be tuned? Contact us at support@drillster.com (opens in new tab).