| Item Writing Guidelines:
Do We Agree?
by Steven B. Just
In my workshop on Best Practices in Test Validation
I spend some time talking about guidelines for question creation,
since a test can’t be valid unless the questions themselves
are valid. I have eight multiple-choice question writing guidelines
that I discuss:
- One piece of information
per question
- All information should
appear in stem
- All choices should
be parallel in form
- Do not use double negatives
- Avoid “all of
the above”
- Arrange responses in
logical order
- Correct choice and distractors
should be of the same length
- All choices should be
in syntactic and semantic agreement with the stem
I am aware that there are other guidelines,
some of which make sense to me:
When a question calls for a personal pronoun,
write questions in the second person, never in the impersonal
third person:
Bad Question: One should always …
Good Question: You should always…
and others with which I disagree:
Do not use “None of the above”
as a choice.
All this begs the questions: How many question-writing
guidelines are there? Are they all correct? Who makes up the
guidelines anyway?
The answers to these three questions are, respectively:
31. No. Anyone can.
The answer to the third question implies the
answer to the second. Since there is no central authority
adjudicating question-writing guidelines, just about anyone
is free to create his or her own guidelines. Thus many “guidelines”
are not guidelines per se, but one person’s opinions.
Now for the first answer: How do I know that
there are 31 guidelines? Because in a paper I came across
recently “A Review of Multiple-Choice Item-Writing Guidelines
for Classroom Assessment” Applied Measurement in Education,
I5(3) 309-334, the authors Thomas Haladyna, Steven Dowling
and Michael Rodriguez reviewed 27 textbooks and 27 research
studies and compiled all the multiple-choice question writing
guidelines they could find.
They then tabulated in what percentage of the
references each guideline was:
- Cited and supported
- Cited and not supported
- Not cited
They reasoned, with some justification I believe,
that those guidelines that are cited AND supported in the
vast majority of studies are valid guidelines.
Let’s look at some of their results:
- The vast majority of
the guidelines (24), if they were cited, were always supported,
which implies that there is a strong consensus on at least
24 item-writing guidelines.
- The only guideline that
was always cited and always supported was: “Keep the
central idea of the question in the stem.”
- 48% of the references
believe “None of the Above” should not be used.
44% said it could be used, but sparingly.
- Avoid “All of
the Above,” to me one of the cardinal guidelines,
was actually opposed by 22% of the references.
- Make distractors plausible
was cited positively by 96% of the references. 4% didn’t
cite it at all. No one opposed it.
- Many obvious guidelines
were uncited in a substantial percentage of the references.
I imagine the authors thought these guidelines were so obvious
they weren’t worth mentioning. So “Edit and
proof”,” Use correct grammar”, and “Use
humor sparingly” were uncited in 67%, 48% and 85%
of the studies, respectively.
- 15% were against “using
humor sparingly.” Did this mean they were opposed
to using humor at all or they thought humor should be used
more often? The answer according to the authors is the latter,
though they do warn against humor in high stakes testing.
My next article will discuss some other guidelines
and the authors’ findings in some more detail, some
of which are rather counterintuitive. (Read Item
Writing Guidelines: Part 2 )
|
