| Question Writing Common Sense
By Steven B. Just
One of the services provided by Pedagogue is
helping our clients develop valid and reliable tests. An important
part of this process is ensuring that questions are developed
according to the rules of good question writing (questions
can’t be valid if they are not constructed properly).
During the past year we have reviewed thousands of questions
for construction validity.
On average, we suggest changes to about one-third
to one-half the questions our clients send us. Many of the
questions violate basic rules of good question construction
(e-mail us if you would like the list). To some extent we
can excuse these. The rules are not difficult, in fact they
are quite straightforward, but even many writers with advanced
degrees in instructional design are not trained in the basics
of question writing. But many of the errors have nothing to
do with knowing or not knowing question writing rules. Many
of the errors are plain sloppiness and/or laziness on the
question writer’s part. I suspect question writing is
seen as an unpalatable chore by many subject matter experts
so they just dash off the questions as quickly as they can
without much thought. Here’s a sampling of the kinds
of errors we’ve encountered recently:
1. Lack of agreement between
stem and choices
Often the stem will ask for a plural answer and some
or all of the choices will be singular. In some cases the
stem is plural and all of the choices are singular except
the correct answer, which is plural — an unfortunate
give-away of the correct response.
2. All of the Above questions
This is one of those rules of question writing that inexperienced
authors might not know. Fair enough. But don’t create
a 20 question test in which eight questions have “All
of the Above” as a choice, and have it be the correct
answer in all eight questions. Common sense tells you not
to do that.
3. True-False questions
There is nothing inherently wrong with True-False questions,
but everyone knows that even an unknowledgeable test taker
will guess the correct answer 50% of the time, so don’t
create a test where half the questions are True-False.
4. Brain teasers
We recently encountered the following question
type:
All of the following statements
are true EXCEPT:
Statement 1
Statement 2
Statement 3
Statement 4
Statement 5
None of the above
What does EXCEPT “None of the Above”
mean?! We still haven’t figured it out.
5. Positive lists with one negative choice
in a stem that asks for a negative
It took us a while to see this pattern
but we recently validated a test in which a dozen or so questions
had the following format:
All of the following are
animals EXCEPT:
Horse
Cat
Dog
Bird
Not fish
Apparently what the question writer had done
was find paragraphs in the training program in which there
were lists (for example “All of the following are side
effects of Drug A:”). Then the writer created questions
of the form:
All of the following are
side effects of Drug A EXCEPT:
List of all side
effects except one
Not the final side effect
Even one question of this type is invalid (use
of a double negative) but a dozen questions where the correct
answer was always the NOT choice is a serious violation of
test validity.
6. Violation of basic rules of grammar
A surprising number of writers don’t
know the difference between affect and effect, it’s
and its, and compose and comprise; nor do they know how to
use semi-colons properly.
You may not be an experienced question writer,
you may not enjoy question writing, but just following basic
rules of grammar, common sense and logic will help you avoid
many of the common question writing problems.
|
