| Some Recent “Survey” Results
by Steven B. Just
Recently I chaired a panel discussion on testing
at the annual meeting of the Society of Pharmaceutical and
Biotech Trainers (SPBT). On the panel with me were four people,
each with deep experience in using assessments to enhance
the learning process and deliver tangible business results:
Steve Wells, Senior Director, Commercial Learning and Development,
Wyeth; Vicki Colman, Senior Manager, US Learning and Development,
Schering Plough; Greg Sapnar, Associate Director Metrics and
Certification. Bristol-Myers Squibb; and Bridget Siebenson,
Manager, Instructional Design, Ortho-McNeil Janssen Pharmaceuticals.
There were about 35 attendees from a cross-section of life
science companies at the workshop.
We were fortunate to have audience response
systems at the workshop and used them to poll the audience
about their companies’ assessment practices. With the
caveat that this was not a scientific sampling we thought
the results would be useful to you in benchmarking your own
assessment practices.
Survey Question 1
True or False: We Fully Understand How to Write Valid Questions
Comments: 69%
is an astonishingly high percentage of trainers who don’t
feel confident in their question writing abilities. Writing
valid questions is a requirement for creating valid assessments.
The rules are actually very straightforward. We’ve written
about them on several occasions: “Item
Writing Guidelines: Do We Agree?” , “Item
Writing Guidelines: Do We Agree?” (Part 2) and "Question
Writing Common Sense."
Survey Question
2
Our Assessment Strategy Has Limitations, Which Are Primarily
Due To:
Comments:
There’s a nice spread in answers to this question.
20% of the respondents said they had no limitations on their
assessment strategy. But nearly one-quarter of respondents
(23%) said their primary limitation was lack of expertise.
This does not surprise me. We work with a lot of training
departments and very few have internal assessment expertise—and
this is even true in large training departments with instructional
designers who have advanced degrees on staff. In my experience
instructional design programs give relatively short shrift
to evaluation.
Survey Question 3
Our Passing Score Is:
Comments:
78% of the respondents set their passing scores between
80% and 90%, inclusive, with 90% getting a plurality of votes.
This is consistent with what we see at our clients. Interestingly,
this is the only question that actually has a “correct”
answer. If you’ve been reading our Best Practices articles
"Subjectivity
and the Angoff Process" or attended any of our Best
Practices workshops (available live or via Webinar) you know
that the Angoff method is the generally recommended legally
defensible method for setting passing scores—and you
further know that the Angoff method delivers passing scores
that will vary from test to test.
Survey Question 4
In Our Department We Report Test Results To:
Comments: Let’s
end on a positive note. 41% of respondents report scores to
the employee’s direct supervisor and 44% report scores
to senior management in addition to the direct supervisors.
Reporting results to direct supervisors is essential for coaching
and remediation – not to mention that supervisors need
to know what their employees know (and don’t know).
And reporting results to senior management is critical for
making the business case for training. Senior executives are
increasingly demanding actionable metrics from training departments.
If you are not providing metrics beyond “number of students
trained” you are not making the business case for your
function.
Conclusion
In the 18 years that we have been providing
online testing software and professional services that help
corporations create and deliver fair, valid and reliable tests
much progress has been made. More and more companies take
measurement seriously and are committed to a scientific approach
to assessment. Clearly, however, many trainers understand
that their companies still are not where they should be.
|
