So You Want to Be a U.S. Citizen?
(First You Have to Pass the Test)
By Steven B. Just

“The test is too hard.” “The test is too easy.” “It doesn’t cover the material properly.” “It covers the wrong material.” Is this the transcript of a test validation process at one of our clients? No. It’s from the debate over the new U.S. Citizenship test. Rarely does the issue of test validity make the front page of the New York Times but there it was on September 28 ("New Test Asks: What Does ‘American’ Mean?").

Of course the Times didn’t identify the debate as being about test validity—they were more interested in the political angle—but it was most definitely a debate over validity.

Let’s look at the issues—while trying to avoid the third rail of the immigration debate (Note: italicized text below is quoted from the article):

The questions were submitted to four months of pilot testing this year with more than 6,000 immigrants who were applying for naturalization.

Can’t fault them on this. That’s a significant pilot group. Our clients could learn a lesson here.

At least 15 questions were eliminated as a result of the pilot because they proved too difficult.

A proper pilot often results in the modification or elimination of a number of questions.

In a one-on-one oral examination, an immigration officer asks the applicant 10 questions of varying degrees of difficulty selected from the list of 100.

Without detailed knowledge of the testing process we assume that the item difficulties are well calibrated (based on the pilot) and the examiners have been properly trained to select a representative range of item difficulties. Perhaps the sets of 10 questions are pre-selected for the examiners to ensure that this is the case. The questions are objective but open-ended, meaning that the examiners need to judge if the stated answers are correct. One hopes they are well trained and that during the pilot inter-rater reliability was examined.

To pass, the applicant must answer 6 of those 10 questions correctly. In the pilot runs of the revised test, Mr. Aguilar said, the pass rates improved over the current tests, with 92 percent of participants passing on the first try, as opposed to 84 percent now.

Setting a passing score is one of the most important elements of validity and often one of the most contentious. It’s not mentioned in the article why 60% was set as a passing score. Did they use a valid process?

The questions released yesterday will remain public along with their answers.

This puzzled me. I once worked with a dental school professor who regularly made available to his students all of the questions (and answers) he “might” use on his final exam. He had been teaching for a long time and had an item bank of a few thousand questions (for a 100 question test). His rationale was that it was impossible to memorize the answers to all the questions, and if someone actually could he would have learned everything he was supposed to have learned in the course (which was strictly didactic). Is this the rationale for publishing all 100 questions and answers — which a reasonably determined person could certainly memorize?

Immigration officials said they sought to move away from civics trivia to emphasize basic concepts about the structure of government and American history and geography.… “This test genuinely talks about what makes an American citizen.”

Content validity! I have to take their word that they are testing what’s important, since I am not a subject matter expert, but this is the very definition of content validity.

Of course, not everyone agreed:

The Illinois Coalition for Immigrant and Refugee Rights… said the test included “more abstract and irrelevant questions” and John Fonte, a senior fellow at the conservative Hudson Institute… said it should have included questions about the meaning of the oath of allegiance that new citizens swear. “ I would like to see an even more vigorous emphasis on Americanization.”

Fortunately our clients can ensure content validity by adhering to the learning objectives in the training materials, although discussions about the relative importance of different learning objectives often occur.

Want to see how you would do? Here’s the test.