February 26, 2014

Examining Your Multiple-Choice Questions

By: in Teaching Professor Blog

Add Comment

As Ron Berk (known for his pithy humor) observes, the multiple-choice question “holds world records in the categories of most popular, most unpopular, most used, most misused, most loved and most hated.” According to one source I read, multiple-choice questions were first used around the time of World War I to measure the abilities of new Army recruits. As class sizes have grown and the demands on teacher time expanded, they have become the favorite testing tool in higher education.

However, even those who use them often aren’t all that crazy about them and with good reason. How many problems faced by professionals come in a multiple-choice format? Answering multiple-choice questions doesn’t teach students how to formulate answers; it teaches them how to select answers. And sometimes choosing the correct answer is more a function of literary skill than content knowledge. Multiple-choice questions encourage guessing, and if the guess is correct, students get credit for something they didn’t know. Answer options contain misinformation—that’s what makes them wrong. As students read and consider all the possible answers, they are exposed to incorrect content, which some research has shown influences subsequent thinking about the topic.

Many multiple-choice questions don’t challenge students to think but rather encourage them to memorize. Test bank questions are the worst. Analysis of questions in a variety of disciplines documents that around 85% of the questions test lower-level knowledge. And I’ve already referenced in the blog and lots of other places a very rigorous analysis of test questions on biology exams. “Of the 9713 assessment items submitted … by 50 faculty teaching introductory biology, 93% were rated at Bloom’s level 1 or 2—knowledge and comprehension.” (p. 437)

Clegg and Cashin in a classic (but still very helpful) paper write, “It is true that many multiple-choice items are superficial, but that is the result of poor test craftsmanship and not an inherent limitation of the item type. A well designed multiple-choice item can test high levels of student learning.” And that’s not all multiple-choice questions can do well. They can ascertain student knowledge of a wide range of topics in a timely manner. Best of all, they are quick and easy to grade.

The problem is writing those kinds of challenging questions. It takes careful thought and creativity to write a question that requires thinking to answer, and for most teachers that equates to time. Professional test question writers like those writing items for the SAT, ACT, and GRE devote hours to creating and testing questions.

The reasons and realities that motivate using multiple-choice questions are compelling. So the question is, how do we help faculty write good ones? I’m going to follow this post with one that reviews the do’s and don’ts of writing good questions. But before that, I encourage you to take a long hard look at the multiple-choice questions on your exams. When the low level of multiple-choice test questions is discussed, most faculty quickly assert that their questions are rigorous and do challenge thinking. A number of years ago, a cross-disciplinary faculty cohort reported that a third of their questions measured complex cognitive skills. An analysis showed that only 8.5% of their questions did, with the remaining testing basic comprehension and recall. Being objective isn’t easy, but it’s necessary.

Jacobs and Chase recommend that faculty write multiple-choice questions as they go (not all at once, right before the exam). If the questions are filled with the content pertaining to them, they could be reviewed, revised, or rewritten as part of class preparation. It isn’t always necessary to re-do the whole question. Sometimes an answer option is all that needs to be replaced or revised. And there’s no need for every question to test higher order thinking skills. Most of us could use a few more, but sometimes we do need to test basic knowledge.

References:
Momsen, J. L., Long, T. M., Wyse, S. A., and Ebert-May, D. (2010) Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. Cell Biology Education, 9 (Winter), 435-440.

Clegg, V. L, and Cashin, W. E. (1986). Improving Multiple-Choice Tests. IDEA Paper No. 16. Available for free at: www.theideacenter.org.

Jacobs, L. C. and Chase, C. I. Developing and Using Tests Effectively: A Guide for Faculty. San Francisco: Jossey-Bass, 1992.

© 2014 Faculty Focus, Magna Publications. All Rights Reserved. Use of any content without permission is strictly prohibited.

email
Add Comment

Tags: , , ,


Comments

Ken Mellendorf | February 26, 2014

I agree that multiple-choice is a poor form of test when memorization is not the goal. One aspect of teaching that is eliminated with multiple-choice is partial credit. A calculator error can be identified as such in a complete answer, and graded as such. In multiple-choice, a calculator error can sometimes produce one of the wrong answers by coincidence. This often happens when a button such as (x^2) is not pushed hard enough to register. Also, a student can understand the situation thoroughly and have an excellent setup, with the only problem being the wrong constant, perhaps a forgotten (1/2).

Jossie V. de Varona | February 26, 2014

“In multiple-choice, a calculator error can sometimes produce one of the wrong answers by coincidence.” This not necessarily true. It should not be by coincidence. If you are testing if the student knows how to push buttons for a particular order of operations or computation, then it should not be by coincidence. The answers should be carefully written an reflect all possible mistakes. Assessment of the results is crucial for the evaluator or the teacher.
In a well design multiple choice exam you should have different types of questions with different levels of difficulty that absolutely responds to Blooms Taxonomy. The premise should always be clearly stated and the answers should be well defined and thought. The teacher should know all possible mistakes that students can make, not for pushing a wrong key in the calculator but for misused of the definition, poor understanding of the concept, or not understanding the process at all. Also, is wise to have five alternatives to decrease the probability of guessing the answer. Three of them should be discarded if the student really knows about of what is being asked. Then you end up with two alternatives that will discriminate between the correct answer and the other one that is missing something so it is not correct, but reveals that the process is not completely wrong. Maybe it is a missing sign, a wrong coefficient result of a miscalculation. You can add weight to these alternatives, absolutely 100% to the correct one and some partial weight to those that are nearly correct but something is left out. As you say: “a calculator error can be identified as such in a complete answer, and graded as such”… with partial credit. Of course, when using multiple choice questions in my exams, I have the technology needed for creating the exam and establishing the criteria for doing this when correcting it.

brianehunt | February 26, 2014

Multiple-choice tests work well for some areas of study and work less well for others I'm sure. I teach physiology, and find that it works very well. I am able to test all levels of thinking from knowledge to synthesis and application. Test formatting is critical. My multiple-choice tests can have more than 1 "correct" answer. All questions have at least one correct or best answer, but can have as many as 5 correct answers. I give credit for correctly identifying both correct and incorrect answers. Thus, they can receive partial credit of a sort. Moreover, one can give credit for every correct answer identified and take away points for every incorrect answer. This is done to encourage students to minimize guessing, and only select those answers they strongly believe are correct. I agree that one should write their questions as they cover the material in class. This allows me to format my distractor questions in such a way as to play off of common misconceptions voiced and discussed in class. I understand that there is no ideal testing format. All have strengths and weaknesses. But I have found that with some work, I can make my multiple-choice tests work very well, helping me to discern how well students have mastered the material.

Paul Lowrey | February 26, 2014

"How many problems faced by professionals come in a multiple-choice format?"
I believe most of them do.

Jerry D. Harris | February 26, 2014

I can't say I'm a fan of multiple-choice questions–I'd greatly prefer to have the students PROVIDE an answer rather than try to recognize it–but how many teachers these days, especially in lower-level, introductory-style classes, can afford to risk plummeting assessment scores by assuming that the students are capable of providing an answer without a prompt?

Dr. Erasmus Chirume | February 26, 2014

If the promise of education to society lies in the prognostic and and prophylactic roles of schools and universities, how does assessment of learning through multiple choice help education to deliver on its promise to society?

Randy Connolly | February 27, 2014

Some of my exams feature some multiple choice questions. I think in the contemporary world, the fact that MC is about recognizing rather than providing an answer is quite important and useful. Students now (and graduates later) search for answers all the time on the internet. Being able to recognize a "best" answer actually is pretty darn practical and functional.

Neil Haave | March 6, 2014

MCQs can work in the sense that they can test for student understanding at all levels of Bloom's taxonomy, but good MCQs do require a lot of time and effort to construct. Another piece of advice I recently received from a colleague was to not write MCQs alone – write them with a group of colleagues preferably in the same room while they are preparing their own so that you can try out the MCQs on each other. This is a good way of inserting quality control into the question writing process. The Team-Based Learning Collaborative has some good resources listed on their website: http://www.teambasedlearning.org/Default.aspx?pag… (it appears that one of the links is dead and the other requires a search for the authors).


Trackbacks

  1. “Examining Your Multiple-choice Questions” | THE CMUOnline Blog
  2. Examining Your Multiple-Choice Questions | My Educational Technology Blog: A Place of Resources and Tools for Educators
  3. Final Friday 4 for February
  4. “Tips for Writing Good Multiple-Choice Questions” (Teaching Professor Blog) | Augie Prof in Progress


website security