Articles by Patti Shank, PhD, CPT

01:22:2010

Course and Instructor Evaluations: Misconceptions and Realities

If evaluation sounds good in theory but feels bad in practice, it may be that you or others are operating under some common misconceptions.

Misconception: Outcomes are the only things worth measuring.

Reality: Outcomes, such as numbers of new courses developed, enrollments, retention, and satisfaction levels, are important and they should be measured. But it’s also important to evaluate critical processes, such as support for faculty course development, relationships between course designers and developers and faculty, and student ability to get help as needed.

The processes that are involved in producing and delivering online courses and instruction should be evaluated alongside the outcomes of these processes so it’s possible to see what changes would allow for better outcomes.

I recently worked with an institution that had an adversarial relationship between faculty and the online course development team, and both spent time pointing fingers to explain why the results weren’t optimal. What they didn’t see was that this adversarial relationship created bottlenecks and course development problems. Obvious solution? Build a better process and fix the damage caused by the old one.

If the process of producing and delivering online courses and instruction is problematic, courses and instruction are also likely to be problematic—and these problems are unlikely to improve without improving the process. So, while evaluating outcomes, it’s also important to evaluate the processes that impact those outcomes. You will find inefficiencies, poor relationships, rework, contention, and more that are making better outcomes difficult or impossible.

Misconception: Evaluation is a CYA activity to be endured.

Reality: The purpose of evaluation should be to continuously improve, not to check off boxes on a checklist and then breathe a sigh of relief until evaluation needs to be done again.

Most higher education institutions conduct end-of-course evaluations, but this kind of evaluation often doesn’t result in significant improvements to courses and cannot impact courses in progress.

Because end-of-course evaluations may be required but often aren’t sufficient, some online instructors have begun to implement weekly or bimonthly anonymous evaluations by students so they can make changes to the course and the process in the here and now.

Bottom line? The purpose of evaluating online courses and instruction should be improvement, not pain. And improvement efforts are most successful when they are valuable to all concerned. So analyze whether the misconceptions described in this article apply to your institution—and if some do, consider how to change them for the better.

Course and Instructor Evaluation: If It’s So Good, Why Does It Feel So Bad? excerpted from Online Classroom, January 2008.

10:09:2009

Using Self-Check Exercises to Assess Online Learning

The intermediate statistics class I took quite a number of years ago had two types of learners at the outset—those who were worried about passing the course and those who were sure they couldn’t pass it. The professor clearly understood the “fear-of-stats” phenomenon and used a number of instructional techniques to help learners gain confidence and skills.

One especially valuable technique was consistent use of self-check exercises. These were handed out at the end of each class along with an answer sheet. Class started each time with a question-and-answer period about the self-check exercises from the previous session. Doing the exercises was optional and they weren’t handed in or graded, but nearly everyone did them, and the folks who did easily gained confidence and passed the course.

What are self-check exercises, exactly? They are problems (with answers) given to learners that allow them to assess how they are doing on an ongoing basis. Doing them online with self-grading provides immediate feedback. Links to additional materials can be provided to help anyone who is having difficulties. Online learners can do these exercises and submit questions they have, which the instructor can aggregate and respond to for the benefit of all learners.

Studies show that these types of activities help learners keep tabs on their progress and adjust their efforts, know when to seek help, and stay on track. These outcomes are especially important in online courses.

Some of the most important benefits of self-check exercises for online learning include:

• Helping learners determine what they do and do not understand so they can target where extra study is needed.
• Providing immediate feedback to learners and an option to link to additional materials (which may reduce the number of unfocused questions sent to the instructor).
• Providing feedback to the instructor about where learners are having difficulties so immediate interventions can be implemented.
• Increasing learner satisfaction with the instructor and the course.

Getting Started
Consider how self-check exercises can be used in the courses you teach. Are there concepts that learners consistently have problems understanding? Are there terms that learners need to memorize or concepts that they need to understand? These might be the best places to start.

Patti Shank, PhD, CPT, is a widely recognized instructional designer and instructional technologist, and writer who builds and helps others build good online and blended courses and facilitates learning. She can be reached through her website: http://www.learningpeaks.com.

Excerpted from Online Classroom, Feb. 2007.

05:01:2009

Creating Better Multiple-Choice Tests for Online Courses

Multiple-choice tests are commonly used to assess achievement of learning objectives because they can be efficient. Despite their widespread use, they’re often poorly designed. Poorly written multiple-choice tests are equally damaging in classroom-based and online courses, but in online courses learners often have to contend with more challenges, and poor assessments can add insult to injury.

Some plusses and minuses to multiple-choice tests
Multiple-choice tests can be developed for many different types of content and, if the test items are well written, can measure achievement of multiple levels of learning objectives, from simple recall and comprehension to more complex levels, such as ability to analyze a situation, apply principles, discriminate, interpret, judge relevance, select best solutions, and so on.

Multiple-choice tests are easy to administer and can be improved using item analysis in order to eliminate or correct poorly written items. They are easy to score and less susceptible to scoring subjectivity than short-answer or essay-type items. They don’t measure writing ability (which can be a plus or minus) and often do assess reading ability (another potential plus or minus, but in reality often a minus). They are more subject to guessing than many other types of learning assessments.

Multiple-choice tests are often promoted as “objective.” Although scoring them doesn’t involve subjectivity, humans do judge what questions to ask and how to ask them. These are very subjective decisions!

When multiple-choice is appropriate
Multiple-choice test items call for learners to select an answer or answers from a list of alternatives. Because they do not ask learners to construct an answer or actually perform, they tend to measure knowing about rather than knowing how.

Multiple-choice items cannot assess learners’ ability to construct, build, or perform. They are best used for objectives that can be assessed by selecting the correct answer from a list of choices rather than supplying the answer or performing a task. Think for a moment about how different selecting is from constructing and performing and you’ll recognize the limitations of multiple-choice testing.

Writing better multiple-choice items
Confusing and ambiguous language and poorly written or implausible distractors are very common errors when writing multiple-choice test items. Here’s a to-do list to help you avoid these mistakes and write better multiple-choice test items.

• Provide clear directions. Group questions with the same directions together.
• Include as much of the question as possible in the stem, and reduce wordiness of alternatives.
• Include words in the stem that would otherwise be repeated in each of the alternatives.
• Make sure language is precise, clear, and unambiguous. Include qualifiers as needed, but don’t add unnecessary information or irrelevant sources of difficulty.
• Avoid highly technical language or jargon unless technical knowledge and jargon are part of the assessment.
• Avoid negatives and these words: always, often, frequently, never, none, rarely, and infrequently. When a negative is used, it should be CAPITALIZED, underlined, or bolded to call attention to it.
• Don’t use double negatives or double-barreled questions (asking two things in one question).

Although it takes time and practice to write good items, this time and effort is well spent.

Patti Shank, PhD, CPT, is a widely recognized instructional designer and technologist, writer, and author who builds and helps others build good online courses and facilitate learning. She can be reached through her website: http://www.learningpeaks.com/.