There are many reasons why student feedback is not particularly helpful, but there are things we can do to make it better. Here are a few suggestions on how to extract more value from course evaluations:

We don’t learn a lot from student feedback when we don’t ask good questions. At the top of my list of bad questions I’d put the ever-popular “What did you like most/least about the course/instructor” kind of questions. I wish I could make those questions illegal. Since when did the goal of education become providing learners with what they like? I know teachers can’t remove these questions from institutionally mandated forms. We can object, though, and we can ask students better questions on our own. If teachers want to make changes that improve teaching and learning, we need to ask about the impact of a policy, practice, behavior, technique, assignment, or instructional approach on students’ efforts to learn.

We don’t learn as much as could when we ask after it’s too late to make a difference. For me that means the end of the course when students are busy and stressed. They’ve got more important things on their minds. And at many institutions they are asked to evaluate every course every semester, which is not what the research recommends. Then there’s the reality that the feedback they provide isn’t going to benefit them—the course is over. The feedback that helps teachers make good choices about what and how to change doesn’t emerge from those overall, global assessments of how the course compares with all other courses on the planet. It’s found in responses to smaller segments of instruction, or course events like assignments or group activities, and it’s solicited right after the fact while students clearly remember what happened and the teacher has time to implement alterations.

We compromise the learning potential of student feedback when we don’t teach the principles of constructive feedback. The benefits of doing so go in both directions. Teachers get feedback that is more helpful than hurtful and students start developing a skill they can use in virtually every profession. Students deliver more constructive feedback when they understand what teachers do and don’t have the power to change, and what is and isn’t relevant to learning. Most of the time we don’t decide when the class meets or who enrolls in it. Moreover, our selection of ties or the types of earrings we wear don’t merit commentary in feedback that addresses learning experiences. Constructive feedback doesn’t preclude students from identifying things about the course and instruction that compromised their efforts to learn. It’s about how those comments are delivered. The golden rule of feedback is that teachers and students should give each other feedback in the form they’d like to have feedback given unto them.

We don’t learn much from student feedback when they don’t take the process seriously. And the reason students don’t take the process seriously is because they don’t think we do. They complain about some teachers, assignments, and courses year after year and nothing changes. Teachers can convince students that their feedback does matter by soliciting it and then talking about it during the course. Responding to student suggestions does not obligate the teacher to do whatever students recommend. If a course activity or assignment is essential to achieving certain learning outcomes, then removing it would be irresponsible. Teachers can help students understand by explaining the educational rationale behind the decision to continue the activity or assignment, and then by exploring what could be done that might help them do better. If a teacher makes a change that students recommend, they often feel vested in making the change successful.

It’s true that students don’t always provide good, helpful feedback, but that doesn’t mean they can’t or won’t. It’s up to faculty to solicit and respond to their feedback in ways that make it a learning experience for both parties.

Yes, students can learn from activities that involve them in providing instructional feedback, especially if it’s focused on their learning experiences in class. Most students have little insight into themselves as learners. So, if the assessment activity gets them thinking about how they learn and what teaching policies, practices, and behavior expedite their efforts to learn, it can be a beneficial activity for them as well as for the teacher.

The trick is coming up with feedback activities that garner these benefits and I just found a great example. Professor La Lopa, who teaches hospitality and tourism management at Purdue University, has students in his 200-level Human Resource Management course write a reflective paper on quality teaching and its assessment. (I can hear some of you wondering about the appropriateness of the assignment. His article, referenced below, explains the context which more than justified it for me.) What’s most creative about the assignment are some of the prompts students respond to in the paper. Here’s a condensed and slightly edited version of some of them.

• How would you describe your ideal professor? Include a description of the classroom setting (number of students, physical space, etc). Paint as clear a picture for me as possible so I can envision your ideal college professor and class.
• Now describe the typical teacher you have actually experienced in your courses here. What is the typical classroom setting like?
• If you could put one question on a course evaluation what would it be and why would you ask it?
• If you were the president of your college or university, what method would you use to evaluate the [teaching] performance of college professors?

The article is worth reading for the quotes excerpted from the student papers alone. Their observations demonstrate just how well an assignment like this gets students thinking about good teaching, its assessment, and its relationship to learning.

There are lots of potential spin-offs from an activity framed around these questions. The most frequently mentioned characteristics of the “ideal” professor could be shared and discussed. Why these characteristics? Are these characteristics that support efforts to learn? How? Why? How about the teacher writing a short description of the “ideal” student followed by another short description of the “typical” student? I wonder if the one question teachers would add to the course evaluation would be anything like the question students would add. Maybe the best way to evaluate professors is by how well their students learn. Is that a good idea? Why? Why not?

There’s lots of research documenting that students don’t believe that their feedback is taken seriously by institutions or instructors, which in part explains the poor quality of the feedback they provide. And there’s lots of research documenting that if faculty talk with students about assessment feedback it improves end-of-course ratings. It’s a visible sign that teachers care and are willing to work with students, even if we don’t make all the changes they propose. Good feedback activities like the one described here have one final benefit: they can be learning experiences for students.

Please share the ways you collect, respond to, and use feedback from students. We’re especially interested in those ways that also encourage students to encounter themselves as learners.

Reference: La Lopa, J. (2011). Student reflection on quality teaching and how to assess it in higher education. Journal of Culinary Science & Technology, 9 (4), 282-292.

How Can I Use Frequent Student Feedback to Improve My Courses?

Program includes a CD with the video presentation, plus supplemental materials, PowerPoint slides, and complete transcript • $99

It’s the end of the semester, and you’ve just given out the course evaluations. You find that a significant portion of your class is dissatisfied with the course. Your students aren’t grasping the learning objectives, and they are struggling.

What’s wrong with this picture?

If you are only asking for feedback at the end of the semester, there’s not much you can do to improve the learning for your current students. It’s too late. That is why it’s important to get student feedback early in the semester — and multiple times throughout — to understand your students and meet their learning needs.

In How Can I Use Frequent Student Feedback to Improve My Courses?, Mary Clement, Ed.D., shares the five times during the semester when getting student feedback is most valuable and provides practical ways to obtain that information.

You’ll learn techniques you can implement right away, such as:

• Conducting a “Student Interest Inventory” early on, to learn about students’ subject-specific knowledge and how they like to learn
• Using ungraded “One-Minute Papers” to find out what your students feel they’d need to study if there was a test tomorrow
• Surveying students after the first exam or paper to see how they prepared for it and to find out whether their grades matched their expectations
• Sharing feedback with students when you’re ready to implement changes based on their input

During this information-packed session, you’ll learn how to:

• Select optimal times to solicit student feedback
• Develop easy-to-use instruments for student feedback
• Ascertain the quality of student feedback
• Use student feedback to generate collegial discussions about teaching
• Make course improvements based on student feedback

How Can I Use Frequent Student Feedback to Improve My Courses? includes a CD with the video presentation, supplemental materials, a critical reflection worksheet, a copy of the PowerPoint slides and the program’s transcript.

Want to make this program available for your entire campus?

Consider ordering a Campus Access License for an additional $100. It allows the purchasing institution to upload the CD of the seminar onto the institution’s password-protected internal website for unlimited access by the entire campus community.

However, with the right approach, these course evaluations can be a very constructive tool. One of the keys is to solicit feedback throughout the course, rather than waiting until the end when it’s too late to make improvements that will make a difference to the current roster of students.

As both a teacher and an instructional designer for more than 25 years, I have been blessed to see things from a variety of perspectives. I have also learned some lessons the hard way! Here are 10 strategies for administering multiple informal course surveys that have proven very successful for me:

1. Ask the questions clearly. Perhaps one of the reasons student comments aren’t clear is because we’re asking somewhat ambiguous questions. Give the exercise the same time and attention you do when writing learning goals. Ask yourself, “What am I trying to assess?” Have a colleague you respect review your questions the first time out.

2. Ask the right questions. If you want to know if one of your primary learning outcomes is being/was achieved, ask. Sure it’s the student’s perspective, but it can be a valuable piece of information and you will spot a trend quickly. These questions work nicely in a Likert-scale multiple choice format. Other questions can be the same as, or similar to, the department’s formal end-of-course evaluations.

3. Ask for written comments. I have always found that written comments can be the most insightful, if interpreted fairly. As Weimer (2012) pointed out, two students can take two different meanings from the same question, so keep it simple.

Four questions I use are:

1. What is one thing you like about this course (so far)?
2. What is one thing you do not like about this course (so far)?
3. What is one thing that could be improved in this course?
4. Do you have any additional comments you would like to share?

4. Use an LMS or some other way of automating the process of administering the survey. This makes it easy for you and your students. Use the reporting and statistical analysis tools to help you interpret the results.

5. Share the results with your students. This is a critical step. Identify the items that you can deal with and follow through. Tell your students how you will address their concerns. Explain to the students the items that you cannot fix and pass these along to the appropriate departments, if necessary. I also told my students that I would publish the results, including my comments on my faculty homepage, and share them with their program coordinator and dean or academic manager, which I did. There were multiple short and long-term benefits from doing this: accountability, student buy-in, and trust from my academic managers. Risky perhaps, but well worth it in the long run.

6. Survey more than once. Perhaps one-third or halfway through the semester, and once more. Assess any progress you made from the comments on the first survey. This can even be an additional survey question. Don’t wait until the department’s formal end-of-course evaluation – by then it is too late. Your students will also appreciate your genuine interest in improving the course for them. If your course gets off track a little at the beginning, you will have ample time to get things back on track.

7. Don’t survey unless you are prepared to deal with the feedback. If you ask your students for feedback, summarize and share the results, and implement the things that are constructive, your students will feel empowered. They will take a different look at the course and become part of the growth process, your included. This was one of my course activities that my students appreciated the most.

8. Tell your students what you are looking for. Let them know it’s ok to be negative, as long as the criticism is constructive and not mean-spirited. I don’t allow them to name fellow students or other teachers, or make comments unrelated to the course.

9. If you are not inclined to use a survey/questionnaire approach, use a simpler tool. Try a “start-stop-continue” approach, a “one-minute” paper, or a simple poll. These can be done almost weekly or at the end of each module/lesson.

10. Keep your perspective (and humor) on things. Once you get the results/comments, interpret them carefully, but don’t overanalyze or internalize any criticism. Teachers whom I have assisted with this activity initially get very worried because students can be blunt. My advice? Treat the comments like judges scoring figure skaters in an international competition. Throw out the high one and the low one, and what’s left over is usually a good overall indicator. It might not be all positive feedback the first time, but as long as it is constructive, you have something to build upon.

At the start of every one of my courses, as one of the ice-breaker activities and introductions, I would ask my students to visit my faculty homepage and look at the course feedback results for previous cycles of this course. On the page they can see not only the feedback from students, but my responses to the individual comments. By doing this, I think I demonstrated that I was serious about the quality of the course and my performance. I would also point out that I expected the same level of quality insights and constructive feedback from them two or three times during the semester.

Greg Cooper, M.A. Ed. is an instructional designer with the Teaching and Learning Centre (TLC) at the University of Calgary, Alberta. He previously worked at Cambrian College for 27 years as an elearning designer and as a professor in face-to-face and online courses, and at the University of Wisconsin-Madison as a learning technology consultant.

Reference:
Weimer, Maryellen (2012, November 28). End-of-course evaluations: making sense of student comments. Faculty Focus, retrieved from http://www.facultyfocus.com/articles/teaching-professor-blog/end-of-course-evaluations-making-sense-of-student-comments/

I think part of the reason for the vague feedback is that students don’t believe that the evaluations are taken all that seriously, not to mention they’re in the middle of the usual end-of-semester stress caused by having lots of big assignments due and final exams to face. It’s just not the best time to be asking for feedback and so students dash off a few comments which instructors are left to decipher.

In most end-of-course evaluations, students tend to comment about some of the same aspects of instruction. They frequently address issues of organization, whether students were treated fairly and the challenging aspects of the course. Carol Lauer wondered if faculty and students defined some of these common terms similarly and so she asked a faculty and student cohort to say what they meant when they saw or used the term on course evaluations.

Would you be surprised to learn that faculty and students define the terms differently, or that students themselves don’t agree on definitions? Probably not, I’m thinking. Even so, some of the specifics are interesting. Take “not organized,” for example. Almost a third of the faculty think students use that term when the teacher changes or doesn’t follow the syllabus. Just over 11% of students said that’s what the term meant to them. Seventeen percent of the students equated it with the instructor not being prepared, 15% said they used it when the teacher had no apparent plan for the day and almost 13% equated it with getting student work graded and returned slowly.

“Not fair” refers to problematic grading according to almost 50% of the faculty surveyed, but to just over 2% of the students. To students “not fair” gets written on an evaluation when the teacher plays favorites and doesn’t treat all students the same way. Students and faculty are closer in their understanding of what “challenging” means when it’s applied to a course. It means hard work and lots of it.

The point here isn’t terribly profound but it merits a reminder, especially at the end of courses when teachers are tired. Many of the terms used to describe teaching on rating forms and in student comments are abstractions. “Organized” is something teachers are and deciding whether a teacher is or isn’t depends on what the teacher does. Various behaviors, actions and inaction can be what any given individual sees as the presence or absence organization.

There is good news here. If you’re interested in improving something like organization, if you define it behaviorally, you can change what you, do which is a lot easier than changing what you are. Organization has never been one of my strong suits and I didn’t make much progress trying to “be” organized. But when I started putting a skeleton outline on the board, when I stopped five minutes before the end of period and used the outline to summarize, when I began class working with students to create a list of points to remember from last class, I was seen by students as being more organized.

But it isn’t all good news. A collection of dashed off student comments collected at the end of the semester doesn’t easily translate into an action-based improvement agenda. What the student comments mean is probably not what you think they mean. Communication about the impact of teaching policies and practices on efforts to learn needs to be ongoing so there’s an opportunity for clarifying feedback, adjustments and then more feedback. We can and should make efforts to change the way our institutions collect student assessments, but, until that glacier melts, we need to take matters into our own hands and solicit a different kind of feedback and at different times during the course.

Reference: Lauer, C. (2012). A comparison of faculty and student perspectives on course evaluation terminology. To Improve the Academy, 31, 195-211.

This question was explored in a study that involved a 4,000-student population and 848 undergraduate courses. The students have a two-week window during which they can electronically submit their anonymous course evaluations, one for each course in which they are registered. During that two-week period, they receive three email reminders.

The collected data enabled the faculty researcher to identify several characteristics that differentiated students who completed the course evaluations from those who did not. First-term beginning students respond more, as do students who are evaluating a course that is a requirement for their major. The author suggests that new students may be more enthusiastic about participating in university life. More seasoned students may think that the evaluations are not taken seriously by the instructor or institution and therefore are less motivated to complete them. It makes sense that students would consider courses in their major more important than other courses. Interestingly, course size was not a variable that reliably predicted who would complete the surveys.

The data also revealed that men, students with light course loads, and students with low cumulative GPAs and low course grades were less likely to evaluate the course. Why are women more likely to evaluate their courses? The researcher refers to this result as “puzzling.” (p. 22) The course load variable “appears to be a measure of student attachment to the university.” (p. 23); those taking fewer courses tend to be less committed to the institution.

Certainly the most interesting finding is the data indicating that students doing poorly in the course are less likely to complete the course evaluations: “A matched pairs test that completely controls for class- and instructor-invariant student characteristics confirms the finding that students who do better in the course are more likely to participate in SET (student evaluation of teaching).” (p.28) Add to this another finding documenting that students who are more likely to have strong opinions about the course (indicated by how quickly within the two-week window they submitted their evaluations) had, on average, positive views about the course and instructor. The author concludes that based on this data, online course evaluations do not attract disproportionately more dissatisfied students. In fact, they do the opposite, giving credence to the contention that the results may be biased in favor of the instructor and course as opposed to against them.

Reference: Kherfi, S. (2011). Whose opinion is it anyway? Determinants of participation in student evaluation of teaching. Journal of Economic Education, 42 (1), 19-30.

Reprinted from “Who Participates in End-of-Course Ratings?”The Teaching Professor, 25.9 (2011): 4,5.

For more on Faculty Evaluation, download a FREE copy of Effective Strategies for Improving College Teaching and Learning!

Sign In Create an Account

Yet, some areas of higher education have undergone very little change. In particular, the methods used to evaluate and ensure that quality teaching is occurring in traditional and online classes. Yes, there are those required course evaluations dutifully and anonymously completed by students at the end of every term, but these are typically viewed quantitatively and do little to transform instruction. The quality of teaching is rarely given serious attention. Basically, if a professor simply shows up to teach class (and sometimes even if she doesn’t) she receives a satisfactory rating without consideration of how well she did her job. In other words, the methods used to evaluate teaching in higher education are as outdated as cassette players and floppy disks.

Routinely taking for granted that satisfactory teaching at the college level is acceptable requires us to ignore a significant body of research on this subject. Scholars agree that good teaching requires time, effort, commitment, knowledge, presence, and ingenuity (Weimer, 2010). Good teaching does not just happen. So how do we invoke change in a system that has been in place since dissertations were written on manual typewriters?

Faculty can bring attention to the need for a better system of evaluating teaching by demonstrating excellence even when it is not required or even expected. Documenting and sharing evidence of quality teaching serves as a reminder to administrators (as well as colleagues) that there is more to teaching than showing up. Submitting a report with additional teaching-related information is a good first step. For example, faculty members can report on course load, number of students enrolled in each class, grade distribution, and the number of tests and other graded assignments. Further documentation may provide information on less quantifiable items such as the type and quality of feedback given to students, use of technology to teach content, and creative delivery methods.

Faculty members can take the voluntary evaluation process one step further by implementing supplemental evaluations of their teaching. The following section offers five suggestions for supplemental evaluations that provide meaningful feedback that can be used to improve teaching.

Midterm Course Evaluation – Student learning and experience with the class can be gauged halfway through the course through an anonymous evaluation. The instructor can review the feedback and immediately make adjustments if there is a general consensus about an issue in the class. The midterm course evaluation may also provide information about the type of lessons and assignments that students feel are beneficial. Surveys may be formal with specific questions answered on a Likert scale or it may ask open-ended questions which allow students to elaborate in more detail. This type of evaluation has obvious advantages over those required at the end of the term because students can benefit directly from the feedback provided.

Questionnaire Following Exam – Feedback from students about the format, content, and level of difficulty for specific questions can be useful in making changes to an exam or the lessons that were given in preparation for the exam. This same technique can also be used at the conclusion of a major assignment or project. Ask students: “1) Were instructions clear? If not, what additional information would have been helpful? 2) Was the exam or assignment beneficial to learning and retaining course content? 3) Please share other constructive comments.” Faculty can then use the information to make changes prior to the next exam or project.

Interview Students – Invite the highest achieving students to participate in an exit interview at the end of the semester. Why the highest achieving? These students will provide the most accurate and honest perspective of how to improve the course. Prepare questions in advance and be specific. For example: “If one assignment had to be removed, which should it be? Which assignment should definitely be kept in the course? Why?” Reflect on their responses and decide how information can be used to improve the course.

Peer Evaluations – Peer feedback typically takes the form of classroom observations but can also include reviewing course materials and assignments with a colleague who can give suggestions. Faculty should choose a colleague whom they trust and respect as a teacher to conduct the peer evaluation.

Self-Evaluation – When examining one’s own teaching behavior it is imperative that the faculty member realistically assess areas of strength and weakness. Videotaping a class to view and evaluate later is an excellent tool. The purpose of the self-evaluation is not just to show teaching ability in a positive light, but to demonstrate how feedback can be used to improve instruction.

There are many options for documenting teaching through records and supplemental evaluations. Hopefully, these suggestions have sparked interest in developing a voluntary evaluation system.

Now it’s your turn. What are some of the ways you evaluate your teaching effectiveness? Please share in the comment box.

Reference: Weimer, M. (2010). Inspired college teaching: A career-long resource for professional growth. Hoboken, NJ: Wiley, John & Sons, Incorporated.

Dr. Karyn W. Tunks is an associate professor in College of Education at the University of South Alabama.

1. Understand and accept today’s college students. First and foremost, students want us to know who they are. They want us to know their names and to know about their world. Today’s students are busy, technologically savvy, and multitaskers. Some are prepared for college work; others are not. Regardless of their backgrounds, all students have lofty ambitions and want to succeed. To help them, we can provide background knowledge in our subject areas. We also need to share the rationale behind what we do and ask students to do. I recommend making invisible expectations explicit. I regularly start class by saying, “We are learning this because …” When students understand why and how the material is relevant to them, they find more motivation to study and end up rating the course more highly.

2. Establish clear criteria for grading. All students want good grades, and they want to know exactly how to get those grades. College students today have experienced criteria sheets and rubrics since elementary school, and they want the same in college. They want to know where they stand on any given day in the semester.

After 20 years of college teaching, I have learned that telling students that their grades are based on percentages (20 percent homework, 25 percent quizzes, etc.) does not have meaning for them. They cannot figure their grades with a percentage system. A total point grading scale provides that clarity. Each assignment, quiz, lab paper, project, or exam has a certain number of points. These point values can be listed in the syllabus as well as the total points needed for the final grade. To help students keep track of their points, I give them a worksheet and explain that if they record their points, they will know exactly how many they have and how many they still need to earn.

I get rave reviews about my “no-mystery” approach to grading on course evaluations, and I believe that the good reviews are due to the clarity and ease of the total point system. I re-explain the grading system after the first assignments/quizzes/tests are returned. On the first day of classes, students are overloaded with information. It’s not until after an assignment has been graded that they are ready to understand how the grading system works.

3. Get formative feedback early. The end-of-course evaluation is a summative one. Although it aims to help us improve future courses, it does not enable us to respond to the needs of the students currently enrolled in the course. Formative feedback collected early in the course accomplishes that goal.

The first major paper or exam is a great time to collect formative feedback. I recommend attaching a page to the back of the exam, or asking students to respond to questions like these on the day papers are due:

1. How long did you study for this exam or work on preparing this paper?
2. How/where did you study/write?
3. Which class activities (lectures, discussions, reviews, online notes) helped you the most in learning this material? Why?
4. Which class activities helped you the least? Why?
5. Which topics remain the most difficult for you?
6. What has a professor done in the past that helped you learn?

If I have students answer these questions on a page attached to the exam, I let them know they can tear that page off and submit it anonymously when they turn in the exam.

You can also ask for feedback once that first exam or paper has been graded and returned. I like to ask questions then about improvement goals—what the student wants to do better and what else could be done in class to support their efforts to improve. If you don’t want to deal with open-ended questions, students can rate declarative statements such as “I would prefer more discussion of assigned readings.”

If you ask for their input, students will want to see that you listened to them. They may expect some changes. A short report back to them can be used to explain what you are willing and not willing to change, and why.

Students aren’t qualified to comment about all aspects of instruction, but they can rate how they feel about the classroom climate. They are good evaluators of what helps them learn and what confuses them. Getting their feedback early in the semester enables you to tailor the course to their learning needs.

Finally, it helps to talk about your evaluation results with a trusted colleague. Sharing student responses can help us see patterns in evaluations and sort out the “outliers” or just plain wacky comments. Knowing what our students need helps us teach in ways that promote their learning, and that means better results on the end-of-course evaluations.

Dr. Mary C. Clement is the director of the Center for Teaching Excellence at Berry College.

Excerpted from The Teaching Professor, 25.4 (2011): 1,3.

“There’s a great deal of faculty dissatisfaction out there with the over-reliance on student ratings to evaluate teaching effectiveness,” said Linda B. Nilson, PhD, founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. “In fact these ratings are very good for measuring customer satisfaction, no doubt about that, but really, under the best of circumstances, there’s only a mild correlation between student learning and an instructor’s rating.”

During the recent online seminar Measuring Learning: The Ultimate Teaching Evaluation, Nilson outlined a number of different tests and instruments faculty could use to serve as solid evidence of student learning as a result of the course. Faculty could then attach the results of the measurement along with their student ratings.

Some of the measurements discussed during the seminar include:
Perceived learning gains – Developed in 1997, the Student Assessment of their Learning Gains (SALG) instrument was originally used to assess learning in undergraduate chemistry, but was later updated to meet the needs of a broader range of courses. The SALG asks students to assess and report on their own learning, and on the degree to which specific aspects of the course have contributed to that learning.

First week writing and corrections – Give the students an ungraded writing assignment on key concepts, principles and techniques, processes, and cause/effect issues related to your course content. At the end of the course, have students write a letter to their “pre-course self” correcting errors, poor reasoning, misconceptions they may have written about in the first assignment.

First-week final exam – One of the more controversial methods of measuring student learning is to have students take the final exam during the first week in class, but don’t grade them on it. At the end of the semester give them that same exam again and compare the results. While letting students see their final exam makes some faculty nervous, Nilson says most students won’t remember any of the questions, and if they do what’s the harm? It will simply help them focus in on what you feel is important for them to know.

Targeted essay – In order to demonstrate the relevance of the course content, as an end-of-course assignment you could ask students to write an essay based on this question: Pretend you are on a job interview and the interviewer asked “What are the three most important things you learned in your X course?”

Nilson says she would like to see student satisfaction play a lesser role in the evaluation of faculty, noting that the real goal of higher education is not to please students but to facilitate student learning. Interestingly, a side benefit of simply asking students to think about and articulate what they learned in a course could be higher student ratings.

“Really what it does is make students more aware of their learning,” said Nilson. “Yes, it could make the evaluations more positive, but it’s also making them more accurate. When students have to reflect on their learning — where they were versus where they are now — they’re going to give a more valid assessment of your teaching and the effectiveness of the course and that’s not a bad thing… They can’t say ‘I didn’t learn anything in this course.’”

Measuring Learning: The Ultimate Teaching Evaluation

Student evaluations of faculty have been around for decades, but never before have they carried so much weight, or been so divisive. The use of student evaluations for faculty review can be problematic because student ratings aren’t the best measures of student learning. What’s more, when careers depend on student satisfaction, many may feel compelled to keep students happy, rather than ensure academic rigor.

Do you ever wonder what’s more important – educating your students or producing satisfied customers?

When student ratings are the sole measurement of teaching assessment, many faculty start to wonder.

Order a copy of Measuring Learning: The Ultimate Teaching Evaluation, and learn about Dr. Linda B. Nilson’s approach to measuring educator effectiveness based on what really counts: student learning. Dr. Nilson is the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University, and author of Teaching at Its Best: A Research-Based Resource for College Instructors, now in its third edition (Jossey-Bass, 2010).

What You’ll Learn
Stressing the importance of measuring student learning and aggregating the results, Dr. Nilson shows you:

• How to adapt tests and instruments to serve as evidence of students’ learning
• How to give this evidence a number, so others can use it easily
• How to submit this evidence for faculty review, to balance student ratings.

Order the CD + Transcript package

Different Methods for Different Disciplines
During the online video seminar, you’ll learn measures of student learning you can tailor to your subject matter and course level, such as the following end-of-course measurements:

• Integrative essay or journal entry
• Targeted essay questions
• Survey of students’ perceived learning.

Tests for the beginning and end of your course include:

• A first, ungraded assignment, later corrected for a grade
• Giving your final the first week of class (ungraded), repeating it at the end of the session for a grade, and comparing results
• Surveying students’ confidence about knowledge of course material.

You’ll also receive sample documents and additional resources with in-depth recommendations.

When you order the recording on CD, you also receive a copy of the transcript, all for the same price as attending the seminar live.

An optional Campus Access License is available for an additional $200. It allows the purchasing institution to upload the CD of the seminar onto the institution’s password-protected internal web site for unlimited access by members of the campus community.

Order the CD + Transcript package

Who Will Benefit:
Measuring Learning: The Ultimate Teaching Evaluation, is an intermediate level seminar and will help all higher education faculty members interested in improving assessment of teaching effectiveness. In addition, deans, provosts, academic vice presidents and department chairs who evaluate their faculty’s teaching effectiveness will find this seminar invaluable.

All seminars include a discussion guide for facilitators
Participating in a Magna Online Seminar as a team can help leverage unique insights, foster collaboration, and build momentum for change. Each seminar now includes a Discussion Guide for Facilitators which provides step-by-step instructions for generating productive discussions and thoughtful reflection. You’ll also get guidelines for continuing the conversation after the event, implementing the strategies discussed, and creating a feedback loop for sharing best practices and challenges.

The collection of items on student rating forms can be thought of as an operational definition of good teaching and we all know that good teaching can be defined in many different ways. One interesting thing you can do with these instruments is to go through and cross off or change the items that don’t fit with your definition. I’m not suggesting you revise the instrument and then administer it, but making the changes for your own edification enables you to see where you agree and disagree with the definition proposed by the instrument.

Too many rating instruments remind me of that old expression that a camel looks like a horse designed by a committee. They are assembled via a political process where those with labs want questions about labs and those with studio courses want questions about those. Usually the compromise involves including both.

When an instrument is empirically constructed the process of deciding what goes on it involves something called validity. David Kember and Doris Leung offer a simple description of validity saying it is established “if an instrument actually provides a measure of what it purports to measure.” (p. 342) There are two kinds of validity; face validity and content validity. Face validity means the wording of the items refers what is being measured. That’s pretty straightforward and not really a problem on most course evaluation instruments. Content validity implies that an instrument includes all the dimensions, aspects or parts of the construct and that those parts are represented in a balanced way. That’s a problem with something like good teaching where the definitions are different and not universally agreed upon.

Empirically developed rating instruments assemble the collection of items based on the reports of various interested parties. In the case of teaching, that has meant students (current and former), teachers with a special emphasis on the views of good teachers, and administrators. In the mid 70s, Ken Feldman, whose meta-analyses on various aspects of ratings are legendary, reviewed the research on the ingredients and components of effective teaching (as reported by these groups) and from that large and not well-organized data base derived a set of 19 characteristics. His work and others like it justifies the inclusion of items that commonly appear on rating forms—things like teacher’s preparation and organization of the course; teacher’s enthusiasm for the subject; and teacher’s availability and helpfulness.

Times change and as Kember and Leung point out the characteristics that emerged out of Feldman’s analysis of the literature focus on teaching. Eleven of Feldman’s 19 characteristics begin with “teacher’s” and four more deal with the content and its presentation. “The model is of the teacher-centred content-oriented type [of instruction]. The dimensions fit well with didactic teaching, but it is hard to see the applicability of many of the dimensions to other more student-centred forms of teaching.” (p. 342) Kember and Leung’s new instrument is more attuned to the goals and objectives of learner-centered teaching. If that’s of interest, their article can be consulted.

My goal here is to make sure when you take a look at your results, you consider how the instrument is defining good teaching. How closely does that definition correspond with your own? And second, you recognize that definitions of teaching are not all equally acceptable. If you’re using an instrument to acquire feedback for yourself, then you can and should ask students for feedback in areas relevant to your definition. But if the instrument is being used to assess teaching across the institution, then the item selection process should be governed by what is known about aspects of instructional practice that can be linked to learning outcomes.

Reference: Kember, D. and Leung, D. Y. P. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education, 33 (4), 341-353.

The researchers surveyed this group, asking the faculty members to respond to several questions, including this most important one: “Your student ratings have increased for at least three consecutive semesters during the last three years in your [Course Name] class. What factors led to this change in your teaching performance?”

The slightly more than 200 respondents most frequently attributed the increase in ratings to changes made in one or several of these five areas:

1. more active/practical learning, including efforts to make the content’s relevance apparent to students;
2. better teacher/student interactions, exemplified by learning students’ names and having individual conferences with them;
3. making expectations for learning outcomes clearer while still maintaining high standards;
4. being better prepared for class; and
5. revising the evaluation policies and procedures used to assess student work.

The first three of these categories accounted for almost 50 percent of the faculty responses. A bit surprisingly, 5 percent of the respondents whose scores had improved didn’t list anything they’d done or they indicated that they were not aware of having implemented any changes.

This cohort of faculty included full-time tenured faculty (actually this was the largest group, 56 percent), full-time nontenured faculty (12 percent), and part-time appointees (35 percent). The researchers note that this indicates how faculty in all kinds of positions can improve. That so many in the already-tenured and part-time categories did so is especially noteworthy and encouraging.

In addition to the survey, 30 faculty from 10 of 12 colleges at the institution were interviewed “to gain a better understanding of the change process.” (p. 167) Several interesting findings emerged from the interviews. For many faculty members, the most difficult part of the process was being willing to admit that they needed to change. “Humbling” was an adjective used to describe the feeling. Often there was some sort of triggering event—frequently it involved end-of-course ratings results. After teaching a course seven times, one faculty member received his lowest-ever overall course rating. He was shocked but reported that he decided to find out why. Others talked about an overall lack of excitement in the course and their own motivation to change and do better.

In the interviews, almost 80 percent of the faculty indicated that the effort required to implement the changes was minimal. It seemed that for most it was more a matter of fine-tuning their teaching. The researchers conclude, “The results of this study should be encouraging to faculty members who feel they cannot improve.” (p. 171)

Reference: McGowan, W. R., and Graham, C. R. (2009). Factors contributing to improved teaching performance. Innovative Higher Education, 34, 161-171.

Reprinted from “Teachers Who Improved.” The Teaching Professor, 23.10 (2009): 2.

Misconception: Outcomes are the only things worth measuring.

Reality: Outcomes, such as numbers of new courses developed, enrollments, retention, and satisfaction levels, are important and they should be measured. But it’s also important to evaluate critical processes, such as support for faculty course development, relationships between course designers and developers and faculty, and student ability to get help as needed.

The processes that are involved in producing and delivering online courses and instruction should be evaluated alongside the outcomes of these processes so it’s possible to see what changes would allow for better outcomes.

I recently worked with an institution that had an adversarial relationship between faculty and the online course development team, and both spent time pointing fingers to explain why the results weren’t optimal. What they didn’t see was that this adversarial relationship created bottlenecks and course development problems. Obvious solution? Build a better process and fix the damage caused by the old one.

If the process of producing and delivering online courses and instruction is problematic, courses and instruction are also likely to be problematic—and these problems are unlikely to improve without improving the process. So, while evaluating outcomes, it’s also important to evaluate the processes that impact those outcomes. You will find inefficiencies, poor relationships, rework, contention, and more that are making better outcomes difficult or impossible.

Misconception: Evaluation is a CYA activity to be endured.

Reality: The purpose of evaluation should be to continuously improve, not to check off boxes on a checklist and then breathe a sigh of relief until evaluation needs to be done again.

Most higher education institutions conduct end-of-course evaluations, but this kind of evaluation often doesn’t result in significant improvements to courses and cannot impact courses in progress.

Because end-of-course evaluations may be required but often aren’t sufficient, some online instructors have begun to implement weekly or bimonthly anonymous evaluations by students so they can make changes to the course and the process in the here and now.

Bottom line? The purpose of evaluating online courses and instruction should be improvement, not pain. And improvement efforts are most successful when they are valuable to all concerned. So analyze whether the misconceptions described in this article apply to your institution—and if some do, consider how to change them for the better.

Course and Instructor Evaluation: If It’s So Good, Why Does It Feel So Bad? excerpted from Online Classroom, January 2008.