There are many reasons why student feedback is not particularly helpful, but there are things we can do to make it better. Here are a few suggestions on how to extract more value from course evaluations:

We don’t learn a lot from student feedback when we don’t ask good questions. At the top of my list of bad questions I’d put the ever-popular “What did you like most/least about the course/instructor” kind of questions. I wish I could make those questions illegal. Since when did the goal of education become providing learners with what they like? I know teachers can’t remove these questions from institutionally mandated forms. We can object, though, and we can ask students better questions on our own. If teachers want to make changes that improve teaching and learning, we need to ask about the impact of a policy, practice, behavior, technique, assignment, or instructional approach on students’ efforts to learn.

We don’t learn as much as could when we ask after it’s too late to make a difference. For me that means the end of the course when students are busy and stressed. They’ve got more important things on their minds. And at many institutions they are asked to evaluate every course every semester, which is not what the research recommends. Then there’s the reality that the feedback they provide isn’t going to benefit them—the course is over. The feedback that helps teachers make good choices about what and how to change doesn’t emerge from those overall, global assessments of how the course compares with all other courses on the planet. It’s found in responses to smaller segments of instruction, or course events like assignments or group activities, and it’s solicited right after the fact while students clearly remember what happened and the teacher has time to implement alterations.

We compromise the learning potential of student feedback when we don’t teach the principles of constructive feedback. The benefits of doing so go in both directions. Teachers get feedback that is more helpful than hurtful and students start developing a skill they can use in virtually every profession. Students deliver more constructive feedback when they understand what teachers do and don’t have the power to change, and what is and isn’t relevant to learning. Most of the time we don’t decide when the class meets or who enrolls in it. Moreover, our selection of ties or the types of earrings we wear don’t merit commentary in feedback that addresses learning experiences. Constructive feedback doesn’t preclude students from identifying things about the course and instruction that compromised their efforts to learn. It’s about how those comments are delivered. The golden rule of feedback is that teachers and students should give each other feedback in the form they’d like to have feedback given unto them.

We don’t learn much from student feedback when they don’t take the process seriously. And the reason students don’t take the process seriously is because they don’t think we do. They complain about some teachers, assignments, and courses year after year and nothing changes. Teachers can convince students that their feedback does matter by soliciting it and then talking about it during the course. Responding to student suggestions does not obligate the teacher to do whatever students recommend. If a course activity or assignment is essential to achieving certain learning outcomes, then removing it would be irresponsible. Teachers can help students understand by explaining the educational rationale behind the decision to continue the activity or assignment, and then by exploring what could be done that might help them do better. If a teacher makes a change that students recommend, they often feel vested in making the change successful.

It’s true that students don’t always provide good, helpful feedback, but that doesn’t mean they can’t or won’t. It’s up to faculty to solicit and respond to their feedback in ways that make it a learning experience for both parties.

However, with the right approach, these course evaluations can be a very constructive tool. One of the keys is to solicit feedback throughout the course, rather than waiting until the end when it’s too late to make improvements that will make a difference to the current roster of students.

As both a teacher and an instructional designer for more than 25 years, I have been blessed to see things from a variety of perspectives. I have also learned some lessons the hard way! Here are 10 strategies for administering multiple informal course surveys that have proven very successful for me:

1. Ask the questions clearly. Perhaps one of the reasons student comments aren’t clear is because we’re asking somewhat ambiguous questions. Give the exercise the same time and attention you do when writing learning goals. Ask yourself, “What am I trying to assess?” Have a colleague you respect review your questions the first time out.

2. Ask the right questions. If you want to know if one of your primary learning outcomes is being/was achieved, ask. Sure it’s the student’s perspective, but it can be a valuable piece of information and you will spot a trend quickly. These questions work nicely in a Likert-scale multiple choice format. Other questions can be the same as, or similar to, the department’s formal end-of-course evaluations.

3. Ask for written comments. I have always found that written comments can be the most insightful, if interpreted fairly. As Weimer (2012) pointed out, two students can take two different meanings from the same question, so keep it simple.

Four questions I use are:

1. What is one thing you like about this course (so far)?
2. What is one thing you do not like about this course (so far)?
3. What is one thing that could be improved in this course?
4. Do you have any additional comments you would like to share?

4. Use an LMS or some other way of automating the process of administering the survey. This makes it easy for you and your students. Use the reporting and statistical analysis tools to help you interpret the results.

5. Share the results with your students. This is a critical step. Identify the items that you can deal with and follow through. Tell your students how you will address their concerns. Explain to the students the items that you cannot fix and pass these along to the appropriate departments, if necessary. I also told my students that I would publish the results, including my comments on my faculty homepage, and share them with their program coordinator and dean or academic manager, which I did. There were multiple short and long-term benefits from doing this: accountability, student buy-in, and trust from my academic managers. Risky perhaps, but well worth it in the long run.

6. Survey more than once. Perhaps one-third or halfway through the semester, and once more. Assess any progress you made from the comments on the first survey. This can even be an additional survey question. Don’t wait until the department’s formal end-of-course evaluation – by then it is too late. Your students will also appreciate your genuine interest in improving the course for them. If your course gets off track a little at the beginning, you will have ample time to get things back on track.

7. Don’t survey unless you are prepared to deal with the feedback. If you ask your students for feedback, summarize and share the results, and implement the things that are constructive, your students will feel empowered. They will take a different look at the course and become part of the growth process, your included. This was one of my course activities that my students appreciated the most.

8. Tell your students what you are looking for. Let them know it’s ok to be negative, as long as the criticism is constructive and not mean-spirited. I don’t allow them to name fellow students or other teachers, or make comments unrelated to the course.

9. If you are not inclined to use a survey/questionnaire approach, use a simpler tool. Try a “start-stop-continue” approach, a “one-minute” paper, or a simple poll. These can be done almost weekly or at the end of each module/lesson.

10. Keep your perspective (and humor) on things. Once you get the results/comments, interpret them carefully, but don’t overanalyze or internalize any criticism. Teachers whom I have assisted with this activity initially get very worried because students can be blunt. My advice? Treat the comments like judges scoring figure skaters in an international competition. Throw out the high one and the low one, and what’s left over is usually a good overall indicator. It might not be all positive feedback the first time, but as long as it is constructive, you have something to build upon.

At the start of every one of my courses, as one of the ice-breaker activities and introductions, I would ask my students to visit my faculty homepage and look at the course feedback results for previous cycles of this course. On the page they can see not only the feedback from students, but my responses to the individual comments. By doing this, I think I demonstrated that I was serious about the quality of the course and my performance. I would also point out that I expected the same level of quality insights and constructive feedback from them two or three times during the semester.

Greg Cooper, M.A. Ed. is an instructional designer with the Teaching and Learning Centre (TLC) at the University of Calgary, Alberta. He previously worked at Cambrian College for 27 years as an elearning designer and as a professor in face-to-face and online courses, and at the University of Wisconsin-Madison as a learning technology consultant.

Reference:
Weimer, Maryellen (2012, November 28). End-of-course evaluations: making sense of student comments. Faculty Focus, retrieved from http://www.facultyfocus.com/articles/teaching-professor-blog/end-of-course-evaluations-making-sense-of-student-comments/

I think part of the reason for the vague feedback is that students don’t believe that the evaluations are taken all that seriously, not to mention they’re in the middle of the usual end-of-semester stress caused by having lots of big assignments due and final exams to face. It’s just not the best time to be asking for feedback and so students dash off a few comments which instructors are left to decipher.

In most end-of-course evaluations, students tend to comment about some of the same aspects of instruction. They frequently address issues of organization, whether students were treated fairly and the challenging aspects of the course. Carol Lauer wondered if faculty and students defined some of these common terms similarly and so she asked a faculty and student cohort to say what they meant when they saw or used the term on course evaluations.

Would you be surprised to learn that faculty and students define the terms differently, or that students themselves don’t agree on definitions? Probably not, I’m thinking. Even so, some of the specifics are interesting. Take “not organized,” for example. Almost a third of the faculty think students use that term when the teacher changes or doesn’t follow the syllabus. Just over 11% of students said that’s what the term meant to them. Seventeen percent of the students equated it with the instructor not being prepared, 15% said they used it when the teacher had no apparent plan for the day and almost 13% equated it with getting student work graded and returned slowly.

“Not fair” refers to problematic grading according to almost 50% of the faculty surveyed, but to just over 2% of the students. To students “not fair” gets written on an evaluation when the teacher plays favorites and doesn’t treat all students the same way. Students and faculty are closer in their understanding of what “challenging” means when it’s applied to a course. It means hard work and lots of it.

The point here isn’t terribly profound but it merits a reminder, especially at the end of courses when teachers are tired. Many of the terms used to describe teaching on rating forms and in student comments are abstractions. “Organized” is something teachers are and deciding whether a teacher is or isn’t depends on what the teacher does. Various behaviors, actions and inaction can be what any given individual sees as the presence or absence organization.

There is good news here. If you’re interested in improving something like organization, if you define it behaviorally, you can change what you, do which is a lot easier than changing what you are. Organization has never been one of my strong suits and I didn’t make much progress trying to “be” organized. But when I started putting a skeleton outline on the board, when I stopped five minutes before the end of period and used the outline to summarize, when I began class working with students to create a list of points to remember from last class, I was seen by students as being more organized.

But it isn’t all good news. A collection of dashed off student comments collected at the end of the semester doesn’t easily translate into an action-based improvement agenda. What the student comments mean is probably not what you think they mean. Communication about the impact of teaching policies and practices on efforts to learn needs to be ongoing so there’s an opportunity for clarifying feedback, adjustments and then more feedback. We can and should make efforts to change the way our institutions collect student assessments, but, until that glacier melts, we need to take matters into our own hands and solicit a different kind of feedback and at different times during the course.

Reference: Lauer, C. (2012). A comparison of faculty and student perspectives on course evaluation terminology. To Improve the Academy, 31, 195-211.

Yet, some areas of higher education have undergone very little change. In particular, the methods used to evaluate and ensure that quality teaching is occurring in traditional and online classes. Yes, there are those required course evaluations dutifully and anonymously completed by students at the end of every term, but these are typically viewed quantitatively and do little to transform instruction. The quality of teaching is rarely given serious attention. Basically, if a professor simply shows up to teach class (and sometimes even if she doesn’t) she receives a satisfactory rating without consideration of how well she did her job. In other words, the methods used to evaluate teaching in higher education are as outdated as cassette players and floppy disks.

Routinely taking for granted that satisfactory teaching at the college level is acceptable requires us to ignore a significant body of research on this subject. Scholars agree that good teaching requires time, effort, commitment, knowledge, presence, and ingenuity (Weimer, 2010). Good teaching does not just happen. So how do we invoke change in a system that has been in place since dissertations were written on manual typewriters?

Faculty can bring attention to the need for a better system of evaluating teaching by demonstrating excellence even when it is not required or even expected. Documenting and sharing evidence of quality teaching serves as a reminder to administrators (as well as colleagues) that there is more to teaching than showing up. Submitting a report with additional teaching-related information is a good first step. For example, faculty members can report on course load, number of students enrolled in each class, grade distribution, and the number of tests and other graded assignments. Further documentation may provide information on less quantifiable items such as the type and quality of feedback given to students, use of technology to teach content, and creative delivery methods.

Faculty members can take the voluntary evaluation process one step further by implementing supplemental evaluations of their teaching. The following section offers five suggestions for supplemental evaluations that provide meaningful feedback that can be used to improve teaching.

Midterm Course Evaluation – Student learning and experience with the class can be gauged halfway through the course through an anonymous evaluation. The instructor can review the feedback and immediately make adjustments if there is a general consensus about an issue in the class. The midterm course evaluation may also provide information about the type of lessons and assignments that students feel are beneficial. Surveys may be formal with specific questions answered on a Likert scale or it may ask open-ended questions which allow students to elaborate in more detail. This type of evaluation has obvious advantages over those required at the end of the term because students can benefit directly from the feedback provided.

Questionnaire Following Exam – Feedback from students about the format, content, and level of difficulty for specific questions can be useful in making changes to an exam or the lessons that were given in preparation for the exam. This same technique can also be used at the conclusion of a major assignment or project. Ask students: “1) Were instructions clear? If not, what additional information would have been helpful? 2) Was the exam or assignment beneficial to learning and retaining course content? 3) Please share other constructive comments.” Faculty can then use the information to make changes prior to the next exam or project.

Interview Students – Invite the highest achieving students to participate in an exit interview at the end of the semester. Why the highest achieving? These students will provide the most accurate and honest perspective of how to improve the course. Prepare questions in advance and be specific. For example: “If one assignment had to be removed, which should it be? Which assignment should definitely be kept in the course? Why?” Reflect on their responses and decide how information can be used to improve the course.

Peer Evaluations – Peer feedback typically takes the form of classroom observations but can also include reviewing course materials and assignments with a colleague who can give suggestions. Faculty should choose a colleague whom they trust and respect as a teacher to conduct the peer evaluation.

Self-Evaluation – When examining one’s own teaching behavior it is imperative that the faculty member realistically assess areas of strength and weakness. Videotaping a class to view and evaluate later is an excellent tool. The purpose of the self-evaluation is not just to show teaching ability in a positive light, but to demonstrate how feedback can be used to improve instruction.

There are many options for documenting teaching through records and supplemental evaluations. Hopefully, these suggestions have sparked interest in developing a voluntary evaluation system.

Now it’s your turn. What are some of the ways you evaluate your teaching effectiveness? Please share in the comment box.

Reference: Weimer, M. (2010). Inspired college teaching: A career-long resource for professional growth. Hoboken, NJ: Wiley, John & Sons, Incorporated.

Dr. Karyn W. Tunks is an associate professor in College of Education at the University of South Alabama.

1. Understand and accept today’s college students. First and foremost, students want us to know who they are. They want us to know their names and to know about their world. Today’s students are busy, technologically savvy, and multitaskers. Some are prepared for college work; others are not. Regardless of their backgrounds, all students have lofty ambitions and want to succeed. To help them, we can provide background knowledge in our subject areas. We also need to share the rationale behind what we do and ask students to do. I recommend making invisible expectations explicit. I regularly start class by saying, “We are learning this because …” When students understand why and how the material is relevant to them, they find more motivation to study and end up rating the course more highly.

2. Establish clear criteria for grading. All students want good grades, and they want to know exactly how to get those grades. College students today have experienced criteria sheets and rubrics since elementary school, and they want the same in college. They want to know where they stand on any given day in the semester.

After 20 years of college teaching, I have learned that telling students that their grades are based on percentages (20 percent homework, 25 percent quizzes, etc.) does not have meaning for them. They cannot figure their grades with a percentage system. A total point grading scale provides that clarity. Each assignment, quiz, lab paper, project, or exam has a certain number of points. These point values can be listed in the syllabus as well as the total points needed for the final grade. To help students keep track of their points, I give them a worksheet and explain that if they record their points, they will know exactly how many they have and how many they still need to earn.

I get rave reviews about my “no-mystery” approach to grading on course evaluations, and I believe that the good reviews are due to the clarity and ease of the total point system. I re-explain the grading system after the first assignments/quizzes/tests are returned. On the first day of classes, students are overloaded with information. It’s not until after an assignment has been graded that they are ready to understand how the grading system works.

3. Get formative feedback early. The end-of-course evaluation is a summative one. Although it aims to help us improve future courses, it does not enable us to respond to the needs of the students currently enrolled in the course. Formative feedback collected early in the course accomplishes that goal.

The first major paper or exam is a great time to collect formative feedback. I recommend attaching a page to the back of the exam, or asking students to respond to questions like these on the day papers are due:

1. How long did you study for this exam or work on preparing this paper?
2. How/where did you study/write?
3. Which class activities (lectures, discussions, reviews, online notes) helped you the most in learning this material? Why?
4. Which class activities helped you the least? Why?
5. Which topics remain the most difficult for you?
6. What has a professor done in the past that helped you learn?

If I have students answer these questions on a page attached to the exam, I let them know they can tear that page off and submit it anonymously when they turn in the exam.

You can also ask for feedback once that first exam or paper has been graded and returned. I like to ask questions then about improvement goals—what the student wants to do better and what else could be done in class to support their efforts to improve. If you don’t want to deal with open-ended questions, students can rate declarative statements such as “I would prefer more discussion of assigned readings.”

If you ask for their input, students will want to see that you listened to them. They may expect some changes. A short report back to them can be used to explain what you are willing and not willing to change, and why.

Students aren’t qualified to comment about all aspects of instruction, but they can rate how they feel about the classroom climate. They are good evaluators of what helps them learn and what confuses them. Getting their feedback early in the semester enables you to tailor the course to their learning needs.

Finally, it helps to talk about your evaluation results with a trusted colleague. Sharing student responses can help us see patterns in evaluations and sort out the “outliers” or just plain wacky comments. Knowing what our students need helps us teach in ways that promote their learning, and that means better results on the end-of-course evaluations.

Dr. Mary C. Clement is the director of the Center for Teaching Excellence at Berry College.

Excerpted from The Teaching Professor, 25.4 (2011): 1,3.

This semester I have one student who thanks me every time he leaves class. I’m not sure of his motivation, but the impact of his words makes me feel that he values the class and what we do there. The other 19 students leave without saying a word, which I find frustrating. I want to know more about what they experience in class and if they too find it valuable.

Kember, Leung, & Kwan, writing in Assessment & Evaluation in Higher Education (2002) indicate that formal faculty questionnaires completed by students at the end of the semester are not always effective in improving faculty performance, for many different reasons. Part of this problem is that the evaluations occur after the fact, after the class is completed and the professor and students have gone their separate ways. Hesketh & Laidlaw, writing in Medical Teacher (2002), state that feedback is most effective when it is well-timed according to daily work and is as close to the event that it evaluates as possible.

That’s why I like to use something called a “plus/delta” evaluation. The plus/delta is a brief, half-page form that I hand out at the beginning of class. It was first developed by Dr. Marj Davis and Dr. Helen Grady at Mercer University. I ask students during class to think of a “plus” – something they like about our class, and a “delta” – something they’d like to change. When class is over, I ask them to leave their completed forms (with no name) by the door, and I collect them and read the anonymous answers. After being sure to tell the students to give me substantive feedback, and not to mention that the room is too cold or that they are hungry after lunch – things I cannot control – I usually get good, solid comments that I can use to improve my teaching.

I conducted a plus/delta in two of my classes recently, and learned that my students liked the PowerPoint presentations I was giving, but felt I wasn’t using the textbook enough. I also learned that they wanted more hands-on assignments so that they could apply what they were learning. This was immediate, timely feedback that enabled me to redirect my lesson planning to accommodate their interests.

It’s not a perfect solution, of course. Not everything the students write is valuable. Sometimes they write a plus but leave the delta blank. And sometimes they comment that everything is fine the way it is. Nevertheless, the plus/delta is a quick and easy way to receive valuable feedback from students on a regular basis. It takes very little class time, keeps the responses anonymous, and points me to small changes I can make to improve the class. This in turn makes the class experience more valuable for everyone!

Susan Codone, PhD is an associate professor of technical communication in the School of Engineering at Mercer University.

“There’s a great deal of faculty dissatisfaction out there with the over-reliance on student ratings to evaluate teaching effectiveness,” said Linda B. Nilson, PhD, founding director of the Office of Teaching Effectiveness and Innovation at Clemson University. “In fact these ratings are very good for measuring customer satisfaction, no doubt about that, but really, under the best of circumstances, there’s only a mild correlation between student learning and an instructor’s rating.”

During the recent online seminar Measuring Learning: The Ultimate Teaching Evaluation, Nilson outlined a number of different tests and instruments faculty could use to serve as solid evidence of student learning as a result of the course. Faculty could then attach the results of the measurement along with their student ratings.

Some of the measurements discussed during the seminar include:
Perceived learning gains – Developed in 1997, the Student Assessment of their Learning Gains (SALG) instrument was originally used to assess learning in undergraduate chemistry, but was later updated to meet the needs of a broader range of courses. The SALG asks students to assess and report on their own learning, and on the degree to which specific aspects of the course have contributed to that learning.

First week writing and corrections – Give the students an ungraded writing assignment on key concepts, principles and techniques, processes, and cause/effect issues related to your course content. At the end of the course, have students write a letter to their “pre-course self” correcting errors, poor reasoning, misconceptions they may have written about in the first assignment.

First-week final exam – One of the more controversial methods of measuring student learning is to have students take the final exam during the first week in class, but don’t grade them on it. At the end of the semester give them that same exam again and compare the results. While letting students see their final exam makes some faculty nervous, Nilson says most students won’t remember any of the questions, and if they do what’s the harm? It will simply help them focus in on what you feel is important for them to know.

Targeted essay – In order to demonstrate the relevance of the course content, as an end-of-course assignment you could ask students to write an essay based on this question: Pretend you are on a job interview and the interviewer asked “What are the three most important things you learned in your X course?”

Nilson says she would like to see student satisfaction play a lesser role in the evaluation of faculty, noting that the real goal of higher education is not to please students but to facilitate student learning. Interestingly, a side benefit of simply asking students to think about and articulate what they learned in a course could be higher student ratings.

“Really what it does is make students more aware of their learning,” said Nilson. “Yes, it could make the evaluations more positive, but it’s also making them more accurate. When students have to reflect on their learning — where they were versus where they are now — they’re going to give a more valid assessment of your teaching and the effectiveness of the course and that’s not a bad thing… They can’t say ‘I didn’t learn anything in this course.’”

Measuring Learning: The Ultimate Teaching Evaluation

Student evaluations of faculty have been around for decades, but never before have they carried so much weight, or been so divisive. The use of student evaluations for faculty review can be problematic because student ratings aren’t the best measures of student learning. What’s more, when careers depend on student satisfaction, many may feel compelled to keep students happy, rather than ensure academic rigor.

Do you ever wonder what’s more important – educating your students or producing satisfied customers?

When student ratings are the sole measurement of teaching assessment, many faculty start to wonder.

Order a copy of Measuring Learning: The Ultimate Teaching Evaluation, and learn about Dr. Linda B. Nilson’s approach to measuring educator effectiveness based on what really counts: student learning. Dr. Nilson is the founding director of the Office of Teaching Effectiveness and Innovation at Clemson University, and author of Teaching at Its Best: A Research-Based Resource for College Instructors, now in its third edition (Jossey-Bass, 2010).

What You’ll Learn
Stressing the importance of measuring student learning and aggregating the results, Dr. Nilson shows you:

• How to adapt tests and instruments to serve as evidence of students’ learning
• How to give this evidence a number, so others can use it easily
• How to submit this evidence for faculty review, to balance student ratings.

Order the CD + Transcript package

Different Methods for Different Disciplines
During the online video seminar, you’ll learn measures of student learning you can tailor to your subject matter and course level, such as the following end-of-course measurements:

• Integrative essay or journal entry
• Targeted essay questions
• Survey of students’ perceived learning.

Tests for the beginning and end of your course include:

• A first, ungraded assignment, later corrected for a grade
• Giving your final the first week of class (ungraded), repeating it at the end of the session for a grade, and comparing results
• Surveying students’ confidence about knowledge of course material.

You’ll also receive sample documents and additional resources with in-depth recommendations.

When you order the recording on CD, you also receive a copy of the transcript, all for the same price as attending the seminar live.

An optional Campus Access License is available for an additional $200. It allows the purchasing institution to upload the CD of the seminar onto the institution’s password-protected internal web site for unlimited access by members of the campus community.

Order the CD + Transcript package

Who Will Benefit:
Measuring Learning: The Ultimate Teaching Evaluation, is an intermediate level seminar and will help all higher education faculty members interested in improving assessment of teaching effectiveness. In addition, deans, provosts, academic vice presidents and department chairs who evaluate their faculty’s teaching effectiveness will find this seminar invaluable.

All seminars include a discussion guide for facilitators
Participating in a Magna Online Seminar as a team can help leverage unique insights, foster collaboration, and build momentum for change. Each seminar now includes a Discussion Guide for Facilitators which provides step-by-step instructions for generating productive discussions and thoughtful reflection. You’ll also get guidelines for continuing the conversation after the event, implementing the strategies discussed, and creating a feedback loop for sharing best practices and challenges.

The collection of items on student rating forms can be thought of as an operational definition of good teaching and we all know that good teaching can be defined in many different ways. One interesting thing you can do with these instruments is to go through and cross off or change the items that don’t fit with your definition. I’m not suggesting you revise the instrument and then administer it, but making the changes for your own edification enables you to see where you agree and disagree with the definition proposed by the instrument.

Too many rating instruments remind me of that old expression that a camel looks like a horse designed by a committee. They are assembled via a political process where those with labs want questions about labs and those with studio courses want questions about those. Usually the compromise involves including both.

When an instrument is empirically constructed the process of deciding what goes on it involves something called validity. David Kember and Doris Leung offer a simple description of validity saying it is established “if an instrument actually provides a measure of what it purports to measure.” (p. 342) There are two kinds of validity; face validity and content validity. Face validity means the wording of the items refers what is being measured. That’s pretty straightforward and not really a problem on most course evaluation instruments. Content validity implies that an instrument includes all the dimensions, aspects or parts of the construct and that those parts are represented in a balanced way. That’s a problem with something like good teaching where the definitions are different and not universally agreed upon.

Empirically developed rating instruments assemble the collection of items based on the reports of various interested parties. In the case of teaching, that has meant students (current and former), teachers with a special emphasis on the views of good teachers, and administrators. In the mid 70s, Ken Feldman, whose meta-analyses on various aspects of ratings are legendary, reviewed the research on the ingredients and components of effective teaching (as reported by these groups) and from that large and not well-organized data base derived a set of 19 characteristics. His work and others like it justifies the inclusion of items that commonly appear on rating forms—things like teacher’s preparation and organization of the course; teacher’s enthusiasm for the subject; and teacher’s availability and helpfulness.

Times change and as Kember and Leung point out the characteristics that emerged out of Feldman’s analysis of the literature focus on teaching. Eleven of Feldman’s 19 characteristics begin with “teacher’s” and four more deal with the content and its presentation. “The model is of the teacher-centred content-oriented type [of instruction]. The dimensions fit well with didactic teaching, but it is hard to see the applicability of many of the dimensions to other more student-centred forms of teaching.” (p. 342) Kember and Leung’s new instrument is more attuned to the goals and objectives of learner-centered teaching. If that’s of interest, their article can be consulted.

My goal here is to make sure when you take a look at your results, you consider how the instrument is defining good teaching. How closely does that definition correspond with your own? And second, you recognize that definitions of teaching are not all equally acceptable. If you’re using an instrument to acquire feedback for yourself, then you can and should ask students for feedback in areas relevant to your definition. But if the instrument is being used to assess teaching across the institution, then the item selection process should be governed by what is known about aspects of instructional practice that can be linked to learning outcomes.

Reference: Kember, D. and Leung, D. Y. P. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education, 33 (4), 341-353.

The researchers surveyed this group, asking the faculty members to respond to several questions, including this most important one: “Your student ratings have increased for at least three consecutive semesters during the last three years in your [Course Name] class. What factors led to this change in your teaching performance?”

The slightly more than 200 respondents most frequently attributed the increase in ratings to changes made in one or several of these five areas:

1. more active/practical learning, including efforts to make the content’s relevance apparent to students;
2. better teacher/student interactions, exemplified by learning students’ names and having individual conferences with them;
3. making expectations for learning outcomes clearer while still maintaining high standards;
4. being better prepared for class; and
5. revising the evaluation policies and procedures used to assess student work.

The first three of these categories accounted for almost 50 percent of the faculty responses. A bit surprisingly, 5 percent of the respondents whose scores had improved didn’t list anything they’d done or they indicated that they were not aware of having implemented any changes.

This cohort of faculty included full-time tenured faculty (actually this was the largest group, 56 percent), full-time nontenured faculty (12 percent), and part-time appointees (35 percent). The researchers note that this indicates how faculty in all kinds of positions can improve. That so many in the already-tenured and part-time categories did so is especially noteworthy and encouraging.

In addition to the survey, 30 faculty from 10 of 12 colleges at the institution were interviewed “to gain a better understanding of the change process.” (p. 167) Several interesting findings emerged from the interviews. For many faculty members, the most difficult part of the process was being willing to admit that they needed to change. “Humbling” was an adjective used to describe the feeling. Often there was some sort of triggering event—frequently it involved end-of-course ratings results. After teaching a course seven times, one faculty member received his lowest-ever overall course rating. He was shocked but reported that he decided to find out why. Others talked about an overall lack of excitement in the course and their own motivation to change and do better.

In the interviews, almost 80 percent of the faculty indicated that the effort required to implement the changes was minimal. It seemed that for most it was more a matter of fine-tuning their teaching. The researchers conclude, “The results of this study should be encouraging to faculty members who feel they cannot improve.” (p. 171)

Reference: McGowan, W. R., and Graham, C. R. (2009). Factors contributing to improved teaching performance. Innovative Higher Education, 34, 161-171.

Reprinted from “Teachers Who Improved.” The Teaching Professor, 23.10 (2009): 2.

In fact, in the Kindred and Mohammed (reference below) analysis of evaluation for 626 professors, 41.5 percent had only one rating listed. The site does not prevent students from evaluating a given instructor more than once. It does not ensure that they have even taken the course they are evaluating. And there’s nothing that prevents an instructor from entering high scores and making nice comments.

Despite these problems, use of the site continues to grow. Brown, Baillie, and Fraser (reference below) report that in January 2009 the site boasted more than 6.8 million ratings for more than a million instructors from over 6,000 colleges and universities in the United States, Canada, England, Scotland, and Wales. The numbers show just how interested students are in finding out as much as they can about a course before taking it—that was also one of the findings from the focus group interviews conducted by Kindred and Mohammed. Eighty-three percent of the students surveyed by Brown, Baillie, and Fraser reported that they had visited the site. They go there because most institutions do not give students access to end-of-course rating data.

Thirty-six percent of the students in Brown, Baillie, and Fraser’s sample said they had posted on the site. Students in Kindred and Mohammed’s focus groups said that students posted because they wanted to share information about the course with other students. They also indicated that students are especially motivated to offer assessments when a teacher is really good or really bad.

Kindred and Mohammed did a content analysis of 788 comments—just about 75 percent of students who rated professors made comments. “Approximately 42% of the ratings coded in the present sample (437 ratings) contained statements pertaining to the competence of the instructor.” About 30 percent of those comments were negative, with the remaining 70 percent positive. The researchers conclude, “While issues such as personality and appearance did enter into the postings, these were secondary motivators compared to more salient issues such as competence, knowledge, clarity and helpfulness.”

What do students think about the ratings and comments that appear on the site? In Brown, Baillie, and Fraser’s survey, 71 percent of the students said they avoided taking an instructor based on the ratings that appeared on the site. Interestingly, 58 percent of that sample said they thought students were more honest in the evaluations posted on the site than on the evaluations collected by the institution at the end of the course. However, in the focus group interviews conducted by Kindred and Mohammed, students said they trusted the opinions of their friends more than what they read on the RateMyProfessors site. They also reported that they took evaluations on the site less seriously if a lot of the comments contradicted each other. They said they could tell when a comment was just angry venting and took those comments with a grain of salt.

The RateMyProfessors site does not appear as though it is going away any time soon and its astounding popularity attests to how hungry students are for information about courses. Are there ways instructors could share this information about their courses? Doing so helps to ensure that students get accurate information about the course and its instructor.

References: Kindred, J. and Mohammed, S. N. (2005). “He will crush you like an academic ninja!”: Exploring teaching ratings on ratemyprofessors.com. Journal of Computer-Mediated Communication, 10 (3), article 9. http://jcmc.indiana.edu/vol10/issue3/kindred.html.

Brown, M. J., Baillie, M., and Fraser, S. (2009). Rating ratemyprofessors.com: A comparison of online and official student evaluations of teaching. College Teaching, 57 (2), 89-92.

Excerpted from Despite Shortcomings Popularity of RateMyProfessors.com Grows, The Teaching Professor, vol. 23, no. 6, pg. 8.

Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning

In today’s competitive online learning landscape, students have more options and higher expectations than ever before. Ensuring quality is not just important, it’s critical … and it requires constant vigilance. Simply having an online program is no longer good enough, if it ever was. So what are you doing to ensure the quality of online courses and programs at your institution?

Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning

Download your copy of this report today! It's FREE to Faculty Focus members.

Sign In Create an Account

Creating surveys is easy, but creating good surveys take more thought and effort. This special report will show you how to build surveys that ask the right questions, the right way, to get the information you need.

In the report you will find 10 articles from Online Classroom, including a three-part and a five-part series that provides step-by-step guidance on how to use surveys and evaluations to improve online courses, programs, and instruction. You’ll learn when to use surveys, how to design effective survey questions, why it’s important to ensure anonymity, and the advantages and disadvantages of Web-based surveys.

Articles in Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning include:

• Online Teaching Fundamentals: What to Evaluate, parts 1-3
• Course and Instructor Evaluation: If It’s So Good, Why Does It Feel So Bad?
• Getting Evaluation Data through Surveys: What to Consider before Getting Started
• Using Surveys to Improve Courses, Programs, and Instruction, parts 1-5

Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning

Download your copy of this report today! It's FREE to Faculty Focus members.

Sign In Create an Account

In order to improve online programs, courses, and instruction, you have to first determine your goals, select metrics that will tell you what we want to know, analyze these metrics for clues about needed changes, and then make those changes. It may sound simple, but it isn’t.

If you’re dedicated to continuous improvement, this special report is loaded with practical advice that will help you create more effective surveys before, during, and after your course ends.

Shouldn’t you be a part of the Faculty Focus community?

Sign up today for the Faculty Focus e-newsletter to gain access to all of our free reports. Like our special reports, the e-newsletter is completely free and will give you the strategies, tips, and insight on the topics that impact your students, your school, and your work, including:

• Instructional Design
• Faculty Development
• Teaching Strategies
• Distance Learning
• Classroom Management
• Educational Assessment
• Faculty Evaluation
• Learning Styles
• Curriculum Development
• Trends in Higher Education
• And much, much more.

Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning

Download your copy of this report today! It's FREE to Faculty Focus members.

Sign In Create an Account

Misconception: Outcomes are the only things worth measuring.

Reality: Outcomes, such as numbers of new courses developed, enrollments, retention, and satisfaction levels, are important and they should be measured. But it’s also important to evaluate critical processes, such as support for faculty course development, relationships between course designers and developers and faculty, and student ability to get help as needed.

The processes that are involved in producing and delivering online courses and instruction should be evaluated alongside the outcomes of these processes so it’s possible to see what changes would allow for better outcomes.

I recently worked with an institution that had an adversarial relationship between faculty and the online course development team, and both spent time pointing fingers to explain why the results weren’t optimal. What they didn’t see was that this adversarial relationship created bottlenecks and course development problems. Obvious solution? Build a better process and fix the damage caused by the old one.

If the process of producing and delivering online courses and instruction is problematic, courses and instruction are also likely to be problematic—and these problems are unlikely to improve without improving the process. So, while evaluating outcomes, it’s also important to evaluate the processes that impact those outcomes. You will find inefficiencies, poor relationships, rework, contention, and more that are making better outcomes difficult or impossible.

Misconception: Evaluation is a CYA activity to be endured.

Reality: The purpose of evaluation should be to continuously improve, not to check off boxes on a checklist and then breathe a sigh of relief until evaluation needs to be done again.

Most higher education institutions conduct end-of-course evaluations, but this kind of evaluation often doesn’t result in significant improvements to courses and cannot impact courses in progress.

Because end-of-course evaluations may be required but often aren’t sufficient, some online instructors have begun to implement weekly or bimonthly anonymous evaluations by students so they can make changes to the course and the process in the here and now.

Bottom line? The purpose of evaluating online courses and instruction should be improvement, not pain. And improvement efforts are most successful when they are valuable to all concerned. So analyze whether the misconceptions described in this article apply to your institution—and if some do, consider how to change them for the better.

Course and Instructor Evaluation: If It’s So Good, Why Does It Feel So Bad? excerpted from Online Classroom, January 2008.

1. Ask students to write a letter to a future student about the course. This technique helps students to reflect on the course with students’ needs in mind rather than the instructor’s.

Here are some prompts that Hall uses to get students to produce this letter:

• What did you find interesting about this course?
• What did you not like in this course?
• Provide some insight about problems you had in the course.
• What does it take to succeed in this course?

2. Seek feedback from a few select students. Hall used to ask the entire class for critical feedback, but he has since recruited just one or two people two or three times per course to reflect on the negatives.

“I ask them to skip the praise and ask, ‘What’s one negative thing you can find about the course?’ They hate that, but sometimes they’re able to spot something,” Hall says.

Excerpted from Tips From the Pros: Two Creative Means of Eliciting Student Feedback. Online Classroom, January 2007.

I am just about to retire from Penn State and leave my faculty position teaching undergraduates. I’ll still be working; there’s this newsletter to edit and a world of faculty who still need advice, ideas, and encouragement to do their very best in the classroom. But you don’t end 33 years of college teaching without thinking about those things that will and won’t be missed on campus. Here’s my list.

Things I’ll miss:

• The nervous anticipation of going to class, rehearsing my lines as I drive to campus, thinking about all that’s possible, believing that I just might be able to make some of it happen.
• Those days in class when students get it. Sometimes that new understanding shines from their faces, sometimes they make a comment that attests to how well they’ve got it, and sometimes they report the details in a paper. Sometimes they give you credit. Even if they don’t, it’s still an event worthy of witness.
• Those days in class when I get it. When I see how to connect content to students; efforts to learn to appropriate processes; and students to the insights, ideas, and motivation of other students.
• Seeing seniors at graduation and remembering how they looked that first day of their first semester in college.
• Watching students who started out failing or doing poorly learning to succeed.
• Colleagues whose passion for teaching spreads enough hot coals to light new fires and rekindle others when their embers burn low or die out.
• Colleagues who use their fine minds, keen intellects, and inquisitive sensibilities to tackle teaching and learning with intellectual robustness.

Things I won’t miss

• Those bright, capable students who don’t care and won’t make an effort. Those students full of potential who happily do work just barely above the line that marks acceptable.
• Colleagues who have given up on teaching and are doing time in the classroom—the ones who’ve locked themselves out of meaningful, trusting relationships by using policies and practices that render all encounters with students adversarial.
• Colleagues who blame students for what they aren’t accomplishing as teachers.
• End-of-course student evaluations that ask irrelevant questions and give administrators data from which to draw dubious conclusions.
• Peer reviews where the Lake Wobegon effect devalues any teaching that is truly above average.
• Grading papers so full of grammatical errors that it’s difficult to see beyond them to the ideas behind them.
• Students so full of excuses there’s no room left for learning.
• Students with whom conversations never get past the points—those taken off, missed, totaled, awarded for extra credit, given, earned, offered as bonus, secured surreptitiously, or bought on the black market.
• Those days in class when I can’t make it happen, when my best efforts don’t make a difference. Those days when passivity, like fog, settles over the classroom, when students yawn and nod off and no amount of enthusiasm cuts through the chill of complacency—those days when only the cold signifies that this place isn’t teaching hell.

If you’re retiring this year, please share some of the things you will and won’t miss by clicking on the Add Comment button.

While it is certainly true that faculty know much more than students about the subject matter taught in the classroom, academic leaders have a vital interest in learning how well that material is being communicated to students. Who better other than students to provide that perspective? Faculty peers or advisory committee members who advise the chair about the quality of instruction being offered can provide a helpful perspective, of course, but “spot” visits from outsiders are notoriously hit-or-miss and lack the longitudinal consistency of the student who is on the receiving end of instruction day in and out.

Student Evaluations Reveal Patterns
There is no perfect evaluation instrument to assess instruction, what we do have are evaluation instruments that provide potentially useful feedback. Chairs and deans who read student evaluations annually know that, in the distribution of scores on quantitative sections of evaluations, certain broad patterns often emerge. Accumulated over several semesters, the scores often confirm a pattern of either very good or very bad teaching while revealing improvement (or not) for the “mixed-bag” instructors comprising the majority of the data set. You may not want to go to court with such evidence, but at least you have something other than possibly self-serving faculty self-assessments.

And while it is certainly true that student evaluations could threaten academic freedom or be misused by administrators, my own experience as a chair and dean for 28 years suggests that this is an outside possibility at best. In arguments about evaluations, statistics tend to be used by instructors more than by administrators.

Instructors with low student ratings have been known to aver that the only explanation is something like “Of course I have low scores. Unlike my colleagues, I have high standards, am a tough grader, and refuse to engage in a popularity contest.” Yet narrative comments from students almost never complain about low grades or high standards; rather, the complaints are usually in the areas of unclear grading and classroom policies, inability to answer questions effectively, lateness in returning papers and exams, and uninspiring classroom presentations.

What Do You Think About Student Evaluations?
Are student evaluations a good thing? Reservations aside, maybe so. Perhaps this time Martha got it right. What do you think? Please enter your comments below.

Excerpted from September 2006, Academic Leader.

These student evaluations are so much a part of our system and have become so routine for our students and faculty that I have seldom questioned their value or necessity. Indeed, one study of 600 liberal arts colleges found that the number of schools asking students to evaluate their instructors had escalated from 29 percent to 86 percent over the course of a decade. But are they really (as Martha Stewart might say) “a good thing?”

I am not as sure as I once was. In recent years, academics and researchers have identified a host of concerns, objections, and questions about student evaluations of faculty, including these:

1. Are students qualified to judge the quality of a professor’s pedagogy and academic expertise? And it is not just a matter of competence. There is a kind of consumer mentality at work when we ask students to provide their anonymous “customer satisfaction” ratings for courses. It may not be in the best interest of faculty or students to assume this right and such a level of competence. These evaluations may have an effect on the faulty-student relationship that is disturbingly negative.

2. Are students evaluating teaching effectiveness—or something else? That might depend on the evaluation instrument, the insight of the student, the personality of the faculty member, the motivation and fairness of the young evaluator, and myriad other variables that make the fundamental validity of the process doubtful.

3. Are faculty rights to academic freedom compromised by the pressures to secure favorable student evaluations? Some faculty critics point out that the power students exercise through the evaluation of courses tends to make teaching a popularity contest resulting in easier assignments, grade inflation, and entertainment values that supersede rigorous academic standards and inhibit faculty freedom to advance controversial or unpopular ideas.

4. Are administrators using student evaluations to intrude on the privacy of the classroom and to manipulate faculty behavior? Heaven forbid! (I say); but the director of the office of educational assessment at one large state university reviewed the research and opined that “if student ratings are to qualify as evidence in support of faculty employment decisions, questions concerning their reliability and validity must be addressed.”

So you see, administrator colleagues, why I have new-found reservations about student evaluations of faculty. Are these a “good thing”? Maybe not. That’s my view—what’s yours? Please share your comments below.

Thomas R. McDaniel, professor of education, is a senior vice president at Converse College.

Excerpted from August 2006, Academic Leader.