By Doug Ward
Gauging the effectiveness of teaching solely on student evaluations has always been a one-dimensional “solution” to a complex issue. It is an approach built on convenience and routine rather than on a true evaluation of an instructor’s effectiveness.
And yet many universities routinely base promotion and tenure decisions on those evaluations, or, rather, a component of those evaluations in the form of a single number on a five-point scale. Those who rank above the mean for a department get a thumbs-up; those below the mean get a thumbs-down. It’s a system that bestows teaching with all the gravitas of a rounding error.
A new meta-analysis of research into student course evaluations confirms this weakness, underscoring the urgency for change. The authors of that study argue that student evaluations of teaching are not only a questionable tool but that there is no correlation between evaluations and student learning.
That’s right. None.
“Despite more than 75 years of sustained effort, there is presently no evidence supporting the widespread belief that students learn more from professors who receive higher SET ratings,” the authors of the study write, using SET for student evaluations of teaching.
The study, titled “Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related,” has been accepted for publication in Studies in Educational Evaluation. It was written by Bob Uttl, Carmela A. White, and Daniela Wong Gonzalez of Mount Royal University in Calgary, Alberta.
As part of their analysis, they challenge the validity of a seminal 1981 study that is often held up as evidence of the importance of teaching evaluations. That study and subsequent studies, they say, suffered from small sample sizes and “multiple methodological flaws that render their conclusions unwarranted.”
Course evaluations, they say, provide little more than a score for student perceptions, arguing that if student learning is important, we need other methods for evaluating teaching.
Their findings fall in line with a 2014 study by the statisticians Philip B. Stark and Richard Freishtat of the University of California, Berkeley. That study argues that course evaluations are fraught with statistical problems and “pernicious distortions that result from using SET scores as a proxy for teaching quality and effectiveness.” Among those distortions: low response rates, and failure to account for factors such as size and format of class, and academic discipline.
This is all damning evidence, especially because universities rely heavy on student evaluations in making decisions about instruction, and about instructors’ careers. It is especially problematic for the growing number of adjunct instructors, who are often rehired – or not – based solely on student evaluations; and for graduate teaching assistants, who are often shoved into classes with little pedagogical instruction and forced to make decisions about their teaching solely through the lens of end-of-semester evaluations.
All this points to the need for swift and substantial change in the way we evaluate teaching and learning. That does not mean we should abandon student evaluations of courses, though. Students deserve to be heard, and their observations can help instructors and administrators spot problem areas in courses.
The non-profit organization IDEA makes a strong case for using student evaluations of teaching, and has been one of its staunchest proponents. IDEA has created a proprietary system for course evaluations, one that it says accounts for the many biases that creep into most surveys, so its defense of course evaluations must be viewed with that in mind.
Nonetheless, it makes a strong case. In a paper for IDEA earlier this year, Stephen L. Benton and Kenneth R. Ryalls make a point-by-point rebuttal to criticisms of student evaluations of teaching, saying that “students are qualified to provide useful, reliable feedback on teacher effectiveness.” They acknowledge faculty frustration with the current system, saying that course evaluations are often poorly constructed, created in ways that ask students to make judgments they are not qualified to make, and “overemphasized in summative decisions about teaching effectiveness.”
“Those institutions who employ an instrument designed by a committee decades ago, or worse yet allow each department to develop its own tool, are at risk of making decisions based on questionable data,” they write.
So what can we do? I suggest two immediate steps:
Expand the evaluation system. This means de-emphasizing student evaluations in making decisions about teaching effectiveness. No department should rely solely on these evaluations for making decisions. Rather, all departments should rely on range of factors that provide a more nuanced measurement of faculty teaching. I’ve written previously about CTE’s development of a rubric for evaluating teaching, and that rubric can be a good first step in making the evaluation system fairer and more substantial. The goal with that rubric is to help departments identify a variety of means for judging teachers – including student evaluations – and to give them flexibility in the types of discipline-specific evidence they use. It is a framework for thinking about teaching, not a rigid measurement tool.
Revisit student evaluations of teaching. As I said, students’ opinions about courses and instructors deserve to be heard. If we are going to poll students about their courses, though, we should use a system that helps filter out biases and that provides valid, meaningful data. The IDEA model is just one way of doing that. Changing the current system will require an investment of time and money. It will also require the will to overcome years of entrenched thinking.
The problems in student evaluations of teaching are simply a visible component of a much larger problem. At the root of all this is a university system that fails to value effective and innovative teaching, and that rewards departments for increasing the number students rather than improving student learning. If the university system hopes to survive, it simply must give teaching the credit it deserves in the promotion and tenure process. Moving beyond reliance on course evaluations would be a solid first step.
Doug Ward is the associate director of the Center for Teaching Excellence and an associate professor of journalism. You can follow him on Twitter @kuediting.
Recent Comments