If you’re here in Italy, terms run later than many other places, but finally, finally, your students are counting down the hours until they can put their English on the shelf and you out of their mind and get down to full-on summer fun. Actually living life! But wait, before you say goodbye, there’s one more thing you’ve got to get from them.
It’s time for student evaluations… of your teaching!
If there’s one thing you actually do have to prepare for in the academic year, it’s student evaluations of teaching (SET). So before you set foot into that classroom again, go check out Russ Mayne’s awesome list of sure-fire ways to ace your SETs this year. Go on, do it.
Now that you’re back, read on. I’ve distilled some insights from my reading of behavioral economist and Nobel Prize winner Daniel Kahneman’s 2011 bestseller Thinking Fast and Slow to produce one big fat evidence-based footnote to Russ’s article.
Go out with a (happy) bang
Before you hand out your SET forms, consider this: what’s the most memorable thing you did before asking them to fill out the SET? Was it a game? Was it a pizza party? Or was it the dreaded final exam? How you answer could make all the difference in the world for your students and, perhaps, your future career.
In his book, Daniel Kahneman describes the relationship between our experiencing self and our remembering self. Whatever the first self has to say about an experience while we’re in it, it’s the second one that gets the final say in how we’ll think about and judge that experience for the rest of our lives. It seems that we always confuse memory with experience.
And what happens at the end of the experience is what we remember most. To illustrate, Kahneman relates the story of an audience member at one of his lectures who reported listening to a wonderful production of a classical symphony on a CD only to find it interrupted by a terrible scratch at the end. Despite the fact that the first 30 or 40 minutes of the symphony had been a joyous experience ― “musical bliss” ― the memory of the experience was defined by the scratch heard at the end.
What you can learn from a colonoscopy
Kahneman tried to formulate rules to predict how we remember pain based on some patients’ experience and memory of a painful colonoscopy procedure.
The peak-end rule
This rule states that how we remember pain ― our official memory of it, so to speak ― is the average of peak state (i.e. the worst level) of pain plus the end state of pain, or (peak + end)/2.
Duration neglect
This rule states that duration of pain is entirely irrelevant to our after-the-fact perception of pain. Whether it’s six minutes or six hours doesn’t matter; only the peak-end rule conditions our memory.
Implications for SET
What that means, of course, is that you want their pre-SET memories to be good ones. Which means ending not with a test, but a pizza party.
If you’re trying to manipulate positively influence your students’ memories of the experience, you then put some time between that painful final exam and the actual questionnaire. Don’t let that exam be the scratch on the otherwise lovely classical symphony.
But don’t just fill the time in between with pizza parties: as Kahneman argues that “gradual relief may be preferable to abrupt relief” when it comes to the end. So you might consider having a series of increasingly less painful tasks before the big pizza-filled pre-SET sendoff.
And when planning for next year, with your end-of-year pizza party far in the distance, you might consider leveling the peak-pain experience. Rather than a high stakes mid-term and final opt instead for continuous assessment. Assessment based on a series of low stakes tests will ensure that the peak pain level is never very great. And what they’ll remember from your class is not a series of short sharp shocks but one fairly smooth (or only mildly uncomfortable) ride.
As Kahneman concludes, the remembering self is the one that “keeps score”. If you want to get a good one yourself, better hold the test and double down on the pizza margherita.
The best SETs money can buy (only 10 cents!)
When it comes to making judgments or predictions about anything from the frequency of terrorist attacks to estimates of our own life satisfaction, we often don’t do the difficult task of digging up all the relevant examples before reaching a conclusion. Rather we base our judgments on the nearest available information, a shortcut called the availability heuristic.
One of the dangers of the availability heuristic is that it operates by unconsciously reformulating the question in light of the most available evidence. By substituting one question for another, it can produce readily produce an answer that seems like the right answer to original question.
To illustrate, Kahneman reports on experiment from Norbert Schwartz in which participants were asked to complete a questionnaire on life happiness. Just before they took the questionnaire, however, half the group was sent out to make a photocopy where they found ― as luck would have it ― a dime laying by the copy machine. When they completed the questionnaire, these same students ranked their happiness markedly higher.
In a general sense, this is the availability heuristic at work, as when asked to consider our general happiness we reach for the readily available memory of the very minor stroke of good luck. Kahneman further defines this as a mood heuristic, or a short of shortcut that extrapolates our general well being from our feeling in the specific moment. In effect, participants were unconsciously answering the much easier-to-answer question: How happy are you right now?
And all it took was 10 cents.
Implications for SET
Make it your job to sow happiness before you hand out those evaluation forms. Fill your pockets with ten cents in your local currency, and have those students rushing to the photocopier.
You could also (for no cost at all) announce that you’d made a mistake marking their last exam, and that you’d added an extra two points to their score.
It’s not clear how much a feeling of just “getting lucky” and a sense of exclusiveness are essential to this increase in happiness (if everybody found a dime on their seat, and they all knew about it, would the effect be the same?), but to be safe you might announce exam-mark increases by way of private note.
The important thing to remember is that it shouldn’t cost you an arm and a leg (I mean, they’re not paying you that much, right?) to make sure your students are substituting any unrelated positive feeling for how they actually feel about your course when completing their SET.
Want less criticism? Ask for more
When it comes to our personal beliefs, the more evidence or examples we can marshal in support, the more confident we are. This is the availability heuristic at work again, as we make judgments based the information closest at hand. Which also means that the less evidence we can quickly access, the less confident we feel.
This is especially true when we expect the task to be easy: when we suddenly can’t think of examples, and we’re surprised by it (what Kahneman calls “unexplained unavailability”) our confidence in our own belief gets dealt a serious blow.
Implications for SETs
If you want to set yourself up for success, precede any general evaluation by varying the number of examples you ask for:
Think of three ways you’ve improved your English this year.
Do you feel that your English has improved this year?
Thinking of three way you’ve improved should be easy, and produce a more positive general feeling when answering the second question. Ask for 12 ways, however, and as the students run out steam producing enough of the requested answers, they’ll start wondering how, and if, any improvement was actually made.
But this is exactly what you want them doing when asking them for criticism. Compare a question like: Think of three ways this course could be improved to Think of 12 ways this course could be improved. The relative difficulty of retrieving enough ideas for the second question ― even if they produce six or seven ― should leave your students struggling to reach the target number.
This means that even the most troublesome students, waiting to tear you to pieces in the evaluation form, may find themselves a victim of “unexplained unavailability”: unable to produce enough negative examples, they’ll suddenly be left thinking that, relative to the expectations, there’s not much about the class that needs improving.
Kahneman cites a UCLA professor who tested this theory by varying the amount of suggested improvements students were required to produce. Students who had to produce more rated the class higher: even though they actually listed more improvements, they failed to reach the (higher) minimum number.
Game on
In summary, by exploiting cognitive bias you’ve got a much better chance of getting good SET results than actually hoping they’ll be able to fairly and accurately judge the quality of the course, their learning and your teaching (this is science, people!). To get the SETs you deserve, just make sure that
- their memory of peak pain (e.g. the test) is comfortable behind them
- good fortune smiles on them immediately preceding the SET
- it’s easy to produce enough praise but hard to produce enough criticism to meet questionnaire requirements.
Sure, some people claim that SETs are biased against female instructors, don’t correlate to effective learning, or even downright worthless. But I say they just aren’t trying hard enough!
So what are you waiting for? Get out there and put a Nobel Prize winner’s insights to work for you today.
Got any other smart ideas for how to game your SET? I’d love to hear about them in the comments.
DYNAMITE ELT 