Oh boy. I didn’t see this one coming, though I suppose I should have: Students Used for Cheap Labour. This is a link to our student newspaper, and possibly it loads better in your browser than it does in mine, but I had to view its source to get at the content, so I will explain. Steve Joordens, a psychology prof at UTSC, has been working on a piece of software that has students engaging in not just reading and responding to articles, but actually grading each other’s work:

The program PeerScholar is currently being used to mark two written assignments, which are worth 5 percent each. After writing their own answers in the program, students are asked to log in later during the week to read over other students’ answers. Students are then asked to grade each answer based on criterion available on the website. All student work is graded by five students, to provide fairness in the marking, Joordans [sic]claims.

I’ve met Steve. I went over to UTSC a few months ago to talk with him about what he’s doing and get a demo. He’s a very nice guy, very smart guy, and while he’s taken a very different approach to instructional technology than I have, his work is very interesting. I found myself very challenged by what he’s doing because it’s so radically different and yet so similar to the work I’m doing myself. The pool of data he’s gathered means that he can do some serious statistical analysis on how students grade, the numbers of students who will try to game the system, how to account for gaming the system, etc. It hit my like a brick wall; stats. Instructional technology as a thing that gathers stats, from which we can extrapolate and learn something about the user group. It’s just not in my repetoire of goals, what can I say, that’s what a background in english, history and theological studies gets you. Seeing a demo of PeerScholar showed me my biases very, very clearly. It was like looking into a mirror for the first time. Revealing and a little unsettling.

My focus has always been more touchy-feely, more humanities than social sciences, in that I’m more interested in using “web 2.0” to create a culture of feedback inside a class, to use comment features as a way to train students to work up a response to everything they read, to make reading scholarly work simply another form of dialogue rather than monologue. As a way to help build a sense of community, because community always needs to be built and strengthened. I generally steer clear of grading per se; assessment is a grey area for me in a lot of ways, and while I have ideas about it, I still feel that the instructor is the best judge when it comes to assessing student work. When it comes to interactive work, it seems to me that grading less rather than more (grading the whole experience, the whole process, rather than a single instance) is the way to go. So it wouldn’t have occurred to me to include students grading each other as a feature. Reacting to each other? Yes. Leaving feedback, starting a discussion, quoting each other, definitely. But grading seems so…formal. Final. Mercenary, somehow. But Professor Joordens is a working instructor, with a huge class to teach, so I can easily see how he would stop to consider how technology could help automate the process. If they don’t automate it, students in those classes will only be able to express themselves through scantron sheets. I appreciate what he’s trying to do. I can absolutely understand and respect the desire to get those students getting more engaged and doing more writing about what they’re reading. I can’t think of a more passive and limiting educational experience than nothing but multiple choice exams for assessment. So I see where he’s coming from.

I didn’t see this coming, though:

However, according CUPE 3902, since marking and grading of student work is a paid position at U of T, the students are subsequently covered by the Collective Agreement for Teaching Assistants, which also makes them members of the union. As a result of this, CUPE 3902 is arguing that students are being made to work for free, which CUPE 3902 Chair Anil Varughese claims is to “compensate for the failure to hire enough trained and qualified teaching assistants to evaluate them.”

Ack! Slippery slope, isn’t it. Reading an article and responding to it is coursework, but reading another student’s response and assigning it a grade is paid labour. I absolutely see CUPE’s point, though, and so does Professor Joordens:

On the UTSC’s PSYA01 website, Joordans [sic] goes on to say, “I will be completely honest. The original reason for seriously considering a peer-to-peer evaluation process was financial. We cannot afford to pay a large team of TAs to mark written answers for large classes. Moreover, it would take them so long to do the marking that it also just wouldn’t be practical. Peer-to-peer evaluation, when combined with great internet programming, is fast and cheap.”

Oops.

The Star has weighed in on this issue as well: Peer Marking Gets a Negative Grade:

Jemy Joseph, 20, “absolutely loved the idea” when she found out her course at the University of Toronto Scarborough also featured short, written assignments that would be returned with assessments of ability to write and think critically.
Her problem was that the marking — worth 10 per cent of her final grade — was done by her 1,500 classmates, as part of peerScholar, an online evaluation program in limited use at the school.

“The idea behind it is great because you’re not just getting graded but you’re also getting some sort of feedback,” said Joseph, who took the course in 2004. “But I’m not comfortable with getting marks from random students who have no experience in grading and may not put a lot of work into it.”

If I recall correctly, the statistics indicate that students are getting roughly the same grade from each other than they would get from a graduate TA. Though possibly that’s an aggregate statistic, I’m not sure. (Stats: really not my territory.) I don’t think this student is actually complaining about the grade she got, but more about the relative emptiness behind it. She feels cheated out of not getting that feedback from the person teaching the course, or someone who is part of the authority of the course staff. There’s a piece missing there that we need to define. I think it’s easy to see the value that faculty bring to courses, but often the shift into using more technology in the classroom makes people forget about that value, or think it can be replaced by something automated. But students clearly still value the experience and knowledge of instructors themselves. You can give them the grades they want, give them a relatively easy and quick way to get those grades, but they still want more of the faculty member’s time and thoughts. This is a good thing; students aren’t necessarily just here to pick up a grade.

More from the Star article:

“We’re not opposed to finding ways to move beyond multiple-choice testing,” Chantal Sundaram, a representative with Local 3902 of the Canadian Union of Public Employees, said yesterday. “But we think the best way to do that, to have more critical thinking and more long, written answers in introductory courses, is by hiring more teaching assistants. …

“This practice raises issues around our collective agreement and our workplace, but we believe it’s also an issue around the quality of education for the undergraduate students.”

Again, the union has a point. If multiple choice is not desireable and we accept Steve Joordens’ mission, what are the options when faced with 1500 students per term who want to take PSY100?

The basic structure of the system Steve Joordens created is, I think, sound; students can still read and evaluate each other’s work, I think, it just can’t translate directly into a grade. It seems to me. I hadn’t considered how very carefully we need to tread when moving interactive internet applications into the classroom in a deeply unionized environment. I’ve always been on the side of hiring more TAs when technology is involved rather than fewer; the more feedback from official, experienced sources, the better.

This grievance is definitely one to grow on.