Labour kills Educational Innovation

Turns out the union didn’t like PeerScholar, a system for facilitating peer-reviewing in large classes.

From the National Post:

Congratulations to the Canadian Union of Public Employees (CUPE), which has achieved another milestone in its ongoing quest to propagate the belief that public-sector unions are inherent, dedicated enemies of growth and innovation. In Friday’s Post, Emily Senger and Kathryn Blaze Carlson described CUPE’s successful grievance against the developers of PeerScholar, a new online application created at the University of Toronto that allows students to file short papers which are then graded and annotated, in an environment of mutual anonymity, by fellow class members who have completed the same assignment.

CUPE, which represents teaching assistants at U of T, took the view that, in the words of spokesman Mikael Swayze, “If students are doing marking, then they’re in our bargaining unit and must be paid.” In

short, the union — in blocking the use of PeerScholar within an enormous first-year psychology class — was merely defending its own interests against a low-cost alternative to having TAs grade written material.

Perhaps no one could hold that against CUPE if the only harm done was to make higher education more expensive. As a society, we claim to care a great deal about keeping university accessible to the poor and affordable for the middle class; sometimes do-gooders will even say that our future competitiveness and prosperity depend on it. But these people can be ignored. If we did care we would never, under any circumstances, give a labour cartel a veto over a professor’s conduct of his class.

We also claim to care a great deal about the quality of education — another premise that CUPE has pretty well refuted. In the case of Professor Steve Joordens’ psych class, the alternative to having written assignments peer-marked was not to have written assignments marked by unionized TAs, but to have no written assignments at all. He will now have to return to the same traditional practice followed in most introductory psych classes: basing grades entirely on machine-readable multiple-choice exams. This has always been a feature of psych classes that lazy, sly undergraduates liked, because it favours guessing ability over grasp of the course content and skill at writing and argumentation.

The very possibility of bringing about change to the factory-like environment of undergraduate survey courses was a major breakthrough for Joordens, who developed Peer-Scholar with his grad student Dwayne Pare. Peer grading may sound like a loopy idea, but early research by Joordens and Pare showed that for a simple assignment involving a written reaction to a set text, peer grading is statistically indistinguishable from “expert” grading by trained graduate students — if you have enough peers.

The magic number for maximum validity appears to be about six, so the PeerScholar interface makes each student grade six colleagues’ papers after they hand in their own. The peer grading is formally made part of the assignment, and students get the benefit of comparing other work to their own. Students who felt their grade was unfair were permitted to appeal to a teaching assistant, on condition that they had to live with the new grade whether it was higher or lower.

PeerScholar was a research project worth pursuing for its own sake, but the Ontario Superior Court’s support for CUPE’s grievance means that the Joordens/Pare research will be very difficult to reproduce scientifically, under real-world conditions, inside Ontario. It also means that PeerScholar, as a made-in-Ontario software application, will be hard to sell to colleges and universities where teaching assistants are unionized. And the potential for Peer-Scholar to improve the quality of large first-year survey classes may never be realized. How often does a labour union, with one single action, harm science, education and business all at once? What an astonishing hat trick of ignorance and greed; doff your lids, readers.


7 thoughts on “Labour kills Educational Innovation

  1. I just wanted to thank you for sharing this article with your readership. I am very happy that, amidst the legal and political issues, we can also discuss deeper issues like better education. I look forward to seeing the discussion.

  2. Hi Steve,

    Where I currently work, we put a lot of emphasis on students learning from each other. My role as an instructor there is not to be the talking head, but the guide who walks beside the student – a party of explorers, rather than a class. If we’re all explorers, then learning from each other only seems natural. One section of every course is a group project that students design and facilitate for each other, so I was very intrigued by your work in PeerScholar. It seems like a brilliant way to add a new dimension to learning, especially for those of us in the online sector of education.

    I hadn’t come across it yet in archaeological circles, so the National Post editorial was the first I’d heard of it, I’m afraid. But it caused me to search it out, and I’m very excited by what I saw.

    Keep up the good work,

  3. The more research we do on peer-assessment, the more I’ve come to believe that there are two distinct ways in which we learn things … either we consult some expert to receive a “formal” education, or we hang out with peers and experience some cyclical experience of peer learning. Peer learning might be less “formal” but that doesn’t make it less powerful. Consider the following example. At a recent conference on technology and education the teachers kept mentioning that their students know technology better than they do. All older people feel this push of youth technology literacy. Where do the young people learn these tech skills? From experts? Formal education? No, they hang out with their peers and, together, they learn more and more.

  4. Indeed. It is very powerful – but something to be careful about peer learning is an ‘echo chamber’ effect. To continue with the example you’ve raised, in the work that I’ve done with students, I find they have a very superficial knowledge of tech and tech issues. They can use lots of it – but some they have no idea about, or rather, they use it uncritically (see posts on teaching at ‘Ancient History Ramblings’ by Scott Moore, or Bill Caraher’s The Archaeology of the Ancient Mediterranean, especially his ‘teaching thursday’ posts).

    What I like about PeerScholar is that since it is in the context of a course, you mitigate that echo chamber effect – you have the ‘informal’ embedded in the ‘formal’, so there is guidance there. That I think leverages the best of both worlds.

  5. Could student services or the alumni association host the service to take it off the hands of the professor and give it at least an arms length of separation for such a corrosive environment? That way such a great service would still be available to the students for self learning.


  6. Hi Michael,

    Really, this grievance is more about students marking, then it is about the peer-assessment process. That is, if I did everything the same … even asked students to give each other comments, but didn’t have them give a mark, then there would be no issue with the union. So yeah, in self-help kinds of contexts it can be used without problem. In some classes it has been provided for students who wish to share essays (before a due date) and give one another comments. A sort of revision tool available to those who want to. They then use the comments to form their final draft which is graded in a conventional manner.

    The trick is, if you’re going to ask a large classroom to use it as a compulsory part of a course, you need marks associated with it to get the students to “buy in”. That is, if you say … “this is just for you … write an essay, then you’ll get comments from your peers that will help you learn to write and think more clearly … oooh, but there are no marks associated with this”, many – maybe most – students just won’t take either the writing or marking seriously. So those who do end up getting noisy feedback from those who don’t, and it all balls up real fast like. We are currently working on other ways of achieving “buy in” without the marks counting … just some other ideas forced by the union’s position on this … but it remains a shame that the most simple implementation is now a little harder to employ.

    Oh, and with respect to Shawn’s point … I do think more formal learning has a place, and I agree that peerScholar – employed in a class context – does offer a really nice combination of the two. Thanks for that.

    If you haven’t seen it yet, the saga continues today in the National Post with two op-eds … one from me and one from the union.


    1. The response to the Editorial, from the Union :

      The National Post’s June 22 editorial ( “Proudly stopping progress”) got one thing right about the University of Toronto teaching assistants of CUPE 3902. We are proud. We are proud of the work we do and the contribution we make to maintaining quality education at the university. That’s precisely why we opposed the administration’s attempts to replace teaching assistants with software that had students grading other students’ work for free.

      The “innovation” in question — software created at the university called peerScholar — allows students to write short written pieces and have them “graded” by fellow students. The issue isn’t so much

      the merits of this peer-review grading approach. It’s the use of the technology to replace, rather than supplement, grading by qualified professionals. Students deserve this feedback to better equip them to move forward in their studies.

      Imagine if we took the principle a step further. Why not allow peer assessment in hospitals? What if five patients were surveyed about the appropriate medication to prescribe to another patient? What if “on average” they were as good as doctors? Would that be good health care? Or just good luck?

      Joking aside, this grading software is not helping us maintain and improve the quality of education. It may make grading fast and cheap, but that’s not what students and their families pay so dearly for when they enroll.

      Which brings us to the real issue. Money. Some 1,500 students pay about $600 each for the course (some are prohibited from attending in person, and must watch the course on the Web because the room only seats 600 students). The province kicks in about the same amount per student, bringing total revenue of about $1,800,000 to the university. Ten-thousand dollars worth of teaching-assistant hours are currently assigned to the course. Assuming about $20,000 goes to pay the professor’s salary, net revenues from the course amount to about $1,770,000 — more than enough to hire enough teaching assistants to do all the grading. Even hiring 16 more would only cost another $160,000, leaving plenty of funds for other university purposes.

      The administration’s defence of the use of this software is nothing more than another cash grab on the backs of students and their families, not just the workers in our bargaining unit.

      Perhaps we should be looking at why there are classes of 1,500 students in the first place. Or why, in some courses, there aren’t enough seats for the students who pay for the course.

      A trade union can’t negotiate these issues. What we can do is force institutions to confront the true costs of their decisions and to recognize there are other choices. In this case, the institution starved a critical course of important resources despite driving enrolment through the roof. The professor thought peer grading was the only alternative to mass multiple-choice assignments.

      But using software like this is not the only option available for administrations and professors. Instead, we should be giving students and their families what they deserve, and investing enough in these courses to ensure that professors have the resources they need, and students get the feedback they deserve from trained and qualified instructors. – Mikael Swayze is the senior staff representative, CUPE 3902. Alison Norman is the grievance officer, CUPE 3902.

      Mikael jests that we should peer review medicine… well, why not? The ‘wisdom of crowds’, if properly harnessed, can work wonders… and that’s what this really boils down to: who controls academic teaching, research, and discovery: the prof, the union, the student, the government?

      The debate continues with Steve’s position, from the same issue of the National Post:

      In an article in Friday’s Post (“Peer-grading system fails court scrutiny), Emily Senger and Kathryn Blaze Carlson described the legal aspects of the peerScholar debate; it was a turf war wherein the union argued, and the Superior Court agreed, that anyone who marks in the university context is part of their union. The Post’s editorial on Monday (“Proudly stopping progress”) followed up with a pointed attack against the union. I would like to add two more facets to this issue; first, a brief discussion of the education benefits of peer-assessment, and second, a description of the “everybody wins” solution that could have been. I hope the latter is not yet out of our grasp.

      This is not the place for a detailed discussion of peer-assessment’s education benefits. I encourage readers to go to for a complete presentation along with the supporting research. Instead, I want to express how common and powerful peer learning is with a simple example.

      Our kids know technology better than we do. How did they come to possess this complex and deep knowledge of iPhones, Facebook and computers in general? They did not take formal courses in which experts presented a programmatic curriculum, tested their knowledge along the way and eventually provided feedback on the tests. Rather, they learned by interacting with peers — teaching those with lesser knowledge and learning from those with more. That is, by comparing their knowledge with that of their peers, a cyclical learning process occurred that is both natural and self-guided.

      Most of us learn sports, language and almost all the things we know in this way. Formal education is actually the oddity in life, the core approach of schools and universities. Should such institutions stick only to formal education? Well, not if it is inferior, and when it comes to teaching large classes about thinking and writing, it is indeed inferior. Thus, if this debate were only about education, peerScholar would win hands-down.

      So what could have been and still could be? A simple vision really: a deal between the union and the university wherein the union agrees to allow peerScholar to be used as long as, whenever it is used, TA hours are slightly increased.

      The results of this simple agreement could be staggering. Students could be asked to think and write in every class irrespective of class size, and could experience an extremely rich context for learning these skills. Teaching assistants could spend less time marking hundreds of papers, and more time directly interacting with undergraduates in ways that both enhance the graduate and undergraduate experience, and teach graduate students how to teach, something that just doesn’t happen enough at the university level.

      Finally, taxpayer money would be used in an efficient and powerful way to ensure that our children, with whom the future of our country lies, are being given the best possible education.

      Thus, by “renting out” the little piece of turf the union has now claimed, it is possible to create a situation whereby everyone wins. If instead they choose to keep it to themselves, I will be forced to come up with silly ways to tip-toe around that piece of turf while still giving my students the education I believe I am charged — by the taxpayers — to provide.

      Rest assured, progress has not been stopped; it’s only a question of how straight a line it can travel, and how intelligently we can all work together to make it happen in the best manner.
      National Post

      Steve Joordens is professor of psychology, University of Toronto Scarborough. He and his business partner Dwayne Paré developed and run peerScholar.

Comments are closed.