My REF story; or, Confessions of a REFFail and REFugee

This Thursday, 18 December 2014, at the end of a long term and just in time for Christmas, sees publication of the results of the Research Excellence Framework (formerly called the Research Assessment Exercise). If the name doesn’t immediately tell non-academics precisely what this thing is, as indeed it may not, then it’s a survey of the quantity and quality of research done by academics in UK universities. Except for the fact that it isn’t. But it pretends to be, and based on that pretense it does a vast amount to damage to many individual academics, to most universities, and to all of UK Higher Education in general. It diverts research and writing away from the risky and the long-term in favour of the safely and the rapidly publishable. Even then, problems with assessment of “outputs” means that many results are wrong, enough of them to call the whole vastly time-consuming and expensive exercise into question. Yet those results will nevertheless be used to justify career-ending decisions for some academics by senior managers who are exempt from the REF by virtue of being senior managers, and institution-breaking decisions by ministers who are either ignorant of or hostile to academia or both. Follow the following link to see Derek Sayer of Lancaster University explain in detail what a hilarious-if-it-weren’t-so-serious farrago the REF is: http://www.timeshighereducation.co.uk/features/one-scholars-crusade-against-the-ref/1/2017405.article

One of the points that Professor Sayer makes is that the REF has become such an all-consuming monster that universities now run on-going internal REFs throughout the five-or six-year intervals between censuses to determine who will be entered in those national censuses. These internal REFS are, as Professor Sayer says, “highly divergent,” but a common feature in the run up to 2014 is that universities use your own departmental colleagues, and a “critical friend” from outside, to peer review your putative entries (of which there must be four—books or articles, whichever, but the key thing is there must be four of them), and then grade them on a scale of 1 to 4 to determine your REFability. The problem is that this involves “frequently ad hoc and generally anything-but-transparent staff selection procedures [by] individual institutions.” As Professor Sayer explains, this means “peer” reviewers are often not, and indeed usually not, actually peers.  They may be colleagues whose fields are closest to your own, but usually not close enough that they would be asked to review your book or your article for an academic journal. The institutional pool is just too small. And the “critical friend” from outside may happen to be in the same field as one or two members of the relevant department, but cannot possibly be qualified to comment on the quality of the work of every member of any department or even subsection of a department (medieval, early modern, or modern). Plus, book and article reviewing for publishing houses and academic journals, besides being done by actual peers who know your field, is normally also done double-blind, to prevent any kind of abuse. Not for internal REFs, though, where you don’t know who the reviewer is—but they do know who you are.  In short, then, your REFability rests on the judgments of people who are mostly inexpert in your field and who may have something against you personally or more generally against your sex, gender, sexual orientation, skin colour, religion, class background or whatever else. And that REF judgment may affect your future career development, even to the point of determining whether you have a future career or don't.  Yet, as Professor Sayer also says, the process’s “victims are often reluctant to speak on the record and universities hide their selection practices behind firewalls of confidentiality.”

It was this point, and the Times Higher Education twitter hashtag #MyREFstory, that got me thinking I should tell my REF story.  Professor Sayer tells his, from the point of view of someone who was all set to be a REF success, but, to his enormous credit, requested not to be entered into an exercise he considers so flawed as to be a fraud--his university denied his request and then rejected his appeal against the denial, so he was entered after all.  My REF story, though, is different.  It is one of a REF reject, one of the REF Riffraff, if you will.  One of the victims who has been reluctant so far to speak on the record.  But Thursday’s coming, and there are many who are still too vulnerable to speak out.  I’m not vulnerable any more, for reasons explained below.  So this is for all the REF Riffraff who still are and who can't speak for themselves. 

My former university adopted exactly the same internal REF procedures described above, with all its inherent problems. I submitted my publications to it: an eight-volume edited collection of documents about the British-American Empire published in Pickering and Chatto’s Major Works series; an article published in a well-rated journal in my field; and a monograph published by Palgrave Macmillan. The problem with the document collection was a problem with the REF in general—that edited document collections don’t fit into the REF definition of research publications. In fact, the collection required a lot of research, has three historical essays in it, headnotes for every document, explicatory footnotes, and it’s a published thing; eight things in fact.  But whatever....  So I was on the back foot there already. The article got a 3, which made it REFable.  But the book was rated a 2 by the internal reviewers, not REFable for most universities, including my former employer. And here’s where problems with the internal reviewing system begin.  First, the article, which got a 3, was based on some of the same material as one chapter of the book, although the book and indeed the relevant chapter in the book took a different approach, with more research included and wider conclusions drawn than was the case with the article. Yet the book got a lower mark of 2.  But these anomalies are often the outcomes of subjective judgments (a subjectivity that is admitted in the REF rules, but only, as Professor Sayer shows, as a pretext for disallowing appeals against decisions).  So I want to leave that aside now and focus instead on the processes by which judgments, whatever they are, are arrived at. 

As is generally the case (I am not singling out my former institution), my peer reviewers weren’t peers. In a department of just over 30, with expertise ranging from classics and ancient to contemporary history, the early modern cohort to which I belonged numbered seven. Include the nineteenth century, into which my book went, and we’re talking 10 people max.  None of them are specialists in American history, none of them would be asked to review my work for book publishers or journals. I thus refused to participate in REF reviewing. If I'm not qualified to review my colleagues' work for a journal, how can I be qualified to review it for the REF?  Two of my "colleagues" were not of the same mind, however, and gave my book that non-REFable score of 2, with all the career-threatening possibilities that carries with it. Career-threatening for me, that is, not for them: they will gain a marginal career advantage for gamely collaborating with the REF. And, as above, as is common, they did this anonymously. At best, this creates a work environment that is utterly inimical to collegiality—you don’t know who among your colleagues did this to you, so you can’t trust anyone.  Except of course that you pretty much do know who did it.  You can’t be absolutely sure, but in a field of 10 at most--where you can eliminate some on the grounds that they had other roles that precluded peer reviewing and others who you know are too honorable to ignorantly do a “number two” on anyone else’s career prospects--then you’re not left with many candidates to choose from.  The department and university cannot guarantee the anonymity it promises to even its closest collaborators, and anyone who thinks it can is being very foolish indeed.  No matter how much paper-trail burning they are urged to do to protect themselves from the Freedom of Information Act (as detailed in the Sayer article), the departmental world is simply too small to hide in. These potentially corrupt and corrupting factors are other reasons wny I refused to particpate in the REF reviewing process.  

But it’s all worse than this anyway. One-way anonymity, no matter how imperfect, still allows for all sorts of abuses. Or at least suspicions of abuses, which is still deeply corrosive.  I’m not saying I was judged on anything other than the real or perceived quality of my work. I accept that my “peers” genuinely believe my book is shit. But if they wanted a “professional” disguise with which to stiff a “colleague” for some other reason, they had the perfect opportunity to do so. And in turn they handed the perfect opportunity for the non-anonymous then Pro-Vice Chancellor for Research to stiff someone he has more than once referred to as a “Midlands peasant,” had he wanted to.  Of course I am not saying he did anything other than simply follow the guidance of my peers in the internal REF, or that his comments about my background were anything other than mere inconsequential banter. My rejection may have been for another entirely professional reason too; my refusal to participate in REF reviewing, for example.   

To my particular ex-university’s credit, senior managers promised that REF entry (and presumably non-entry) would not affect people’s future careers. Given the same senior managers’ form for changing their policies, however, and given that they know that we know they move goalposts all the time and do it with impunity, these promises have no more practical (or moral) value than barefaced lies.  And indeed already my former colleagues are being asked to include their REF entries on Professional Development forms, and have been told that promotion boards can take these forms into account.

So, anyway, predicting these changes, I was not accepting that 2 for my book.  Luckily for me, the department’s “critical friend” had recommended a 3 (though, given their equal inexpertise, this is of no more worth than the 2s), and so had the two members of the department charged with overseeing the internal REF (both among the department’s early modernists and nineteenth-century historians, but still not Americanists). The Head of College confirmed the 3, and that’s the matter of record, but I was out of the REF anyway--due to some crossover between the article and book (notwithstanding that this happens in the normal course of academic publishing, and notwithstanding there being differences between the article and book anyway), and due to the inability of this massive and vastly expensive Research Assessment Framework to categorise an eight-volume document collection as a research publication.

As a result of being a REF reject, I could now be facing a very uncertain future.  But because of REF, and for other reasons I’ve blogged about here before, I decided a while ago to seek academic work overseas.  Happily for me, in 2013, I got the French "habilitation" that qualified me to apply for professor posts there, and in 2014, a few weeks after learning ly REF rejection, I got my current job as Professor of American Civilisation at Jean Moulin University in Lyon, all of which rather helped with the old self-worth issues that might otherwise have come with that rejection. Of course, not everyone can escape as I have.  But I hope my REF story adds a little bit to the due discreditation of this appalling exercise, or Framework, or whatever stupid thing they call it in the future.  And that if you too are a REF reject then it doesn't really mean a thing about you. I hope others will add their stories.  And then, as much as I can’t believe this thing has gone on as it has as long as it has, equally, I cannot believe it cannot be stopped.