This Thursday,
18 December 2014, at the end of a long term and just in time for Christmas, sees publication of
the results of the Research Excellence Framework (formerly called the Research Assessment Exercise). If the name doesn’t immediately
tell non-academics precisely what this thing is, as indeed it may not, then it’s
a survey of the quantity and quality of research done by academics in UK
universities. Except for the fact that it isn’t. But it pretends to be, and
based on that pretense it does a vast amount to damage to many individual
academics, to most universities, and to all of UK Higher Education in general. It
diverts research and writing away from the risky and the long-term in favour of
the safely and the rapidly publishable. Even then, problems with assessment of “outputs”
means that many results are wrong, enough of them to call the whole vastly
time-consuming and expensive exercise into question. Yet those results
will nevertheless be used to justify career-ending decisions for some academics by senior
managers who are exempt from the REF by virtue of being senior managers, and
institution-breaking decisions by ministers who are either ignorant of or hostile
to academia or both. Follow the following link to see Derek Sayer of Lancaster
University explain in detail what a hilarious-if-it-weren’t-so-serious farrago the
REF is: http://www.timeshighereducation.co.uk/features/one-scholars-crusade-against-the-ref/1/2017405.article
One of the points that Professor Sayer makes is that the REF has become such an all-consuming monster that universities now run on-going internal REFs throughout the five-or six-year intervals between censuses to determine who will be entered in those national censuses. These internal REFS are, as Professor Sayer says, “highly divergent,” but a common feature in the run up to 2014 is that universities use your own departmental colleagues, and a “critical friend” from outside, to peer review your putative entries (of which there must be four—books or articles, whichever, but the key thing is there must be four of them), and then grade them on a scale of 1 to 4 to determine your REFability. The problem is that this involves “frequently ad hoc and generally anything-but-transparent staff selection procedures [by] individual institutions.” As Professor Sayer explains, this means “peer” reviewers are often not, and indeed usually not, actually peers. They may be colleagues whose fields are closest to your own, but usually not close enough that they would be asked to review your book or your article for an academic journal. The institutional pool is just too small. And the “critical friend” from outside may happen to be in the same field as one or two members of the relevant department, but cannot possibly be qualified to comment on the quality of the work of every member of any department or even subsection of a department (medieval, early modern, or modern). Plus, book and article reviewing for publishing houses and academic journals, besides being done by actual peers who know your field, is normally also done double-blind, to prevent any kind of abuse. Not for internal REFs, though, where you don’t know who the reviewer is—but they do know who you are. In short, then, your REFability rests on the judgments of people who are mostly inexpert in your field and who may have something against you personally or more generally against your sex, gender, sexual orientation, skin colour, religion, class background or whatever else. And that REF judgment may affect your future career development, even to the point of determining whether you have a future career or don't. Yet, as Professor Sayer also says, the process’s “victims are often reluctant to speak on the record and universities hide their selection practices behind firewalls of confidentiality.”
It was this point,
and the Times Higher Education twitter hashtag #MyREFstory, that got
me thinking I should tell my REF story. Professor
Sayer tells his, from the point of view of someone who was all set to be a REF success,
but, to his enormous credit, requested not to be entered into an exercise he
considers so flawed as to be a fraud--his university denied his request and then rejected his appeal against the denial, so he was entered after all.
My REF story, though, is different. It is one of a REF reject, one of the REF Riffraff, if you
will. One of the victims who has been
reluctant so far to speak on the record.
But Thursday’s coming, and there are many who are still too vulnerable to
speak out. I’m not vulnerable any
more, for reasons explained below. So
this is for all the REF Riffraff who still are and who can't speak for themselves.
My former university adopted
exactly the same internal REF procedures described above, with all its inherent
problems. I submitted my publications to it: an eight-volume edited collection
of documents about the British-American Empire published in Pickering and
Chatto’s Major Works series; an article
published in a well-rated journal in my field; and a monograph published by
Palgrave Macmillan. The problem with the document collection was a problem with
the REF in general—that edited document collections don’t fit into the REF definition of
research publications. In fact, the collection required a lot of
research, has three historical essays in it, headnotes for every document, explicatory
footnotes, and it’s a published thing; eight things in fact. But whatever.... So I was on the back foot there
already. The article got a 3, which made it REFable. But the book was
rated a 2 by the internal reviewers, not REFable for most universities, including my former employer. And
here’s where problems with the internal reviewing system begin. First, the article, which got a 3, was based on some of the same material as one chapter of the book, although the book and
indeed the relevant chapter in the book took a different approach, with more
research included and wider conclusions drawn than was the case with the
article. Yet the book got a lower mark of 2. But these
anomalies are often the outcomes of subjective judgments (a subjectivity that
is admitted in the REF rules, but only, as Professor Sayer shows, as a pretext
for disallowing appeals against decisions). So I want to leave that aside now and focus instead
on the processes by which judgments, whatever they are, are arrived at.
As is generally the
case (I am not singling out my former institution), my peer reviewers weren’t
peers. In a department of just over 30, with expertise ranging from classics
and ancient to contemporary history, the early modern cohort to which I
belonged numbered seven. Include the nineteenth century, into which my book went, and
we’re talking 10 people max. None of
them are specialists in American history, none of them would be asked to review
my work for book publishers or journals. I thus refused to participate in REF reviewing. If I'm not qualified to review my colleagues' work for a journal, how can I be qualified to review it for the REF? Two of my "colleagues" were not of the same mind, however, and gave my book that non-REFable
score of 2, with all the career-threatening possibilities that carries with it.
Career-threatening for me, that is, not for them: they will gain a marginal career advantage for gamely
collaborating with the REF. And, as above, as is common, they did this anonymously. At best, this creates a work environment that is utterly inimical
to collegiality—you don’t know who among your colleagues did this to you, so
you can’t trust anyone. Except of course
that you pretty much do know who did it. You can’t be absolutely sure, but in a field
of 10 at most--where you can eliminate some on the grounds that they had
other roles that precluded peer reviewing and others who you know are too honorable to ignorantly do a “number two” on anyone else’s career prospects--then you’re not left with many candidates to choose from. The department and university cannot guarantee
the anonymity it promises to even its closest collaborators, and anyone who thinks
it can is being very foolish indeed. No
matter how much paper-trail burning they are urged to do to protect themselves from the
Freedom of Information Act (as detailed in the Sayer article), the departmental
world is simply too small to hide in. These potentially corrupt and corrupting factors are other reasons wny I refused to particpate in the REF reviewing process.
But it’s all worse
than this anyway. One-way anonymity, no matter how imperfect, still allows for
all sorts of abuses. Or at least suspicions of abuses, which is still deeply
corrosive. I’m not saying I was judged
on anything other than the real or perceived quality of my work. I accept that
my “peers” genuinely believe my book is shit. But if they
wanted a “professional” disguise with which to stiff a “colleague” for some
other reason, they had the perfect opportunity to do so. And in turn they handed the perfect
opportunity for the non-anonymous then Pro-Vice Chancellor for Research to
stiff someone he has more than once referred to as a “Midlands peasant,” had he
wanted to. Of course I am not saying he
did anything other than simply follow the guidance of my peers in the internal REF,
or that his comments about my background were anything other than mere inconsequential banter. My rejection may have been for another entirely professional reason too; my refusal to participate in REF reviewing, for example.
To my particular ex-university’s
credit, senior managers promised that REF entry (and presumably non-entry) would
not affect people’s future careers. Given the same senior managers’ form for
changing their policies, however, and given that they know that we know they move goalposts all the time and do it with impunity, these promises have no more practical
(or moral) value than barefaced lies. And indeed already my former colleagues are being
asked to include their REF entries on Professional Development forms, and have
been told that promotion boards can take these forms into account.
So, anyway, predicting
these changes, I was not accepting that 2 for my book. Luckily for me, the department’s “critical
friend” had recommended a 3 (though, given their equal inexpertise, this is of
no more worth than the 2s), and so had the two members of the department charged
with overseeing the internal REF (both among the department’s early modernists
and nineteenth-century historians, but still not Americanists). The Head of
College confirmed the 3, and that’s the matter of record, but I was out of the
REF anyway--due to some crossover between the article and book (notwithstanding that this
happens in the normal course of academic publishing, and notwithstanding there being differences between the
article and book anyway), and due to the inability of this massive
and vastly expensive Research Assessment Framework to categorise an eight-volume
document collection as a research publication.
As a result of being a REF reject, I could
now be facing a very uncertain future.
But because of REF, and for other reasons I’ve blogged about here before, I decided
a while ago to seek academic work overseas.
Happily for me, in 2013, I got the French "habilitation" that qualified me to apply for professor posts there, and in 2014, a few weeks after learning ly REF rejection, I got my current job as Professor of American Civilisation at Jean Moulin University in Lyon, all of which rather helped with the old self-worth issues that might otherwise have come with that rejection. Of course, not everyone can escape as I have. But I hope my REF story adds a little bit to
the due discreditation of this appalling exercise, or Framework, or whatever
stupid thing they call it in the future. And that if you too are a REF reject then it doesn't really mean a thing about you. I hope others will add their stories. And then, as much as I can’t believe this
thing has gone on as it has as long as it has, equally, I cannot believe it cannot be stopped.
