Thursday, January 05, 2012

The Bearable Light of Openness: Renovating Obsolete Peer-Review Bottlenecks

This is a talk I gave at the Modern Language Association annual conference in Seattle 1/5/2012:

When I was opening my café in the early nineties, I redid the plumbing for the entire space. This meant, at the time, that a New York City plumbing inspector would have to sign off on the work before we could open. My licensed plumber was there, of course. We followed the inspector around, his face growing more and more glum as he looked at the new waste lines, the sinks, the hook-up to the espresso machine. Finally, when he had seen everything, he turned to the plumber and shrugged. The plumber suggested that they take a little walk around the block. When they returned, the inspector was smiling. He signed the necessary forms and left. When I asked the plumber how much I owed him for the bribe, he laughed and said I had already paid; it had been worked into his estimate.
A month or so after we opened, the inspector re-appeared. He sat on a stool and I gave him a complimentary cup of coffee. We chatted.
He told me a little about inspecting, including that he could always find problems enough to fail any installation. He gave me three or four examples from my own establishment, none of which was a problem, but all of which diverged from the letter of the law. He also told me that the law would change, that inspectors would no longer be required; the plumbers themselves would be permitted to conduct self-inspections. He was philosophical about it, telling me he was ready to retire anyhow, and that he had a nice bit put aside.
It has been many years and a housing boom since—and plumbing in New York City does not seem to have suffered. Just the opposite.
Had I been involved in academia at the time of my talk with the inspector, and had I been so inclined, I could have told him that I understood completely what he was talking about. There’s not a book or an article that a serious academic can’t make look foolish. You don’t even have to be particularly dishonest, just look to the details and forget the whole, forget the purpose, forget the possible effectiveness. If you don’t like the conclusion, the scholarly trail, the particular school of thought or the place where the scholar works, you can even ignore those and hide your bias, making a case against the work solely on petty grounds.
Until recently, the scholar whose work you are reviewing had best make the right payment, just as my plumber had done with the inspector. The payment’s not in money but in conformity. Peer reviewers, particularly those conducting blind peer review for academic journals, are picked because of status in their fields. They are the ones who have already made it; they define what is legitimate and are rarely open to challenge from those who have not yet reached “peer” status—the state of many of those writing for such journals. We all know this, and understand the corruption. Yet we continue to participate in the system. Well, some of us do.
Last month, I was asked to review an article for a rather prestigious venue on a topic relating to one of my books. The general editor had suggested to the area editor that I be asked. It was something of a set-up: the article contained no reference to my book, even though it purported to be an overview of the specialization. Clearly, the general editor had noticed this. For whatever reason, that editor did not want a positive review; I could be pretty well guaranteed not to give one. So, the burden of rejection was being passed on to someone the author could never identify.
I refused the opportunity, pointing out the omission and saying I did not believe it would fair for me to review the article. The area editor wrote back, clearly surprised, but thanking me for being so honest. I got the feeling that my response was a rare one. To respond otherwise, however, would have been just as corrupt as that plumbing inspector.
I’ve also been on the other side recently. An article I wrote was panned by a blind reviewer, one who, for whatever reason, had taken umbrage. To make a point, I had illustrated a parallel progression in two American institutions; the reviewer scathingly pointed out that I made no convincing case that the two are the same. Quite rightly; I wasn’t trying to, for they were tangential. The reviewer then listed a number of statements I had made, claiming I had not substantiated them. Again, quite rightly—but I hadn’t been trying to. At no point did the reviewer address my argument or refute my claims. There was no advice for the author in the review, and none for the editor, except “reject.”
So, I wrote back to the editor—who had praised the piece before sending it for blind review—and withdrew it, though the second review was not yet in. I love the editorial process, and love comments that can help me improve what I write—but not this. So, I published the piece myself, on my blog…
where I’ve already had a scathing comment from the right-wing political agitator David Horowitz: “Aaron why do you continue to peddle this horseshit about me, which you know not to be true?” Frankly, though Horowitz addresses my points no more than did the blind reviewer, at least he is willing to do it in public. I can respect that: Horowitz is nothing if not open about his prejudices.
All of this brings me to my main topics: bottlenecks and the dark.
In an earlier time, when there was limited space for publication, a rigorous pre-publication peer-review process could have been justified—to some degree, at least. Keeping it blind, though, had even less justification. An attempt to allow openness and honesty without consequence, it just as often produced pettiness. Today, when it is possible for anyone to publish at any time, we don’t need either the bottleneck or the darkness, especially since the process is so easily corrupted. Just as advances have made plumbing self-inspection safe and efficient, we now have open enough and strong enough post-publication review possibilities to make blind peer-review prior to publication unnecessary. As we all know, whether we admit it or not, it continues simply because we have made appearance in peered-review journals the standard for advancement. It continues because our committees on re-appointment, tenure, and promotion want an easy benchmark. We have not yet institutionalized post-publication review, though why that is baffles me. Wouldn’t the number of citations, reviews, and other responses give as strong an indication of the value of an essay as its original venue? Stronger, I’d say.
Especially since, in a digital environment, even the published essay can be improved in light of comments and criticism. And should be.
In response to an article by Richard Smith in Breast Cancer Research, Developmental Neuropsychologist Deborah Bishop wrote of “the real function of peer review, which should be to offer advice to the editor and the author.”[1] Sure, but why should this not be post-publication, in a milieu where change, unlike in the days of reliance on print, is easy? Bishop’s criticism of peer review is that it is often an easy way for editors to avoid making decisions—and she is right. This is why I prefer writing book chapters: anthology editors are quite focused on their topics and don’t pass things off. The function of any editor, today, should be to help strengthen essays that he or she has selected, for whatever reason. This should be the case in journals as well. Certainly, a peer-review process with this in mind could still start before publication, but there’s no reason it should become a bottleneck or a means for evaluation from the dark.
Journals like Cheryl-Ball edited Kairos have developed open and productive systems of review, in this case a three-tier process revolving around a named editorial board and the clear purpose of working with creators to strengthen their work. But even Kairos, a respected journal presenting the best, most cutting-edge of its field, is sometimes looked at askance by those evaluating careers. It is not, after all, a traditional, blind-reviewed journal. It’s system, though superior to blind peer review, is still sometimes seen as suspect.
My plumbing inspector had made his money, legitimate and otherwise. There’s money in peer review, too—and not only for the relics of the past. Perhaps this is part of the problem. Companies like Sage and Routledge make a great deal off of the peer-reviewed journals they own and continue to protect. Zoë Corbyn asks:
have these gatekeepers for what counts as acceptable… become too powerful? Is the system of reward that has developed around them the best?...
Unpicking the power of academic and scholarly journals, with their estimated global turnover of at least $5 billion (£3 billion) a year, is a complex business. There are an estimated 25,000 scholarly peer-reviewed journals in existence….
It is these - particularly the elite titles… - that are at the heart of the recognition-and-reward system…. From career progression to grant income, "wealth" within the academy is determined by the production of… knowledge as recorded in peer-reviewed scholarly journals.[2]
The owners of the journals make their money from the fact that scholars are beholden to them for creation of that “’wealth’ within the academy.” Those with that “wealth” have too little incentive to give it up—and they are generally the ones evaluating the advancement of others.
On the other hand, as Richard Smith writes:
what happens after publication can also be called peer review, and that, I believe, is the peer review that really matters - the process whereby the world decides the importance and place of a piece of research…. Many studies are never cited once, most disappear within a few years, and very few have real, continuing importance.
And the correlation between what is judged important in pre-publication peer review and what has lasting value seems to be small…. Many papers get very high marks from their peer reviewers but have little effect on the field. And on the other hand, many papers get average ratings but have a big impact'[3]
So what to do? Well, we’re already doing it. Joe Pickrell suggests we aim for:
1. Immediate publication without peer review….
2. One-click recommendation of papers….
3. Connection to a social network….
4. Effective search based on the collective opinion on a paper.[4]
I would add in a number of modifications, such as something akin to the political blog Daily Kos’s “trusted user” status for readers of any particular journal. And I would spotlight social-networking sites like Academia.edu, where scholars can post their own work, connect, and search keywords for pieces that might interest them.
For anyone still wondering why blind peer-review should be jettisoned, Smith provides a list of reasons:
Firstly, it is very expensive in terms of money and academic time….
Secondly, peer review is slow….
Thirdly, peer review is largely a lottery.....
A fourth problem with peer reviews is that it does not detect errors….
The fifth problem with pre-publication peer review is bias….
Finally, peer review can be all too easily abused…. [5]
Frankly, I find it odd that reliance on blind peer-review continues, especially in career evaluation—but it does. My own institution has recently instituted a third-year deans’ review where one of the benchmarks is at least one peer-reviewed publication.
Still, blind peer review is dead. It just doesn’t know it yet. It certainly doesn’t warrant renovation; the structure is collapsing. The bottleneck it once created is no longer necessary. It The situation in this regard is much like that of journalism, just five or six years ago, when the blogs were first seriously challenging a profession that had become almost terminally inward-looking and almost completely averse to change or innovation. Though I saw—and still occasionally see—old-style journalism’s stalwarts pounding the podium, red-faced, claiming that the lack of gatekeeping they imagined in the challenge of the blogs would destroy journalism forever, that has not happened. If anything, the profession is much more vigorous today, more varied and experimental—even though many of its older structures have collapsed. The plumber I used a couple of years ago said much the same thing about the state of his profession, post paid inspectors.
Kairos and Academia.edu are only two examples of what will replace those 25,000 peer-reviewed journals if those journals don’t begin to change. The money and other wealth generated by peer review will dwindle, as will the numbers. What is now 25,000 will soon be 2,500, and those will be the one that have changed, that have embraced openness and digital possibilities and the new sorts of post-publication review that seem to pop up every day. As journalism has found, gatekeeping does not die when new venues can be established cheaply and by anyone. It simply changes. Aggregators funnel the best or the selected, citations rise in import, and choice, such as that represented on WorldCat.org of library inclusions, becomes a significant benchmark. A deliberate and controlled editorial bottleneck becomes irrelevant.
This is probably the last year a panel like this will seem necessary. No matter how much we try, academics cannot hold the fort any more than journalists were able to. What we have to do is adapt, or we will be superseded—and there are few of us who, like my plumbing inspector, are willing to be pulled down that route. Whether we like it or not, we are going to have to bear the light of openness—and quite soon, now.


[1] Deborah Bishop, “Comment on ‘Classical Peer Review: An Empty Gun’ by Richard Smith,’ Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13/comments.
[2] Zoë Corbyn, “A threat to scientific communication,” The Times Higher Education, August 13, 2009, http://www.timeshighereducation.co.uk/story.asp?storycode=407705.
[3] Richard Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.
[4] Joe Pickrell, “Why publish science in peer-reviewed journals?” Genomes Unzipped: Public Personal Genomics, July 13, 2011, http://www.genomesunzipped.org/2011/07/why-publish-science-in-peer-reviewed-journals.php.
[5] Richard Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.

No comments: