This is a talk I gave at the Modern Language Association annual conference in Seattle 1/5/2012:
When
I was opening my café in the early nineties, I redid the plumbing for the
entire space. This meant, at the time, that a New York City plumbing inspector
would have to sign off on the work before we could open. My licensed plumber
was there, of course. We followed the inspector around, his face growing more and
more glum as he looked at the new waste lines, the sinks, the hook-up to the
espresso machine. Finally, when he had seen everything, he turned to the
plumber and shrugged. The plumber suggested that they take a little walk around
the block. When they returned, the inspector was smiling. He signed the
necessary forms and left. When I asked the plumber how much I owed him for the
bribe, he laughed and said I had already paid; it had been worked into his
estimate.
A
month or so after we opened, the inspector re-appeared. He sat on a stool and I
gave him a complimentary cup of coffee. We chatted.
He
told me a little about inspecting, including that he could always find problems
enough to fail any installation. He gave me three or four examples from my own
establishment, none of which was a problem, but all of which diverged from the
letter of the law. He also told me that the law would change, that inspectors
would no longer be required; the plumbers themselves would be permitted to
conduct self-inspections. He was philosophical about it, telling me he was
ready to retire anyhow, and that he had a nice bit put aside.
It
has been many years and a housing boom since—and plumbing in New York City does
not seem to have suffered. Just the opposite.
Had
I been involved in academia at the time of my talk with the inspector, and had I
been so inclined, I could have told him that I understood completely what he
was talking about. There’s not a book or an article that a serious academic
can’t make look foolish. You don’t even have to be particularly dishonest, just
look to the details and forget the whole, forget the purpose, forget the
possible effectiveness. If you don’t like the conclusion, the scholarly trail,
the particular school of thought or the place where the scholar works, you can even
ignore those and hide your bias, making a case against the work solely on
petty grounds.
Until
recently, the scholar whose work you are reviewing had best make the right
payment, just as my plumber had done with the inspector. The payment’s not in
money but in conformity. Peer reviewers, particularly those conducting blind
peer review for academic journals, are picked because of status in their
fields. They are the ones who have already made it; they define what is
legitimate and are rarely open to challenge from those who have not yet reached
“peer” status—the state of many of those writing for such journals. We all know
this, and understand the corruption. Yet we continue to participate in the
system. Well, some of us do.
Last
month, I was asked to review an article for a rather prestigious venue on a
topic relating to one of my books. The general editor had suggested to the area
editor that I be asked. It was something of a set-up: the article contained no
reference to my book, even though it purported to be an overview of the specialization.
Clearly, the general editor had noticed this. For whatever reason, that editor
did not want a positive review; I could be pretty well guaranteed not to give
one. So, the burden of rejection was being passed on to someone the author
could never identify.
I
refused the opportunity, pointing out the omission and saying I did not believe
it would fair for me to review the article. The area editor wrote back, clearly
surprised, but thanking me for being so honest. I got the feeling that my
response was a rare one. To respond otherwise, however, would have been just as
corrupt as that plumbing inspector.
I’ve
also been on the other side recently. An article I wrote was panned by a blind
reviewer, one who, for whatever reason, had taken umbrage. To make a point, I had
illustrated a parallel progression in two American institutions; the reviewer
scathingly pointed out that I made no convincing case that the two are the
same. Quite rightly; I wasn’t trying to, for they were tangential. The reviewer
then listed a number of statements I had made, claiming I had not substantiated
them. Again, quite rightly—but I hadn’t been trying to. At no point did the
reviewer address my argument or refute my claims. There was no advice for the
author in the review, and none for the editor, except “reject.”
So,
I wrote back to the editor—who had praised the piece before sending it for
blind review—and withdrew it, though the second review was not yet in. I love
the editorial process, and love comments that can help me improve what I
write—but not this. So, I published the piece myself, on my blog…
where
I’ve already had a scathing comment from the right-wing political agitator
David Horowitz: “Aaron why do you continue to peddle this horseshit about me,
which you know not to be true?” Frankly, though Horowitz addresses my points no
more than did the blind reviewer, at least he is willing to do it in public. I
can respect that: Horowitz is nothing if not open about his prejudices.
All
of this brings me to my main topics: bottlenecks and the dark.
In
an earlier time, when there was limited space for publication, a rigorous
pre-publication peer-review process could have been justified—to some degree,
at least. Keeping it blind, though, had even less justification. An attempt to
allow openness and honesty without consequence, it just as often produced
pettiness. Today, when it is possible for anyone to publish at any time, we
don’t need either the bottleneck or the darkness, especially since the process
is so easily corrupted. Just as advances have made plumbing self-inspection
safe and efficient, we now have open enough and strong enough post-publication
review possibilities to make blind peer-review prior to publication
unnecessary. As we all know, whether we admit it or not, it continues simply
because we have made appearance in peered-review journals the standard for advancement.
It continues because our committees on re-appointment, tenure, and promotion
want an easy benchmark. We have not yet institutionalized post-publication review,
though why that is baffles me. Wouldn’t the number of citations, reviews, and
other responses give as strong an indication of the value of an essay as its
original venue? Stronger, I’d say.
Especially
since, in a digital environment, even the published essay can be improved in
light of comments and criticism. And should be.
In
response to an article by Richard Smith in Breast
Cancer Research, Developmental Neuropsychologist Deborah Bishop wrote of “the
real function of peer review, which should be to offer advice to the editor and
the author.”[1]
Sure, but why should this not be post-publication, in a milieu where change,
unlike in the days of reliance on print, is easy? Bishop’s criticism of peer
review is that it is often an easy way for editors to avoid making
decisions—and she is right. This is why I prefer writing book chapters:
anthology editors are quite focused on their topics and don’t pass things off.
The function of any editor, today, should be to help strengthen essays that he
or she has selected, for whatever reason. This should be the case in journals
as well. Certainly, a peer-review process with this in mind could still start
before publication, but there’s no reason it should become a bottleneck or a means
for evaluation from the dark.
Journals
like Cheryl-Ball edited Kairos have
developed open and productive systems of review, in this case a three-tier
process revolving around a named editorial board and the clear purpose of
working with creators to strengthen their work. But even Kairos, a respected journal presenting the best, most cutting-edge
of its field, is sometimes looked at askance by those evaluating careers. It is
not, after all, a traditional, blind-reviewed journal. It’s system, though
superior to blind peer review, is still sometimes seen as suspect.
My
plumbing inspector had made his money, legitimate and otherwise. There’s money
in peer review, too—and not only for the relics of the past. Perhaps this is
part of the problem. Companies like Sage and Routledge make a great deal off of
the peer-reviewed journals they own and continue to protect. Zoë Corbyn asks:
have these
gatekeepers for what counts as acceptable… become too powerful? Is the system
of reward that has developed around them the best?...
Unpicking the
power of academic and scholarly journals, with their estimated global turnover
of at least $5 billion (£3 billion) a year, is a complex business. There are an
estimated 25,000 scholarly peer-reviewed journals in existence….
It is these -
particularly the elite titles… - that are at the heart of the recognition-and-reward
system…. From career progression to grant income, "wealth" within the
academy is determined by the production of… knowledge as recorded in
peer-reviewed scholarly journals.[2]
The
owners of the journals make their money from the fact that scholars are
beholden to them for creation of that “’wealth’ within the academy.” Those with
that “wealth” have too little incentive to give it up—and they are generally
the ones evaluating the advancement of others.
On
the other hand, as Richard Smith writes:
what happens
after publication can also be called peer review, and that, I believe, is the
peer review that really matters - the process whereby the world decides the
importance and place of a piece of research…. Many studies are never cited
once, most disappear within a few years, and very few have real, continuing
importance.
And the
correlation between what is judged important in pre-publication peer review and
what has lasting value seems to be small…. Many papers get very high marks from
their peer reviewers but have little effect on the field. And on the other
hand, many papers get average ratings but have a big impact'[3]
So
what to do? Well, we’re already doing it. Joe Pickrell suggests we aim for:
1. Immediate publication without peer review….
2. One-click recommendation of papers….
3. Connection to a social network….
4. Effective search based on the collective
opinion on a paper.[4]
I
would add in a number of modifications, such as something akin to the political
blog Daily Kos’s “trusted user” status for readers of any particular journal. And
I would spotlight social-networking sites like Academia.edu, where scholars can
post their own work, connect, and search keywords for pieces that might
interest them.
For
anyone still wondering why blind peer-review should be jettisoned, Smith
provides a list of reasons:
Firstly, it is
very expensive in terms of money and academic time….
Secondly, peer
review is slow….
Thirdly, peer
review is largely a lottery.....
A fourth problem
with peer reviews is that it does not detect errors….
The fifth
problem with pre-publication peer review is bias….
Finally, peer
review can be all too easily abused…. [5]
Frankly,
I find it odd that reliance on blind peer-review continues, especially in
career evaluation—but it does. My own institution has recently instituted a
third-year deans’ review where one of the benchmarks is at least one
peer-reviewed publication.
Still,
blind peer review is dead. It just doesn’t know it yet. It certainly doesn’t
warrant renovation; the structure is collapsing. The bottleneck it once created
is no longer necessary. It The situation in this regard is much like that of
journalism, just five or six years ago, when the blogs were first seriously
challenging a profession that had become almost terminally inward-looking and
almost completely averse to change or innovation. Though I saw—and still
occasionally see—old-style journalism’s stalwarts pounding the podium,
red-faced, claiming that the lack of gatekeeping they imagined in the challenge
of the blogs would destroy journalism forever, that has not happened. If
anything, the profession is much more vigorous today, more varied and
experimental—even though many of its older structures have collapsed. The plumber
I used a couple of years ago said much the same thing about the state of his
profession, post paid inspectors.
Kairos and Academia.edu
are only two examples of what will replace those 25,000 peer-reviewed journals
if those journals don’t begin to change. The money and other wealth generated
by peer review will dwindle, as will the numbers. What is now 25,000 will soon
be 2,500, and those will be the one that have changed, that have embraced openness
and digital possibilities and the new sorts of post-publication review that
seem to pop up every day. As journalism has found, gatekeeping does not die
when new venues can be established cheaply and by anyone. It simply changes.
Aggregators funnel the best or the selected, citations rise in import, and choice,
such as that represented on WorldCat.org of library inclusions, becomes a
significant benchmark. A deliberate and controlled editorial bottleneck becomes
irrelevant.
This
is probably the last year a panel like this will seem necessary. No matter how
much we try, academics cannot hold the fort any more than journalists were able
to. What we have to do is adapt, or we will be superseded—and there are few of
us who, like my plumbing inspector, are willing to be pulled down that route.
Whether we like it or not, we are going to have to bear the light of
openness—and quite soon, now.
[1]
Deborah Bishop, “Comment on ‘Classical Peer Review: An Empty Gun’ by Richard
Smith,’ Breast Cancer Research,
Volume 12, Supplement 4: Controversies in
Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13/comments.
[2] Zoë Corbyn, “A threat to scientific communication,”
The Times Higher Education, August
13, 2009, http://www.timeshighereducation.co.uk/story.asp?storycode=407705.
[3]
Richard Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.
[4]
Joe Pickrell, “Why publish science in peer-reviewed journals?” Genomes Unzipped: Public Personal Genomics,
July 13, 2011, http://www.genomesunzipped.org/2011/07/why-publish-science-in-peer-reviewed-journals.php.
[5] Richard
Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.
No comments:
Post a Comment