Saturday, January 21, 2012

"Data Driven Instruction"

Mark Naison, a tireless proponent of common sense in education and politics, has reminded me of the Orwellian aspects of "Data Driven Instruction." He writes:
Anyone who thinks this approach is going to improve the quality of instruction, and create better relationships between teachers, students and parents, is sorely mistaken. It will increase the stress level on all concerned and squeeze out compassion, empathy and community building along with creative instruction. But the school reformers don't care. They are determined to bring a "business atmosphere" into public education, with teachers poring over test scores the way executives pore over sales data!
That's only the consequence. There's also the internal nonsense of the phrase itself.

As one website describes it, "Data Driven Instruction":
emphasizes frequent testing and focused attention on what the children are actually learning allows educators to effectively and realistically pinpoint, assess and remedy weaknesses and reinforce success.
The assumptions here are as dangerous as they are mind-boggling (to say nothing of the grammar--but we all err in that, sometimes). There's the assumption that frequent testing is a good thing, that testing can tell us what students are "actually learning," that testing provides useful information, that testing leads to success. None of these things is true, but we've been conditioned to believe all of them in an Orwellian bombardment of misinformation masked by false equivalencies ("war is peace").

The basis of the problem here is the assumption that testing is an effective measure of learning. However, I have yet to see that really important first step, discussion of just how a test gives useful information. Hell, the fact of being a test means nothing: tests can be designed so that any one of us fails--or any particular group among us succeeds. Most tests are designed to do no more than show a temporary mastery of a particular set of "facts." That's no way of evaluating education.

"Learning" is not something that can be assessed through testing alone, for testing is a tool of limited utility. It is a tool, yes, but it cannot be the only one.

Neither can "data" be the sole judge of effective instruction. Much of learning only becomes apparent years after it has taken place. Immediate assessment, the provenance of educational data, covers only one small part of real education. Reliance on assessment data alone (or even primarily) limits education. In fact, it debases it. Certainly, it does not enhance it.

GIGO--Garbage In, Garbage Out. What we are asking students to produce on tests is garbage--if it is considered a sign of the effectiveness of education. Not only is testing necessarily backward-looking, but it is reductive. You can't test the future; you can't reduce learning to Scantron sheets.

Friday, January 20, 2012

PIPA and SOPA: Just Whose "Property" Is It, Anyway?

Harvard economist Greg Mankiw writes on his blog that:
In a free society, you don't have the freedom to steal your neighbor's property. And that should include intellectual property.
Should it?  Why? Because Intellectual Property is called "property"?

What we have here is homonyms being mistaken for synonyms.

Simply put, calling an apple an orange doesn't make it one.

Intellectual Property (IP) has a completely different status both in law and culture than does physical property. Conflating the two simply confuses issues of the provenance of creative activity and rights for profiting from it, rights limited from the get-go by the US Constitution.

So let's stop these analogies, making IP the same as physical property, can we, please?

Moving on:

I've a suggestion for those trying to craft anti-piracy laws: Why not, while finding ways of strengthening protection for new creative output, make a trade-off? Roll back copyright extensions to, say, the creator's lifetime only or, for corporate holders, to a maximum of 75 years. That way, the intellectual commons won't be nearly so threatened and it will appear that those wanting greater protection now are also ceding something. I think many of us who are anti-SOPA and PIPA right now would feel a lot better about the desire to protect if there were a more magnanimous aspect to the argument, too.

SOPA and PIPA: Who Are the Pirates?

We forget, in our discussions of piracy of Intellectual Property, that those complaining loudest about piracy are pirates themselves.

Or, at least, are the descendants of pirates, still profiting by the piracy of their ancestors.

The initial patent and copyright laws in England, in the 17th and 18th centuries, allowed for "ownership" for a period of 14 years, renewable once. US laws, by 1800, had followed suit. This limitations codified an intellectual commons of material open for use by everyone. The particular creators, who had made use of the commons in their own activities, were provided an incentive, the right to profit from their work for up to 28 years. After that, what came from the commons went back to the commons.

Nothing, of course, is created out of thin air. Everything is based on what went before. Creativity in the future can only occur if 'what went before' is available for use.

The system made sense.

Oh, and it had another factor: things weren't in copyright by virtue of publication. It took an effort on the part of the creator. If someone didn't want a work copyrighted, it was quite easy: don't copyright it. Newspapers, for example, which were generally political organs, wanted to see their stories reprinted as often as possible. Automatic copyright would have added an unwanted barrier to that, reprinters having to make sure they had clear right to republish instead of just going ahead.

That also made sense.

But, soon, that sensible system was encroached upon, by people who had begun to see the creator as the individual genius and the creation as "property" like any other. They chafed at what they saw as limitation of their rights over their creations.  By 1831 (earlier, in England), the duration of copyright had been doubled.

It has gotten longer (and stronger) ever since. It would last forever in the US, except that would require a Constitutional Amendment (the Constitution specifically calls for copyright of limited duration).

Why? Because the companies and corporations (who hold most of the valuable copyrights) have the money to push Congress to expand their ownership rights.

To steal from the intellectual commons.

That's right, to steal.

Under the law as it was established during the infancy of our republic, there was a robust intellectual commons, a wide range of work "owned" by all of us. Today, that commons is quite a bit smaller, much of it stolen from us and given to copyright holders. Today, The Jazz Singer belongs not to the culture that created it and sustains it, but to a corporation that "protects" it, a corporation that has no connection to the initial act of creation but that has stolen our common right in order to maintain continued profit. Today, Mickey Mouse, whose popularity is ours as a whole and who has become a part of our broader culture, place maintained by us, is not owned by us, but by people with no connection at all (except in law) to his creation. Our rights to Mickey have been stolen from us.

This is very real theft, though we have been taught to look at it the other way around. What we once had is no longer ours, but belongs to corporations who continue to profit on our cultural legacy, making "theirs" what should be "ours."

There's reason for strengthening protections of Intellectual Property for new creations. I am all for that. But, at the same time, let's start trying to get some of our own back, our cultural heritage, the commons that makes future creativity possible.

Wednesday, January 18, 2012

SOPA, PIPA, and the Strangling of Creativity

There's not much I want to add to this today, except to applaud those sites going black to draw attention to what happens when we constrict the commons, as we have been doing slowly over the past two centuries, expanding copyright protections and contracting just what one can do in terms of new creative work.

The Stop Online Piracy Act (SOPA) and the Protect Intellectual Property Act (PIPA) are bills now making their way through Congress, bills that will allow tighter control of Intellectual Property (IP) by corporations than is already possible--and control is now way too tight. If you will, please, sign this petition against PIPA. Here's a Daily Kos piece on it and on the websites going black for the day.

Here's a link, though, to the first pages of my chapter "Intellectual Property in a Digital Age" in Robert Leston and my Beyond the Blogosphere: Information and Its Children. In the chapter, I discuss the history and implications of the way we look at IP and at the concept of "copy."

Thursday, January 12, 2012

The Old Red Truck

In 1986, I bought this truck in Ouagadougou, Burkina Faso. I got it from a Dutchman who had used it to travel down from the Mediterranean.  He had put a Peugeot diesel engine in it.

I used the truck for over a year, often sleeping in the back, traveling across Burkina Faso and, once, down into Togo (the right-side windshield fell out, that trip, requiring the passenger to wear goggles).

This picture was taken in the north of Burkina Faso during the rainy season (obviously), just an hour east of Aribinda on the road from Djibo to Dori.

By the time I left, there was little left of the truck. The roads had taken quite a toll on it. Just shortly after I took this, we got stuck in the mud of that road. It took all of us (I think there were five or six) in the truck plus maybe twenty people from a nearby village to get the truck out of the mud and back on solid roadway, even using the tracks that I carried on top.

The tubes on top on each side held water that fed a sink inside. Quite the motor home!

Monday, January 09, 2012

Lessons for Academics: What Journalists Know About Gatekeeping (Part 2)


Journalists today know how to use the internet for research so well that they don’t even know they know it. That is, digital tools now come so naturally to hand, and have been tailored so expertly to the needs of the specific individuals and projects, that the journalists don’t even think about them—they just use them. In this, most academics are a decade or so behind.

“Traditional” news aggregators (“newspapers” and the like), sources, and depositories of information no longer suffice, in journalism. At least, not in the forms they once exhibited. Nobody trying to keep up with international affairs will rely on The New York Times, for example. They will read it, yes, but differently than they read it twenty years ago. Then, it was often the only available source; now, it is one of a tight weave of information possibilities. This fact has changed The Times as much as it has changed how the newspaper is read.

Today’s news stories aren’t over once they see “print.” They are changed, updated as new information is found, as errors are pointed out, as events unfold. The writer constantly looks to comments and to related stories, among other things, to make to make the story stronger. Unlike the way it was when I was trained as a reporter in the 1970s, stories aren’t over once they are in the paper.

In most areas of news gathering and news utilization, everything has changed (to use the cliché) over the past decade. In fact, in most areas of media, everything has changed. Book publishing is moving beyond reliance on a single (paper) platform. Magazines no longer center on print and the demands of print timetables. Television shows aren’t schedule bound. Movies exist everywhere.

Academic publishing is one of the only areas where digitally sparked changes aren’t yet universally manifest. There are at least two reasons for this: First, academic structures are inherently conservative, loathe to change, and academic publishing is tied directly to those structures. Second, academic communities are walled off, to some degree, from the forces at work in the broader culture. As a result, they don’t feel the pressure towards change that entities in journalism, for example, have had to respond to. In addition, and (rather ironically) partly as a result of digital possibilities, the amount of money being made through academic publishing has grown substantially over the past decades, making the publishers (many of them outside of academia itself) protective of the way things are and unlikely to tolerate experiment.

At the same time, some of the most interesting experiments in publishing are occurring within academia. There are new types of journals with flexible and inclusive editorial structures and openness to multimedia presentation, book creation that begins online and that embraces contribution from various quarters, and much beyond. Change is happening. It is just slow in comparison with much of the rest of the publishing, media, and information worlds.

When an assistant professor hears that re-appointment, tenure, and promotion are greatly influenced by publication in ‘significant’ peer-reviewed journals, she or he almost instinctively pulls back from work that tends toward the experimental or new—for self-preservation. And even their supporters on the various committees find themselves retreating to defense of the candidate’s work in traditional venues, recognizing that as the safest ways of getting the candidates through the process successfully. This ends up providing push-back against any pressure towards change, keeping the new and wonderful work being done on the sidelines.

So, it is not sufficient that the journals and publishers change, though that is part of the equation. Until those responsible for hiring, promotion, and tenure processes willingly back away from emphasis on peer review as a requirement, academic publishing will continue to lag far behind. Already, the continued focus on peer review seems anachronistic; soon, it will seem bizarre.

Why?

The means for replacing peer review are already present, and are in use in many other areas of the media world. These allow academics to engage more actively in a world of scholarship, to experiment with avenues of research, and to demonstrate the contribution of their work.

Social networking can become academic networking, for one thing, as is happening at academia.edu, on ‘faculty commons’ sites across the country, on area-specific websites, and even through the communities that online academic journals are building. Academic organizations are doing much the same, even setting up digital versions of their conferences, allowing the papers presented to be housed in places where they can become parts of on-going conversations.

Each scholar becomes something of an aggregator, much in the way each journalist does, utilizing certain pieces and ignoring others. When each scholar has a vigorous online presence (something that is coming), those who react to them positively can look at their connections, at the papers and books they ‘follow,’ and at the work they are cited in to then make their own decisions without having to look at each paper or other item purporting to deal with a particular area of interest. Communities of scholars are most certainly becoming the center of much of our activity—and these communities are increasingly anchored online.

As younger scholars become more and more involved in public research and writing, it also becomes easier to evaluate what they are doing. Who has cited them? Even now, that is easy to find. And how often? Who links to their work? It is becoming easier and easier to discover the impact of the scholarship even of someone outside of our own particular specialties.

Soon, as has already happened with journalists, we academics will have each developed online methodologies for our own work, ways of skipping over that which we can safely ignore, ways of pointing ourselves to work we really should be examining. Soon, we will be able to use these skills in evaluating the contributions of each other in our institutional capacities.

Soon, and probably (for many of us) without even knowing it, we will find that we have by-passed blind peer review altogether, replacing it with open (to the writer, at least) systems of editorial evaluation and improvement and public means of viewing just how much, or how little, each of us has accomplished.

Sunday, January 08, 2012

Lessons for Academics: What Journalists Know About Gatekeeping (Part 1)


In preparing my talk on peer review for the MLA conference in Seattle last week, I forgot that few of my fellow academics have much familiarity with ‘gatekeeping,’ certainly not to the extent that journalists have, especially after the upheavals of the past decade. Though the situations are different (journalists working with a responsibility to the public sphere directly where academics look to the needs of specific disciplines away from more generalized discussions), academics should know as much about the responsibilities and ramifications of gatekeeping as do journalists. But they don’t.

In journalism, the discussion of the problems of gatekeeping has been public and even heated ever since Benjamin Franklin’s 1731 “Apology for Printers.” The responsibility of the venue, both to the public and to the author, is keenly felt and its implications hotly contested. As this is something I have written about, both in the first two of my blogosphere books (The Rise of the Blogosphere and Blogging America: The New Public Sphere) and in “The Citizen Journalism as Gatekeeper: A Critical Evolution” (for Public Journalism 2.0: The Promise andReality of a Citizen-Involved Press), I made the mistaken assumption that editors and writers in academia have parallel concerns.

In a way, of course, they do. But the discussion has never been as public as in journalism. The academic gatekeepers, also, have never been challenged from outside, as gatekeepers in journalism, given the nature of the profession, always have been. In addition, the field of journalism, in keeping with the concept of freedom of the press, is not limited to credentialed professionals. As Gary Hudson and Mick Temple write in “We Are All Journalists Now”:
Despite the belief that the forums that debate the question of “what is journalism” are controlled by communities (for example, the journalism academic community) with a vested interest in a limited definition, we have no wish to limit access to “the profession”. Indeed, such a wish would be ludicrous in today’s world. The blogger, the online pundit, the producer of an online community newsletter can call themselves journalists, but unless they are committed to writing new and accurate material they have no right to do so. (73-74)
Things are, of course, quite different in academia. In addition to entry qualifications, scholars have found themselves beholden to the gatekeepers in other ways never the case in journalism, where stepping outside the establishment has always been easier than in academia. It is much simpler to start a newspaper than a college; an academic journal needs more behind it than The Journal from Joe’s Garage.

Gatekeeping is a major topic in journalism today, and has been for quite some time. It should be in academia, too, but the conversation has yet to be either as broad or as deep as in journalism. One reason for this is that the gatekeepers have tremendous impact on scholars—through editorial and review functions for journals and presses and through procedures for tenure and promotion. People are loathe to challenge, for fear for their own careers. Gatekeeping has greater impact on individual scholars than is found in journalism on individual reporters, for journalism has less formulaic structures and more alternatives.

As I write in “The Citizen Journalism as Gatekeeper: A Critical Evolution”:
During the second half of the twentieth century, theories of gatekeeping began to appear, generally extending the work of Kurt Lewin (1947) whose explorations of leadership and group dynamics provided a starting point and initially were applied to journalism by David Manning White (1950). Examinations of gatekeeping continue today in the work of Pamela Shoemaker, whose recent book (with Timothy Vos) is Gatekeeping Theory (2008). Shoemaker has provided the framework for study of gatekeeping on a theoretical level since the 1980s, with her 1991 work (written with Stephen Reese), Mediating the message: Theories of influences on mass media content, examining the gap between experienced and mediated version of events. (46)
Over the past half century, journalism has developed a complex understanding of gatekeeping that, perhaps, still eludes most academic publishing. This goes back at least to White, who studied the gatekeeping of an editor he calls “Mr. Gates”:
It is a well-known fact in individual psychology that people tend to perceive as true only those happenings which fit into their own beliefs concerning what is likely to happen. It begins to appear (if Mr. Gates is a fair representative of his class) that in his position as “gatekeeper” the newspaper editor sees to it (even though he may never be consciously aware of it) that the community shall hear as a fact only those events which the newsman, as the representative of his culture, believes to be true. (171) 
Though, even in academia, as Shoemaker and Reese write, “gatekeeping involves the selection, shaping, and repetition of information” (255), we don’t really question the ability of academic gatekeepers to do this competently and honestly, referring to academic credentials as proof enough that they can and will. Journalism has never been quite so complacent, recognizing that, as Shoemaker and Reese also claim:
the media gatekeeper must winnow down a larger number of potential messages to a few. The book publisher chooses from many possible titles; the network programmer selects from among several ideas for sitcoms, serials, and dramas to compose a prime-time schedule; and the newspaper editor must decide on a handful of stories to run on the front page. These decisions directly affect the media content that reaches the audience. But are those decisions made at the whim of the individual? (100)
Their question needs to be addressed in academia also, especially by anyone who is going to argue in favor of retaining blind pre-publication peer review, where the possibility of “whim” decision-making can warp a field of study in all sorts of unanticipated ways. This is especially important when “whim” can have great impact on careers, far more than in journalism, as well as on the course of future research.

Unlike journalism, where traditional forms of gatekeeping are being re-assessed in light of changes brought on through new digital possibilities, the grip of traditional academic gatekeepers is still strong. The forces granting tenure and promotion continue to use “peer review” as a shortcut in their decision-making and there is no alternate system of higher education for scholars to turn to. That is beginning to change, and it will have to, as digital possibilities make themselves even more strongly felt in scholarship and its publication.

The question is, can we academics learn enough quickly enough to replace reliance on peer review with a fair and open system that promotes genuine effort and scholarship without placing limitations on either means or avenues of exploration? I’ll try to provide a few suggestions for such a system in Part 2.

Friday, January 06, 2012

Reaction to Our Peer Review Panel

After delivering my talk as part of a panel on academic peer review at the Modern Language Association annual convention yesterday, I felt extremely let down. I did not feel I had read well, and had fumbled the questions afterwards. Part of the problem was that I delivered the talk via Skype and was unable to see or hear the audience (the computer's camera in the room was directed to the podium and away from the audience). That was disorienting, to say the least.

Another problem was that I followed Cheryl Ball, editor of Kairos and at least six times as competent as I am. And was followed by Allen Mendenhall, a lawyer and PhD student in English, also better than I am. Even the moderator, my colleague Sean Scanlan, knows more about the issue than I do--and did a great job handling the panel.

Today, then, I was a little shocked to read Scott Jashik's article on the panel on Inside Higher Ed. He makes me wish I could have been there in person. I am pleased that he could see beyond my stuttering delivery to the point I was trying to make.

The issues surrounding blind peer review are complex and daunting. Perhaps our panel, along with Jashik's article, do provide some small movement towards resolving them.

I hope so.

Thursday, January 05, 2012

The Bearable Light of Openness: Renovating Obsolete Peer-Review Bottlenecks

This is a talk I gave at the Modern Language Association annual conference in Seattle 1/5/2012:

When I was opening my café in the early nineties, I redid the plumbing for the entire space. This meant, at the time, that a New York City plumbing inspector would have to sign off on the work before we could open. My licensed plumber was there, of course. We followed the inspector around, his face growing more and more glum as he looked at the new waste lines, the sinks, the hook-up to the espresso machine. Finally, when he had seen everything, he turned to the plumber and shrugged. The plumber suggested that they take a little walk around the block. When they returned, the inspector was smiling. He signed the necessary forms and left. When I asked the plumber how much I owed him for the bribe, he laughed and said I had already paid; it had been worked into his estimate.
A month or so after we opened, the inspector re-appeared. He sat on a stool and I gave him a complimentary cup of coffee. We chatted.
He told me a little about inspecting, including that he could always find problems enough to fail any installation. He gave me three or four examples from my own establishment, none of which was a problem, but all of which diverged from the letter of the law. He also told me that the law would change, that inspectors would no longer be required; the plumbers themselves would be permitted to conduct self-inspections. He was philosophical about it, telling me he was ready to retire anyhow, and that he had a nice bit put aside.
It has been many years and a housing boom since—and plumbing in New York City does not seem to have suffered. Just the opposite.
Had I been involved in academia at the time of my talk with the inspector, and had I been so inclined, I could have told him that I understood completely what he was talking about. There’s not a book or an article that a serious academic can’t make look foolish. You don’t even have to be particularly dishonest, just look to the details and forget the whole, forget the purpose, forget the possible effectiveness. If you don’t like the conclusion, the scholarly trail, the particular school of thought or the place where the scholar works, you can even ignore those and hide your bias, making a case against the work solely on petty grounds.
Until recently, the scholar whose work you are reviewing had best make the right payment, just as my plumber had done with the inspector. The payment’s not in money but in conformity. Peer reviewers, particularly those conducting blind peer review for academic journals, are picked because of status in their fields. They are the ones who have already made it; they define what is legitimate and are rarely open to challenge from those who have not yet reached “peer” status—the state of many of those writing for such journals. We all know this, and understand the corruption. Yet we continue to participate in the system. Well, some of us do.
Last month, I was asked to review an article for a rather prestigious venue on a topic relating to one of my books. The general editor had suggested to the area editor that I be asked. It was something of a set-up: the article contained no reference to my book, even though it purported to be an overview of the specialization. Clearly, the general editor had noticed this. For whatever reason, that editor did not want a positive review; I could be pretty well guaranteed not to give one. So, the burden of rejection was being passed on to someone the author could never identify.
I refused the opportunity, pointing out the omission and saying I did not believe it would fair for me to review the article. The area editor wrote back, clearly surprised, but thanking me for being so honest. I got the feeling that my response was a rare one. To respond otherwise, however, would have been just as corrupt as that plumbing inspector.
I’ve also been on the other side recently. An article I wrote was panned by a blind reviewer, one who, for whatever reason, had taken umbrage. To make a point, I had illustrated a parallel progression in two American institutions; the reviewer scathingly pointed out that I made no convincing case that the two are the same. Quite rightly; I wasn’t trying to, for they were tangential. The reviewer then listed a number of statements I had made, claiming I had not substantiated them. Again, quite rightly—but I hadn’t been trying to. At no point did the reviewer address my argument or refute my claims. There was no advice for the author in the review, and none for the editor, except “reject.”
So, I wrote back to the editor—who had praised the piece before sending it for blind review—and withdrew it, though the second review was not yet in. I love the editorial process, and love comments that can help me improve what I write—but not this. So, I published the piece myself, on my blog…
where I’ve already had a scathing comment from the right-wing political agitator David Horowitz: “Aaron why do you continue to peddle this horseshit about me, which you know not to be true?” Frankly, though Horowitz addresses my points no more than did the blind reviewer, at least he is willing to do it in public. I can respect that: Horowitz is nothing if not open about his prejudices.
All of this brings me to my main topics: bottlenecks and the dark.
In an earlier time, when there was limited space for publication, a rigorous pre-publication peer-review process could have been justified—to some degree, at least. Keeping it blind, though, had even less justification. An attempt to allow openness and honesty without consequence, it just as often produced pettiness. Today, when it is possible for anyone to publish at any time, we don’t need either the bottleneck or the darkness, especially since the process is so easily corrupted. Just as advances have made plumbing self-inspection safe and efficient, we now have open enough and strong enough post-publication review possibilities to make blind peer-review prior to publication unnecessary. As we all know, whether we admit it or not, it continues simply because we have made appearance in peered-review journals the standard for advancement. It continues because our committees on re-appointment, tenure, and promotion want an easy benchmark. We have not yet institutionalized post-publication review, though why that is baffles me. Wouldn’t the number of citations, reviews, and other responses give as strong an indication of the value of an essay as its original venue? Stronger, I’d say.
Especially since, in a digital environment, even the published essay can be improved in light of comments and criticism. And should be.
In response to an article by Richard Smith in Breast Cancer Research, Developmental Neuropsychologist Deborah Bishop wrote of “the real function of peer review, which should be to offer advice to the editor and the author.”[1] Sure, but why should this not be post-publication, in a milieu where change, unlike in the days of reliance on print, is easy? Bishop’s criticism of peer review is that it is often an easy way for editors to avoid making decisions—and she is right. This is why I prefer writing book chapters: anthology editors are quite focused on their topics and don’t pass things off. The function of any editor, today, should be to help strengthen essays that he or she has selected, for whatever reason. This should be the case in journals as well. Certainly, a peer-review process with this in mind could still start before publication, but there’s no reason it should become a bottleneck or a means for evaluation from the dark.
Journals like Cheryl-Ball edited Kairos have developed open and productive systems of review, in this case a three-tier process revolving around a named editorial board and the clear purpose of working with creators to strengthen their work. But even Kairos, a respected journal presenting the best, most cutting-edge of its field, is sometimes looked at askance by those evaluating careers. It is not, after all, a traditional, blind-reviewed journal. It’s system, though superior to blind peer review, is still sometimes seen as suspect.
My plumbing inspector had made his money, legitimate and otherwise. There’s money in peer review, too—and not only for the relics of the past. Perhaps this is part of the problem. Companies like Sage and Routledge make a great deal off of the peer-reviewed journals they own and continue to protect. Zoë Corbyn asks:
have these gatekeepers for what counts as acceptable… become too powerful? Is the system of reward that has developed around them the best?...
Unpicking the power of academic and scholarly journals, with their estimated global turnover of at least $5 billion (£3 billion) a year, is a complex business. There are an estimated 25,000 scholarly peer-reviewed journals in existence….
It is these - particularly the elite titles… - that are at the heart of the recognition-and-reward system…. From career progression to grant income, "wealth" within the academy is determined by the production of… knowledge as recorded in peer-reviewed scholarly journals.[2]
The owners of the journals make their money from the fact that scholars are beholden to them for creation of that “’wealth’ within the academy.” Those with that “wealth” have too little incentive to give it up—and they are generally the ones evaluating the advancement of others.
On the other hand, as Richard Smith writes:
what happens after publication can also be called peer review, and that, I believe, is the peer review that really matters - the process whereby the world decides the importance and place of a piece of research…. Many studies are never cited once, most disappear within a few years, and very few have real, continuing importance.
And the correlation between what is judged important in pre-publication peer review and what has lasting value seems to be small…. Many papers get very high marks from their peer reviewers but have little effect on the field. And on the other hand, many papers get average ratings but have a big impact'[3]
So what to do? Well, we’re already doing it. Joe Pickrell suggests we aim for:
1. Immediate publication without peer review….
2. One-click recommendation of papers….
3. Connection to a social network….
4. Effective search based on the collective opinion on a paper.[4]
I would add in a number of modifications, such as something akin to the political blog Daily Kos’s “trusted user” status for readers of any particular journal. And I would spotlight social-networking sites like Academia.edu, where scholars can post their own work, connect, and search keywords for pieces that might interest them.
For anyone still wondering why blind peer-review should be jettisoned, Smith provides a list of reasons:
Firstly, it is very expensive in terms of money and academic time….
Secondly, peer review is slow….
Thirdly, peer review is largely a lottery.....
A fourth problem with peer reviews is that it does not detect errors….
The fifth problem with pre-publication peer review is bias….
Finally, peer review can be all too easily abused…. [5]
Frankly, I find it odd that reliance on blind peer-review continues, especially in career evaluation—but it does. My own institution has recently instituted a third-year deans’ review where one of the benchmarks is at least one peer-reviewed publication.
Still, blind peer review is dead. It just doesn’t know it yet. It certainly doesn’t warrant renovation; the structure is collapsing. The bottleneck it once created is no longer necessary. It The situation in this regard is much like that of journalism, just five or six years ago, when the blogs were first seriously challenging a profession that had become almost terminally inward-looking and almost completely averse to change or innovation. Though I saw—and still occasionally see—old-style journalism’s stalwarts pounding the podium, red-faced, claiming that the lack of gatekeeping they imagined in the challenge of the blogs would destroy journalism forever, that has not happened. If anything, the profession is much more vigorous today, more varied and experimental—even though many of its older structures have collapsed. The plumber I used a couple of years ago said much the same thing about the state of his profession, post paid inspectors.
Kairos and Academia.edu are only two examples of what will replace those 25,000 peer-reviewed journals if those journals don’t begin to change. The money and other wealth generated by peer review will dwindle, as will the numbers. What is now 25,000 will soon be 2,500, and those will be the one that have changed, that have embraced openness and digital possibilities and the new sorts of post-publication review that seem to pop up every day. As journalism has found, gatekeeping does not die when new venues can be established cheaply and by anyone. It simply changes. Aggregators funnel the best or the selected, citations rise in import, and choice, such as that represented on WorldCat.org of library inclusions, becomes a significant benchmark. A deliberate and controlled editorial bottleneck becomes irrelevant.
This is probably the last year a panel like this will seem necessary. No matter how much we try, academics cannot hold the fort any more than journalists were able to. What we have to do is adapt, or we will be superseded—and there are few of us who, like my plumbing inspector, are willing to be pulled down that route. Whether we like it or not, we are going to have to bear the light of openness—and quite soon, now.


[1] Deborah Bishop, “Comment on ‘Classical Peer Review: An Empty Gun’ by Richard Smith,’ Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13/comments.
[2] Zoë Corbyn, “A threat to scientific communication,” The Times Higher Education, August 13, 2009, http://www.timeshighereducation.co.uk/story.asp?storycode=407705.
[3] Richard Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.
[4] Joe Pickrell, “Why publish science in peer-reviewed journals?” Genomes Unzipped: Public Personal Genomics, July 13, 2011, http://www.genomesunzipped.org/2011/07/why-publish-science-in-peer-reviewed-journals.php.
[5] Richard Smith, “Classical Peer Review: An Empty Gun,” Breast Cancer Research, Volume 12, Supplement 4: Controversies in Breast Cancer Research 2010, http://breast-cancer-research.com/content/12/S4/S13.

Tuesday, January 03, 2012

What the Dickens?

As subway reading, I've turned to Victorian literature: Austen, Trollope, Eliot, Dickens.... Great stuff, and perfectly absorbing, an hour each way.

This morning, I was reading Charles Dickens' Dombey and Son and came across this:
'But he's chockful of science,' he observed, waving his hook towards the stock-in-trade. 'Look'ye here! Here's a collection of 'em. Earth, air, or water. It's all one. Only say where you'll have it. Up in a balloon? There you are. Down in a bell? There you are. D'ye want to put the North Star in a pair of scales and weigh it? He'll do it for you.'
It may be gathered from these remarks that Captain Cuttle's reverence for the stock of instruments was profound, and that his philosophy knew little or no distinction between trading in it and inventing it.
'Ah!' he said, with a sigh, 'it's a fine thing to understand 'em. And yet it's a fine thing not to understand 'em. I hardly know which is best. It's so comfortable to sit here and feel that you might be weighed, measured, magnified, electrified, polarized, played the very devil with: and never know how.'
Somehow, this seems more appropriate to today than to a century-and-a-half ago.