Peer and non-peer review
Andrew Odlyzko
Digital Technology Center
University of Minnesota
http://www.dtc.umn.edu/~odlyzko
Fears about possible damage to the classical peer review system are
slowing down the evolution of scholarly communication. In particular,
they are inhibiting the development of freely accessible article archives.
I am convinced that these fears are unjustified. Although the peer review
system will change substantially with the spread of such archives, it will
change for the better. New modes of communication will offer more freedom
of choice to scholars, and will provide faster, more complete, and more
flexible feedback mechanism about the quality of available information.
A good overview of the history and current state of the peer review
system is provided by the book [1]. (See also the survey [2],
which does not cover the subject as deeply, but has the advantage of
being shorter and, even more important, is freely available online.)
The peer review system is really a collection of many different
systems, of uneven effectiveness. They guarantee neither correctness
nor novelty of the published results, even among the most selective and
prestigious journals. However, traditional peer review (with anonymous
referees selected by an editor evaluating submissions to a journal)
does perform a valuable screening function. Still, it is just a part
of the entire communication system. Evaluation of the merit of an
article is never truly complete, as sometimes historians will revisit
this question centuries after publication. It is the presence of such
self-correcting features in the entire scholarly communication system
that makes the deficiencies of the current peer review system tolerable.
However, it is natural to expect evolution to occur.
In the Gutenberg era of print journals, placing heavy reliance on
traditional peer review was sensible. Printing and distributing journals
was very expensive. Furthermore, providing additional feedback after
publication was hard and slow. Therefore it was appropriate to devote
considerable attention to minimizing the volume of published material,
and making sure it was of high quality. With the development of more
flexible communication systems, especially the Internet, we are moving
towards a continuum of publication. I have argued, starting with [3],
that this requires a continuum of peer review, which will provide feedback
to scholars about articles and other materials as they move along the
continuum, and not just in the single journal decision process stage.
We can already see elements of the evolving system of peer review in
operation.
Many scholars, including Stevan Harnad [4], one of the most prominent
proponents of open access archives, argue for a continuing strong role
for the traditional peer review system at the journal level. I have no
doubt that this system will persist for quite a while, since sociological
changes in the scholarly arena are very slow [5]. However, I do expect
its relative importance to decline. The reason is that there is a
continuing growth of other types of feedback that scholars can rely on.
This is part of the general trend (described in [6]) in which traditional
journals are continuing as before, but the main action is in novel and
often informal modes of communication that are growing much more rapidly.
The growing flood of information does require screening. Some of this
reviewing can be done by non-peers. Indeed, some of it has traditionally
been done by non-peers, for example in legal scholarship, where U.S. law
reviews are staffed by students. The growing role of interdisciplinary
research might lead to a generally greater role for non-peers in reviewing
publications. However, in most cases only peers are truly qualified
to review technical results. Still, peer evaluations can be obtained,
and increasingly are being obtained, much more flexibly than through the
traditional anonymous journal refereeing process. Some can come from
use of automated tools to harvest references to papers, in a much more
flexible and comprehensive way than the Science Citation Index provided
in the old days. Other, more up-to-date evaluations, can be obtained
from a variety of techniques, such as those described in [6].
An example of how evolving forms of peer review function is provided by
the recent proof that testing whether a natural number is prime (that
is, divisible only by 1 and itself) can be done fast. (The technical
term is in "polynomial time.") This had been an old and famous open
problem of mathematics and computer science. On Sunday, August 4, 2002,
Maninda Agrawal, Neeraj Kayal, and Nitin Saxena of the Indian Institute
of Technology in Kanpur sent out a paper with their astounding proof of
this result to several of the recognized experts on primality testing.
(Their proof was astounding because of its unexpected simplicity.)
Some of these experts responded almost right away, confirming the validity
of the proof. On Tuesday, August 6, the authors then posted the paper
on their Web site and sent out email announcements. This prompted many
additional mathematicians and computer scientists to read the paper, and
led to extensive discussions on online mailing lists. On Thursday, August
8, the New York Times carried a story announcing the result and quoting
some of the experts who had verified the correctness of the result.
Review by peers played a central role in this story. The authors first
privately consulted known experts in the subject. Then, after getting
assurance they had not overlooked anything substantial, they made their
work available worldwide, where it attracted scrutiny by other experts.
The New York Times coverage was based on the positive evaluations
of correctness and significance by those experts. Eventually the
authors will submit their paper to a conventional journal, where it
will undoubtedly undergo conventional peer review, and be published.
The journal version will probably be the main one cited in the future,
but will likely have little influence on the development of the subject.
Within weeks of the distribution of the Agrawal-Kayal-Saxena article,
improvements on their results had been obtained by other researchers,
and future work will be based mainly on those. Agrawal, Kayal, and
Saxena will get proper credit for their breakthrough. However, although
their paper will go through the conventional journal peer review and
publication system, that will be almost irrelevant for the intellectual
development of their area.
One can object that only potentially breakthrough results are likely
to attract the level of attention that the Agrawal-Kayal-Saxena result
attracted. But that is not a problem. It is only the most important
results that require this level of attention and at this rapid a rate.
There will be a need for some systematic scrutiny of all technical
publications, to ensure that the literature does not get polluted to
erroneous claims. However, we should expect a much more heterogeneous
system to evolve, in which many of the ideas mentioned in [3] will play
a role. For example, the current strong prohibition of simultaneous
publication in multiple journals is likely to be discarded as another
relic of the Gutenberg era when print resources were scarce. Also,
we are likely to see separate evaluations of significance and correctness.
This note is a personal perspective on how peer review is likely to evolve
in the future. It is based primarily on my experience in areas such
as mathematics, physics, computing, and some social sciences. However,
I believe there is nothing special about those areas. Although health
sciences have moved towards electronic publishing more slowly than the
fields I am familiar with, I do not see much that is special about
their needs. In particular, I believe that the frequently voiced
concerns about need for extra scrutiny of research results that might
affect health practices are a red herring. Yes, decision about medical
procedures or even diet should be based on solidly established research.
However, the extra levels of scrutiny are more likely to be obtained by
more open communication and review systems than we have today.
[1] F. Godlee and T. Jefferson, eds., "Peer Review in Health Sciences,"
BMJ Books, 1999.
[2] F. Rowland, The peer-review process, Learned Publishing, vol. 15,
no. 4, Oct. 2002, pp. 247-258. Available online at
.
[3] A. M. Odlyzko, Tragic loss or good riddance? The impending demise
of traditional scholarly journals, Intern. J. Human-Computer Studies
(formerly Intern. J. Man-Machine Studies) 42 (1995), pp. 71-122.
Available online at .
[4] S. Harnad, The invisible hand of peer review, Exploit Interactive,
issue 5, April 2000. Available online at
and
.
[5] A. M. Odlyzko, The slow evolution of electronic publishing, in
"Electronic Publishing '97: New Models and Opportunities," A. J. Meadows
and F. Rowland, eds., ICCC Press, 1997, pp. 4-18. Available online at
.
[6] A. M. Odlyzko, The rapid evolution of scholarly communication,
Learned Publishing, 15(1) (Jan. 2002), pp. 7-19. Available online
at and
.