Despite the raising of voices over the years expressing disenchantment with law reviews (from Fred Rodell, Goodbye to Law Reviews, 23 Va. L. Rev. 38 (1936) to Jonathan Mermin, Remaking Law Review, 56 Rutgers L. Rev. 603 (2004)), their proliferation remains “an awe-inspiring scholastic phenomenon…. Currently there are over seven hundred active law reviews in the United States, of which nearly two hundred are general-interest and the remainder are specialized.” (Perry 1-2) The endeavor to rank law reviews has a similarly long history, dating back at least to 1930 (Perry fn. 8) and shows no signs of abating; nor does the effort to evaluate the ranking methods. (See the article reviewed herein as well as Gregory Scott Crespi, Judicial and Law Review Citation Frequencies for Articles Published in Different “Tiers” of Law Journals: An Empirical Analysis, 44 Santa Clara L. Rev. 897 (2004), among others.) We continue to ask the questions: how do scholars and librarians assess the quality of law reviews? And is the attempt to do so worthwhile?
According to Dr. Ronen Perry, L.L.D., in The Relative Value of American Law Reviews: A Critical Appraisal of Ranking Methods, 10 Virginia Journal of Law and Technology (2005), available at http://ssrn.com/abstract=806144, the ranking of law reviews, if carried out by proper methods, has important benefits to the scholarly and legal communities. These benefits include increased competition among law reviews leading to improved quality; assistance to legal scholars in deciding where to submit and to ultimately publish their scholarship; and benefit to law reviews themselves in providing a measure of their success or failure to be used in improving their product. Other advantages include the benefits that accrue to students who serve on the editorial boards of highly ranked law reviews and to their respective law schools; assistance to the consumers of legal literature in selecting which of the many published articles to read; usefulness to librarians in making acquisition decisions (although Dr. Perry recognizes that this factor is limited by the increased reliance upon electronic databases such as Lexis, Westlaw, or Hein Online for law review articles); and just the plain old American love of rankings of any kind. After all, don’t most of us feel compelled to look at the year-end “Top Ten” lists of books, movies, and music, the ubiquitous “best and worst dressed” lists, and just about anything of that nature that catches our eye?
Dr. Perry sets forth the basic requirements of a law review ranking methodology that would achieve the identified objectives. The ranking method must be based on a “quality-sensitive” set of criteria and be “responsive to quality changes” (Perry 8); it must be based on objective criteria; it must be practical; and it must be based on data that are “readily verifiable” and not prone to manipulation by interested parties. Having articulated these requirements, Dr. Perry ultimately concludes that there is no ranking method that satisfies them all. The bulk of his article is devoted to a thorough analysis and evaluation of the various ranking methods currently in use, which fall into three categories.
The first category - methods based on direct evaluations of quality such as evaluation by a committee of experts or a general survey of quality - is rejected by Dr. Perry based on such methods’ vulnerability to subjectivity and bias, as well as their impracticality based on both the sheer numbers of law reviews, even within specialized subject areas, and the difficulty of selecting a class of respondents.
The second category of methods, those based on indirect quality evaluation, presents its own set of weaknesses. For example, Dr. Perry rejects as entirely unreliable methods based on author prominence, article rejection rate, and law review editors’ academic aptitude. He also spurns, as inaccurate and subject to manipulation, journal usage studies based on library or electronic database statistics.
Dr. Perry also reasons that the third type - complex ranking methods that employ several quality measures - are also intrinsically flawed, as they do not transcend the weaknesses of each component measure. Further, because the determination of the weight assigned to each component measure involves subjective judgment, no complex ranking method can ever be wholly objective.
Dr. Perry devotes a large portion of his article to a comprehensive survey and evaluation of law review ranking methods based on citation analysis. He begins by explaining that the various impacts of a particular law review could be evaluated by measuring the frequency of its citation in sources other than law reviews: for example, in texts, by appellate or trial courts, and on law school reading lists. He goes on to debunk, in a debate-provoking fashion, various criticisms of citation-based rating methods. He also thoroughly explores the adjustments that must be made to a citation-based ranking system to control for the distorting factors of self-citation, cross-citation in symposia issues, the publication time span of the journals to be ranked, subject matter distribution, and publication volume. Dr. Perry contrasts the reliability of a total journal citation frequency measure with a standardized criteria based on citations-per-word method. He concedes that citation-based ranking methods seem to be inherently biased in favor of the journals of the most highly ranked law schools. Dr. Perry concludes, however, that citation analysis is the most promising method of assessing the impact of law reviews on academic and professional discourse, and posits that the best ranking criterion is “an adjusted and standardized citation rate.” (Perry 56, 66)
Along with its analysis of methodology, Dr. Perry’s article provides an excellent review of the literature on law review ranking methods, and opens the door to further inquiry about subject specialized journals and comparison to ranking methods used in other disciplines. It also broaches the topic - well worth exploring further - of the citation of law review articles outside the academic community, a topic provocatively addressed by David Hricik and Victoria S. Salzmann, Why There Should Be Fewer Articles Like This One: Law Professors Should Write More For Legal Decision-Makers and Less For Themselves, 38 Suffolk U. L. Rev. 761 (2005).