Art Imitates Life

Which of these texts is an April Fool joke? How can you tell? Links will be posted in a few days.

TEXT 1
The Federal Intelligence Service discovered a Ponzi scheme of academic citations lead by an unemployed particle physicist. A house search conducted in Berlin last week revealed material documenting the planning and administration of a profitable business of trading citations for travel reimbursement.

According to the Federal Intelligence Service, the hint came from researchers at Michigan University, Ann Arbor, who were analyzing the structure of citation networks in the academic community. In late 2010, their analysis pointed towards an exponentially growing cluster originating from a previously unconnected researcher based in Germany's capital. A member of the Ann Arbor group, who wants to remain unnamed, inquired about the biography of the young genius, named Al Bert, sparking such amount of activity. The researcher was easily able to find Dr. Bert scheduled for an unusual amount of seminars in locations all over the world, sometimes more than 4 per week. However, upon contacting the respective institutions, nobody could remember the seminars, which according to Prof. Dr. Dr. Hubert at The Advanced Institute is "Not at all unusual." The network researcher from Ann Arbor suspected Dr. Bert to be a fictitious person and notified the university whose email address Dr. Bert was still using.

It turned out Dr. Bert is not a fictitious person. Dr. Bert's graduated in 2006, but his contract at the university run out in 2008. After this, colleagues lost sight of Dr. Bert. He applied for unemployment benefits in October 2008. As the Federal Intelligence Service reported this Wednesday, he later founded an agency called 'High Impact' (the website has since been taken down) that offered to boost a paper's citation count. A user registered with an almost finished, but not yet published, paper and agreed to pay EUR 10 to Dr. Bert's agency for each citation his paper received above the author's average citation count at the time of registration. The user also agreed to cite 5 papers the agency would name. A registered user would earn EUR 10 for each recruitment of a new paper, possibly their own.

This rapidly created a growing network of researchers citing each others papers, and encouraged the authors to produce new papers, certain they would become well cited. Within only a few months, the network had spread from physics to other research fields. With each citation, Dr. Bert made an income. The algorithm he used to assign citations also ensured his own works became top cites. Yet, with many researchers suddenly having papers with several hundred citations above their previously average citation count, their fee went into some thousand dollars. On several instances Dr. Bert would suggest they invite him for a seminar at their institution and locate it in a non-existent room. He would then receive reimbursement for a fraudulent self-printed boarding pass, illegible due to an alleged malfunctioning printer.

Names of researchers subscribed to Dr. Bert's agency were not accessible at the time of writing.


TEXT 2

The Case of IJNSNS

The field of applied mathematics provides an illuminating case in which we can study such impact-factor distortion. For the last several years, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS) has dominated the impact-factor charts in the “Mathematics, Applied” category. It took first place in each year 2006, 2007, 2008, and 2009, generally by a wide margin, and came in second in 2005. However, as we shall see, a more careful look indicates that IJNSNS is nowhere near the top of its field. Thus we set out to understand the origin of its large impact factor.

In 2008, the year we shall consider in most detail, IJNSNS had an impact factor of 8.91, easily the highest among the 175 journals in the applied math category in ISI’s Journal Citation Reports (JCR). As controls, we will also look at the two journals in the category with the second and third highest impact factors, Communications on Pure and Applied Mathematics (CPAM) and SIAM Review (SIREV), with 2008 impact factors of 3.69 and 2.80, respectively. CPAM is closely associated with the Courant Institute of Mathematical Sciences, and SIREV is the flagship journal of the Society for Industrial and Applied Mathematics (SIAM).  Both journals have a reputation for excellence.

Evaluation based on expert judgment is the best alternative to citation-based measures for journals. Though not without potential problems of its own, a careful rating by experts is likely to provide a much more accurate and holistic guide  journal quality than impact factor or similar metrics. In mathematics, as in many fields, researchers are widely in agreement about which are the best journals in their specialties. The Australian Research Council recently released such an evaluation, listing quality ratings for over 20,000 peer reviewed journals across disciplines. The list was developed through an extensive review process involving learned academies (such as the Australian Academy of Science), disciplinary bodies (such as the Australian Mathematical Society), and many researchers and expert reviewers.11 This rating is being used in 2010 for the Excellence in Research Australia assessment initiative and is referred to as the ERA 2010 Journal List. The assigned quality rating, which is intended to represent “the overall quality of the journal,” is one of four values: 

• A*: one of the best in its field or subfield

• A: very high quality

• B: solid, though not outstanding, reputation

• C: does not meet the criteria of the higher tiers.

The ERA list included all but five of the 175 journals assigned a 2008 impact factor by JCR in the category “Mathematics, Applied”. Figure 1 shows the impact factors for journals in each of the four rating tiers. We see that, as a proxy for expert opinion, the impact factor does rather poorly. There are many examples of journals with a higher impact factor than other journals that are one, two, and even three rating tiers higher. The red line is drawn so that 20% of the A* journals are below it; it is notable that 51% of the A journals have an impact factor above that level, as do 23% of the B journals and even 17% of those in the C category. The most extreme outlier is IJNSNS, which, despite its relatively astronomical impact factor, is not in the first or second but, rather, third tier.

The ERA rating assigned its highest score, A*, to 25 journals. Most of the journals with the highest impact factors are here, including CPAM and SIREV, but, of the top 10 journals by impact factor, two were assigned an A, and only IJNSNS was assigned a B. There were 53 A-rated journals and 69 B-rated journals altogether. If IJNSNS were assumed to be the best of the B journals, there would be 78 journals with higher ERA ratings, whereas if it were the worst, its ranking would fall to 147. In short, the ERA ratings suggest that IJNSNS is not only not the top applied math journal but also that its rank should be somewhere in the range 75–150. This remarkable mismatch between reputation and impact factor needs an explanation.

Makings of a High Impact Factor

A first step to understanding IJNSNS’s high impact factor is to look at how many authors contributed substantially to the counted citations and who they were. The top-citing author to IJNSNS in 2008 was the journal’s editor-in-chief, Ji-Huan He, who cited the journal (within the two-year window) 243 times. The second top citer, D. D. Ganji, with 114 cites, is also a member of the editorial board, as is the third, regional editor Mohamed El Naschie, with 58 cites. Together these three account for 29% of the citations counted toward the impact factor.

For comparison, the top three citers to SIREV contributed only 7, 4, and 4 citations, respectively, accounting for less than 12% of the counted citations, nd none of these authors is involved in editing the journal. For CPAM the top three citers (9, 8, and 8) contributed about 7% of the citations and, again, were not on the editorial board.

Another significant phenomenon is the extent to which citations to IJNSNS are concentrated within the two-year window used in the impact factor calculation. Our analysis of 2008 citations to articles published since 2000 shows that 16% of the citations to CPAM fell within that two-year  window and only 8% of those to SIREV did; in contrast, 71.5% of the 2008 citations to IJNSNS fell within the two-year window. In Table 1, we show the 2008 impact factors for the three journals, as well as a modified impact factor, which gives the average number of citations in 2008 to articles the journals published not in 2006 and 2007 but in the preceding six years. Since the cited half-life (thetime it takes to generate half of all the eventual citations to an article) for applied mathematics is nearly 10 years,12 this measure is at least as reasonable as the impact factor. It is also independent, unlike JCR’s 5-Year Impact Factor, as its time period does not overlap with that targeted by the Journal.

Table 1.  2008 impact factor with normal 2006–7 window      Modified 2008 “impact factor” with 2000–5 window

IJNSNS 8.91 1.27

CPAM 3.69 3.46 S

IREV 2.8 10.4

2008 impact factors computed with the usual two-preceding-years window, and with a window going back eight years but neglecting the two immediately preceding.

Note that the impact factor of  JNSNS drops precipitously, by a factor of seven, when we consider a different citation window. By contrast, the impact factor of CPAM stays about the same, and that of SIREV increases markedly. One may simply note that, in distinction to the controls, the citations made to IJNSNS in 2008 greatly favor articles published in precisely the two years that are used to calculate the impact factor. Further striking insights arise when we examine the high-citing journals rather than high-citing authors. The counting of journal self-citations in the impact factor is frequently criticized, and indeed it does come into play in this case. In 2008 IJNSNS supplied 102, or 7%, of its own impact factor citations.

The corresponding numbers are 1 citation (0.8%) for SIREV and 8 citations (2.4%) for CPAM. The disparity in other recent years is similarly large or larger. However, it was Journal of Physics: Conference Series that provided the greatest number of IJNSNS citations. A single issue of that journal provided 294 citations to IJNSNS in the impact factor window, accounting for more than 20% of its impact factor. What was this issue? It was the proceedings of a conference organized by IJNSNS editor-in-chief He at his home university. He was responsible for the peer review of the issue. The second top-citing journal for IJNSNS was Topological Methods in Nonlinear Analysis, which contributed 206 citations (14%), again with all citations coming from a single issue. This was a special issue with Ji-Huan He as the guest editor; his co-editor, Lan Xu, is also on the IJNSNS editorial board. J.-H. He himself contributed a brief article to the special issue, consisting of three pages of text and thirty references. Of these, twenty were citations to IJNSNS within the impact-factor window. The remaining ten consisted of eight citations to He and two to Xu.

Continuing down the list of IJNSNS high-citing journals, another similar circumstance comes to light: 50 citations from a single issue of the Journal of Polymer Engineering (which, like IJNSNS, is published by Freund), guest edited by the same pair, Ji-Huan He and Lan Xu. However, third place is held by the journal Chaos, Solitons and Fractals, with 154 citations spread over numerous issues. These are again citations that may be viewed as subject to editorial influence or control. In 2008 Ji-Huan He served on the editorial board of CS&F, and its editor-in-chief was Mohamed El Naschie, who was also a coeditor of IJNSNS. In a highly publicized case, the entire editorial board of CS&F was recently replaced, but El Naschie remained coeditor of IJNSNS.

Many other citations to IJNSNS came from papers published in journals for which He served as editor, such as Zeitschrift für Naturforschung A, which provided forty citations; there are too many others to list here, since He serves in an editorial capacity on more than twenty journals (and has just been named editor-in-chief of four more journals from the newly formed Asian Academic Publishers). Yet another source of citations came from papers authored by IJNSNS editors other than He, which accounted for many more. All told, the aggregation of such editor-connected citations, which are time-consuming to detect, account for more than 70% of all the citations contributing to the IJNSNS impact factor.