What do you do if you hold a quality initiative and nobody comes?

The Higher Education Commission of  Pakistan is proposing to rank all private and public universities in the country. Unfortunately, the universities do not seem very keen on the idea and most of them are not submitting data.


"An official of HEC told APP that all public and private Higher Education Institutions (HEIs) were asked to submit their data by July 15 for carrying out ranking process but around 10 out of 132 universities have submitted their data. HEC is taking the initiative of ranking the universities to help strengthen their indigenous quality culture and improve international visibility. The HEC has already directed the universities to meet the deadline for providing authentic data and those which failed to provide data will be ranked zero by placing them at the bottom in the ranking list to be published through print media.

HEC’ initiative to carry out quality based ranking of Higher Education Institutions (HEIs) is aimed at international compatibility, primarily based on the QS Ranking System acceptable widely across countries. The commission has taken various initiatives to bring HEIs of Pakistan at par with international standards and ranking is one of the measures to scale the success of efforts to achieve international competitiveness in education, research and innovation."


The problem of conscientious objectors and of universities that might simply not be able to collect data is one that has plagued global and national rankers from the beginning. Times Higher and Thomson Reuters allow universities to opt out but that is risky if those opting out include the likes of Texas at Austin. On the other hand, QS will collect data from third party and national sources if universities fail to cooperate.

What Global Rankings Ignore
(at least some of them)

Inside Higher Ed has an article by Indira Samarasekera, president and vice-chancellor of the University of Alberta, that voices some fairly conventional complaints about international university rankings. She has some praise for two of the rankers:

"The problems with national and international rankings are numerous and well known. So well known, in fact, that the world’s most powerful ranking organizations — the World’s Best Universities Rankings conducted by U.S. News & World Report in partnership with Quacquarelli Symonds and the Times Higher Education Rankings — have been working diligently to revise ranking measures and their methods in an attempt to increase the accuracy and objectivity of the rankings.

It should be pointed out that U.S. News & World report does not conduct any world rankings: it just publishes those prepared by QS. And I wonder how successful those diligent attempts will be.

She goes on:

"From my perspective, rankings are also missing the mark by failing to shine a light on some of the most significant benefits that universities bring to local, national and global societies. The focus of most rankings is on academic research outputs — publications, citations and major awards — that stand in as proxies for research quality and reach. While these outputs do a fairly good job of pinpointing the impact of a university’s contributions to knowledge, especially in science, technology, engineering and health sciences, they provide little indication of what kind of impact these advancements have on factors that the global community generally agrees are markers of prosperous and secure societies with a high quality of life.

Let me give you an example of what I mean: governments and policy makers everywhere now consider universities as economic engines as well as educational institutions. Public investments in research are increasingly directed toward research with the potential to translate into products, processes and policies — even whole new industries. This trend in research funding reveals a lot about the ways in which universities matter to governments, policy makers, regions and the public today, but the rankers aren’t paying attention.

Consider Israel. According to data on NASDAQ’s website, Israel has more companies listed on the NASDAQ stock exchange than any other country in the world except the U.S., and major companies such as Intel, Microsoft, IBM and Google have major research and development centers in Israel. Why? If you look at the data, you see a correlation between this entrepreneurial activity and the investments in and outputs from Israel’s universities.

Israel is among a handful of nations with the highest public expenditure on educational institutions relative to GDP, and it has the highest rate of R&D investment relative to GDP in the world. It also has the highest percentage of engineers in the work force and among the highest ratio of university degrees per capita. Many of the companies listed on NASDAQ were started by graduates of Israel’s universities: Technion, Tel Aviv University, Weizmann Institute and Hebrew University of Jerusalem, to mention a few. Do international university rankings capture these economic impacts from research and postsecondary education in Israel? The answer is no. In spite of their tremendous impact and output, Israel’s universities are ranked somewhere in the 100 to 200 range."

In fact, the Shanghai rankings had the Hebrew University of Jerusalem in 72nd position in 2010 and the percentage of Israeli universities in the Shanghai 500 was higher than any other country. So, the vice-chancellor's logic leads to the conclusion that Shanghai does at a better job at capturing this aspect of excellence than QS or THE.

Tel Aviv University and the Hebrew University of Jerusalem were not in the THE 200 or indeed the THE top 400. What happened is that Thomson Reuters either did not receive or did not ask for the information.

This WUR had such promise

The new Times Higher Education World University Rankings of 2010 promised much, new indicators based on income, a reformed survey that included questions on postgraduate teaching, a reduction in the weighting given to international students.

But the actual rankings that came out in September were less than impressive.  Dividing the year's intake of undergraduate students by the total of academic faculty looked rather odd. Counting the ratio of doctoral students to undergraduates, while omitting masters programs, was an invitation to the herding of marginal students into substandard doctoral degree programmes.

The biggest problem though was the insistence on giving a high weighting – somewhat higher than originally proposed -- to citations. Nearly a third of the total weighting was assigned to the average citations per paper normalized by field and year. The collection of statistics about citations is the bread and butter of Thomson Reuters (TR), THE’s  data collector, and one of their key products is the Incites system, which apparently was the basis for their procedure during the 2010 ranking exercise. This compares the citation records of academics with international scores benchmarked by year and field. Of course, those who want to find out exactly where they stand have to find out what the benchmark scores are and that is something that cannot be easily calculated without Thomson Reuters.

Over the last two or three decades the number of citations received by papers, along with the amount of money attracted from funding agencies, has become an essential sign of scholarly merit. Things have now reached the point where, in many universities, research is simply invisible unless it has been funded by an external agency and then published in a journal noted for being cited frequently by writers who contribute to journals that are frequently cited. The boom in citations has begun to resemble classical share and housing bubbles as citations acquire an inflated value increasingly detached from any objective reality.

It has become clear that citations can be manipulated as much as, perhaps more than, any other indicator used by international rankings. Writers can cite themselves, they can cite co-authors, they can cite those who cite them. Journal editors and reviewers can  make suggestions to submitters about who to cite. And so on.

Nobody, however, realized quite how unrobust citations might become until the unplanned intersection of THE’s indicator and a bit of self citation and mutual citation by two peripheral scientific figures raised questions about the whole business.

One of these two was Mohamed El Naschie who comes from a wealthy Egyptian family. He studied in Germany and took a Ph D in engineering at University College London. Then he taught in Saudi Arabia while writing several papers that appear to have been of an acceptable academic standard although not very remarkable. 

But this was not enough. In 1993 he started a new journal dealing with applied mathematics and theoretical physics called Chaos, Solitons and Fractals (CSF), published by the leading academic publishers, Elsevier. El Naschie’s journal published many papers written by himself. He has, to his credit, avoided exploiting junior researchers or insinuating himself into research projects to which he has contributed little. Most of his papers do not appear to be research but rather theoretical speculations many of which concern the disparity between the mathematics that describes the universe and that which describes subatomic space and suggestions for reconciling the two.

Over the years El Naschie has listed a number of universities as affiliations. The University of Alexandra was among the most recent of them. It was not clear, however, what he did at or for the university and it was only recently, after the publication of the 2010 THE World University Rankings, that there is documentation of any official connection.

El Naschie does not appear to be highly regarded by physicists and mathematicians, as noted earlier in this blog,  and he has been criticized severely in the physics and mathematics blogosphere.  He has, it is true, received some very vocal support but he is not really helped by the extreme enthusiasm and uniformity of style of his admirers. Here is a fairly typical example, from the comments in Times Higher Education: 

“As for Mohamed El Naschie, he is one of the most original thinkers of our time. He mastered science, philosophy, literature and art like very few people. Although he is an engineer, he is self taught in almost everything, including politics. Now I can understand that a man with his charisma and vast knowledge must be the object of envy but what is written here goes beyond that. My comment here will be only about what I found out regarding a major breakthrough in quantum mechanics. This breakthrough was brought about by the work of Prof. Dr. Mohamed El Naschie”

Later, a professor at Donghua University, China, Ji-Huan He, an editor at El Naschie’s  journal, started a similar publication, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS), whose editorial board included El Naschie. This journal was published by the respectable and unpretentious Israeli company, Freund of Tel Aviv. Ji-Huan He’s journal has published 29 of his own papers and 19 by El Naschie. The  two journals have contained articles that cite and are cited by articles in the other. Since they deal with similar topics some degree of cross citation is to be expected but here it seems to be unusually large.

Let us look at how El Naschie worked. An example is his paper, ‘The theory of Cantorian spacetime and high energy particle physics (an informal review)’, published in Chaos, Solitons and Fractals,41/5, 2635-2646, in  September  2009.

There are 58 citations in the bibliography. El Naschie cites himself 24 times, 20 times to papers in Chaos, Solitons and Fractals and 4 in IJNSNS.  Ji-Huan He is cited twice along with four  other authors from CSF. This paper has been cited 11 times, ten times in CSF in issues of the journal published later in the year.

Articles in mathematics and theoretical physics do not get cited very much. Scholars in those fields prefer to spend time thinking about an interesting paper before settling down to comment. Hardly any papers get even a single citation in the same year. Here we have 10 for one paper. That might easily be 100 times the average for that discipline and that year.

The object of this exercise had nothing to do with the THE rankings. What it did do was to push El Naschie’s  journal into the top ranks of scientific journals as measured by the Journal Impact Factor, that is the number of citations per paper within a two year period. It also meant that for a brief period El Naschie was listed by Thomson Reuters’ Science Watch as a rising star of research.

Eventually, Elsevier appointed a new editorial board at CSF that did not include El Naschie. The journal did however continue to refer to him as the founding editor. Since then the number of citations has declined sharply.

Meanwhile, Ji-huan  He was also accumulating a large number of citations, many of them from conference proceedings that he had organized. He was launched into the exalted ranks of the ISI Highly Cited Researchers and his journal topped the citation charts in mathematics. Unfortunately, early this year Freund sold off its journals to the reputed German publishers De Gruyter, who appointed a new editorial board that did not include either him or El Naschie.

El Naschie, He and a few others have been closely scrutinized by Jason Rush, a mathematician formerly of the University of Washington. Rush was apparently infuriated by El Naschie s unsubstantiated claims to have held senior positions at a variety of universities including Cambridge, Frankfurt, Surrey and Cornell. Since 2009 he has closely, perhaps a little obsessively, published a blog that chronicles the activities of El Naschie and those associated with him. Most of what is known about El Naschie and He was unearthed by his blog, El Naschie Watch.

Meanwhile, Thomson Reuters were preparing their analysis of citations for the THE rankings. They used the Incites system and compared the number of citations with benchmark scores representing the average for year and field.

This meant that for this criterion a high score did not necessarily represent a large number of citations. It could simply represent more citations than normal in a short period of time in fields where citation was infrequent and, perhaps more significantly since we are talking about averages here, a small total number of publications. Thus, Alexandria, with only a few publications but listed as the affiliation of an author who was cited much more frequently than usual in theoretical physics or applied mathematics, did spectacularly well.


This is rather like declaring Norfolk (very flat according to Oscar Wilde) the most mountainous county in England because of a few hillocks that were nonetheless relatively much higher than the surrounding plains.

Thomson Reuters would have done themselves a lot of good if they had taken the sensible course of using several indicators of research impact, such as total citations, citations per faculty, the h-index or references in social media or if they had allocated a smaller weighting to the indicator or if they had imposed a reasonable  threshold number of publications instead of just 50 or if they had not counted self-citations, or citations within journals or if they had figured out a formula to detect mutual citations..

So, in September  THE published its rankings with University of Alexandria in the top 200 overall and in fourth place for research impact, ahead of Oxford, Cambridge and most of the Ivy league. Not bad for a university that had not even been counted by HEEACT, QS or the Shanghai rankings and that in 2010 had lagged behind two other institutions in Alexandria itself in Webometrics.

When the rankings were published THE pointed out that Alexandria had once had a famous library and that a former student had gone on to the USA to eventually win a Nobel prize decades later. Still, they did concede that the success of Alexandria was mainly due  to one "controversial" author.

Anyone with access to the Web of Science could determine in a minute precisely who the controversial author was. For a while it was unclear exactly how a few dozen papers and a few hundred citations could put Alexandria among the world’s elite. Some observers wasted time wondering if  Thomson Reuters had been counting papers from a community college in Virginia or Minnesota, a branch of the Louisiana State University or federal government offices in the Greater Washington area. Eventually, it was clear that El Naschie could not, as he himself asserted, have done it by himself: he needed the help of the very distinctive features of Thomson Reuters’ methodology.

There were  other oddities in the 2010 rankings. Some might have accepted a high placing for Bilkent University in Turkey. It was well known for its Academic English programs. It also had one much cited article whose apparent impact was increased because it was classified as multidisciplinary, usually a low cited category, thereby scoring well above the world benchmark. However, when regional patterns were analyzed, the rankings began to look rather strange, especially the research impact indicator. In Australia, the Middle East, Hong Kong and Taiwan the order of universities, looked rather different from what local experts expected. Hong Kong Baptist University the third best in the SAR? Pohang University of Science and Technology so much better than Yonsei or KAIST? Adelaide the fourth best Australian university?

In the UK or the US these placings might seem plausible or at least not worth bothering about. But in the Middle East the idea of Alexandria as top university even in Egypt is a joke and the places awarded to the others look very dubious.

THE and Thomson Reuters tried to shrug off the complaints by saying that there were just a few outliers which they were prepared to debate and that anyone who criticized them had a vested interest in the old THE-QS rankings which had been discredited. They  dropped hints that the citations indicator would be reviewed but so far nothing specific has emerged.

A few days ago, however,  Phil Baty of THE seemed to imply that there was nothing wrong with the citations indicator.

Normalised data allow fairer comparisons, and that is why Times Higher Education will employ it for more indicators in its 2011-12 rankings, says Phil Baty.

One of the most important features of the Times Higher Education World University Rankings is that all our research citations data are normalised to take account of the dramatic variations in citation habits between different academic fields.

Treating citations data in an “absolute manner”, as some university rankings do, was condemned earlier this year as a “mortal sin” by one of the world’s leading experts in bibliometrics, Anthony van Raan of the Centre for Science and Technology Studies at Leiden University. In its rankings, Times Higher Education gives most weight to the “research influence” indicator – for our 2010-11 exercise, this drew on 25 million citations from 5 million articles published over five years. The importance of normalising these data has been highlighted by our rankings data supplier, Thomson Reuters: in the field of molecular biology and genetics, there were more than 1.6 million citations for the 145,939 papers published between 2005 and 2009; in mathematics, however, there were just 211,268 citations for a similar number of papers (140,219) published in the same period.

To ignore this would be to give a large and unfair advantage to institutions that happen to have more provision in molecular biology, say, than in maths. It is for this crucial reason that Times Higher Education’s World University Rankings examine a university’s citations in each field against the global average for that subject.

Yes, but when we are assessing hundreds of universities in very narrowly defined fields we start running into quite small samples that can be affected by deliberate manipulation or by random fluctuations.

Another point is that if there are many more journals, papers, citations and grants in oncology or genetic engineering than in the spatialization of gender performativity or the influence of Semitic syntax on Old Irish then perhaps society is telling us something about what it values and that is something that should not be dismissed so easily.

So, it could be  we are going to get the University of Alexandria in the top 200 again, perhaps joined by Donghua university.

At the risk of being repetitive, there are a few simple  things that Times Higher  and TR could do to make the citations indicator more credible. There are also  more ways of measuring research excellence.Possibly they are thinking about them but so far there is no sign  of this.

The credibility of last year's rankings has  declined further with the decisions of the judge presiding over the libel case brought by El Naschie against Nature (see here for commentary). Until now it could be claimed that El Naschie was a wll known scientist by virtue of the large numbers of citations that he had received or at least an interesting and controversial maverick.

El  Naschie is pursuing a case against  Nature for publishing an article that suggested his writings were not of a high quality and that those published in his journal did not appear to be properly peer reviewed

The judge has recently ruled  ruled that  El Naachie cannot proceed with a claim for specific damages since he has not brought any evidence for this. He can only go ahead with a claim for general damages for loss of reputation and hurt feelings. Even here, it looks like it will be tough going. El Naschie seems to be unwilling or unable to find expert witnesses to testify to the scientific merits of his papers.

"The Claimant is somewhat dismissive of the relevance of expert evidence in this case, largely on the basis that his field of special scientific knowledge is so narrow and fluid that it is difficult for him to conceive of anyone qualifying as having sufficient "expert" knowledge of the field. Nevertheless, permission has been obtained to introduce such evidence and it is not right that the Defendants should be hindered in their preparations."

He also seems to have problems with locating records that would demonstrate that his many articles published in Chaos, Solitons and Fractals were adequately reviewed.

46. The first subject concerns the issue of peer-review of those papers authored by the Claimant and published in CSF. It appears that there were 58 articles published in 2008. The Claimant should identify the referees for each article because their qualifications, and the regularity with which they reviewed such articles, are issues upon which the Defendants' experts will need to comment. Furthermore, it will be necessary for the Defendants' counsel to cross-examine such reviewers as are being called by the Claimant as to why alleged faults or defects in those articles survived the relevant reviews.
    
    
47. Secondly, further information is sought as to the place or places where CSF was administered between 2006 and 2008. This is relevant, first, to the issue of whether the Claimant has complied with his disclosure obligations. The Defendants' advisers are not in a position to judge whether a proportionate search has been carried out unless they are properly informed as to how many addresses and/or locations were involved. Secondly, the Defendants' proposed expert witnesses will need to know exactly how the CSF journal was run. This information should be provided.

It would therefore  seem to be getting more and more difficult for anyone to argue that TR's methodology has uncovered a pocket of excellence in Alexandria.

Unfortunately, it is beginning to look as though THE will not only use much the same method as last time but will apply normalisation to other indicators as well.

But what about the other performance indicators used to compare institutions? Our rankings examine the amount of research income a university attracts and the number of PhDs it awards. For 2011-12, they will also look at the number of papers a university has published that are co-authored by an international colleague.

Don’t subject factors come into play here, too? Shouldn’t these also be normalised? We think so. So I am pleased to confirm that for the 2011-12 World University Rankings, Times Higher Education will introduce subject normalisation to a range of other ranking indicators.
This is proving very challenging. It makes huge additional demands on the data analysts at Thomson Reuters and, of course, on the institutions themselves, which have had to provide more and richer data for the rankings project. But we are committed to constantly improving and refining our methodology, and these latest steps to normalise more indicators evidence our desire to provide the most comprehensive and rigorous tables we can.

What this might mean is that universities that spend modest amounts of money in fields where little money is usually spent would get a huge score. So what would happen if an eccentric millionaire left millions to establish a lavishly funded research chair in continental philosophy at Middlesex University?  There are no doubt precautions that Thomson Reuters could take but will they? The El Naschie business does not inspire very much confidence that they will.

The reception of the 2010 THE WUR rankings suggests that the many in the academic world have doubts about the wisdom of using normalised citation data without considering the potential for gaming or statistical anomalies. But the problem may run deeper and involve citations as such. QS, THE 's rival and former partner, have produced a series of subject rankings based on data from 2010. The overall results for each subject are based on varying combinations of the scores for academic opinion, employer opinion and citations per paper (not per faculty as in the general rankings).

The results are interesting. Looking at citations per paper alone we see that Boston College and Munich are jointly first in Sociology. Rutgers is third for politics and international studies. MIT is third for philosophy (presumably Chomsky and co). Stellenbosch is first for Geography and Area studies. Padua is first for linguistics. Tokyo Metropolitan University is second for biological sciences and Arizona State University first.

Pockets of excellence or statistical anomalies? These results may not be quite as incredible as Alexandria in the THE rankings but they are not a very good advertisement for the validity of citations as a measure of research excellence.

It appears that THE have not made their minds up yet. There is still time to produce a believable and rigorous ranking system. But whatever happens, it is unlikely that citations,  normalized or unnormalized, will continue to be the unquestionable gold standard of academic and scientific research.

The Coming Ascendancy of China

Matthew Reisz in Times Higher Education reports that Wen Jiabao, Prime Minister of China, has been awarded the King Charles II medal by the Royal Society for an ambitious national research program.

"The scale and success of Chinese investment in research was reflected in findings released last month by Thomson Reuters - drawing on data collected for the Times Higher Education World University Rankings - which showed that the country's elite C9 League now generates more income per academic staff member than the UK's Russell Group.

The top Chinese universities also award the highest number of doctoral degrees per academic.
Despite a vast increase in output over the past decade, there has been no discernible dip in standards, and the quality of the research produced by Chinese universities has remained at about the world average."

Some warnings are necessary. It is debatable whether the generation of research income is always a good indicator of quality. For one thing, note that the report talks about "per academic staff". Getting rid of or forgetting about "unproductive" departments like philosophy or languages could boost scores as easily as getting grants.

Still, it seems likely that the Chinese are on the way to scientific supremacy, at least in the natural sciences. There are obstacles ahead such as centralised control that might one day slow down the growth of research. They might even  read  Times Higher and stop being so unpleasantly aggressive and competitive, but at the moment that seems unlikely.

Incidentally, do the data collected for the THE World University Rankings tell us anything that we couldn't learn from the Shanghai rankings?

QS Subject Rankings for the Social Sciences

QS have released their subject rankings for the social sciences based on data gathered during last year's rankings.

The overall rankings are not surprising. Here are top three in each subject.

Sociology
1.  Harvard
2.  UC Berkeley
3.  Oxford

Statistics and Operational Research
1.  Stanford
2.  Harvard
3.  UC Berkeley

Politics and International Studies
1.  Harvard
2.  Oxford
3.  Cambridge

Law
1.  Harvard
2.  Oxford
3.  Cambridge

Economics and Econometrics
1.  Harvard
2.  MIT
3. Stanford

Accounting and Finance
1.  Harvard
2.  Oxford
3.  MIT

The top three in the citations per paper indicator is, in most cases, rather different. Are these pockets of excellence or something else?

Sociology
1=  Boston College
1=  Munich
3.   Florida State University

Statistics and Operational Research
1.  Aarhus
2.  Helsinki
3.  Erasmus University Rotterdam

Politics and International Studies
1.  Yale
2.  Oslo
3.  Rutgers

Law
1.  Victoria University of Wellington
2.  Koln
3.  Munster

Economics and Econometrics
1.  Dartmouth
2.  Harvard
3.  Princeton

Accounting and Finance
1.  University of Pennsylvania
2=  Harvard
2=  North Carolina at Chapel Hill

Worth Reading

Andrejs Rauhvargers,  Global University Rankings and their Impact (European University Association).
Rauhvargers

Interesting News

U.S. News are getting ready to start ranking American online colleges.

The THE Survey

Times Higher Education and its partner Thomson Reuters have announced the completion of their survey of academic opinion. There were 17,554 responses from 137 countries, nearly a third more than last year. That means nearly 31,000 responses over the last two years but THE, in contrast to their rivals, QS, will only count responses to this year's survey.

QS have still not closed their survey so it looks as though they might well be push the number of responses to their survey over 17,500 and claim victory. THE, no doubt, will point out that all of their respondents are new ones and that QS are counting respondents from 2010 and 2009.

THE have indicated the number of responses but not the number of survey forms that were sent out. So, the response rate for the survey is still unknown. This is more important for judging the validity of the survey than just the number of responses.

QS Latin American Rankings

QS have published the results of their preliminary study for a Latin American university ranking. This would be the second in their series of regional rankings after the Asian rankings, now in the third year.

The methodology suggested by the rankings is as follows:

Latin American Academic Reputation  30%
Papers per Faculty  10%
Citations per Paper 10%
Student Faculty Ratio 10%
Staff with Ph D 10%
Latin American Employer Reputation 20%
International Faculty 2.5%
International Students 2.5%
Inbound Exchange Students 2.5%
Outbound Exchange Students  2.5%

QS's surveys have been criticised on several grounds, including low response rates. However, the employer survey is valuable as an external  assessment of universities, while the academic survey might be considered a complement to citations-based indicators which in both the THE and QS rankings have thrown up some odd results.

There are two indicators that are directly research based. The apparent ease with which citations can be manipulated means that a variety of indicators could be used here, including citations per paper, h-index, total publications and citations, proportion of funded research and publications in high impact journals.QS have missed an opportunity here.

Student faculty ratio is allocated 10% instead of 20 % as in the international ranking. This is an admittedly crude proxy for teaching quality. QS are apparently experimenting with a student satisfaction survey which might produce more valid results.

Ten per cent goes to the proportion of staff with Ph Ds. This may well encourage the further and pointless over-production of substandard doctorates.

Five per cent goes to international students and international faculty. I am not sure that this will mean very much especially in the smaller Central American republics. Counting exchange students is definitely not a good idea. This is something that can be easily manipulated. In the Asian rankings there were some large and puzzling increases in the numbers of exchange students between 2009 and 2010.

The QS Arts and Humanities Rankings

See here for the complete rankings.

Here are the top five in each indicator, academic survey, employer survey, citations per paper of the QS subject rankings.

There is nothing surprising about the leaders in the two surveys. But the citations indicator is another matter. Perhaps, QS has followed Times Higher in uncovering "clear pockets of excellence". Would any specialists out there like to comment on Newcastle University (the English one, not the Australian) and Durham as first for history -- something to do with proximity to Hadrian's Wall? What about Brown for Philosophy, Stellenbosch for Geography and Area Studies and Padua for linguistics?

English Language and Literature
Academic survey
1.  Harvard
2.  Oxford
3.  Cambridge
4.  UC Berkeley
5.  Yale

Employer Survey
1.  Oxford
2.  Cambridge
3.  Harvard
4.  MIT
5.  UC Los Angeles

No ranking for citations

Modern Languages
Academic Survey
1.  Harvard
2,  UC Berkeley
3.  Oxford
4.  Cambridge
5.  Cornell

Employer Survey
1.  Harvard
2.  Oxford
3.  Cambridge
4.  MIT
5.  Stanford

No rankings for citations

History
Academic Survey
1.  Harvard
2.  Cambridge
3.  Oxford
4.  Yale
5.  UC Berkeley

Employer Survey
1. Oxford
2.  Harvard
3.  Cambridge
4.  University of Pennsylvania
5. Yale

Citations per Paper
1=  Newcastle (UK)
1=  Durham
3.   Liverpool
4.   George Washington
5.   University of Washington

Philosophy
Academic Survey
1.  Oxford
2.  Harvard
3.  Cambridge
4.  UC Berkeley
5.  Princeton

Employer Survey
1.  Cambridge
2.  Harvard
3.  Oxford
4.  MIT
5.  UC Berkeley

Citations per Paper
1.  Brown
2.  Melbourne
3.  MIT
4=  Rutgers
4=  Zurich


Geography and Area Studies
Academic survey
1.  UC Berkeley
2.  Cambridge
3.  Oxford
4.  Harvard
5.  Tokyo

Employer Survey
1.  Harvard
2.  Cambridge
3.  Oxford
4.  MIT
5.  UC Berkeley

Citations per Paper
1.  Stellenbosch
2. Lancaster
3.  Durham'
4.  Queen Mary London
5.  University of Kansas


Linguistics
Academic Survey
1.  Cambridge
2.  Oxford
3.  Harvard
4.  UC Berkeley
5.  Stanford

Employer Survey
1.  Harvard
2.  Oxford
3.  MIT
4.  UC Berkeley
5.  Melbourne

Citations per Paper
1.  Padua
2.  Boston University
3.  York University (UK)
4.  Princeton
5.  Harvard

Asia: Japan Falling, Korea and China Rising

See my article on the QS Asian Rankings 2011  in University World News

The QS Life Sciences Ranking Continued

Looking at the scores for the three indicators, academic survey, employer survey and citations per paper, we find the situation is similar to that of the engineering rankings released last month. There is a reasonably high correlation between the scores for the two surveys:

Medicine                     .720
Biological Sciences      .747
Psychology                  .570

The correlations between the score for citations per paper and the academic survey are low but still significant:
Medicine                          .290
Biological Sciences           .177
Psychology                       .217

The correlations between the indicator citations and the employer survey are low or very low and insignificant:
Medicine                               .129
Biological Sciences                .015 
Psychology                           -027


Looking at the top five universities for each indicator, there are no surprises as far as the surveys are concerned but some of the universities in the top five for citations do cause some eyebrow raising. Arizona State university? University of Cinncinati? Tokyo Metropolitan University? Perhaps these are hitherto unnoticed pockets of excellence of the Alexandrian kind?

Top Five in Medicine

Academic Survey

1.    Harvard
2.    Cambridge
3.    Oxford
4.    Stanford
5.    Yale

Employer Survey

1.     Harvard
2.     Cambridge
3.     Oxford
4.     MIT
5.     Stanford

Citations per Paper 

1.    MIT
2.    Rockefeller University
3.    Caltech
4.    The University of Texas M. D. Anderson Cancer Center
5.     Harvard


Top Five in Biological Sciences

Academic Survey

1.    Cambridge
2.    Harvard
3.    UC Berkeley
4.    Oxford
5.    MIT

Employer Survey


1.  Harvard
2.  Cambridge
3.  MIT
4.  Oxford
5.  Stanford

Citations per Paper

1.  Arizona State university
2.   Tokyo Metropolitan University
3.   MIT
4.   Rockefeller University
5.   Harvard

Top Five in Psychology

Academic Survey

1.    Harvard
2.   Stanford
3.    UC Berkeley
4.    Cambridge
5.    Oxford

Employer Survey 

1.     Cambridge
2.     Harvard
3.     Oxford
4.     Stanford
5.     UC Berkeley

Citations per Paper

1.     UC Irvine
2.     Emory
3.     Unuversity of Cinncinati
4.     Princeton
5.     Dartmouth College

Inappropriate Analogy Watch

Times Higher Education of April 21st has a rather disconcerting cover, a close up picture of a bonobo ape. Inside there is a long article by a graduate student at the University of British Columbia that argues that humans may have been too hasty in assuming that their current aggressive behavior is rooted in their ancestry. He suggests that humanity is more closely related to the bonobos than to the common chimpanzees. The former are peaceful, promiscuous, egalitarian, dominated by females and without hang ups about homosexuality. They sound rather like a mix between a hippie commune and a humanities faculty at an American state university or least like those places would imagine themselves to be. Common chimpanzees on the other hand are notorious for behaving like a gang of skinheads on a Saturday night.

This is a variant of a common theme in popularized social science writing. For a long time, western feminists and leftists have looked to contemporary or historical pre-modern societies for validation only to find disappointment. Margaret Mead’s free loving Samoans tuned out to be rather different while the search for mother earth worshipping matriarchies has been equally futile. Now, it seems they are forced to go back several million years. Perhaps the bonobo really are what primatologists say they are. But it would be unsurprising if they turn out to be  as politically incorrect, competitive and unpleasant as the chimpanzees.

In any case, it is pseudo-science to suggest that humanity can take any other species as a model or inspiration . There are dozens of extinct species and subspecies between us and the bonobos who may have been even more gentle and promiscuous than the bonobo or even more violent and competitive than the chimpanzee.

 The point of the article is found in an editorial by Ann Mroz in the same issue.

In higher education, we appear to moving from an approach based on cooperation to one based on competition, from the bonobo compact to the chimp reforms, if you like. The Browne Review launches us into a quasi-market world, which in itself has far-reaching implications. Unfortunately, it comes on top of a range of pre-existing and co-existing factors: the concentration of research funding; tighter immigration rules; cuts in teacher training and NHS cash; and internationalisation.

Some post-1992 institutions facing immediate financial constraints are moving swiftly to deal with their problems. London Metropolitan University, for example, is cutting about 400 of its 557 degree courses, and the University of East London is planning to axe its School of Humanities and Social Sciences.

Staff at the former institution describe the move as "an attempted reversal of widening participation...of everything that London Met...came into existence to promote". Staff at the latter describe its social sciences and humanities as high-performing areas. "Are UEL's non-traditional students going to be denied an academic education on the basis of managers' assumption that all such students are good for - and will be willing to pay for - is training?" they ask.

She therefore concludes.

UK universities have survived for 800 years through successful evolution in a relatively stable habitat, a context they share with the cooperative bonobo. The competitive chimpanzee, however, has had to adapt to more hostile conditions. In shaping the next stage of its evolution, the academy has the choice of emulating either the aggressive ape or the better angels of our nature.

There is a problem with this. The bonobo are close to extinction. There are only 10,000 of them left, compared with 300,000 common chimpanzees and the only reason those 10,000 have survived is that they are separated by the Congo river from the chimpanzees.

If Ann Mroz thinks British universities have evolved though cooperation over 800 years she should start  by reading the novels of C. P. Snow. No doubt they have become thoroughly cooperative over the last few years as diversity workshops, collaborative projects, performance appraisals, quality audits and professional development seminars have eradicted most signs of individuality in their faculty.

But there is no Congo river separating British universities from all those nerds and buffs in Korea, China and Singapore who work 80 hours a week  and refuse to cooperate and are quite uninterested in diversity, safe and comfortable environments and collegiality.  

And just what is so bad about training?

New QS Rankings

QS have just released their Life Sciences rankings based on their employer and academic surveys and citations per paper.

Here are the top five for medicine, biology and psychology.

Medicine

1.  Harvard
2.  Cambridge
3.  MIT
4.  Oxford
5.  Stanford

Biological Sciences

1.  Harvard
2.  MIT
3.  Cambridge
4.  Oxford
5.  Stanford

Psychology

1.  Harvard
2.  Cambridge
3.  Stanford
4.  Oxford
5.  UC Berkeley

Rising and Falling in Asia-Pacific

One problem with most international rankings is that they tend to measure historical quality and are not much use for predicting what will happen in the near future. The Shanghai rankings' alumni and awards criteria allow Oxbridge and some German universities to live off intellectual capital generated decades ago. The surveys of the QS rankings inevitably favour big, old, wealthy universities with years of alumni and endowments behind them. It will take a long time for any rapidly developing school to score well on the eleven year criteria in the HEEACT rankings.

I have compiled a list of the percentage change in the number of publications in ISI databases of universities in the Asia Pacific region between 2009 and 2010. The ranking includes all the universities listed in the 2009 Shanghai ARWU from the Asia Pacific region.

King Saud University is at the top of the table, almost doubling its output of papers between 2009 and 2010. Six out of the top 10 are from Greater China. Some major Japanese universities seem to be shrinking and Israel and Australia do not seem to be doing very well.

Some caveats. This is basically a measure of quantity not quality of research. Also, the results may reflect organisational changes such as the acquisition or loss of a medical school. The data were collected over several weeks, during which there could be additions to the databases so the scores were rounded out to whole numbers.

Percentage change in publications in the ISI Databases, 2009-2010

1.   King Saud University, Saudi Arabia,  95
2.   Shandong University, China,  16
3.   National Yang Ming University, Taiwan  15
4.   Tainjin University, China,  13
5.    Nanyang Technological University, Singapore,  13
6.    Sun Yat Sen University, China,   11
7.    Fudan University, China,  10
8.    Victoria University Wellington, New Zealand 9
9.    Niigata University,  Japan, 9
10.  Gunma University, Japan, 9
11.  Nihon University, Japan, 9
12.  University of Tasmania,  Australia, 9
13.  Tokyo Medical and Dental University, Japan,  8
14.  Jilin University, China, 8
15.  Chang Gung University, Taiwan, 7
16.  Chinese University of Hong Kong, 7
17.  Massey University of New Zealand, 7
18.  University of Auckland, New Zealand, 6
19.  Deakin University, Australia,  6
20.  La Trobe University, Australia, 6
21.  Seoul National University, Korea, 6
22.  Nanjing University, China,  6
23.  Nankai University, Japan, 6
24   University of Canterbury, New Zealand, 6
25.  Swinburne University of Technology, Australia, 6
26.  Weizman Institute of Science, Israel, 6
27.  Kumamoto University, Japan, 6
28.  Osaka Prefecture University, Japan, 5
29.  Yonsei University, Korea, 4
30.  Peking University, China, 4
31.  University of Haifa, Israel, 4
32.  Lanzhou University, China, 3
33.  James Cook University, Australia, 3
34.  Hong Kong Polytechnic University, 3
35.  University of Hong Kong, 3
36.  University of Queensland, Australia, 3
37.  China Agricultural University, 3
38.  Curtin University of Technology, Australia, 3
39.  University of Otago, New Zealand, 3
40.  Kyungpook National University, Korea, 3
41.  Korea Advanced Institute of Science and Technology, 2
42.  Sichuan University, China, 3 
43.  Korea University, 2
44.   University of Adelaide, Australia, 2
45.   Dalian University of Technology, China, 2
46.   Macquarie University, Australia, 2
47.   Sunkyunkwan University, Korea,  2
48.   Pusan National University, Korea, 1
49.   Flinders University, Australia, 1
50.   Shanghai Jiao Tung University, China, 1
51.   University of New South Wales, Australia, 1
52.   National Taiwan University, 1
53.   Osaka City University, Japan, 1
54.   Monash University, Astralia, 0
55.   Gifu University, Japan, 0
56.   Tsinghua University, China, -1
57.  Hiroshima University, Japan, -1
58.  Zhejiang University, China, -1
59.  Pohang University of Science and Technology, Korea, -1
60.  Bar Ilan University, Israel, -1
61.  National Chiao Tung University, Taiwan, -1
62.  Kobe University, Japan,-1
63.  University of Tehran, Iran, -1
64.  University of Western Australia, -2
65.  University of Western Sydney, Australia,  -2
66.  National Tsinghua University, Taiwan, -2
67.  National University of Singapore, -2
68.  Harbin Institute of Technology, China,  -2
69.  Kanazawa  University, Japan, -3
70.  City University of Hong Kong, -3
71.  University of Tokushima, Japan,  -4
72.  University of Newcastle, Australia  -4
73.  University of Melbourne, Australia.  -4
74.  Ben Gurion University of the Negev, Israel,  -4
75.  Indian Institute of Science,  - 4
76.  Osaka University, Japan, -4
77.  Kyushu Uniersity, Japan,  -4
78.  University of Wollongong, Australia,  -4
79.  Hokkaido University,  Japan, -5
80.  Huazhong University of Science and Technology, China,  -5
81.  University of Tsukuba, Japan,  -5
82.  University of Tokyo, Japan,  -5
83.  Yamaguchi University, Japan, -5
84.  Hanyang University, Korea,  -6
85.  Hong Kong university of Science and Technlogy,  -6
86.  Technion - Israel Institute of Technology,   -6
87.   Hebrew University of Jerusalem, Israel,  -6
88.   Nagasaki  University,  Japan,  -6
89.   Kyoto University, Japan,  -6
90.   Chiba University, Japan,  -6
91.   Australian National University, -6
92.   Kagoshima University, Japan, -7
93.   Tel Aviv University, Israel.   -7
94.   Nagoya University, Japan, -7
95.   National Cheng Kung University, Taiwan, -8
97.    Keio University, Japan, -8
98.    Okayama University, Japan, -8
99.    National Central University, Taiwan, -9
100.  Nara Institute of Science and Technology, Japan,-9
101.  Ehime University, Japan,  -10
102.  Tohoku University, Japan, -10
103.  Tokyo Institute of Technology, Japan,  -10
104.  Indian Institute of Technology Kharagpur,  -13

Queen's Will be Ranked This Year

A bit of good news for Times Higher Education.  Queen's University, Canada, has decided to take part in this year's THE World University Rankings.

'Rankings methodologies had come under scrutiny in recent years. Some universities including Queen’s were concerned that inconsistent criteria and data used for comparing institutions did not accurately reflect their objectives, and some have participated in rankings selectively or not at all.
Last year, Queen’s decided not to submit information to the Times Higher Education ranking because of concerns about its methodology. As a result, Queen’s was not included in the Top 200 list. The Times [sic] has since changed its methodology.

“Queen’s is still concerned because the rankings focus mainly on research volume and intensity, and although Queen’s is one of Canada’s top research universities, our quality undergraduate student experience and out-of- classroom experience are not fully captured,” says Chris Conway, Director, Institutional Research and Planning. “This just means we need to work hard to tell the other side of our story – that we’re a balanced academy, excelling in both research and the student experience.” '

The implication that Queen's has decided to take part this year because of a change in methodology is difficult to accept. THE has talked about revising their citations indicator but nothing definite has emerged. The real reason might be this:

'Although both global and domestic rankings struggle with standardizing data collection and interpretation, they provide one of the few tools available to prospective undergraduate students and their families for evaluating universities.

“With so many options, rankings help to reassure parents and students about their decision to attend a given university,” says Andrea MacIntyre, the university’s international admission manager.
Queen’s position in rankings is one of the top three concerns among prospective undergraduate students, particularly in China and India, where the national education systems focus heavily on class standings from the early stages of education.'

300: The Iranian Version

I have always thought that university ranking organisations are unimaginative in their choice of names. Surely they could do better than ARWU, WUR and HEEACT?  I admit though that SCIMAGO sounds a bit better. What about renaming one of the tables the Comparative Ranking of Academic Performance?

For a moment, when reading the text below, I thought QS had come with a slightly more interesting name for their rankings, SUE,  but it is, it seems, just the QS WUR somewhat mutated by translation in and out of Farsi.

"QS World University Ranking asks head of ISC (Islamic World Science Citation Center) for cooperation in electing high ranked universities of the world.

IBNA: According to the public relations of ISC, each year the QS World University Ranking releases the list of top world universities and for the year 2011, Dr Jafar Mehrad is asked for cooperation in electing high-ranked universities.

Meanwhile, ten out of total state and non-state Iranian universities and 30 universities from Asia and the Middle East are to be sorted in this election according to the criteria of this ranking system.

For the election of top world universities, 300 experts are annually invited and this year Dr Jafar Mehrad is one of the jurors.
The results of QS World University ranking will be released in autumn 2011. "

I wonder where the bit about 300 jurors came from.

Worth Reading

Retraction Watch is a blog that deals with the retraction of scientific papers because of plagiarism, duplication, fabrication of data and so on.

 Second Edition

The National Research Council in the US has released a new version  of its 2010 doctoral program rankings. It seems that there were a large number of errors first time around  According to an article by David Glenn in the Chronicle of Higher Education:

The National Research Council released on Thursday a revised edition of its 2010 rankings of American doctoral programs that corrects four types of errors discovered in the original report, which was issued last September. But the new rankings do not deal with certain other concerns that scholars have raised about the project.

In the revised edition, almost all programs' positions on the council's "ranges of rankings" have changed at least slightly, but in most cases the changes are not substantial. In a few academic fields, however, the numbers have changed significantly for at least 20 percent of the programs. Those include geography, linguistics, and operations research.

A spreadsheet of the new rankings is available for download at the council's Web site. The council has also released a separate, much smaller spreadsheet that summarizes the changes in programs' "R" and "S" rankings. (R rankings reflect how similar a program is to the programs in its field with the strongest reputations. S rankings more directly reflect a program's performance on variables that scholars in the field say are most important, such as faculty research productivity or student diversity.)

The new edition makes four kinds of corrections. The original report in many cases undercounted faculty members' honors and awards, the proportion of new graduates who find academic jobs, and the proportion of first-year students who are given full financial support. In nonhumanities fields, the report also used faulty data for faculty members' 2002 publications, which in turn caused errors in calculations of citation counts.

Reviewing the THE Rankings

An article by Phil Baty in Times Higher Education looks at the various components of last year's THE World University Rankings and gives some hints about changes to come this year. Some good points but also some problems. My comments are in red.


We look at research in a number of different ways, examining reputation, income and volume (through publication in leading academic journals indexed by Thomson Reuters). But we give the highest weighting to an indicator of “research influence”, measured by the number of times published research is cited by academics across the globe.

We looked at more than 25 million citations over a five-year period from more than five million articles.

Yes, but when you normalise by field and by year, then you get very low benchmark figures and a few hundred citations to a few dozen articles can acquire disproportionate influence.

All the data were normalised to reflect variations in citation volume between different subject areas, so universities with strong research in fields with lower global citation rates were not penalised.


The lower the global citation rates the more effect a strategically timed and placed citation can have and the greater the possibility of gaming the system.

We also sought to acknowledge excellence in research from institutions in developing nations, where there are less-established research networks and lower innate citation rates, by normalising the data to reflect variations in citation volume between regions. We are proud to have done this, but accept that more discussion is needed to refine this modification.

In principle this sounds like a good idea but it could just mean that Singapore, Israel, South Africa and the south of Brazil might be rewarded for being located in under-achieving regions of which they are not really a part. 

The “research influence” indicator has proved controversial, as it has shaken up the established order, giving high scores to smaller institutions with clear pockets of research excellence and boosting those in the developing world, often at the expense of larger, more established research-intensive universities.

Here is a list of universities that benefited disproportionately from high scores for the "research influence" indicator. Are they really smaller, are they really in the developing world? And as for those clear pockets of excellence, that would certainly be the case for Bilkent (you can find out who he is in five minutes) but for Alexandria...?

Boston College
University of California Santa Cruz
Royal Holloway, University of London
Pompeu Fabra
Bilkent
Kent State University
Hong Kong Baptist University
Alexandria
Barcelona
Victoria University WellingtonTokyo Metropolitan University
University of Warsaw

Something else about this indicator that nobody seems to have noticed is that even if the methodology remains completely unchanged, it is capable of producing dramatic changes from year to year. Suppose that an article in a little cited field like applied math was cited ten times in its first year of publication. That could easily be 100 times the benchmark figure. But in the second year that might be only ten times the benchmark. So if the clear pocket of research excellence stops doing research and becomes a newspaper columnist or something like that, the research influence score will go tumbling down.

We judge knowledge transfer with just one indicator – research income earned from industry – but plan to enhance this category with other indicators.

This is a good idea since it represents, however indirectly, an external assessment of universities.

Internationalisation is recognised through data on the proportion of international staff and students attracted to each institution.

Enough has been said about the abuses involved in recruiting international students. Elsewhere THE have said that they are adding more measures of internationalisation.

The flagship – and most dramatic – innovation is the set of five indicators used to give proper credit to the role of teaching in universities, with a collective weighting of 30 per cent.
But I should make one thing very clear: the indicators do not measure teaching “quality”. There is no recognised, globally comparative data on teaching outputs at present. What the THE rankings do is look at the teaching “environment” to give a sense of the kind of learning milieu in which students are likely to find themselves.
The key indicator for this category draws on the results of a reputational survey on teaching. Thomson Reuters carried out its Academic Reputation Survey – a worldwide, invitation-only poll of 13,388 experienced scholars, statistically representative of global subject mix and geography – in early 2010.
It examined the perceived prestige of institutions in both research and teaching. Respondents were asked only to pass judgement within their narrow area of expertise, and we asked them “action-based” questions (such as: “Where would you send your best graduates for the most stimulating postgraduate learning environment?”) to elicit more meaningful responses.

In some ways, the survey is an improvement on the THE-QS "peer review" but the number of responses was lower than the target and we still do not know how many survey forms were sent out. Without knowing the response rate we cannot determine the validity of the survey.

The rankings also measure staff-to-student ratios. This is admittedly a relatively crude proxy for teaching quality, hinting at the level of personal attention students may receive from faculty, so it receives a relatively low weighting of just 4.5 per cent.

Wait a minute. This means the measure is the number of faculty or staff  per student. But the THE web site says "undergraduates admitted per academic", which is the complete opposite. An explanation is needed

We also look at the ratio of PhD to bachelor’s degrees awarded, to give a sense of how knowledge-intensive the environment is, as well as the number of doctorates awarded, scaled for size, to indicate how committed institutions are to nurturing the next generation of academics and providing strong supervision.

Counting the proportion of postgraduate students is not a bad idea. If nothing else, it is a crude measure of the maturity of the students. However, counting doctoral students may well have serious backwash effects as students who would be quite happy in professional or masters programs are coerced or cajoled into Ph D courses that they may never finish and which will lead to a life of ill-paid drudgery if they do.

The last of our teaching indicators is a simple measure of institutional income scaled against academic staff numbers. This figure, adjusted for purchasing-price parity so that all nations compete on a level playing field, gives a broad sense of the general infrastructure and facilities available.

Yes, this is important and it's time someone started counting it.

Art Imitates Life: Update

It is time to bring the unbearable suspense to an end. The first text is the April Fool's Joke. Go here. The second is an article in Notices of the AMS by Douglas Arnold and Kristine Fowler.

Art Imitates Life

Which of these texts is an April Fool joke? How can you tell? Links will be posted in a few days.

TEXT 1
The Federal Intelligence Service discovered a Ponzi scheme of academic citations lead by an unemployed particle physicist. A house search conducted in Berlin last week revealed material documenting the planning and administration of a profitable business of trading citations for travel reimbursement.

According to the Federal Intelligence Service, the hint came from researchers at Michigan University, Ann Arbor, who were analyzing the structure of citation networks in the academic community. In late 2010, their analysis pointed towards an exponentially growing cluster originating from a previously unconnected researcher based in Germany's capital. A member of the Ann Arbor group, who wants to remain unnamed, inquired about the biography of the young genius, named Al Bert, sparking such amount of activity. The researcher was easily able to find Dr. Bert scheduled for an unusual amount of seminars in locations all over the world, sometimes more than 4 per week. However, upon contacting the respective institutions, nobody could remember the seminars, which according to Prof. Dr. Dr. Hubert at The Advanced Institute is "Not at all unusual." The network researcher from Ann Arbor suspected Dr. Bert to be a fictitious person and notified the university whose email address Dr. Bert was still using.

It turned out Dr. Bert is not a fictitious person. Dr. Bert's graduated in 2006, but his contract at the university run out in 2008. After this, colleagues lost sight of Dr. Bert. He applied for unemployment benefits in October 2008. As the Federal Intelligence Service reported this Wednesday, he later founded an agency called 'High Impact' (the website has since been taken down) that offered to boost a paper's citation count. A user registered with an almost finished, but not yet published, paper and agreed to pay EUR 10 to Dr. Bert's agency for each citation his paper received above the author's average citation count at the time of registration. The user also agreed to cite 5 papers the agency would name. A registered user would earn EUR 10 for each recruitment of a new paper, possibly their own.

This rapidly created a growing network of researchers citing each others papers, and encouraged the authors to produce new papers, certain they would become well cited. Within only a few months, the network had spread from physics to other research fields. With each citation, Dr. Bert made an income. The algorithm he used to assign citations also ensured his own works became top cites. Yet, with many researchers suddenly having papers with several hundred citations above their previously average citation count, their fee went into some thousand dollars. On several instances Dr. Bert would suggest they invite him for a seminar at their institution and locate it in a non-existent room. He would then receive reimbursement for a fraudulent self-printed boarding pass, illegible due to an alleged malfunctioning printer.

Names of researchers subscribed to Dr. Bert's agency were not accessible at the time of writing.


TEXT 2

The Case of IJNSNS

The field of applied mathematics provides an illuminating case in which we can study such impact-factor distortion. For the last several years, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS) has dominated the impact-factor charts in the “Mathematics, Applied” category. It took first place in each year 2006, 2007, 2008, and 2009, generally by a wide margin, and came in second in 2005. However, as we shall see, a more careful look indicates that IJNSNS is nowhere near the top of its field. Thus we set out to understand the origin of its large impact factor.

In 2008, the year we shall consider in most detail, IJNSNS had an impact factor of 8.91, easily the highest among the 175 journals in the applied math category in ISI’s Journal Citation Reports (JCR). As controls, we will also look at the two journals in the category with the second and third highest impact factors, Communications on Pure and Applied Mathematics (CPAM) and SIAM Review (SIREV), with 2008 impact factors of 3.69 and 2.80, respectively. CPAM is closely associated with the Courant Institute of Mathematical Sciences, and SIREV is the flagship journal of the Society for Industrial and Applied Mathematics (SIAM).  Both journals have a reputation for excellence.

Evaluation based on expert judgment is the best alternative to citation-based measures for journals. Though not without potential problems of its own, a careful rating by experts is likely to provide a much more accurate and holistic guide  journal quality than impact factor or similar metrics. In mathematics, as in many fields, researchers are widely in agreement about which are the best journals in their specialties. The Australian Research Council recently released such an evaluation, listing quality ratings for over 20,000 peer reviewed journals across disciplines. The list was developed through an extensive review process involving learned academies (such as the Australian Academy of Science), disciplinary bodies (such as the Australian Mathematical Society), and many researchers and expert reviewers.11 This rating is being used in 2010 for the Excellence in Research Australia assessment initiative and is referred to as the ERA 2010 Journal List. The assigned quality rating, which is intended to represent “the overall quality of the journal,” is one of four values: 

• A*: one of the best in its field or subfield

• A: very high quality

• B: solid, though not outstanding, reputation

• C: does not meet the criteria of the higher tiers.

The ERA list included all but five of the 175 journals assigned a 2008 impact factor by JCR in the category “Mathematics, Applied”. Figure 1 shows the impact factors for journals in each of the four rating tiers. We see that, as a proxy for expert opinion, the impact factor does rather poorly. There are many examples of journals with a higher impact factor than other journals that are one, two, and even three rating tiers higher. The red line is drawn so that 20% of the A* journals are below it; it is notable that 51% of the A journals have an impact factor above that level, as do 23% of the B journals and even 17% of those in the C category. The most extreme outlier is IJNSNS, which, despite its relatively astronomical impact factor, is not in the first or second but, rather, third tier.

The ERA rating assigned its highest score, A*, to 25 journals. Most of the journals with the highest impact factors are here, including CPAM and SIREV, but, of the top 10 journals by impact factor, two were assigned an A, and only IJNSNS was assigned a B. There were 53 A-rated journals and 69 B-rated journals altogether. If IJNSNS were assumed to be the best of the B journals, there would be 78 journals with higher ERA ratings, whereas if it were the worst, its ranking would fall to 147. In short, the ERA ratings suggest that IJNSNS is not only not the top applied math journal but also that its rank should be somewhere in the range 75–150. This remarkable mismatch between reputation and impact factor needs an explanation.

Makings of a High Impact Factor

A first step to understanding IJNSNS’s high impact factor is to look at how many authors contributed substantially to the counted citations and who they were. The top-citing author to IJNSNS in 2008 was the journal’s editor-in-chief, Ji-Huan He, who cited the journal (within the two-year window) 243 times. The second top citer, D. D. Ganji, with 114 cites, is also a member of the editorial board, as is the third, regional editor Mohamed El Naschie, with 58 cites. Together these three account for 29% of the citations counted toward the impact factor.

For comparison, the top three citers to SIREV contributed only 7, 4, and 4 citations, respectively, accounting for less than 12% of the counted citations, nd none of these authors is involved in editing the journal. For CPAM the top three citers (9, 8, and 8) contributed about 7% of the citations and, again, were not on the editorial board.

Another significant phenomenon is the extent to which citations to IJNSNS are concentrated within the two-year window used in the impact factor calculation. Our analysis of 2008 citations to articles published since 2000 shows that 16% of the citations to CPAM fell within that two-year  window and only 8% of those to SIREV did; in contrast, 71.5% of the 2008 citations to IJNSNS fell within the two-year window. In Table 1, we show the 2008 impact factors for the three journals, as well as a modified impact factor, which gives the average number of citations in 2008 to articles the journals published not in 2006 and 2007 but in the preceding six years. Since the cited half-life (thetime it takes to generate half of all the eventual citations to an article) for applied mathematics is nearly 10 years,12 this measure is at least as reasonable as the impact factor. It is also independent, unlike JCR’s 5-Year Impact Factor, as its time period does not overlap with that targeted by the Journal.

Table 1.  2008 impact factor with normal 2006–7 window      Modified 2008 “impact factor” with 2000–5 window

IJNSNS 8.91 1.27

CPAM 3.69 3.46 S

IREV 2.8 10.4

2008 impact factors computed with the usual two-preceding-years window, and with a window going back eight years but neglecting the two immediately preceding.

Note that the impact factor of  JNSNS drops precipitously, by a factor of seven, when we consider a different citation window. By contrast, the impact factor of CPAM stays about the same, and that of SIREV increases markedly. One may simply note that, in distinction to the controls, the citations made to IJNSNS in 2008 greatly favor articles published in precisely the two years that are used to calculate the impact factor. Further striking insights arise when we examine the high-citing journals rather than high-citing authors. The counting of journal self-citations in the impact factor is frequently criticized, and indeed it does come into play in this case. In 2008 IJNSNS supplied 102, or 7%, of its own impact factor citations.

The corresponding numbers are 1 citation (0.8%) for SIREV and 8 citations (2.4%) for CPAM. The disparity in other recent years is similarly large or larger. However, it was Journal of Physics: Conference Series that provided the greatest number of IJNSNS citations. A single issue of that journal provided 294 citations to IJNSNS in the impact factor window, accounting for more than 20% of its impact factor. What was this issue? It was the proceedings of a conference organized by IJNSNS editor-in-chief He at his home university. He was responsible for the peer review of the issue. The second top-citing journal for IJNSNS was Topological Methods in Nonlinear Analysis, which contributed 206 citations (14%), again with all citations coming from a single issue. This was a special issue with Ji-Huan He as the guest editor; his co-editor, Lan Xu, is also on the IJNSNS editorial board. J.-H. He himself contributed a brief article to the special issue, consisting of three pages of text and thirty references. Of these, twenty were citations to IJNSNS within the impact-factor window. The remaining ten consisted of eight citations to He and two to Xu.

Continuing down the list of IJNSNS high-citing journals, another similar circumstance comes to light: 50 citations from a single issue of the Journal of Polymer Engineering (which, like IJNSNS, is published by Freund), guest edited by the same pair, Ji-Huan He and Lan Xu. However, third place is held by the journal Chaos, Solitons and Fractals, with 154 citations spread over numerous issues. These are again citations that may be viewed as subject to editorial influence or control. In 2008 Ji-Huan He served on the editorial board of CS&F, and its editor-in-chief was Mohamed El Naschie, who was also a coeditor of IJNSNS. In a highly publicized case, the entire editorial board of CS&F was recently replaced, but El Naschie remained coeditor of IJNSNS.

Many other citations to IJNSNS came from papers published in journals for which He served as editor, such as Zeitschrift für Naturforschung A, which provided forty citations; there are too many others to list here, since He serves in an editorial capacity on more than twenty journals (and has just been named editor-in-chief of four more journals from the newly formed Asian Academic Publishers). Yet another source of citations came from papers authored by IJNSNS editors other than He, which accounted for many more. All told, the aggregation of such editor-connected citations, which are time-consuming to detect, account for more than 70% of all the citations contributing to the IJNSNS impact factor.