Anyone interested in the evolution of global university rankings should take a look at 'Highly Cited researchers and the Shanghai Ranking' by Lawrence Cram of the Australian National University and Domingo Docampo of the University of Vigo, Spain.
- The paper, which analyses the HiCi indicator, based on Thomson Reuters Highly Cited researchers database, in the Shanghai Academic Ranking of World Universities (ARWU), notes that the loss or addition of a single highly cited researcher can have a potentially large impact of the ranking of a university, although this would not apply to the top 100 where universities typically have one or two dozen such researchers.
- The paper also provides some economic context that might explain why Thomson Reuters has been so adamant about allowing even the most sensible deviation of its Citation indicator from its InCites system.
"While the focus of this paper is not the commercial aspects of citations databases it is important to understand how commercial drivers might shape the availability of such data. In this respect, the most recently available shareholder earnings presentation for Thompson Reuters reveals a corporate strategy that includes resetting the cost base (i.e. reducing business costs), product simplification and attractive returns to shareholders. These business considerations may have a bearing on the commercial development of citation databases."
- The paper describes the evolution of the Highly Cited Researchers database and some aspects of the new list introduced in 2013. They note that counting by field means that names are repeated and that the number of names in the new list may exceed the number of actual persons.
- The number of highly cited researchers varies between fields, from 276 in Engineering to 401 in Microbiology.
- For some universities, the score in the HiCi indicator in ARWU is very significant. For 13 universities -- 9 in the USA and 4 in Saudi Arabia -- it accounts for 30% of the total score.
- The paper concludes by suggesting that it is time that ARWU considered changing its measurement of citations.
"Significant acrimony accompanies some published comparisons between ARWU and other rankings (Redden, 2013) driven in part by commercial positioning . Given its status as an academic ranking , it may be prudent for ARWU to consider replacing its HiCi indicator with a measure that is nit sourced from a commercial provider if such a product can be found that satisfies the criteria (objective, open, independent ) used by ARWU."
Five days ago I noted an open access paper that analysed the new list of highly cited researchers published by Thomson Reuters (TR) and discussed the disproportionate number of secondary affiliations to a single institution, King Abdulaziz University (KAU) of Jeddah, Saudi Arabia.
The significance of this is that the number of highly cited researchers is an indicator in the Academic Ranking of World Universities (ARWU) produced by Shanghai Jiao Tong University, contributing 20% of the total score. If all affiliations given in the list are counted, including secondary affiliations, then KAU will get an extremely high score for this indicator in the forthcoming rankings and will do very well in the overall rankings.
Times Higher Education (THE) has now noticed what is going on. An article by Paul Jump reports on the findings by Lutz Bornmann of the Max Planck Society and Johann Bauer of the Max Planck Institute.
The THE article has been republished in Inside Higher Ed and there is more discussion in University World News by Yves Gingras of the University of Quebec.
That THE should publish such an article is a little surprising. The researchers who have listed KAU as a secondary affiliation will probably not stop with the the TR highly cited researchers' list -- if they do it will be a major scandal -- but will also include it in papers published in the highly regarded journals indexed in the Web of Science database. That could include those multi-"authored", hyper-cited publications in fields such as astronomy, genetics and particle physics that have propelled institutions such as Moscow State Engineering Physics Institute and Panjab University to high places in the Citations indicator in the THE World University Rankings.
KAU has already received a high score for citations in the THE 2013 world rankings, one that is disproportionate to its very modest score for the Research: Volume, Reputation and Income indicator. It is not impossible that its secondary affiliations in the old highly cited list will start popping up in the"author" list in one or two of those Multi Author (or contributor?) Publications in this year's rankings as might those on the new lists in the years to come.
If so, one wonders whether it is fair to single out ARWU for criticism. However, TR has announced that In the new highly cited list that they excluded physics papers with more than 30 institutional addresses.
"The methodology described above was applied to all ESI fields with the exception of Physics. The relative large number of Highly Cited Papers in Physics dealing with high- energy experiments typically carried hundreds of author names. Using the whole counting method produced a list of high-energy physicists only and excluded those working in other subfields. For example, the number of Highly Cited Papers required for inclusion in Physics, using the standard methodology, turned out to be a remarkable 63. So, as an expedient, it was decided to eliminate from consideration any paper with more than 30 institutional addresses. This removed 436 out of 10,373 Highly Cited Papers in physics and the problem of overweighting to high-energy physics. An analysis without these papers produced a list in which the threshold for number of Highly Cited Papers was 14. It also produced a ranking in which the 2010 Nobel Prize winner in Physics Andre Geim of the University of Manchester appeared first, with 40 Highly Cited Papers. Fields of physics other than high-energy alone now appear as represented by the scientists selected."
If TR does that for this year's rankings as well they will save THE some embarrassment, although Panjab University may be wondering why its "excellence" has suddenly disappeared. We shall have to wait and see what happens.
There is an important point about the lists and their role in the rankings that needs to be made. The count of affiliations of KAU to in the arxiv paper is probably too high. The authors count the number of primary affiliations, then the total number of affiliations and also the fractionated total so that if a researcher has three affiliations then each counts as one third of an affiliation.
However, as indicated in their methodology section , ARWU have surveyed the current highly cited researchers to determine how their affiliations and where there was no response assign secondary affiliations a 16% fraction, which was the percentage given by those who did respond to the survey. This would lead to a total for KAU less than that suggested by the paper although still implausibly high and well above the scores for the other indicators.
The Center for World University Rankings, based in Jeddah, Saudi Arabia, has produced a global ranking of 1,000 universities. Last year and in 2012, 100 universities were ranked. The Center is headed by Nadim Mahassen, an Assistant Professor at King Abdulaziz University.
The rankings include five indicators that measure various aspects of publication and research: Publications in reputable journals, Influence (research papers in highly influential journals), Citations, Broad Impact (h-index) and Patents (h-index).
Altogether these have a weighting of 25%, which seems on the low side for modern world class research universities. The use of the h-index, which reduces the impact of outliers and anomalous cases, is a useful addition to the standard array of indicators. So too is the use of patents filed as a measure of innovation.
Another 25% goes to Quality of Education, which is measured by the number of alumni receiving major international awards relative to size (current number of students according to national agencies). There would appear to be an obvious bias here towards older institutions. There is also a problem that such awards are likely to be concentrated among relatively few universities so that this indicator would not discriminate among those outside the world elite.
A quarter is assigned to Quality of Faculty measured by the number of faculty receiving such awards and another quarter to Alumni Employment measured by the number of CEOs of top corporations.
The last three indicators are unlikely to be regarded as satisfactory. The number of CEOs is largely irrelevant to the vast majority of institutions.
In general, these are a useful addition to the current array of global rankings but the non-research indicators are narrow and not very meaningful. There is also a very serious problem with reliability as noted below.
Now for the standard presentation of rankings, with the addition of a noise analysis.
Publisher
Center for World University Rankings, Jeddah, Saudi Arabia
Scope
Global. 1,000 universities.
Methodology
Quality of Education (25%) measured by alumni winning international awards relative to size.
Alumni Employment (25%) measured by CEOs of top companies relative to size.
Quality of Faculty (25%) measured by "academics" winning international awards relative to size.
Publications in reputable journals (5%).
Influence measured by publications in highly influential journals (5%).
Citations measured by the number of highly cited papers (5%).
Broad Impact measured by h-index (5%).
Patents measured by the number of international filings (5%)
Top Ten
1. Harvard
2. Stanford
3. MIT
4. Cambridge
5. Oxford
6. Columbia
7. Berkeley
8. Chicago
9. Princeton
10. Yale
Countries with Universities in the Top Hundred
USA 54
Japan 8
UK 7
Switzerland 4
France 4
Germany 4
Israel 3
Canada 3
China 2
Sweden 2
South Korea 1
Russia 1
Taiwan 1
Singapore 1
Denmark 1
Netherlands 1
Italy 1
Belgium 1
Australia 1
Top Ranked in Region
USA: Harvard
Canada: Toronto
Latin America: Sao Paulo
Europe: Cambridge
Africa: Witwatersrand
Asia: Tokyo
Middle East: Hebrew University of Jerusalem
Arab World: King Saud University
Noise Index
Average position change of universities in the top 20 in 2013:
0.9
Comparison
Shanghai Rankings (ARWU): 2011-12 -- 0.15; 2012-13 -- 0.25
THE WUR: 2012-13 -- 1.2
QS WUR 2012-13 -- 1.7
Average position change of universities in the top 100 in 2013
10.59
Comparison
Shanghai Rankings (ARWU): 2011-12 -- 2.01; 2012-13 -- 1.66
THE WUR: 2012-13 -- 5.36
QS WUR 2012-13 -- 3.97
The CWUR rankings, once we leave the top 20, are extremely volatile, even more than the THE and QS rankings. This, unfortunately, is enough to undermine their credibility. A pity since there are some promising ideas here.
Thomson Reuters have published another document, The World's Most Influential Scientific Minds, which contains the most highly cited researchers for the period 2002-13. This one includes only the primary affiliation of the researchers, not the secondary ones. If the Shanghai ARWU rankings, due in August, use this list rather than the one published previously, they will save themselves a lot of embarrassment.
Over at arxiv, Lutz Bornmann and Johann Bauer have produced a ranking of the leading institutions according to the number of highly cited researchers' primary affiliation. Here are their top ten universities, with government agencies and independent research centres omitted.
1. University of California (all campuses)
2. Harvard
3. Stanford
4. University of Texas (all campuses)
5. University of Oxford
6. Duke University
7. MIT
8. University of Michigan (all campuses)
9. Northwestern University
10. Princeton
Compared to the old list, used for the Highly Cited indicator in the first Shanghai rankings in 2003, Oxford and Northwestern are doing better and MIT and Princeton somewhat worse.
Bornmann and Bauer have also ranked universities according to the number of primary and secondary affiliations,counting each recorded affiliation as a fraction). The top ten are:
1. University of California (all campuses)
2. Harvard
3. King Abdulaziz University, Jeddah, Saudi Arabia
4. Stanford
5. University of Texas
6. MIT
7. Oxford
8. University of Michigan
9. University of Washington
10. Duke
The paper concludes:
"To counteract attempts at manipulation, ARWU should only consider primary
institutions of highly cited researchers. "
