Saturday, November 03, 2018

Ranking Rankings: Measuring Stability

I have noticed that some rankings are prone to a large amount of churning. Universities may rise or fall dozens of places over a year, sometimes as a result of methodological changes, changes in the number or type of universities ranked, errors and corrections of errors (fortunately rare these days), changes in data collection and reporting procedures, or because there is a small number of data points.

Some ranking organisations like to throw headlines around about who's up or down, the rise of Asia, the fall of America, and so on. This is a trivialisation of any serious attempt at the comparative evaluation of universities, which do not behave like volatile financial markets. Universities are generally fairly stable institutions: most of the leading universities of the early twentieth century are still here while the Ottoman, Hohenzollern, Hapsburg and Romanov empires are long gone.

Reliable rankings should not be expected to show dramatic changes from year to year, unless there has been radical restructuring like the recent wave of mergers in France. The validity of a ranking system is questionable if universities bounce up or down dozens, scores, even hundreds of ranks every year.

The following table shows the volatility of the global rankings listed in the IREG Inventory of international rankings. U-Multirank is not listed because it does not provide overall ranks and UniRank and Webometrics do not give access to previous editions. 

Average rank change is the number of places that each of the top thirty universities has fallen or climbed between the two most recent editions of the ranking.

The most stable rankings are the Shanghai ARWU, followed by the US News global rankings and the National Taiwan University rankings. The GreenMetric rankings, Reuters Innovative Universities and the high quality research indicator of Leiden Ranking show the highest levels of volatility.

This is a very limited exercise. We might get different results if we examined all of the universities in the rankings or analysed changes over several years.



rank
ranking
address
average rank change
1
Shanghai ARWU
China
0.73
2
US News Best Global Universities
USA
0.83
3
National Taiwan University Rankings
Taiwan
1.43
4
THE World University Rankings
UK
1.60
5
Round University Rankings
Russia
2.28
6
University Ranking by Academic Performance
Turkey
2.23
7
QS World University Rankings
UK
2.33
8
Nature Index
UK
2.60
9
Leiden Ranking Publications
Netherlands
2.77
10
Scimago
Spain
3.43
11
Emerging/Trendence
France
3.53
12
Center for World University Ranking
UAE
4.60
13
Leiden Ranking % Publications in top 1%
Netherlands
4.77
14
Reuters Innovative Universities
USA
6.17
15
UI GreenMetric
Indonesia
13.14

No comments: