Ranking Rankings: Measuring Stability

I have noticed that some rankings are prone to a large amount of churning. Universities may rise or fall dozens of places over a year, sometimes as a result of methodological changes, changes in the number or type of universities ranked, errors and corrections of errors (fortunately rare these days), changes in data collection and reporting procedures, or because there is a small number of data points.

Some ranking organisations like to throw headlines around about who's up or down, the rise of Asia, the fall of America, and so on. This is a trivialisation of any serious attempt at the comparative evaluation of universities, which do not behave like volatile financial markets. Universities are generally fairly stable institutions: most of the leading universities of the early twentieth century are still here while the Ottoman, Hohenzollern, Hapsburg and Romanov empires are long gone.

Reliable rankings should not be expected to show dramatic changes from year to year, unless there has been radical restructuring like the recent wave of mergers in France. The validity of a ranking system is questionable if universities bounce up or down dozens, scores, even hundreds of ranks every year.

The following table shows the volatility of the global rankings listed in the IREG Inventory of international rankings. U-Multirank is not listed because it does not provide overall ranks and UniRank and Webometrics do not give access to previous editions. 

Average rank change is the number of places that each of the top thirty universities has fallen or climbed between the two most recent editions of the ranking.

The most stable rankings are the Shanghai ARWU, followed by the US News global rankings and the National Taiwan University rankings. The GreenMetric rankings, Reuters Innovative Universities and the high quality research indicator of Leiden Ranking show the highest levels of volatility.

This is a very limited exercise. We might get different results if we examined all of the universities in the rankings or analysed changes over several years.

rank	ranking	address	average rank change
1	Shanghai ARWU	China	0.73
2	US News Best Global Universities	USA	0.83
3	National Taiwan University Rankings	Taiwan	1.43
4	THE World University Rankings	UK	1.60
5	Round University Rankings	Russia	2.28
6	University Ranking by Academic Performance	Turkey	2.23
7	QS World University Rankings	UK	2.33
8	Nature Index	UK	2.60
9	Leiden Ranking Publications	Netherlands	2.77
10	Scimago	Spain	3.43
11	Emerging/Trendence	France	3.53
12	Center for World University Ranking	UAE	4.60
13	Leiden Ranking % Publications in top 1%	Netherlands	4.77
14	Reuters Innovative Universities	USA	6.17
15	UI GreenMetric	Indonesia	13.14