An Indisputable Ranking Scorecard? Not Really.

The University of New South Wales (UNSW) has produced an aggregate ranking of global universities, known as ARTU. This is based on the "Big Three" rankers, QS, Times Higher Education (THE) and the Shanghai ARWU. The scores that are given are not an average but a aggregate of their ranks, which is then inverted. Nor surprisingly, Australian universities do well and the University of Melbourne is the best in Australia.

Nicholas Fisk, Deputy Vice Chancellor of Research, hopes that this ranking will become "the international scoreboard, like the ATP tennis rankings" and "the indisputable scoreboard for where people fit in on the academic rankings."

This is not a new idea. I had a go at producing an aggregate ranking a few years ago, called Global Ranking of Academic Performance or GRAPE. It was going to be the Comparative Ranking of Academic Performance: maybe I was right the first time.  It was justifiably criticised by Ben Sowter of QS. I think though that it was quite right to note that some of the rankings of the time underrated the top Japanese universities and overrated British and Australian schools.

The ARTU is an another example of the emergence of a cartel or near cartel of the three global rankings that are apparently considered the only ones worthy of attention by academic administrators and the official media.

There are in fact a lot more and these three are not even the best three rankings, far from it. A pilot study, Rating the Rankers,  conducted by the International Network of Research Management Systems (INORMS), has found that on four significant dimensions, transparency, governance, measuring what matters, and rigour, the performance of six well known rankings is variable and that of the big three is generally unimpressive. That of THE is especially deficient.

Seriously, should  we consider as indisputable a ranking that includes indicators that proclaim Anglia Ruskin University as a world leader for research impact and Anadolu University as tops for innovation, another that counts that long dead winners of Nobel and Fields awards, and another that gives disproportionate weight to a survey with more respondents from Australia than from China?

There does seem to be a new mood of ranking skepticism emerging in many parts of the international research community. Rating the Rankers has been announced in an article in Nature. The critical analysis of rankings will, I hope, do more to create fair and valid systems of comparative assessment than simply adding up a bunch of flawed and opaque indicators.

Announcing GRAPE: Global Ranking of Academic Performance

I am surprised that nobody has thought of doing this before.

There are now six international university ranking systems and five of these, World University Rankings (THE-QS London), Academic Ranking of World Universities (Shanghai Jiao Tong University), International Professional Ranking of Higher Education Institutions (Ecole des Mines de Paris), Performance Ranking of Scientific Papers for World Universities (Taiwan) and Global Ranking of Universities (Russia), provide a numerical score. I have simply added the scores for all universities that were in the top 30 on any one of these, converting the top score for The Paris and Taiwan rankings into 100. The top scorer in the composite ranking was of course Harvard which was awarded a composite score of 100. The other scores were then adjusted accordingly. Yale, Imperial College London, Northewestern and Waseda were not included in the Russian rankings so they were ranked according to their total score for the other four.

There are some interesting results. The University of Tokyo comes in second, with a good record for recent research and for CEOs of big companies. University College London and Imperial College perform poorly. Oxford and Cambridge are slipping a bit and Australian universities do badly.

Here then are the top 30 with the combined scores.

1. Harvard 100

2. University of Tokyo 79.91

3. MIT 74.05

4. Stanford 71.21

5. Columbia 62.61

6. Cambridge 61.87

7. Caltech 59.81

8. Oxford 59.29

9. University of Pennsylvania 57.65

10. Yale 57.00

11. Johns Hopkins 56.7

12. University of California Berkeley 55.22

13. Chicago 54.87

14. Cornell 53.42

15. Kyoto 53.42

16 . UCLA 53.07

17. Duke 52.66

18. Princeton 51.49

19. University College London 50.46

20. Michigan 49.19

21. Imperial College London 47.74

22. University of Washington Seattle 47.08

23. University of California San Diego 45.60

24. Toronto 45.46

25. Northwestern 46.09

26. University of Wisconsin Madison 42.98

27. Manchester 42.49

28. Edinburgh 42.46

29. McGill 42.41

30. University of Illinois Urbana Champagne 41.69

It is also intersting to look at the correlations between the specific rankings and the combined scores. The correlations (top 30 institutions only) are as follows.

Paris .818
Shanghai .815
Taiwan .773
Russia .652
THE-QS .491