Comparison of the THES-QS "Peer Review" and Citations per Faculty Scores

QS Quacquarelli Symonds, the consultants responsible for the THES-QS World University Rankings, have now placed data for 540 universities, complete with scores for the various components, on their topuniversities website (registration required). This reveals more dramatically than before the disparity between scores by some universities on the “peer review” and scores for citations per faculty, a measure of research quality. Below are the top 20 universities in the world according to the THES-QS “peer review” by research-active academics who were asked to select the universities that are best for research. In curved brackets to the right is the position of the universities in the 2006 rankings according to the number of citations per faculty.

Notice that some universities, including Sydney, Melbourne, Australian National University and the National University of Singapore perform dramatically better on the peer review than on citations per faculty. Melbourne, rated the tenth best university in the world for research by the THES-QS peer reviewers, is 189th for citations per faculty while the National University of Singapore, in twelfth place for the peer review, comes in at 170th for citations per faculty. The most devastating disparity is for Peking University, 11th on the “peer review” and 352nd for citations, behind, among others, Catania, Brunel, Sao Paulo, Strathclyde and Jyväskylä. Once again, this raises the question of how universities whose research is regarded so lightly by other researchers could be voted among the best for research. Oxford, Cambridge and Imperial College London are substantially overrated by the peer review. Kyoto is somewhat overrated while the American universities, with the exception of Chicago, have roughly the same place that would be indicated by the citations per faculty position.

Of course, part of the problem could be with the citations per faculty. I am fairly confident that the data for citations, which is collected by Evidence Ltd, is accurate but less certain about the number of faculty. I have noted already that if a university increases its score for student faculty ratio by increasing the number of faculty it would suffer a drop in the citations per faculty score. For most universities the trade-off would be worth it since the gap between the very good and the good is much greater for citations than for student-faculty ratio. So, if there has been some inflating of the number of faculty, however and by whom it was done, than this would have an adverse impact on the figures for citations per faculty.

I have therefore included the positions of these universities according to their score for articles in the Science Citation Index-expanded and Social Science Citation Index in 2005 in the Shanghai Jiao Tong rankings. This is not the same as the THES measure. It covers one year only and is based on the number of papers, not number of citations. It therefore measures overall research output of a certain minimum quality rather than the impact of that research on other researchers. The position according to this index is indicated in square brackets.

We can see that Cambridge and Oxford do not do as badly as they did on citations per faculty. Perhaps they produced research characterised by quantity more than quality or perhaps the difference is a result of inflated faculty numbers. Similarly the performance of Peking, National University of Singapore, Melbourne and Sydney is not as mediocre on this measure as it is THES’s citations per faculty.

Nonetheless the disparity still persists. Oxford, Cambridge, Imperial College and universities in Asia and Australia are still overrated by the THES-QS review.

1. Cambridge (46 ) [15]
2. Oxford (63) [17]
3. Harvard (2) [1]
4. Berkeley ( 7) [9]
5. Stanford (3) [10]
6. MIT (4) [29}
7. Yale (20) [27]
8. Australian National University (83) [125]
9. Tokyo (15) [2]
10. Melbourne (189) [52]
11. Peking (352) [50]
12. National University of Singapore (170) [111]
13. Princeton (10) [96]
14. Imperial College London ( 95) [23]
15. Sydney (171) [46}
16. Toronto (18) [3]
17. Kyoto (42) [8]
18. Cornell (16)
19. UCLA (19) [21]
20. Chicago (47) [55]