The Trinity Affair Gets Worse

Trinity College Dublin (TCD) has been doing extremely well over the last few years, especially in research. It has risen in the Shanghai ARWU rankings from the 201-300 to the 151-200 band and from 174th to 102nd  in the RUR rankings.

You would have thought that would be enough for any aspiring university and that they would be flying banners all over the place. But TCD has been too busy lamenting its fall in the Times Higher Education  (THE) and QS world rankings, which it attributed to the reluctance of the government to give it as much money as it wanted. Inevitably, a high powered Rankings Steering Group headed by the Provost was formed to turn TCD around.

In September last year the Irish Times reported that the reason or part of the reason for the fall  in the THE world rankings was that incorrect data had been supplied.  The newspaper said that:

"The error is understood to have been spotted when the college – which ranked in 160th place last year – fell even further in this year’s rankings.

The data error – which sources insist was an innocent mistake – is likely to have adversely affected its ranking position both this year and last. "

I am wondering why "sources" were so keen to insist that it was an innocent mistake. Has someone been hinting that it might have been deliberate?

It now seems that the mistake was not just a misplaced decimal point. It was a decimal point moved six places to the left so that TCD reported a total income of 355 Euro, a research income of 111 Euro and 5 Euro income from industry instead of 355, 111, and 5 million respectively. I wonder what will happen to applications to the business school.

What is even more disturbing, although perhaps not entirely surprising, is that THE's game-changing auditors did not notice.

The ten smartest university rankings in the world (or lists if you want to be pedantic)

Paul Greatrix at Wonk HE has just published a list of the ten dumbest rankings in the world. Some I would agree with but the choice of others seems a little odd. He objects to U-Multirank because it is expensive which is unfair when you consider the money that universities are spending on summits, consultancies, audits, ranking task forces and the like. I personally find the Webometrics methodology comprehensible although I admit that I am still not sure about exactly what a bad practice is.

Anyway, the dumbest rankings list should be supplemented with a list of the smartest rankings. Criteria for inclusion are innovative and imaginative methodology, inclusion of formerly marginalised institutions, groups or individuals, cutting edge insights, or significant social utility. They are not in order since they are all, like all rankings and all US liberal arts colleges, unique, some of them extremely so.

• The Campus Squirrel Listings. "The quality of an institution of higher learning can often be determined by the size, health and behavior of the squirrel population on campus." Top of the charts with five acorns are Kansas State University, Rice University, Ursinus College, Lehigh University, Susquehanna University, and the US Naval Academy.

• The Fortunate 500 University Rankings by the Higher School of Economics Moscow uses a brilliantly sophisticated methodology that is unbiased by exam results, teaching or research. Linkoping University in Sweden is number one.

• The Worst Universities in the World is obviously very useful if you want to spot future high fliers. It identifies those universities certified by Webometrics to be without Presence, Openness, Impact, or Google Scholar citations. 

• Ben Sowter of QS has said that his favourite ranking is GreenMetrics because it is the only one in which his alma mater, the University of Nottingham, is top. Similarly, I am very fond of the Research Ranking of African Universities (sorry, dead link) in which my former employer, Umar ibn El-Kanemi College of Education, Science and Technology, Nigeria,  is ranked 988th.

• The Wikipedia Ranking of World Universities measures the historical influence of universities. As far as I know it is the only ranking in which a North Korean university makes an appearance.

• The Times Higher Education World University Rankings and spin offs have  done wonderful work over the years in identifying unsuspected pockets of excellence. Last year they had Anglia Ruskin University in Cambridge equal to Oxford for research impact measured by citations and well ahead of that other place in Cambridge.

• This tradition is continued in the 2017 Asian Universities Rankings which has discovered  that Veltech University is the third best university in India and the best in Asia for  research impact.

• Princeton review's Stone Cold Sober Universities (staying off alcohol and drugs) is very predictable. Brigham Young University in Utah is always first and the higher rankings are filled with service academies and Christian schools. As long as the Air Force Academy stays in the top ten the world can sleep safely.

• Three years ago Huffington Post published a list of the coldest colleges in the USA. Number one was not the university of Alaska but Minnesota State University.

• There does not seem to be a formal ranking of universities that produce comedians but if there was then Cambridge, whose graduates include John Cleese, Peter Cook and Richard Ayoade, would surely be at the top. Oxford would obviously be the best for producing dancers.

Will Asia start rising again?

Times Higher Education (THE) has long suffered from the curse of field-normalised citations which without fail produce interesting (in the Chinese curse sense) results every year.

Part of THE's citation problem is the kilo-paper issue, papers mainly in particle physics with hundreds or thousands of authors and hundreds or thousands of citations. The best known case is 'Combined Measurement of the Higgs Boson Mass in pp Collisions    ...   ' in Physical Review Letters which has 5,154 contributors.

If every contributor to such papers is given equal credit for such citations then his or her institution would be awarded thousands of citations. Combined with other attributes of this indicator this means that a succession of improbable places, such as Tokyo Metropolitan University and Middle East Technical University,  have soared to the research impact peaks in the THE world rankings.

THE have already tried a couple of variations to counting citations for this sort of paper. In 2015 they introduced a cap, simply not counting any paper with more than a thousand authors. Then in 2016 they decided to give a minimum credit of 5% of citations to such authors.

That meant that in the 2014 THE world rankings an institution with one contributor to a paper with 2,000 authors and 2,000 citations would be counted as being cited 2,000 times, in 2015 not at all and in 2016 100 times. The result was that many universities in Japan, Korea, France and Turkey suffered catastrophic falls in 2015 and then made a modest comeback in 2016.

But there may be more to come. A paper by Louis de Mesnard in the European Journal of Operational Research  proposes a new formula -- (n+2)/3n -- so that if a paper has two authors each one gets two thirds of the credit. If it has 2,000 authors each one is assigned 334 citations.

Mesnard's paper has been given star billing in an article in THE which suggests that the magazine is thinking about using his formula in the next world rankings.

If so, we can expect headlines about the extraordinary recovery of Asian universities in contrast to the woes of the UK and the USA suffering from the ravages of Brexit and Trump-induced depression. 

Worth Reading 8

Henk F Moed, Sapienza University of Rome

A critical comparative analysis of five world university rankings

ABSTRACT

To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a data set and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.

" Discussion and conclusions

The overlap analysis clearly illustrates that there is no such set as ‘the’ top 100 universities in terms of excellence: it depends on the ranking system one uses which universities constitute the top 100. Only 35 institutions appear in the top 100 lists of all 5 systems, and the number of overlapping institutions per pair of systems ranges between 49 and 75. An implication is that national governments executing a science policy aimed to increase the number of academic institutions in the ‘top’ of the ranking of world universities, should not only indicate the range of the top segment (e.g., the top 100), but also specify which ranking(s) are used as a standard, and argue why these were selected from the wider pool of candidate world university rankings."

Scientometrics DOI 10.1007/s11192-016-2212-y