Trinity College Dublin: A Case of Rankings Abuse

Trinity College Dublin (TCD) is giving itself a public flogging over its fall in the rankings. This is rather odd since in fact it has been doing pretty well over the last few years with one exception.

An article in the Irish Times reveals that the Provost of TCD, Patrick Prendergast, is planning a new business school and student accommodation but finds it difficult to raise the necessary cash.

"The college’s planned expansion comes at a time when many in higher education say the sector faces a funding crisis. Rising student numbers, declining state funding and restrictions on staff recruitment mean that many have had to make dramatic cuts to make ends meet.

Trinity and some of the bigger universities have at least been able to plug many of the funding gaps with private income, such as international students’ fees, research and other commercial sources."

According to the provost things are getting pretty rough.

“We have overcrowded classrooms. Our staff-student ratio is a way out of kilter. The universities have been very resilient; they have managed to keep going successfully. But is it sustainable? I and other university presidents don’t think it is sustainable at current funding levels.”

What does this have to do with rankings.?

"University rankings are considered vital to attracting international students and research funding. However, Trinity, like many Irish universities, has slid down world rankings in recent years as it copes with an increase in students and a reduction in funding.

Last month, TCD found itself at the centre of controversy when one of the main ranking agencies, QS, accused the college of violating its rules by influencing academics involved in its annual survey."

The provost claims:

“If Ireland really wants to be an island known for the talent of its people, and have companies locate here, then we can’t afford to have that one global indicator of the quality of education systems – rankings – decline."

But is it true that TCD is sliding down the rankings?

Let's take a look at the Shanghai Academic Ranking of World Universities (ARWU). This measures research in the natural and social sciences at the highest level. The methodology of these rankings has remained largely stable with some minor exceptions.  In 2004  there were some changes to help social science institutions a bit and in 2014 they began to use a new list of highly cited researchers. The latter move helped TCD a bit in 2014 and, assuming the highly cited researchers do not leave, will help a bit more later this year. 

Therefore, if there has been a significant change in scores in these rankings it is a reasonable assumption that this does actually reflect a change in quality.

Looking at the scores for the separate indicators in the Shanghai rankings, the score for alumni with Nobel and Fields awards fell from 15.4 to 10.3, for faculty with awards from 14.4 to 13.3, and for papers in Nature and Science very slightly from 13.2 to 13.1 between 2004 and 2015.

In contrast, TCD did much better for highly cited researchers (from zero to 12.3), publications in Web of Science journals (27.1 to 31.0) and productivity per capita (the sum of the above indicators divided by number of faculty). For all criteria, the top scoring university, Caltech for productivity and Harvard for everything else, gets 100.

Overall, TCD moved from the 202-300 band to the 151-200 band. A rough guess is that TCD has moved up  about 15 places altogether. About five of these places were gained because of the new highly cited researchers list. At this rate it could be in the top 100 in another century or so. That is very long term but as Basil Fawlty said, "these things take time".

Of course, the Shanghai rankings do not measure teaching, quality of students, income or internationalisation. For a detailed look at a more diverse set of criteria we can turn to the Round University Rankings. These are published by a Russian company but use data from Thomson Reuters, the same source that Times Higher Education (THE) used from 2010 to 2014.  There are, however, 20 indicators compared to the THE's 13 and they cover the period from 2010 to 2016.

Overall, TCD improved significantly, going from 174th to 102th place.

In the teaching dimension, TCD rose slightly from 207th place to 197th, doing slightly better for two doctoral degree indicators, quite a bit worse for two academic staff indicators and staying in exactly the same place for teaching reputation (187th).

For research, TCD improved significantly from 2010 to 2016, going from 193rd to 67th, did much better for papers, citations, completed doctoral degrees and research reputation but did worse for normalised citation impact.

TCD did very well for international diversity in 2010 (31st) but slipped back a bit by 2016 (46th).

For financial sustainability, TCD 's relative position worsened considerably falling from 229th to 412th.

So the data from RUR suggests that TCD's income and faculty resources  were declining relative to other universities over this period.  But so far this has had no significant effect on TCD's teaching profile while research has improved noticeably both in quality and quality.

TCD could quite plausibly claim to be the university with a tiger in its tank, getting more research and educating more students with limited financial and faculty resources.

Turning to the QS scores, in 2004 TCD was ranked 87th in the world and by 2014 had risen to 71st place, but fell back to 78th in 2015. They were still doing better than in 2016. It is not a good idea to draw any conclusions from the decline between 2014 and 2015 because in 2015 QS introduced a number of substantial methodological changes.

To summarise, TCD were doing better over fairly long periods in  the Shanghai,  RUR and QS rankings. Possibly, the fall in the QS rankings between 2014 and 2015 might portend difficulty ahead and perhaps the fall in income documented in RUR may eventually have a knock on effect although so far it has not. Still, it seems that  TCD has on the whole  been doing well. So what  is the Provost talking about?

It is the THE rankings that appear to show TCD in a bad light. In 2010 TCD was in 76th place overall and by 2015 had fallen to 160th. It is difficult to tell exactly what happened because 11 of the 13 indicators are bundled into three super-indicators and it is not clear exactly what contributed to rising or falling scores. In 2015 TCD had higher scores for  international orientation and lower scores for Teaching, Research, Citations and Income from Industry. The biggest  decline was in the Research score, from 45.3 to 30.8.

Clearly, THE is the odd man out as far as TCD is concerned and should not be taken too seriously. Firstly, there were major methodological changes last year which produced upheavals for universities around the world, including those in France, Korea and Turkey. There was another batch of changes in 2011. In addition, these rankings generate a lot of noise because of exchange rate fluctuations, the use of surveys which can be quite volatile outside the top fifty or so, and a citations indicator where (until last year) a single paper or adjunct faculty could produce an enormous change in scores.

THE have said  -- and here they must be given credit -- that:  "Because of changes in the underlying data, we strongly advise against direct comparisons with previous years’ World University Rankings."

It seems that TCD is doing the academic equivalent of taking a dive.