Energetic and Proactive Leadership or a Defective Indicator?

It is now normal for university administrators to use global rankings to market their institutions, reward themselves and, all too often, justify demands for more public funding. Unfortunately, some ranking agencies are more than happy to go along with this.

One example of the misuse of rankings is from July of last year  when the Vice-Chancellor of the University of the West Indies, Sir Hilary Beckles, proclaimed a triple first  in the 2020 Times Higher Education (THE) rankings.

The original  target was to be be in the top 3% of ranked universities by the end of the current strategic planning cycle. The Vice-Chancellor was very happy that as a result of  the work of "an energetic and proactive leadership team of campus principals and others" the university was in the top 1% of the THE Latin America and Caribbean rankings and the top 1% of golden age universities, and was the only ranked university in the Caribbean.

Another article has just appeared lauding the achievements of the university. The Vice-Chancellor has asserted that "(t)he Times Higher Education informed us that what we have achieved is quite spectacular. That many universities had taken 30 years to achieve what we have achieved in a mere three years."

If this is an accurate report of what THE said then the magazine is being very irresponsible. Universities are complex structures, and cannot be turned around with just a few million dollars or a few dozen highly cited researchers. Without drastic restructuring or amalgamation, such a remarkable change is almost invariably the result of methodological changes or methodological defects. The latter is the case here.

UWI 's performance might appear  quite impressive but it seems that we have another case of  THE seeing things that nobody else can. THE is not the only global university ranking. In fact, it is in some important respects not a good one. Let's take a look at some other rankings. UWI is not ranked in Leiden Ranking, the Shanghai Academic Ranking of World Universities, the US News Best Global Universities, or the QS World University Rankings.

It does make an appearance in University Ranking by Academic Performance (URAP) published by Middle East Technical University where it is ranked 1622th and it is 1938th in the Center for World University Rankings.

But in the THE World University Rankings the university is in the top 600 and it is 18th in Latin America and the Caribbean. This is because of a single defective indicator.

UWI has an apparently healthy 81.1 in the THE world rankings citations indicator, supposedly a measure of research influence or impact, which at first sight seems odd since it has a very low score, 10.4, for research. In the Latin American rankings, where competition is less severe, that becomes  93.6 for citations and 75.8 for research. How could a university do so brilliantly for research influence when it has a poor reputation for research, doesn't publish many papers, and has little research income?

What has happened is that UWI has benefitted from THE's peculiar citations indicator that does not use fractional counting for papers with less than a hundred authors, plus hyper-normalisation, plus a country bonus for being located in a low performing country, resulting in a few multi-author, multi-cited papers pushing universities into positions that they could never achieve anywhere else.

UWI has recently contributed to a few papers with a large number of contributors and many citations in genetics and medicine in top journals including Lancet, Science, Nature, and the International Journal of Surgery. Five of these papers are linked with the Global Burden of Disease Study.

UWI is  to be congratulated for having a number of affiliated scientists who are taking part in cutting edge projects. Nonetheless, it is true that these have an impact only because of the technical defects of the THE rankings. In the last few years a succession of unlikely places -- Anglia Ruskin, Peradeniya, Reykjavik, Aswan, Brighton and Sussex Medical School, Durban University of Technology, have risen to the top of the citations charts in the THE rankings.

They have done so nowhere else. The exalted status of UWI in the THE WUR and its various offshoots is due to an eccentric methodology. If THE reforms that methodology, something that they have been talking about for a long time, or abandons the world rankings, the university will once again be consigned to the outer darkness of the unranked.

The sensational fall of Panjab University and the sensational coming rise*

*If present methodology continues

The Hindustan Times has a report on the fall of Panjab University (PU) in the latest THE Emerging Economies Rankings. PU has fallen from 150th in 2019 to 201-250 this year. As expected, rivals and the media have launched a chorus of jeers against the maladministration of a once thriving institution.

But that isn't half of it. In 2014, the first year of these rankings, PU was ranked 14th in the emerging world and first in India ahead even of the highly regarded IITs. Since then the decline has been unrelenting. PU was 39th in 2015, 130th in 2018, 166th in 2020.

Since 2014 PU's scores for Research, Teaching and Industry Income have risen. That for International Outlook has fallen but that did not have much of an effect since it accounts for only 10 per cent of the total weighting.

What really hurt PU was the Citations indicator. In 2014 PU received a score of 84.7 for citations largely because of its participation in the mega-papers radiating from the Large Hadron Collider project, which have thousands of contributors and thousands of citations. But then THE stopped counting such papers and consequently in 2016 PU's citation score fell to 41 and its overall place to 121st. It has been downhill nearly every year since then.

The Director of PU's Internal Quality Assurance Cell said that the university has improved for research and teaching but needed to do better for international visibility and that proactive steps had been taken. The former Vice-Chancellor said that other universities were improving but PU was stagnating and was not even trying to overcome its weaknesses.

In fact, PU's decline had  nothing or very little  to do with international visibility or the policy failures of the administration just as its remarkable success in 2014 had nothing to do with working as a team, the strength brought by diversity, dynamic transformational leadership, or anything else plucked from the bureaucrats' big bag of clichés.

PU bet the farm on citations of particle physics mega-papers and for a couple of years that worked well. But then THE changed the rules of the game and stopped counting them, although after another year they received a reduced amount of credit. PU, along with the  Middle East Technical University and some other Turkish, Korean and French institutions that had overinvested in the CERN projects, tumbled down the THE rankings.

But PU may be about to make a comeback. Readers of this blog will probably guess what is coming next.

When THE stopped favouring papers with thousands of contributors they abolished the physics privilege that elevated places like Tokyo Metropolitan University, Federico Santa Maria Technical University, and Colorado School of Mines to the top of the research impact charts and replaced it with medical research privilege .

In 2018 PU started to publish papers that were part of the Gates-backed Global Burden of Disease Study (GBDS). Those papers have hundreds, but not thousands, of authors and so they are able to slip through the mega paper filter. The GBDS has helped Anglia Ruskin University, Brighton and Sussex Medical School, Aswan University, and Kurdistan University of Medical Sciences rise to the top of the research impact metric.

So far there have not been enough citations to make a difference for PU. But as those citations start to accumulate, providing of course that their impact is not diluted by too many other publications, PU will begin to rise again.

Assuming of course that THE's Methodology does not change.