Problems with the THE Reputation Rankings

THE has spent a lot of time and words proclaiming that it is trusted by administrators, students, sponsors, and the like. Perhaps it is, but whether it deserves to be is another matter. A recent article in THE  suggests that THE has made a mess of its reputation rankings and is scrambling to put things right.

Until 2021, THE used Elsevier to conduct its teaching and research reputation survey. The 2020-21 survey received 10,963  responses and was calibrated to ensure proper representation of regions and subjects. 

The survey was brought in-house in 2022, and since then, the number of responses has increased substantially to 29,606 in 2022, 38,796 in 2023, and 55,689 in 2024.

When the number of responses increases so dramatically, one should wonder exactly how this was achieved. Was it by sending out more surveys, improving the response rate, or institutional efforts to encourage participation? 

When the results were announced in February, THE declared that a number of Arab universities had achieved remarkable results in the reputation survey. THE conceded that this stellar performance was largely a regional affair that did not extend to the rest of the world. 

But that was not all. Several Arab universities have been making big strides and improving citation, publication, and patent scores: Cairo University, King Abdullah University of Science and Technology, UAE University, and Qatar University. 

The universities getting high scores in the THE rankings were less well-known in the Arab region and had received much lower scores for reputation in the US News and QS rankings. However, they are likely to do well in the forthcoming THE world and Arab university rankings.

THE has now admitted that some universities were encouraging researchers to vote for their own institutions and that there may have been "agreed relationships" between universities. THE is now talking about rewarding respondent diversity, that is getting support from more than just a few institutions.

It is regrettable that THE did not notice this earlier. If it does encourage such diversity, then quite a few universities will suffer dramatic falls in the rankings this year and next.

Anyway, THE could do a few things to improve the validity of its reputation survey. It could eliminate self-voting altogether, give a higher weighting to votes from other countries, as QS does, add a separate ranking for regional reputation, and combine scores for a number of years.

The problems with the reputation metrics seem to have begun with THE starting its own survey. It would be a good idea to go back to letting Elsevier do the survey. THE is undeniably brilliant at event management and public relations, although perhaps not jaw-droppingly so. However, it is not so good at rankings or data processing.

  

Imperial Ascendancy

The 2025 QS World University Rankings have just been announced. As usual, when there are big fluctuations in scores and ranks, the media are full of words like soaring, rising, plummeting, and collapsing. This year, British universities have been more plummeting than soaring, and this has generally been ascribed to serious underfunding of higher education by governments who have been throwing money at frivolities like childcare, hospitals, schools, roads, and housing.

There has been a lot of talk about Imperial College London rising to second in the world and first in the UK, ahead of Harvard, Oxford, and Cambridge. Imperial's president, quoted in Imperial News, spoke about quality, commitment, and "interrogating the forces that shape our world."

The article also referred to the university's achievements in the THE world rankings, the Guardian University Guide, and the UK's Research and Teaching Excellence Frameworks. It does not mention that Round University Ranking has had Imperial first in the UK since 2014.

So what exactly happened to propel Imperial ahead of Harvard, Oxford, and Cambridge? Perhaps commitment and the interrogation of forces were there in the background, but the more proximate causes were the methodological changes introduced by QS last year. There have been no further changes this year, but the QS rankings do seem to have become more volatile.

In 2023, QS introduced three new indicators. The first is the International Research Network, which measures the breadth rather than the quantity of international research collaborations. This favored universities in English-speaking countries and led to a reported boycott by South Korean universities. 

That boycott does not seem to have done Korean universities any harm since many of them have risen quite significantly this year.

QS has also added an Employment Outcomes metric that combines graduate employment rates and an alumni index of graduate achievements scaled against student numbers. 

Then there is a sustainability indicator based on over fifty pieces of data submitted by institutions. Some reputable Asian universities get low scores here, suggesting that they have not submitted data or that the data has been judged inadequate by the QS validators.

Imperial rose by exactly 0.7 points between the 2024 and the 2025 world rankings, while Harvard, Oxford, and Cambridge all fell. Its score declined for three indicators, Faculty Student Ratio, Citations per Faculty, and International Students, and remained unchanged for International Faculty.

The improvement in the weighted score of five indicators is listed below:

Employment Outcomes                      0.52

Sustainability                                     0.265

Academic Reputation                        0.15

International Research Network       0.035

Employer Reputation                        0.015.

Imperial has improved for all of the new indicators, very substantially for Employments Outcomes and Sustainability, and also for the reputation indicators. I suspect that the Imperial ascendancy may not last long as its peers, especially in Asia, pay more attention to the presentation of employability and sustainability data

Hungarian universities, this is probably not a good idea

Times Higher Education (THE) has informed us that it has reached a "groundbreaking" agreement with the Hungarian Ministry of Culture and Innovation.

It seems that THE will analyse Hungary's higher education system and benchmark with successful higher education hubs according to the "gold standard" world rankings and provide advice and "unparalleled data insights" to Hungarian universities. The cost of this exercise is not mentioned, but it is unlikely to be trivial.

The Hungarian State Secretary for Innovation and Higher Education referred to the presence of Hungarian universities in the THE rankings. Eleven are now in the THE world rankings whereas five years ago seven were listed. 

That sounds very impressive, but wait a minute.

THE tells us in the 2018-19 rankings, there were 1258 universities, of which 1250 were ranked, and in 2023-24, there were 2671, of which 1906 were ranked. It would be remarkable if the number of Hungarian universities did not increase, and it is no big deal that they did.

What is relevant is the number of universities in the top thousand in each edition. For Hungary, it was six in the 2019 rankings and three in 2024. If the THE rankings mean anything, then the quality of  Hungarian universities has apparently declined over the last five years. 

Hungarian universities, however, have generally been drifting downwards in most rankings, not because they are getting worse in absolute terms but because of the steady rise of Asian, especially Chinese, research-based universities. 

Moreover, the THE world rankings rate Hungarian universities worse than any other global ranking. The latest edition of the THE World University Rankings  (WUR) shows three in the world's top 1000. There are five in the top 1000 in the latest QS rankings, four in the Shanghai rankings, five in Leiden Ranking, four in the US News Best Global Universities, four in URAP, five in CWUR, six in Webometrics, and eight in RUR.

The pattern is clear. THE now consistently underestimates the performance of Hungarian universities compared to other rankers. Not only that but some Hungarian universities have dropped significantly in the THE rankings. Eotvos Lorand University has gone from 601-800 to 801-1000, Pecs University from 601-800 to 1001-1200 and Budapest University of Technology and Economics from 801-1000 to 1201-1500.

On the other hand, a couple of Hungarian universities, Semmelweis and Debrecen, have risen through participation in multi-author multi-citation projects.

It is difficult to see what benefit Hungary will get from paying THE for insights, reports, and targets from an organization that has limited competence in the assessment and analysis of academic performance. Seriously, what insights could you get from an organization that in recent years has declared Anglia Ruskin University to be the world leader for research impact, Anadolu University for knowledge transfer, and Macau University of Science and Technology for International Outlook?

It is true that THE is outstanding in public relations and event management, and the universities will no doubt benefit from high praise at prestigious events and receive favourable headlines and awards. It is hard, though, to see that THE are able to provide the knowledgeable and informed advice that universities need to make difficult decisions in the coming years.