Is something happening in China?

The National Taiwan University rankings have been overlooked by the Western media, which is a shame since they can provide useful and interesting insights. 

For example, there are indicators for articles in the SCIE and the SSCI of the Web of Science database over 11 years and over the current year, which for this year's edition is 2023. For both metrics, the top scorer, which in these cases is Harvard, is assigned a score of 100, and the others are calibrated accordingly.

If a university has a score for the one-year indicator that is significantly higher than the score for eleven years, it is likely that they have made significant progress during 2023 compared to the previous decade. Conversely, if a university does much better for the eleven-year indicator than for the current year, it could mean that it has entered a period of low productivity.

Looking at the current ranking, we notice that most leading US, British, and Australian universities are doing well for the current year, with the notable exceptions of the Los Angeles, Berkeley, San Diego, and Davis campuses of the University of California. Saudi universities also do well, but French universities are down for the year.

The big story here is that Chinese universities do much worse for the current year than the 11-year period. Here are the Article scores for five leading institutions:

Tsinghua University 57.9  for eleven years and  47.2 for the current year

Zhejiang  University 64.7 and 55.4

Shanghai Jiao Tong University 65 and 52.8

Peking University 57.1 and 48

Sun Yat-Sen University 54.1 and 47.1.

And so on and so on.

So what is going on? I can think of several possible explanations. Firstly, we are seeing the temporary effect of the Covid restrictions, and soon we shall see a rebound.

Secondly, this is the beginning of a new period of decline for Chinese sciences, and we shall see a further decline in the next few years.

Thirdly, and I think most plausibly, China has lost interest in engagement with the West, whether this means partnerships with elite institutions, publications in scientific journals, or participation in surveys and rankings. This aligns with the abstention from the THE Impact rankings. the lack of data submission to the TOP 500 international ranking of supercomputers, and low scores in the   QS sustainability rankings, which suggests a lack of interest in those metrics.

Whatever the reason, we should have a better idea over the next year or two.

 

China vs the West: Snow’s ‘two culture’ theory goes global

 

Published today in University World News

In 1959, C P Snow, a British scientist, civil servant and novelist, created a stir with a lecture, “The Two Cultures and the Scientific Revolution”. The two cultures were led by natural scientists and literary intellectuals.

There was no doubt about where Snow stood with regard to the cultures. Scientists, he said, had “the future in their bones”, and he was disdainful of those who were ignorant of the basic laws of physics.

He believed that Britain’s stagnation after the Second World War was the result of the domination of public life by humanities graduates and the marginalisation of natural scientists.

Snow’s lecture was met with an equally famous ad hominem blast from the Cambridge literary critic, F R Leavis, which probably did Snow more good than harm. Leavis may, however, have had a prescient point when he talked about how science had destroyed the organic communities of the pre-industrial world.

At the time, his nostalgia was largely misplaced. Those who lived in the villages and farms of England had little reluctance about moving, as did my forebears, to the cotton mills of Derbyshire and the coal mines of South Wales, but, looking at a world where every human instinct has become digital media fodder, Leavis might have been onto something.

It now looks like we have something like Snow’s two cultures emerging at the global level with their centres in China, and in North America and Western Europe.

India and the THE Impact Rankings

The World Economic Forum (WEF), supposedly the voice of the global economic and political elites, recently published an article by Phil Baty, Chief Global Affairs Officer of Times Higher Education (THE), about Indian universities and their apparent progress towards world-class status, shown by their participation and performance in the THE Impact Rankings, which measure universities’ contributions to the UN’s Sustainable Development Goals (SDGs).

This is misleading and irresponsible. Participation, or even a high score, in the Impact Rankings, whether overall or for specific indicators, has little, if anything, to do with the ability of universities to provide instruction in academic and professional subjects or to pursue research, scholarship, and innovation. Indeed, it is difficult to see how many of the criteria used in the Impact Rankings are relevant to attaining the SDGs.

The article begins by quoting Philip Altbach, who said in  2012   that India was a world-class country without world-class universities. That in itself is an interesting comment. If a country can be world-class without world-class universities, then one wonders if such universities are really essential.

There is a bit of bait and switch here. Whatever Altbach meant by world-class in 2012, I doubt that he was referring to performance in meeting the UN’s SDGs.

Baty goes on to claim that Indian universities are improving, and this is shown by Indian universities submitting data for THE impact rankings, which assess universities' contribution to the SDGs, 125 compared with 100 from Türkiye and 96 from Pakistan, out of a total of  2152 universities around the world.

That sounds impressive. However, submissions to the impact rankings and other THE products are voluntary, as THE often points out. There is no real merit involved in filling out the forms except perhaps showing a need to be ranked for something.

In any case, according to the uniRank site, there are 890 higher education institutions in India, 174 in Türkiye, and 176 in Pakistan. That means that the participation rate is about 14% for India, 57% for Türkiye, and 55% for Pakistan. India's participation in THE Impact Rankings is less than that of Pakistan and Türkiye, and in previous years, it has been much less than that of countries like Algeria, Iran, and Iraq.

Nor does gaining a high score in the Impact Rankings tell us very much. Universities are ranked on their four best scores. Many universities simply submit data for five or six goals and just ignore the others, for which their actual contribution might well be zero or negative.

These rankings rely heavily on data submitted by universities. Even if everybody concerned with the collection, transfer, and processing of information is totally honest and competent, there are often immense obstacles to data curation confronting universities in the Middle East, Africa, and Latin America. These rankings may be, in effect, little more than a measure of the ambitions of university leaders and the efficiency of their data analysts.

Moreover, much of the progress toward these goals is measured not by hard, verifiable data but by targets, programs, initiatives, partnerships, facilities, policies, measures, and projects that are subject to an opaque and, one suspects, sometimes arbitrary validation process.

Also, do the criteria measure progress toward the goals? Does producing graduates in law, civil enforcement, and related fields really contribute to peace, justice, and strong institutions? Does a large number of graduates qualified to teach say much about the quality of education?

It might be commendable that a minority of Indian universities, albeit proportionately less than many other countries, have signed up for these rankings and that a few have done well for one or two of the SDGs. It is helpful to know that JSS Academy of Higher Education and Research is apparently a world beater for good health and well-being, Shoolini University of Biotechnology and Management for clean water and sanitation, and Saveetha Institute of Medical and Technical Sciences for affordable and clean energy, but does this really compensate for the pervasive perceived mediocrity of Indian higher education?

The validity of the Impact Rankings can be checked by comparing them with the UI GreenMetric Rankings, which have measured universities' commitment to environmental sustainability since 2010. Some of the indicators here, such as Energy and Climate Change and Water, are similar, although not identical, to those in the Impact Rankings, but there is almost no overlap between the best-performing universities in the two rankings. No doubt THE would say their rankings are more sophisticated but still, even the least cynical observer might wonder a bit.

The reality is that Indian universities have consistently underperformed in the various global rankings, and this is, on balance, a fairly accurate picture. It is probable that current reforms will bring widespread change, but that is still something on the horizon.

Here, THE has not been helpful. Over the last few years, It has repeatedly exaggerated the achievements of a few Indian institutions that have risen in their world or regional rankings, often due to the dysfunctional citations indicator. These include Panjab University, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, JSS Academy of Higher Education and Research, and the Ropar and Indore Institutes of Technology. This has caused resentment among leading Indian institutions, who are perplexed by such relatively marginal places zooming ahead of the highly reputable Indian Institutes of Technology of Bombay, Madras, and Delhi.

The article ignores the boycott by the leading Indian Institutes of Technology (IITs) of the THE World University Rankings partly because of their opacity, where all the metrics are now bundled into pillars, so it is next to impossible to figure out what is causing movement in the rankings without paying THE for consultation and benchmarking.

Indian universities have not performed well in global rankings. In the Shanghai Rankings, the best performer is the Indian Institute of Science in the 401-500 band, down from 301-400 in 2023. In the CWTS Leiden Ranking, the leading university is the Indian Institute of Technology Kharagpur in 284^th place. Compared to China, Japan, and South Korea, India’s performance is rather tepid. The occasional show of excellence with regard to one or two of the SDGs is hardly sufficient compensation.

The current reforms may put Indian research and higher education on track, but India’s problems go deeper than that. There is widespread evidence that the country is lagging far behind in primary and secondary education, and ultimately, that will matter much more than the exploits of universities on the way to meeting sustainability goals.

 

It seems that self-affirmation isn't such a big deal

 

A few years ago, I wrote about a massively cited study in Science, supposedly a leading scientific journal, that claimed to significantly reduce the racial high school achievement gap. The idea was that having low-achieving students write about values important to themselves would start a recursive process leading to an improvement in their relative academic performance. The positive effect of this self-affirmation intervention was conveniently confined to African-American students, which, I suspect, contributed to the paper's acceptance.

I was sceptical, having once taught English in classrooms, that 15 minutes of writing could have such a remarkable impact, and I wondered about whether the abundance of resources, support, and skills in the school under study might have compromised the anonymity of the subjects.

Now it seems that the study was "seriously underpowered" and "always obviously wrong".

How many more politically convenient studies will turn out to be wrong or perhaps even worse than wrong? 

Forget about the Euros, this is really serious

We are told that the failure at the UEFA final was a tragedy for England. Perhaps, but something else happened early in July that should have caused some debate but passed almost unnoticed, namely the publication of the latest edition of the CWTS Leiden Ranking.

The release of the Times Higher Education (THE) World University rankings and, to a lesser extent, the global rankings from Shanghai, QS, and US News (USN) are often met with fulsome praise from the media and government officials when national favourites rise in the rankings and lamentations when they fall, but other rankings, often much more reliable and rigorous, are largely ignored.

This is partly because the THE and QS rankings are dominated by American and British universities. Oxford, Cambridge, and Imperial College London are in the top ten in the overall tables in these three rankings. This year there was a lot of media talk about Imperial moving ahead of Cambridge and Oxford into second place in the QS rankings, just behind MIT. According to these rankings, British universities are on top of the world and criticism from journalists or politicians would surely be churlish in the extreme. 

It would, however, be a mistake to assume that the big brand rankings are objective judges of academic merit or any other sort. They are biased towards UK universities in a variety of obvious and subtle ways. QS, THE, and USN all include surveys of established academics, and the Shanghai Rankings include Nobel and Fields award winners, some of whom are long gone or retired. THE has three metrics based on income. THE USN, and QS give more weight to citations rather than publications, loading the dice for older and better-funded researchers. 

It seems that British universities have complacently accepted the verdict of these rankings and appear unwilling to consider that they are doing anything less than perfect. When the Sunak government proposed some vague and  bland  changes, the Chief Executive of the London Higher Group of Institutions complained that it was "beyond belief" that the government should have the King speak negatively of the "world-leading higher education and research sector." 

It is perhaps time to look at another ranking, one produced by the Centre for Science and Technology Studies (CWTS) at Leiden University. This provides data on publications with various optional filters for subject group, country, period, and fractional counting. There are also rankings for international and industrial collaboration, open-access publications, and gender equity in research.

CWTS does not, however, publish overall rankings, sponsor spectacular events in prestigious settings, or offer consultations and benchmarking services for non-trivial sums. Consequently, it is usually neglected by the media, university heads, or the grandees of the world economy gathered at WEF forums and the like.

Turning to the latest edition,  starting with the default metric, publications in the Web of Science over the period 2019-2022, we see that Zhejiang University has now overtaken Harvard and moved into first place. In the next few years, it is likely that other Chinese universities like Fudan, Peking, and Tsinghua will join Zhejiang at the peak. 

But the most interesting part of Leiden Ranking is the steady decline of British universities. Oxford is now 25th  in the publications table, down from 14th in 2009-12. That's not too bad, but rather different from the latest QS world ranking, where it is third, US News Best Global Universities, where it is fourth, or THE, where it is first. Oxford is well behind several Chinese universities and also behind, among others, the University of Sao Paulo, Seoul National University, and the University of Pennsylvania.

Of course, you could say that this is a crude measure of research activity and that if we look at other metrics, such as publications in the top 10% and the top 1% of journals, then, yes, Oxford does better. The problem is that the high-quality metrics are usually lagging indicators so we can expect Oxford to start declining there also before too long.

When we look at the broad subject tables for publications, there is further evidence of gradual decline.  For Mathematics and  Computer Science, Oxford is 63rd, behind Purdue University, Beijing University of Technology, and the University of New South Wales. In 2009-12 it was 50th. 

For Physical Sciences and Engineering, it is 72nd behind the  University of Tehran, Texas A & M, and Lomonosov Moscow State University. In 2009-12 it was 29th.

It is 64th in Life and Earth Sciences, behind Lanzhou University, the Swedish University of Agricultural Science, and Colorado State University. In 2009-2012 it was 42nd. 

For Biomedical and Health Sciences, it is 39th, behind Duke, University of British Columbia, and Karolinska Institutet; in 2009-2012, it was 27th.

Finally, when it comes to the Humanities and Social Scientists, Oxford remains at the top. It is fourth in the world, just as it was in 2009-2012. 

A glance at some middling British institutions shows the same picture of steady relative decline. Between 2009-2012 and 2019-2022 Reading went from 489th to 719th, Liverpool from 233rd to 302nd, and Cardiff from 190th to 328th. 

It is perhaps unfair to judge complex institutions based on a single metric. Unfortunately, most science, scholarship, and everyday life are based on assigning numbers that may ignore the fine details of indispensable complex phenomena. 

Also, such data does not tell us the full story about teaching and learning, but there is plenty of anecdotal evidence that British universities are not doing so great there either. 

It seems that the big rankings are exaggerating the merits of British higher education. It is time to take a look at some of the global in  rankings produced in places like the Netherlands (Leiden Ranking), Spain (SCImago), Turkiye (URAP), Georgia (RUR), and Taiwan (NTU rankings).

Are British Universities Really Underfunded?

I noticed this on LinkedIn recently. An item by Phil Baty, Chief Global Affairs Officer at Times Higher Education (THE), claims that British universities are seriously underfunded and that their world-class achievements are endangered.

He reports that a brilliant data analyst has revealed that inflation has eroded the value of the tuition fees that UK universities are allowed to charge and that costs have dramatically increased. 

Then we have a graph from a THE data guru that compares British university performance in the 18 metrics used in the current THE world rankings to that of their international peers. The UK is well ahead of the world average of the top 500 universities in the most recent world university rankings for the international outlook indicators and significantly for research strength, field-weighted citations, and publications. It is slightly ahead for research excellence, research reputation, research influence, and patents.

However, when it comes to institutional income, research income, and industry income, British universities are apparently way behind the rest of the world. So, it seems that THE has conclusively demonstrated that UK universities are seriously short of money.

But there are a few things that need to be considered.

First, the THE income indicators are all divided by the number of academic staff. To do well in these measures, a university could have substantial income, or at least report that it did, or it could reduce the number of faculty reported.

In other words, a university that decided to spend its money recruiting teaching and/or research staff would fall in the THE rankings. If it sacked a lot of teachers and researchers, it would be rewarded with a significant improvement. You might think that is a bit bonkers, but that is the unintended consequence of the THE  methodology. I do not know which applies to British universities in general or specifically, but it would be interesting to see a breakdown of the data.

Also, remember that the income indicators are based on data submitted by institutions. It would be unwise to assume that these are completely valid and accurate. A few years ago Alex Usher of HESA published an article showing that there were some serious problems with THE's industry income indicator. I am not sure whether it has improved since then.

Also, we should note that 55 UK universities are in the current THE world top 500. According to Webometrics, there are 31,657 universities worldwide and 355 in the UK. THE is, in effect, claiming that the top 15.49% of British universities, according to THE's criteria, are underfunded compared to the top 1.58% of world universities in general. 

Before signing off, the graph is instructive in that it shows that the rankings are massively biased toward British universities. Consider the weighting for the various metrics. 

The International Outlook pillar has a 7.5% weighting, research quality, that is citations, 30%, teaching and research reputation 33%, and publications per staff 5.5%. These are all criteria where British higher education does better than the world average.

In contrast, the three income metrics, where UK universities do badly, are given weightings of 2.5%, 5.5%, and 2% respectively. 

If THE decided to shift some of its weighting from reputation to income or to doctoral education, which the UK sector also does badly, its THE rank would fall very noticeably.

Problems with the THE Reputation Rankings

THE has spent a lot of time and words proclaiming that it is trusted by administrators, students, sponsors, and the like. Perhaps it is, but whether it deserves to be is another matter. A recent article in THE  suggests that THE has made a mess of its reputation rankings and is scrambling to put things right.

Until 2021, THE used Elsevier to conduct its teaching and research reputation survey. The 2020-21 survey received 10,963  responses and was calibrated to ensure proper representation of regions and subjects. 

The survey was brought in-house in 2022, and since then, the number of responses has increased substantially to 29,606 in 2022, 38,796 in 2023, and 55,689 in 2024.

When the number of responses increases so dramatically, one should wonder exactly how this was achieved. Was it by sending out more surveys, improving the response rate, or institutional efforts to encourage participation? 

When the results were announced in February, THE declared that a number of Arab universities had achieved remarkable results in the reputation survey. THE conceded that this stellar performance was largely a regional affair that did not extend to the rest of the world. 

But that was not all. Several Arab universities have been making big strides and improving citation, publication, and patent scores: Cairo University, King Abdullah University of Science and Technology, UAE University, and Qatar University. 

The universities getting high scores in the THE rankings were less well-known in the Arab region and had received much lower scores for reputation in the US News and QS rankings. However, they are likely to do well in the forthcoming THE world and Arab university rankings.

THE has now admitted that some universities were encouraging researchers to vote for their own institutions and that there may have been "agreed relationships" between universities. THE is now talking about rewarding respondent diversity, that is getting support from more than just a few institutions.

It is regrettable that THE did not notice this earlier. If it does encourage such diversity, then quite a few universities will suffer dramatic falls in the rankings this year and next.

Anyway, THE could do a few things to improve the validity of its reputation survey. It could eliminate self-voting altogether, give a higher weighting to votes from other countries, as QS does, add a separate ranking for regional reputation, and combine scores for a number of years.

The problems with the reputation metrics seem to have begun with THE starting its own survey. It would be a good idea to go back to letting Elsevier do the survey. THE is undeniably brilliant at event management and public relations, although perhaps not jaw-droppingly so. However, it is not so good at rankings or data processing.

  

Imperial Ascendancy

The 2025 QS World University Rankings have just been announced. As usual, when there are big fluctuations in scores and ranks, the media are full of words like soaring, rising, plummeting, and collapsing. This year, British universities have been more plummeting than soaring, and this has generally been ascribed to serious underfunding of higher education by governments who have been throwing money at frivolities like childcare, hospitals, schools, roads, and housing.

There has been a lot of talk about Imperial College London rising to second in the world and first in the UK, ahead of Harvard, Oxford, and Cambridge. Imperial's president, quoted in Imperial News, spoke about quality, commitment, and "interrogating the forces that shape our world."

The article also referred to the university's achievements in the THE world rankings, the Guardian University Guide, and the UK's Research and Teaching Excellence Frameworks. It does not mention that Round University Ranking has had Imperial first in the UK since 2014.

So what exactly happened to propel Imperial ahead of Harvard, Oxford, and Cambridge? Perhaps commitment and the interrogation of forces were there in the background, but the more proximate causes were the methodological changes introduced by QS last year. There have been no further changes this year, but the QS rankings do seem to have become more volatile.

In 2023, QS introduced three new indicators. The first is the International Research Network, which measures the breadth rather than the quantity of international research collaborations. This favored universities in English-speaking countries and led to a reported boycott by South Korean universities. 

That boycott does not seem to have done Korean universities any harm since many of them have risen quite significantly this year.

QS has also added an Employment Outcomes metric that combines graduate employment rates and an alumni index of graduate achievements scaled against student numbers. 

Then there is a sustainability indicator based on over fifty pieces of data submitted by institutions. Some reputable Asian universities get low scores here, suggesting that they have not submitted data or that the data has been judged inadequate by the QS validators.

Imperial rose by exactly 0.7 points between the 2024 and the 2025 world rankings, while Harvard, Oxford, and Cambridge all fell. Its score declined for three indicators, Faculty Student Ratio, Citations per Faculty, and International Students, and remained unchanged for International Faculty.

The improvement in the weighted score of five indicators is listed below:

Employment Outcomes                      0.52

Sustainability                                     0.265

Academic Reputation                        0.15

International Research Network       0.035

Employer Reputation                        0.015.

Imperial has improved for all of the new indicators, very substantially for Employments Outcomes and Sustainability, and also for the reputation indicators. I suspect that the Imperial ascendancy may not last long as its peers, especially in Asia, pay more attention to the presentation of employability and sustainability data

Hungarian universities, this is probably not a good idea

Times Higher Education (THE) has informed us that it has reached a "groundbreaking" agreement with the Hungarian Ministry of Culture and Innovation.

It seems that THE will analyse Hungary's higher education system and benchmark with successful higher education hubs according to the "gold standard" world rankings and provide advice and "unparalleled data insights" to Hungarian universities. The cost of this exercise is not mentioned, but it is unlikely to be trivial.

The Hungarian State Secretary for Innovation and Higher Education referred to the presence of Hungarian universities in the THE rankings. Eleven are now in the THE world rankings whereas five years ago seven were listed. 

That sounds very impressive, but wait a minute.

THE tells us in the 2018-19 rankings, there were 1258 universities, of which 1250 were ranked, and in 2023-24, there were 2671, of which 1906 were ranked. It would be remarkable if the number of Hungarian universities did not increase, and it is no big deal that they did.

What is relevant is the number of universities in the top thousand in each edition. For Hungary, it was six in the 2019 rankings and three in 2024. If the THE rankings mean anything, then the quality of  Hungarian universities has apparently declined over the last five years. 

Hungarian universities, however, have generally been drifting downwards in most rankings, not because they are getting worse in absolute terms but because of the steady rise of Asian, especially Chinese, research-based universities. 

Moreover, the THE world rankings rate Hungarian universities worse than any other global ranking. The latest edition of the THE World University Rankings  (WUR) shows three in the world's top 1000. There are five in the top 1000 in the latest QS rankings, four in the Shanghai rankings, five in Leiden Ranking, four in the US News Best Global Universities, four in URAP, five in CWUR, six in Webometrics, and eight in RUR.

The pattern is clear. THE now consistently underestimates the performance of Hungarian universities compared to other rankers. Not only that but some Hungarian universities have dropped significantly in the THE rankings. Eotvos Lorand University has gone from 601-800 to 801-1000, Pecs University from 601-800 to 1001-1200 and Budapest University of Technology and Economics from 801-1000 to 1201-1500.

On the other hand, a couple of Hungarian universities, Semmelweis and Debrecen, have risen through participation in multi-author multi-citation projects.

It is difficult to see what benefit Hungary will get from paying THE for insights, reports, and targets from an organization that has limited competence in the assessment and analysis of academic performance. Seriously, what insights could you get from an organization that in recent years has declared Anglia Ruskin University to be the world leader for research impact, Anadolu University for knowledge transfer, and Macau University of Science and Technology for International Outlook?

It is true that THE is outstanding in public relations and event management, and the universities will no doubt benefit from high praise at prestigious events and receive favourable headlines and awards. It is hard, though, to see that THE are able to provide the knowledgeable and informed advice that universities need to make difficult decisions in the coming years. 

What happens to those who leave THE?

Times Higher Education (THE) appears to be getting rather worried about leading universities such as Rhodes University, University of Zurich, Utrecht University, and some of the Indian Institutes of Technology boycotting its World University Rankings (WUR) and not submitting data.

Thriving Rankings?

We have seen articles about how the THE rankings are thriving, indeed growing explosively. Now, THE has published a piece about the sad fate that awaits the universities that drop out of the WUR or their Impact Rankings. 

Declining Universities?

An article by two THE data specialists reports that 611 universities that remained in the THE world rankings from 2018 to 2023 retained, on average, a stable rank in the THE reputation ranking. The 16 who dropped out saw a significant decline in their reputation ranks, as did 75 who are described as never being in the WUR.

The last category is a bit perplexing. According to Webometrics, there are over 30,000 higher education institutions in the world and nearly 90,000, according to Alex Usher of HESA. So, I assume that THE is counting only those that got votes or a minimum number of votes in their reputation ranking. 

We are not told who the 75 never-inners or the 16 defectors are, although some, such as six Indian Institutes of Technology, are well known, so it is difficult to check THE's claims. However, it  is likely that an institution that boycotted the THE WUR would also discourage its faculty from participating in the THE academic survey, which would automatically tend to reduce the reputation scores since THE allows self-voting.

Also, we do not know if there have been changes in the weighting for country and subject and how that might modify the raw survey responses. A few years ago, I noticed that Oxford's academic reputation fluctuated with the percentage of survey responses from the humanities. It is possible that adjustments like that might affect the reputation scores of the leavers. 

The opacity of THE's methodology and the intricacies of its data processing system mean that we cannot be sure about THE's claim that departure from the world rankings would have a negative impact. In addition, there is always the possibility that universities on a downward trend might be more likely to pull out because their leaders are concerned about their rankings, so the withdrawal is a result, not the cause of the decline. 

We should also remember that reputation scores are not everything. If a decline in reputation was accompanied by an improvement in other metrics, it could be a worthwhile trade.

What happened to the IITs in the THE WUR?

Fortunately, we can check THE's claims by looking at a group of institutions from the same country and with the same subject orientation. In the 2019-20 world rankings, twelve Indian Institutes of Technology were ranked. Then, six -- Bombay, Madras, Delhi, Kanpur, Kharagpur, Roorkee --  withdrew from the WUR, and six -- Ropar, Indore, Gandhinagar, Guwahati, Hyderabad, Bhubaneswar --  remained, although two of these withdrew later. 

So, let's see what happened to them. First, look at the overall ranks in the WUR itself and then in Leiden Ranking, the Shanghai Rankings (ARWU), and Webometrics.

Looking at WUR, it seems that if there are penalties for leaving THE, the penalties for remaining could be more serious. 

Among the  IITs in the 2020 rankings, Ropar led in the 301-350 band, followed by Indore in the 351-400 band. Neither of them is as reputable in India as senior IITs such as Bombay and Madras and they had those ranks because of remarkable citation scores, although they did much less well for the other pillars. This anomaly was part of the reason for the six leavers to depart.

Fast-forward to the 2024 WUR. IIT Ropar has fallen dramatically to 1001-1200,  Indore, which had fallen from 351-400 to 601-800 in 2023, has opted out, and Gandhinagar has fallen from 501-600 to 801-1000. Bhubaneswar, which was in the 601-800 band in the 2020 WUR,  fell to 1001-1200 in 2022 and 2023 and was absent in 2024. Guwahati and Hyderabad remained in the 601-800 band.

Frankly, it looks like staying in the THE WUR is not always a good idea. Maybe their THE reputation improved but four of the original remaining IITs suffered serious declines.

IITs in Other Rankings

Now, let's examine the IITs' performance in other rankings. First, the total publications metric in Leiden Ranking. Between 2019 and 2023, four of the six early leavers rose, and two fell. The late leavers, Hyderabad and Indore, were absent in 2019 and were ranked in the 900s in 2023. Remainer Guwahati rose from 536th in 2019 to 439th in 2023.

For Webometrics, between 2019 and 2024, all 12 IITs went up except for Bombay.

Finally, let's check the overall scores in the QS WUR. Between 2021 and 2024, four of the six leavers went up, and two went down. Of the others, Guwahati went up, and Hyderabad went down.

So, looking at overall ranking scores, it seems unlikely that boycotting THE causes any great harm, if any. On the other hand, if THE is tweaking its methodology or something happens to a productive researcher, staying could lead to an embarrassing decline.

IITs' Academic Reputation Scores

Next, here are some academic reputation surveys. The  US News Best Global Universities is not as helpful as it could be since it does not provide data from previous editions, and the Wayback Machine doesn't seem to work very well. However, the Global Research Reputation metric in the most recent edition is instructive. 

The six escapees had an average rank of 272, ranging from 163 for Bombay to 477 for Roorkee.

The remainers' ranks ranged from 702 for Guwahati to 1710 for Bhubaneswar. Ropar was not ranked at all. So, leaving THE does not appear to have done the IITs any harm in this metric

Turning to the QS WUR academic reputation metric, the rank in the academic survey for the leavers ranges from 141 for Bombay to 500 for Roorkee. They have all improved since 2022. The best performing remainer is Guwahati in 523rd place.  Ropar and Gandhinagar are not ranked at all. Bhubaneswar, Indore and Hyderabad are all at 601+.  

Now for Round University Ranking's reputation ranking. Four of the six original leavers were there in 2019. Three fell by 2023 and Delhi rose. Two, Bombay and Roorkee, were absent in 2019 and present in 2023.

This might be considered evidence that leaving THE leads to a loss of reputation. But five of the original remainers are not ranked in these rankings, and Guwahati is there in 2023 with a rank of 417, well below that of the six leavers. 

There is then scant evidence that leaving WUR damaged the academic reputations of those IITs that joined the initial boycott, and their overall rankings scores have generally improved.

On the other hand, for IITs Ropar and Bhubaneswar remaining proved disastrous.  

IITs and Employer Reputation

In the latest GEURS employer rankings, published by Emerging, the French consulting firm, there are four exiting IITs in the top 250, Delhi, Bombay, Kharagpur, and Madras, and no remainers.

In the QS WUR Employer Reputation indicator, the boycotters all perform well. Bombay is 69th and Delhi is 80th. Of the six original remainers two, Ropar and Gandhinagar, were not ranked by QS in their 2024 WUR. Three were ranked 601 or below, and Guwahati was 381st, ahead of Roorkee in 421st place.

Conclusion

Looking at the IITs, there seems to be little downside to boycotting THE WUR, and there could be some risk in staying, especially for institutions that have over-invested in specific metrics. It is possible that the IITs are atypical, but so far there seems little reason to fear leaving the THE WUR. A study of the consequences of boycotting the THE Impact Rankings is being prepared 

THE's Big Bang Ranking

 

Another day, another ranking. 

Times Higher Education (THE) has published a "bang for the bucks" ranking.

THE is taking the scores for institutional income, research income, and income from industry and comparing them with the scores "for research, teaching, and working with industry." This, presumably, is supposed to reveal those universities that are able to process their funding efficiently and turn it into publications, citations, patents, doctorates, and survey responses

There are some methodological issues here. It is not clear exactly how the income scores are calculated. Is it from the raw monetary data that THE collects from universities, or has it been through the THE standardization and normalization machine? Is there some sort of weighting or just an average of the three income categories? 

Also, there is a chart that suggests that all the scores are counted except for the financial metrics, but the text implies that the international pillar is not counted as part of the bang that THE purports to measure.

Another issue is that the financial data in the THE rankings refers to the year two years before the date of publication. However, citation and publication data are from a five—or six-year period before the ranking is published. In effect, THE is claiming that their favored schools have a remarkable ability to send money back in time to the years when research proposals were written, papers published, and citations recorded.

THE lists ten countries as good bang producers, starting with the UK and including Pakistan and Egypt. It does not list China, South Korea, Canada, or Australia, which should make us a little suspicious, 

Then, looking at the list of twenty universities with the biggest bangs, we see a few familiar names, including Sussex and Brighton Medical School,  Babol Noshirvani University of Technology,  and  Vita-Salute San Raffaele University, that have appeared in this blog before because they received remarkably high scores for citations and consequently did well in the overall rankings. Some, including Quaid-i-Islam University, COMSATS University, Auckland University of Technology, Government College University Faisalabad, and University College London, have contributed to citation-rich multi-contributor papers from the Global Burden of Disease Studies or the Large Hadron Collider project. Others, such as Shoolini University of  Biotechnology and Management Sciences and Malaviya National Institute of Technology, have scores for research quality that are disproportionate to those for research environment or teaching. It looks as though a lot of  THE's Big Bang simply consists of getting masses of citations. 

It is also possible that universities might obtain a good bang for the buck score by underreporting their income, perhaps accidentally, which would help here, although not in conventional rankings. This has happened to Trinity College Dublin and probably to Harvard, although the latter case went unnoticed by almost everyone. Probably, the very high scores for Sorbonne University and Universite Paris Cite result from the special features of the French funding system.

I suspect quite a few institutions will take this ranking seriously or pretend to and use it as a pretext to try to obtain more largesse from increasingly impoverished states.

It would seem that THE is engaged in a public relations exercise for upmarket British and perhaps for US and continental universities. These are doing all sorts of amazing, brilliant, and exciting things for which they receive insufficient funds from cheapskate governments.  Just imagine what they could do if they got as much money as Chinese universities do. 

Presentation Slides

I am uploading some conference presentation slides. They are out of date but might have some historical interest.

1. QS and THE Subject Rankings Compared. Moscow April 2015.

2. Global Rankings: Changing Methodologies. Yekaterinburg February 2016

3. International Rankings and the Assessment of Teaching: A Regional Perspective. Doha March 2017

4. International University Rankings and Southeast Asia. Kuala Lumpur March 2019

5. Rankings, Universities and the Media. Bologna May 2019

6. COVID-19 and Developments in Global University Rankings. Beijing (online) October 2020

7. Developments in National and Regional Rankings. Warsaw June 2022

Comments on the THE Reputation Rankings

Times Higher Education (THE) has announced the latest edition of its reputation ranking. The scores for this ranking will be included in the forthcoming World University Ranking and THE's other tables, where they will have a significant or very significant effect. In the Japan University Ranking, they will get an 8% weighting, and in the Arab University Ranking, 41%. Why THE gives such a large weight to reputation in the Arab rankings seems a bit puzzling. 

The ranking is based on a survey of researchers "who have published in academic journals, have been cited by other researchers and who have been published within the last five years," presumably in journals indexed in  Scopus.

Until 2022 the survey was run by Elsevier but since then has been brought in-house. 

The top of the survey tells us little new. Harvard is first and is followed by the rest of the six big global brands: MIT, Stanford, Oxford, Cambridge, and Berkeley. Leading Chinese universities are edging closer to the top ten.

For most countries or regions, the rank order is uncontroversial: Melbourne is the most prestigious university in Australia, Toronto in Canada, Technical University of Munich in Germany, and a greyed-out Lomonosov Moscow State University in Russia. However, there is one region where the results are a little eyebrow-raising. 

As THE has been keen to point out, there has been a remarkable improvement in the scores for some universities in the Arab region. This in itself is not surprising. Arab nations in recent years have invested massive amounts of money in education and research, recruited international researchers, and begun to rise in the research-based rankings such as Shanghai and Leiden. It is to be expected that some of these universities should start to do well in reputation surveys.

What is surprising is which Arab universities have now appeared in the THE reputation ranking. Cairo University, the American University in Beirut, Qatar University, United Emirates University, KAUST, and King Abdulaziz University have achieved some success in various rankings, but they do not make the top 200 here. 

Instead, we have nine universities: the American University in the Middle East, Prince Mohammed Bin Fahd University, Imam Mohammed Ibn Saud Islamic University, Qassim University, Abu Dhabi University,  Zayed University, Al Ain University, Lebanese University, and Beirut Arab University. These are all excellent and well-funded institutions by any standards, but it is hard to see why they should be considered to be among the world's top 200 research-orientated universities.

None of these universities makes it into the top 1,000 of the Webometrics ranking or the RUR reputation rankings. A few are found in the US News Best Global Universities, but none get anywhere near the top 200 for world or regional reputation. They do appear in the QS world rankings but always with a low score for the academic survey.

THE accepts that survey support for the universities comes disproportionately from within the region in marked contrast to US institutions and claim that Arab universities have established a regional reputation but have yet to sell themselves to the rest of the world.

That may be so, but again, there are several Arab universities that have established international reputations. Cairo University is in the top 200 in the QS academic survey, and the RUR reputation ranking, and the American University of Beirut is ranked 42nd for regional research reputation by USN. They are, however, absent from the THE reputation ranking. 

When a ranking produces results that are at odds with other rankings and with accessible bibliometric data, then a bit of explanation is needed.

  

Global rankings: the age of deference is coming to an end

 

Published in today's University World News

Rankings and the Threat from the East

Recently, we have heard a lot about global university rankings' responsibilities. Some have drawn attention to the increasing number of universities included in the rankings or the new rankings that allow universities to showcase the remarkable and interesting things they have been doing for society or the environment. There are claims that the well-known rankers are promoting global equity by including many more African and Asia universities.

Perhaps. But it seems that some rankings, particularly the two big-name ones, THE and QS, have another function, which is to downplay the rise of Chinese and maybe other Asian institutions and maintain the dominant position of the elite schools of the Global North. 

The table below shows the number of universities included in the top 100 universities in various global rankings. The table is arranged in ascending order according to the number of Mainland Chinese universities and refers to the most recent edition. 

Chinese universities are apparently uninterested in the rankings that supposedly assess contributions or commitment to the environment, sustainability, or equity. There are none in the top 100 of the new QS Sustainability Rankings or the GreenMetric Rankings and only one in the THE Impact Rankings. On the other hand, China does very well in Nature Index and in Leiden Rankings Publications and Publications in the top 1% of journals and fairly well in the URAP, Scimago, and National Taiwan University rankings. In short, China does best in those rankings that emphasize recent achievements in research in STEM subjects. 

The UK does best in rankings that include a substantial weighting for reputation, internationalization,  or activity related to sustainability and much less well in research-based rankings. 

The USA hasn't really bothered with the GreenMetric and THE Impact rankings. Its best performance is in UniRank, which is a measure of web activity, and Webometrics, which is half web activity, CWUR, which includes faculty and alumni achievement, and US News Best Global Universities, which has a strong reputation element. It is not so good in Nature Index, URAP, and NTU, which are research-based. 

It seems, to simplify a bit, that British and American universities benefit from indicators that measure or try to measure resources, reputation, web presence, and international activity, Chinese and some other Asian institutions are rapidly moving ahead in research and innovation.

Table: Number of Universities in the Top 100

Ranking	Country of publication	USA	UK	Mainland China
QS Sustainability	UK	12	28	0
UniRank	Australia	75	7	0
GreenMetric	Indonesia	3	3	1
THE Impact	UK	6	25	1
USN Global	USA	41	11	4
MosIUR	Russia	41	15	4
QS World	UK	27	17	5
CWUR	UAE	50	9	6
GEURS	France	18	8	6
Webometrics	Spain	53	9	6
THE World	UK	36	11	7
RUR	Georgia	38	11	7
ARWU	China	38	8	10
NTU	Taiwan	36	10	14
Scimago- universities	Spain	39	7	24
URAP	Turkiye	28	6	23
Leiden P 1%	Netherlands	39	8	28
Leiden P	Netherlands	31	6	36
Nature Index -  academic	UK	37	5	35