China, AI, and Rankings

Recently we have seen the crumbling of many illusions. It now seems hard to believe but only a few weeks ago we were assured that President Biden was as sharp as a fiddle or as fit as a tack or something. Also, the Russian economy was collapsing under the weight of Western sanctions. Or again, the presidential race was running neck and neck, and probably heading for a decisive Democrat vote, foretold by that state-of-the-art poll from Iowa.

An equally significant illusion was the supremacy of Western, especially Anglophone, science and scholarship. The remarkable growth of Asian research has often been dismissed as imitative and uncreative and anyway much less important than the amazing things Western universities are doing for sustainability and diversity.

The two big UK rankings, THE and QS, highly regarded by governments and media, have been instrumental in the underestimation of Chinese science and the overestimation of that of the West. Oxford is in first place in the THE world rankings and no other, while MIT leads the QS world rankings and no other. Indeed, Leiden Ranking, probably the most respected ranking among actual researchers, has them in 25^th and 91^st place for publications. 

The myopia of the Western rankers has been revealed by recent events in the world of AI. The release of the large language model (LLM) DeepSeek has caused much soul searching among western academics and scientists. It looks as good as Chat GPT and the others, probably better, and, it seems, very much cheaper. There will likely be more to come in the near future. The researchers and developers were mainly “researchers and developers from China’s elite universities, with minimal overseas education,” according to DeepSeek itself, including Peking University, Tsinghua University, Zhejiang University, Beihang University, Shanghai Jiao Tong University and Nanjing University. There are some overseas links, Monash, Stanford, Texas, but these are less significant.

Some of the anguish or the excitement may be premature. DeepSeek may inspire another Sputnik moment, although that does seem rather unlikely at the moment, and Western companies and institutions may surge ahead again. Also, I suspect, the cheapness may have been exaggerated. Like its western counterparts, DeepSeek has places that it would prefer not to go to – Tiananmen Square and the Uighurs among others – and that could undermine its validity in the long run.

But it is a remarkable achievement nonetheless and it is yet another example of the emerging technological prowess of the Chinese economy. We have seen China build a network of high-speed railways. Compare that with the infamous Los Angeles to San Francisco railroad. Compare China’s military modernization with the state of European navies and armies.

We might add, compare the steady advance of Chinese universities in the output and quality of research and innovation compared to the stagnation and decline of western academia. The main western rankers, THE and QS, have consistently rated  American and British universities more favourably than those in Asia, especially China. Recently it seems that the two dominant rankers have been doing their best to lend a hand to western universities while holding back those in Asia. THE started their Impact rankings with the intention of allowing universities to show the wonderful things they are doing to promote sustainability, an opportunity that has been seized by some Canadian, Australian, and British universities but totally ignored by China. QS has introduced a new sustainability indicator into its world rankings, in which Chinese universities do not do well.

 

AI Rankings

QS and THE have been especially unobservant about the rise of China in computer science, and more specifically in the field of AI. This is in contrast to those rankings based largely on research and derived from public verifiable data.

There are currently four rankings that focus on AI. QS has a ranking for Data Science and Artificial Intelligence and it is very much dominated by Western universities. The top 20 includes 10 US institutions and none from Mainland China, although it does include the Hong Kong University of Science and Technology and the Chinese University of Hong Kong. Massachusetts Institute of Technology is in first place and the best Mainland university is Tongji in 36^th = place.

Now let’s look at EduRank, a rather obscure firm, probably located in California, whose methodology might be based on publications, citations and other metrics. Here the top 20 for AI has 15 US universities, with Stanford University in first place. The best performing Chinese university is Tsinghua at 9^th place.  

University Ranking of Academic Performance (URAP) is published by the Middle East Technical University in Ankara. Their most recent  AI ranking has Tsinghua in first place with Carnegie Mellon in 10^th. The top 20 has 12 Mainland universities and only three American.

The US News Best Global Universities ranking for AI is even more emphatic in its assertion of Chinese superiority. Twelve out of the top 20 universities for AI are Mainland Chinese, with Tsinghua at number one. The best US university was Carnegie Mellon University in 29^th  place, well behind a few universities from Australia, Hong Kong, and Singapore.

 

Computer Science Rankings

Turning to the broader field of Computer Science, the THE Computer Science rankings have Oxford in first place, MIT in third, and Peking University in twelfth. Similarly, QS has a Computer Science and Information Systems subject ranking, the most recent edition of which shows MIT first, Oxford fourth, and Tsinghua eleventh.

In contrast, in the National Taiwan University Computer Science Rankings Tsinghua is first, Stanford seventh, and Oxford 171^st (!). According to the Scimago Institutions Rankings for universities, Tsinghua is first for Computer Science,  MIT 9^th,  Oxford 22^nd.  The Iran-based ISC World University Rankings for Computer and Information Sciences place Tsinghua first, MIT 11^th, and Oxford 18^th. In the US News Best Global Universities Computer Science and Engineering ranking Tsinghua is first, MIT fifth, and Oxford 18^th.

In the Shanghai subject rankings MIT is still just ahead of Tsinghua, mainly because of the World Class Output metric which includes international academic awards since 1991.

It seems then that QS, THE, and EduRank have significantly exaggerated the capabilities of elite Western universities in AI and Computer Science generally and underestimated those of Chinese and other Asian schools. It seems ironic that THE and, to a lesser extent, QS are regarded as arbiters of excellence while URAP, Scimago, the National Taiwan University rankings, and even US News are largely ignored.

 

 

Comments on the New Edition of the THE Reputation Rankings

Times Higher Education (THE) have just published the latest edition of their World Reputation Rankings. At the top, it is business as usual. We have the big six super brands, Harvard, MIT, Oxford, Stanford, Cambridge, and Berkeley. After that, there are no real surprises. The top fifty includes  Ivy League schools like Princeton and Yale, rising Asian giants like Tsinghua, Tokyo and the National University of Singapore, established European institutions like LMU Munich and KU Leuven, and well-known London colleges, LSE and UCL.

But then things start to get interesting. THE has introduced some drastic methodological changes, and these have led to a significant amount of churning.   

A bit of context, last year, the reputation rankings recorded an apparently remarkable achievement by nine Arab universities that came into the top 200 from nowhere. Later, THE announced that they had discovered a "syndicate" that was trading votes in the reputation surveys and that measures would be taken to stop that and penalise the universities involved.

But THE was not satisfied with that, and they have revamped its methodology to include two new metrics in addition to the simple counting of votes for best universities for teaching and research.

The first of these is pairwise comparison. This means, according to THE, that universities are preselected "informed from their publication history," and respondents then place them in order from 1 to 5, thus encouraging them to consider places other than the super brands. Exactly how that preselection works is not clear.

The second is voter diversity, which rewards universities if they have more countries and more subjects in their respondent base, which, THE claims, indicates a strong reputation. 

Whatever THE's intentions, the overall result of these changes is clear. The USA, UK, Netherlands, Canada, Australia, Germany, Sweden, and Switzerland have all increased the number of universities in the top 200. 

The biggest gainers are the UK, which has increased its representation in the top 200 from 20 to 24; Switzerland, which has gone from 4 to 7; and the Netherlands, which has also added another three universities.

Of the twenty British universities in the top 200, eleven have risen, seven are in the same rank or band, and only two, Birmingham and Sheffield, have fallen. It is hard to believe that there has been such a widespread improvement in the international reputation of British universities. 

In contrast, the new methodology has been disastrous for universities in China, Russia, the Arab region, India, Israel, Japan, and South Korea. 

The number of Chinese universities in the top 200 has fallen from 15 to 8. While Tsinghua and Peking Universities have retained their places, others have fallen, Shanghai Jiao Tong University from 43rd  place to 58th, the University of Science and Technology China from 61-70 to 101-150, and Harbin Institute of Technology from 101-125 to  201-300.

Russia has fared even worse. There were six greyed-out Russian universities in the 2023 rankings. Now, there are just two, Lomonosov Moscow State University, in 83rd place, down from 34th, and Bauman Moscow State Technical University, down from 60-70 to 201-300. The latter gets 1.7 points in the pairwise comparison. All of the others are gone.

In 2023, there were four Indian universities in the rankings, the Indian Institute of Science and the Indian Institutes of Technology Bombay, Delhi, and Madras. Now, IIT Bombay has been removed altogether, and the Indian Institute of Science and the IITs Delhi and Madras have been demoted to the 201-300 band. They are joined by Siksha 'O' Anuhandsan, which has a global research rank of 1900 in the US News Best Global Universities. 

The worst-hit area is the Arab Region. Universities in Kuwait, Lebanon, Saudi Arabia, and the UAE have left. The only Arab university now is King Abdullah University of Science and Technology, which does not teach undergraduates.

A glimpse of the impact of the new metrics can be seen by looking at the universities that come at the bottom for each metric. 

For voter counts, the ten worst universities are all European, followed by a handful of Australian and Canadian universities and Siksha 'O' Anuhandsan, suggesting that it is the new indicators that keep them in the ranking.

For pairwise comparison, the bottom is quite diverse; there is Bauman Moscow State Technical University, Liverpool University, the University of Buenos Aires, Beijing Normal Univerity, Universite Paris Cite, and the University of Cape Town.

The universities that do worst for voter diversity are mainly South Korean, Indian, Turkish, and Japanese. 

It seems then that one function of the new methodology is to slow down the advance of Asian universities and maintain the status of the Western elite. 

THE is shocked, shocked ...

We are repeatedly told that the TimesHigher Education (THE) university rankings are trusted by students, governments, and other stakeholders. Perhaps they are. Whether they should be is another matter. 

Last October, THE announced the results of its World University Rankings, and there was a chorus of lamentation from leading Australian universities, among others, who apparently trusted THE. It seems that the debate over restricting the admission of international students has damaged the country's reputation, and that has been reflected in the THE reputation survey. which contributes disproportionately to THE's teaching and research "pillars." That has led to declining overall scores, which will be the start of a vicious downward spiral. British and American universities also bemoaned the decline in ranking scores, supposedly due to the lack of funding from hard-hearted governments.

For many academics and administrators, THE has become the arbiter of excellence and a credible advisor to the agencies that dominate Western economy and society. It has even become a preferred analyst for the WEF, which is supposed to represent the finest minds of the corporate global world. This is quite remarkable since there is a big mismatch between THE's pretensions to excellence and its actual practice. 

A recent example was the publication of a story about how THE's data analysts had detected collusive activity among some universities in order to boost their scores in the reputation surveys that make up a substantial part of the THE World University Rankings and their various derivatives.

On October 24, David Watkins of THE announced that a "syndicate" had been detected where universities supported each other in the THE Arab reputation survey to the exclusion of non-members. Exactly who those members were was not announced, but it probably included the nine universities that made it to the top 200 in THE World Reputation Survey announced in February 2024, the data for which was included in THE world ranking announced in October  2024. It might also include some universities that had made sudden and surprising gains in the Arab University Rankings announced in November 2023, and the World University Rankings announced last October.

There is a whiff of hypocrisy here. THE is apparently getting upset because universities have probably been doing something that the rankers have condoned or at least ignored. There were signs that something was a bit off as far back as the Arab University Rankings in November 2023. These showed surprisingly good performances from several universities that had performed poorly or not at all in other rankings. In particular, universities in the Emirates were rising while those in Egypt were falling. This was interesting because the results were announced at a summit held in Abu Dhabi that featured several speakers from the Emirates, a development reminiscent of the 2014 summit in Qatar when Texas A and M Qatar was proclaimed the top MENA university based on precisely half a highly cited researcher followed by a similar summit in the UAE in 2015 when that university -- actually a program that has now been wound up -- disappeared, and  United Arab Emirates University advanced to fifth place.

Meanwhile,  between October 2023  and January 2024, THE was conducting their survey of academic opinion for the World University Rankings. Before 2021, they had relied on survey data supplied by Clarivate, but now the survey has been brought in-house. That, it now appears, was not a good idea. The number of survey respondents soared, and there was a disproportionate number of respondents from the UAE. In February 2024, THE published the results of its reputation survey, which would later become a part of the world rankings. 

THE listed only the top 200 universities and gave exact scores for the top fifty.  The interesting thing was that nine Arab universities were included whose reputation scores were below the scores for academic reputation in the QS World University Rankings rankings, the scores for global research reputation in the US News Best Global Universities ranking, or scores in the Round University Rankings, if they were actually ranked at all and below their previous scores.  They were also above the scores achieved by leading universities in the region in Egypt, Saudi Arabia, Qatar, and Lebanon, and they appeared unrelated to other indicators. 

It was probably not only Arab universities. Egor Yablokov of E --  Quadrat Science and Education identified several universities whose reputation score appears disproportionate to the overall scores for the THE world rankings.

When the 2025 WUR rankings appeared in October of last year, there were more signs that something was amiss. Universities in the UAE  including Abu Dhabi University and Khalifa University, also in Abu Dhabi, did much better than in previous editions or in other rankings. There were other apparent anomalies. Al Ahliyaa Amaan University was ahead of the University of Jordan, the Lebanese American University higher than the American University of Beirut,  the American University of the Middle East higher than Kuwait University, Future University in Egypt, and the Egypt-Japan University of Science and Technology higher than Cairo University and Al Azhar. 

Then came the Arab University Rankings. It appears that THE had now taken action against the "syndicate", resulting in them dropping significantly. 

In addition to this, there are some trends that require explanation. Many universities in Saudi Arabia and UAE have fallen significantly, while some in Jordan, Egypt, and Iraq have risen.  Applied Science Private University, Jordan, has risen from 91-100 to 25, Al Ahliyya Amman University, also in Jordan, from 91-100 to  28, Ahlia University in  Bahrain from unranked to 17th, Cairo University from  28 to 8, the University of Baghdad from 40 to 20, Mustansiriyah University, Baghdad from 71-80 to 37, An Najah National University, Palestine, 81-90 to 23, and Dhofar University, Oman, from 101-120 to 49. 

So, THE have allocated a whopping 41% weighting for reputation, of which 23% is for research reputation, for their Arab University Rankings, compared to 25% for their Asian rankings and 33% for the Latin American rankings. They have  introduced a new metric, collaboration within the Arab world, taken over the research and teaching survey from Elsevier, increased the number of respondents, organized prestigious summits, and offered a variety of consultancy arrangements. All of this would create an environment in which exclusive agreements were likely to flourish.

The extreme fluctuations resulting from THE's changes to the reputation indicators have seriously undermined THE's credibility, or at least they ought to. It would be better for everybody if THE simply returned the administration of the reputation survey to Elsevier and stuck to event management, where it is unsurpassed.