Shanghai Rankings: Correlations Between Indicators

This is, I hope, the first of  a series. Maybe THE and QS next week.

If we want to compare  the utility of university rankings one attribute to consider is internal consistency. Here, the correlation between the various indicators can tell us a lot. If the correlation between a pair of indicators is 0.90 or above we can assume that these indicators are essentially measuring the same thing.

On the other hand, if  there is no correlation or one that is low, insignificant or even negative we might have doubts about the validity of one or both of the indicators. It is reasonable that if a university scores well for one metric it will do well for others providing they both represent highly valued attributes. A university producing high quality research or collecting large numbers of citations should also score well for reputation. If it does not there might be a methodological problem somewhere.

So, we can assume that if the indicators are valid and are not measuring the same thing the correlation between indicators will probably be somewhere between 0.5 and 0.9.

Let's have a look at the Shanghai ARWU for 2019. The indicator scores were extracted and analysed using PSPP. (It is very difficult to analyse the 2020 edition because of a recent change in presentation.) These rankings have six indicators: alumni and faculty receiving Nobel and Fields awards, papers in Nature and Science, highly cited researchers, publications in the Web of Science, and productivity per capita.

Looking at all 1000 institutions in the Shanghai Rankings, Alumni, Awards, and Nature and Science all correlate well with each other Highly Cited Researchers correlates well with Nature and Science and Publications but less so with Alumni and Awards. Nature and Science correlates well with all the other indicators.

The Publications indicator does not correlate well with Alumni and Awards. This is to be expected since Publications refers to 2018 while the Alumni and Awards indicators go back several decades.

Overall, the correlations are quite good although there is a noticeable divergence between Publications and Alumni and Awards, which cover very different time periods. 

CORRELATIONS

CORRELATION
	/VARIABLES =  alumni awards highlycited naturescience publications pcp finaltotal
	/PRINT = TWOTAIL NOSIG.

Correlations

		alumni	awards	highlycited	naturescience	publications	pcp	finaltotal
alumni	Pearson Correlation	1.00	.78	.51	.72	.45	.63	.76
	Sig. (2-tailed)		.000	.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
awards	Pearson Correlation	.78	1.00	.57	.75	.44	.67	.82
	Sig. (2-tailed)	.000		.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
highlycited	Pearson Correlation	.51	.57	1.00	.79	.72	.64	.87
	Sig. (2-tailed)	.000	.000		.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
naturescience	Pearson Correlation	.72	.75	.79	1.00	.69	.73	.93
	Sig. (2-tailed)	.000	.000	.000		.000	.000	.000
	N	992	992	992	992	992	992	992
publications	Pearson Correlation	.45	.44	.72	.69	1.00	.50	.81
	Sig. (2-tailed)	.000	.000	.000	.000		.000	.000
	N	1000	1000	1000	992	1000	1000	1000
pcp	Pearson Correlation	.63	.67	.64	.73	.50	1.00	.78
	Sig. (2-tailed)	.000	.000	.000	.000	.000		.000
	N	1000	1000	1000	992	1000	1000	1000
finaltotal	Pearson Correlation	.76	.82	.87	.93	.81	.78	1.00
	Sig. (2-tailed)	.000	.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000

Most observers of ARWU and other global rankings are interested in the top levels where elite schools and national flagships jostle for dominance. Analysing correlations among indicators for the top 200 in ARWU, there are high correlations between Alumni, Awards, Nature and Science, and Productivity per Capita, ranging from .69 to .79.

There is also a high correlation of .72 between Nature and Science and Highly Cited Researchers. It is, however, noticeable that the correlation between Publications and other indicators is low for Highly Cited Researchers and very low for Productivity per Capita, Alumni and Awards.

It seems that, especially among the top 200 places, there is a big gap opening between the old traditional elite of Oxbridge, the Ivy League and the like who continue to get credit for long dead Nobel laureates and the new rising stars of Asia and Europe who are surging ahead for WOS papers and beginning to produce or recruit superstar researchers.

Correlations

		alumni	awards	highlycited	naturescience	publications	pcp	finaltotal
alumni	Pearson Correlation	1.00	.79	.36	.69	.21	.62	.78
	Sig. (2-tailed)		.000	.000	.000	.003	.000	.000
	N	200	200	200	199	200	200	200
awards	Pearson Correlation	.79	1.00	.44	.74	.14	.67	.84
	Sig. (2-tailed)	.000		.000	.000	.044	.000	.000
	N	200	200	200	199	200	200	200
highlycited	Pearson Correlation	.36	.44	1.00	.72	.57	.49	.78
	Sig. (2-tailed)	.000	.000		.000	.000	.000	.000
	N	200	200	200	199	200	200	200
naturescience	Pearson Correlation	.69	.74	.72	1.00	.44	.65	.92
	Sig. (2-tailed)	.000	.000	.000		.000	.000	.000
	N	199	199	199	199	199	199	199
publications	Pearson Correlation	.21	.14	.57	.44	1.00	.12	.55
	Sig. (2-tailed)	.003	.044	.000	.000		.083	.000
	N	200	200	200	199	200	200	200
pcp	Pearson Correlation	.62	.67	.49	.65	.12	1.00	.72
	Sig. (2-tailed)	.000	.000	.000	.000	.083		.000
	N	200	200	200	199	200	200	200
finaltotal	Pearson Correlation	.78	.84	.78	.92	.55	.72	1.00
	Sig. (2-tailed)	.000	.000	.000	.000	.000	.000
	N	200	200	200	199	200	200	200

THE's Caucus Ranking

In Alice in Wonderland there is a "caucus race" in which everyone runs around frantically in different directions and eventually everyone wins a prize. Unfortunately, there are not quite enough sweets to go around as prizes and so poor Alice has to make do with her thimble which she gives to the Dodo who then presents it to her.

It seems that THE has come up with a caucus ranking. In the THE Impact Rankings universities expend a lot of energy, do a lot of amazing, astounding and very different things in very different ways and a lot of them get some sort of prize for something. 

These rankings are another example of the growing complexity of the ranking scene. Global university rankings used to be simple. Shanghai Jiao Tong University started its  ARWU rankings in 2000 with just 500 ranked institutions and six indicators. Since then the number of rankings has proliferated and there have been more and more spin-offs, young university, regional, business schools, national, subject  rankings and so on with more indicators and increasingly complex and often opaque methodologies. We are getting to the point where a university is incompetent or excessively honest if it cannot find a ranking indicator, perhaps finely sliced by age, size, mission, and/or subject, in which they can scrape into the top five hundred, or at the very least the top thousand, and therefore into the top 3 or 4 % in the world. 

Some of the recent rankings seem redundant or pointless, going over the same old ground or making granular distinctions that are of little interest. It is no doubt nice to be acclaimed as the best young university for social science in South Asia, and maybe that can be used in advertising, but is it really necessary?

Now we have the third edition of the THE Impact rankings. These, as THE boasts, are the only rankings to measure universities according to their commitment to the UN's Sustainable Development Goals. But that is not very original. Universitas Indonesia's GreenMetric was doing something similar several years ago although not tied explicitly to the UN goals. They have indicators related to energy, infrastructure, climate change, water, waste, transportation, and education.

It seems a little odd that the UN should be accepted as the authority on the achievement of gender equality when its "peacekeeping" forces have repeatedly been accused of rape and sexual assault. Is the UN really the right body to lay down guidelines about health and well-being considering the dubious performance of the WHO during the pandemic crisis?

One also wonders why THE should venture into ranking contributions to sustainability when after a decade it has still failed to come up with a credible citations indicator, which would seem a much easier task. 

It is noticeable that participation in these rankings is very uneven. There are 1,118 universities in the latest edition but only 13 Chinese and only 45 American, of which precisely two are in California, supposedly the homeland of environmental consciousness. The higher education elite of the USA, UK and China are largely absent. On the other hand, Iraq, Egypt, Brazil and Iran are much better represented here than in the research based rankings.

The top of these rankings is dominated by English-speaking universities outside the USA. The overall top twenty contains seven Australian, five British, three Canadian, and one each from Denmark, Ireland, the USA, New Zealand and Italy.

The popularity of the Impact Rankings seems linked to the current problems of many western universities. Public funding has been drying up, academic standards eroding, research output stagnating. Many universities have resorted to importing international, often Chinese, students and faculty to keep up standards, bring in tuition money, fill up postgraduate classes, and do the work of junior researchers.

The international students and researchers have left or are leaving and may not return in significant numbers, although THE "believes" that they will. This is happening as universities trying to reopen face the prospect of unprepared students, dwindling funds, and a lack of interest from employers. Eventually this will impact the position of universities in the global ranking systems. Those universities once dependent on international researchers for their reputation and ranking scores will start to suffer.

It looks as though western universities are losing interest in research and instruction in professional and academic subjects and and are reinventing themselves as purveyors of transformative experiences to the children of the affluent and ambitious, guardians of the purity of cultural discourse, or as saviours of the planet.     

The Financial Post of Canada has published a caustic comment on the joyful proclamations by Queen's University about its ascent to fifth place in the Impact Rankings. A trustee, John Stackhouse, has claimed that its success there meant that it was fulfilling "the true purpose of a university." The article observes that those "who believe the true purpose of a university is to pursue academic excellence and ensure that students who pass through its doors have the skills to build prosperous lives for themselves as productive members of their community, might differ."  In the THE World University Rankings and others Queen's is doing much less well. 

The methodology of the impact rankings does little to inspire confidence. For each of the indicators there is a weighting of 27% for bibliometric measures, such as the amount of research on hunger, health, water, or clean energy. It is easy to see how this could be gamed. Then there is a variety of data submitted by the institutions. Even if every university administrator is a sea-green incorruptible there are many ways in which such data can be massaged or stretched.

Added to that, THE does not appear to be doing a rigorous validation. Universities are not  assessed  the same things, except for the partnership for the goals indicator. The University of Sydney, overall second this year, is ranked for clean water and sanitation, sustainable cities and communities, and life on land. Clean water and sanitation includes supporting water conservation off campus and the reuse of water across the university.

RMIT University, in third place, is ranked for decent work and economic growth, industry innovation and infrastructure and reduced inequalities.  Decent work and economic growth includes expenditure per employee and policies for ending discrimination. So, essentially THE is trying to figure out whether Sydney is better at reusing water than RMIT is at announcing policies that are supposed to reduce discrimination. Comparing research output and impact across disciplines is, as THE ought to know, far from easy. Comparing performance in using water with discrimination policy would seem close to impossible especially since THE does not always use objective criteria but merely examples of best practices. Evidence "is evaluated against a set oef criteria and decisions are cross-validated where there is uncertainty. Evidence is not required to be exhaustive -- we are looking for examples that demonstrate best practice at the institutions concerned."

But it seems that the a substantial number of universities will find these rankings a useful tool in their quest for income and publicity and there will be more editions, and probably sub-rankings of one sort or another, for years to come. 

 

The Remarkable Revival of Oxford and Cambridge

There is nearly always a theme for the publication of global rankings. Often it is the rise of Asia, or parts of it. For a while it was the malign grasp of Brexit which was crushing the life out of British research or the resilience of American science in the face of the frenzied hostility of the great orange beast. This year it seems that the latest QS world rankings are about the triumph of Oxford and other elite UK institutions and their leapfrogging their US rivals. Around the world, quite a few other places are also showcasing their splendid achievements.

In the recent QS rankings Oxford has moved up from overall fifth to second place and Cambridge from seventh to third while University College London, Imperial College London, and Edinburgh have also advanced. No doubt we will soon hear that this is because of transformative leadership, the strength that diversity brings, working together as a team or a family, although I doubt whether any actual teachers or researchers will get a bonus or a promotion for their contributions to these achievements.

But was it leadership or team spirit that pushed Oxford and Cambridge into the top five? That is very improbable. Whenever there is a big fuss about universities rising or falling significantly in the rankings in a single year it is a safe bet that it is the result of an error, the correction of an error, or a methodological flaw or tweak of some kind.

Anyway, this year's Oxbridge advances had as much to do with leadership,  internationalization, or reputation as goodness had with Mae West's diamonds. It was entirely due to a remarkable rise for both places in the score for citations per faculty, Oxford from 81.3 to 96, and Cambridge from 69.2 to 92.1. There was no such change for any of the other indicators.

Normally, there are three ways in which a university can rise in QS's citations indicator. One is to increase the number of publications while maintaining the citation rate. Another is to improve the citation rate while keeping output constant. The third is to reduce the number of faculty physically or statistically.

None of these seem to have happened at Oxford and Cambridge. The number of publications and citations has been increasing but not sufficiently to cause such a big jump. Nor does there appear to have been a drastic reduction of faculty in either place.

In any case it seems that Oxbridge is not alone in its remarkable progress this year. For citations, ETH Zurich rose from 96.4 to 99.8, University of Melbourne from 75 to 89.7, National University of Singapore from 72.9 to 90.6, Michigan from 58 to 70.5. It seems that at the top levels of these rankings nearly everybody is rising except for MIT which has the top score of 100 but it is noticeable that as we get near the top the increase gets smaller.

It is theoretically possible that this might be the result of a collapse of the raw scores of citations front runner MIT which would raise everybody else's scores if it still remained at the top but there is no evidence of either a massive collapse in citations or a massive expansion of research and teaching staff.

But then as we go to the other end of the ranking we find universities' citations scores falling, University College Cork from 23.4 to 21.8, Universitas Gadjah Mada from 1.7 to 1.5, UCSI University Malaysia from 4.4 to 3.6, American University  in Cairo from 5.7 to 4.2.

It seems there is a bug in the QS methodology. The indicator scores that are published by QS are not raw data but standardized scores based on standard deviations from the mean The mean score is set at fifty and the top score at one hundred. Over the last few years the number of ranked universities has been increasing and the new ones tend to perform less well than the the established ones, especially for citations. In consequence, the  mean number of citations per faculty has declined and therefore universities scoring above the mean will increase their standardized scores which is derived from the standard deviation from the mean. If this interpretation is incorrect I'm very willing to be corrected.

This has an impact on the relative positions of Oxbridge and leading American universities. Oxford and Cambridge rely on their  scores in the academic and employer survey and international faculty and staff to keep in the top ten. Compared to Harvard, Stanford and MIT they are do not perform well for quantity or quality of research. So the general inflation of citations scores gives them more of a boost than the US leaders and so their total score rises.

It is likely that Oxford and Cambridge's moment of glory will be brief since QS in the next couple of years will have to do some recentering in order to prevent citation indicator scores bunching up in the high nineties. The two universities will fall again although  it that will probably not be attributed to a sudden collapse of leadership or failure to work as a team.

It will be interesting to see if any of this year's rising universities will make an announcement that they don't really deserve any praise for their illusory success in the rankings.

The Decline of American Research

Looking through the  data of the latest SCImago Journal and Country Rank is a sobering experience. If you just look at the data for the quarter century from 1996 to 2020 then there is nothing very surprising. But an examination of the data year by year shows a clear and  frightening picture of scientific decline in the United States.

Over the whole of the quarter century the United States has produced 11,986,435 citable documents, followed by China with 7,229,532, and the UK with 3,347,117.

Quantity, of course, is not everything when comparing research capability. We also need to look at quality. SJCR also supplies data on citations which are admittedly an imperfect assessment of quality: they can be easily gamed with self-citations, mutual citations, and so on. They often measure what is fashionable, not that which contributes to public welfare or advances in fundamental science. They are at the moment, however, if collected and analysed carefully and competently, perhaps the least unreliable and subjective metric available for describing the quality of research.

Looking at the number of citations per paper, we have to set  a threshold otherwise the top country for quality would be Anguilla, followed by the Federated States of Micronesia, and Tokelau. So we wrote in a 50,000 paper threshold over the 25 years.

Top place for citations   in 1996-2020 goes to Switzerland, followed by the Netherlands, Denmark, and Sweden, with the USA in fifth place and the UK in tenth. Apart from the US it looks like the leaders in research are Cold Europe, the countries around the North and Baltic seas plus Switzerland.

China is well down the list in 51st place.

Things look very different when we look at the data year by year. In 1996 the USA was well ahead of everybody for  the output of citable documents with 350,258 documents, followed by Japan with 89,430. Then came the UK, Germany, France, Russia. China is a long way behind in nineth place with 30,856 documents.

Fast forward to 2020. Looking at output, China has now overtaken the US for citable documents and India has overtaken Germany and Japan. That is something that has been anticipated for quite some time.

That's quantity. Bear in mind that China and India have bigger populations than the USA so the output per capita is a lot less. For the moment.

Looking at citations per paper published in 1996 the United states had 42.14 citations or 1.62 per year over the quarter century. By 2020 this had shrunk to 1.22 per year.

It is possible that the annual score will pick up in later years as the US recovers from the lockdown and the virus before its competitors. It is equally possible that those papers may get fewer citations as time passes.

But Switzerland, which  was slightly ahead of the US in 1996, is in 2020 well in front, having improved its annual count of citations per paper from 1.65 to 1.68. Then there a cluster of other countries that have overtaken the US -- Australia, the  Netherlands, the UK.

And now China has also overtaken the US for quality with 1.23 citations per paper per year.

It looks as though the current pandemic will just accelerate what was happening anyway. Unless something drastic happens we can look forward to China steadily attaining and maintaining hegemony over the natural sciences.

Energetic and Proactive Leadership or a Defective Indicator?

It is now normal for university administrators to use global rankings to market their institutions, reward themselves and, all too often, justify demands for more public funding. Unfortunately, some ranking agencies are more than happy to go along with this.

One example of the misuse of rankings is from July of last year  when the Vice-Chancellor of the University of the West Indies, Sir Hilary Beckles, proclaimed a triple first  in the 2020 Times Higher Education (THE) rankings.

The original  target was to be be in the top 3% of ranked universities by the end of the current strategic planning cycle. The Vice-Chancellor was very happy that as a result of  the work of "an energetic and proactive leadership team of campus principals and others" the university was in the top 1% of the THE Latin America and Caribbean rankings and the top 1% of golden age universities, and was the only ranked university in the Caribbean.

Another article has just appeared lauding the achievements of the university. The Vice-Chancellor has asserted that "(t)he Times Higher Education informed us that what we have achieved is quite spectacular. That many universities had taken 30 years to achieve what we have achieved in a mere three years."

If this is an accurate report of what THE said then the magazine is being very irresponsible. Universities are complex structures, and cannot be turned around with just a few million dollars or a few dozen highly cited researchers. Without drastic restructuring or amalgamation, such a remarkable change is almost invariably the result of methodological changes or methodological defects. The latter is the case here.

UWI 's performance might appear  quite impressive but it seems that we have another case of  THE seeing things that nobody else can. THE is not the only global university ranking. In fact, it is in some important respects not a good one. Let's take a look at some other rankings. UWI is not ranked in Leiden Ranking, the Shanghai Academic Ranking of World Universities, the US News Best Global Universities, or the QS World University Rankings.

It does make an appearance in University Ranking by Academic Performance (URAP) published by Middle East Technical University where it is ranked 1622th and it is 1938th in the Center for World University Rankings.

But in the THE World University Rankings the university is in the top 600 and it is 18th in Latin America and the Caribbean. This is because of a single defective indicator.

UWI has an apparently healthy 81.1 in the THE world rankings citations indicator, supposedly a measure of research influence or impact, which at first sight seems odd since it has a very low score, 10.4, for research. In the Latin American rankings, where competition is less severe, that becomes  93.6 for citations and 75.8 for research. How could a university do so brilliantly for research influence when it has a poor reputation for research, doesn't publish many papers, and has little research income?

What has happened is that UWI has benefitted from THE's peculiar citations indicator that does not use fractional counting for papers with less than a hundred authors, plus hyper-normalisation, plus a country bonus for being located in a low performing country, resulting in a few multi-author, multi-cited papers pushing universities into positions that they could never achieve anywhere else.

UWI has recently contributed to a few papers with a large number of contributors and many citations in genetics and medicine in top journals including Lancet, Science, Nature, and the International Journal of Surgery. Five of these papers are linked with the Global Burden of Disease Study.

UWI is  to be congratulated for having a number of affiliated scientists who are taking part in cutting edge projects. Nonetheless, it is true that these have an impact only because of the technical defects of the THE rankings. In the last few years a succession of unlikely places -- Anglia Ruskin, Peradeniya, Reykjavik, Aswan, Brighton and Sussex Medical School, Durban University of Technology, have risen to the top of the citations charts in the THE rankings.

They have done so nowhere else. The exalted status of UWI in the THE WUR and its various offshoots is due to an eccentric methodology. If THE reforms that methodology, something that they have been talking about for a long time, or abandons the world rankings, the university will once again be consigned to the outer darkness of the unranked.

The sensational fall of Panjab University and the sensational coming rise*

*If present methodology continues

The Hindustan Times has a report on the fall of Panjab University (PU) in the latest THE Emerging Economies Rankings. PU has fallen from 150th in 2019 to 201-250 this year. As expected, rivals and the media have launched a chorus of jeers against the maladministration of a once thriving institution.

But that isn't half of it. In 2014, the first year of these rankings, PU was ranked 14th in the emerging world and first in India ahead even of the highly regarded IITs. Since then the decline has been unrelenting. PU was 39th in 2015, 130th in 2018, 166th in 2020.

Since 2014 PU's scores for Research, Teaching and Industry Income have risen. That for International Outlook has fallen but that did not have much of an effect since it accounts for only 10 per cent of the total weighting.

What really hurt PU was the Citations indicator. In 2014 PU received a score of 84.7 for citations largely because of its participation in the mega-papers radiating from the Large Hadron Collider project, which have thousands of contributors and thousands of citations. But then THE stopped counting such papers and consequently in 2016 PU's citation score fell to 41 and its overall place to 121st. It has been downhill nearly every year since then.

The Director of PU's Internal Quality Assurance Cell said that the university has improved for research and teaching but needed to do better for international visibility and that proactive steps had been taken. The former Vice-Chancellor said that other universities were improving but PU was stagnating and was not even trying to overcome its weaknesses.

In fact, PU's decline had  nothing or very little  to do with international visibility or the policy failures of the administration just as its remarkable success in 2014 had nothing to do with working as a team, the strength brought by diversity, dynamic transformational leadership, or anything else plucked from the bureaucrats' big bag of clichés.

PU bet the farm on citations of particle physics mega-papers and for a couple of years that worked well. But then THE changed the rules of the game and stopped counting them, although after another year they received a reduced amount of credit. PU, along with the  Middle East Technical University and some other Turkish, Korean and French institutions that had overinvested in the CERN projects, tumbled down the THE rankings.

But PU may be about to make a comeback. Readers of this blog will probably guess what is coming next.

When THE stopped favouring papers with thousands of contributors they abolished the physics privilege that elevated places like Tokyo Metropolitan University, Federico Santa Maria Technical University, and Colorado School of Mines to the top of the research impact charts and replaced it with medical research privilege .

In 2018 PU started to publish papers that were part of the Gates-backed Global Burden of Disease Study (GBDS). Those papers have hundreds, but not thousands, of authors and so they are able to slip through the mega paper filter. The GBDS has helped Anglia Ruskin University, Brighton and Sussex Medical School, Aswan University, and Kurdistan University of Medical Sciences rise to the top of the research impact metric.

So far there have not been enough citations to make a difference for PU. But as those citations start to accumulate, providing of course that their impact is not diluted by too many other publications, PU will begin to rise again.

Assuming of course that THE's Methodology does not change.

An Indisputable Ranking Scorecard? Not Really.

The University of New South Wales (UNSW) has produced an aggregate ranking of global universities, known as ARTU. This is based on the "Big Three" rankers, QS, Times Higher Education (THE) and the Shanghai ARWU. The scores that are given are not an average but a aggregate of their ranks, which is then inverted. Nor surprisingly, Australian universities do well and the University of Melbourne is the best in Australia.

Nicholas Fisk, Deputy Vice Chancellor of Research, hopes that this ranking will become "the international scoreboard, like the ATP tennis rankings" and "the indisputable scoreboard for where people fit in on the academic rankings."

This is not a new idea. I had a go at producing an aggregate ranking a few years ago, called Global Ranking of Academic Performance or GRAPE. It was going to be the Comparative Ranking of Academic Performance: maybe I was right the first time.  It was justifiably criticised by Ben Sowter of QS. I think though that it was quite right to note that some of the rankings of the time underrated the top Japanese universities and overrated British and Australian schools.

The ARTU is an another example of the emergence of a cartel or near cartel of the three global rankings that are apparently considered the only ones worthy of attention by academic administrators and the official media.

There are in fact a lot more and these three are not even the best three rankings, far from it. A pilot study, Rating the Rankers,  conducted by the International Network of Research Management Systems (INORMS), has found that on four significant dimensions, transparency, governance, measuring what matters, and rigour, the performance of six well known rankings is variable and that of the big three is generally unimpressive. That of THE is especially deficient.

Seriously, should  we consider as indisputable a ranking that includes indicators that proclaim Anglia Ruskin University as a world leader for research impact and Anadolu University as tops for innovation, another that counts that long dead winners of Nobel and Fields awards, and another that gives disproportionate weight to a survey with more respondents from Australia than from China?

There does seem to be a new mood of ranking skepticism emerging in many parts of the international research community. Rating the Rankers has been announced in an article in Nature. The critical analysis of rankings will, I hope, do more to create fair and valid systems of comparative assessment than simply adding up a bunch of flawed and opaque indicators.

Indian University Performance to be Judged by Rankings

I have commented on Indian responses to the rankings before and many times on problems with the better known rankings so I apologize for repeating myself. 

The influence of global university rankings continues to expand. There seem to be few areas of higher education or research where they are not consulted or used for appointments, promotion, admissions, project evaluation, grant approval, assessment, publicity and so on.

The latest example is from India.  The Indian Express reports that the Minister of Education has announced that the progress of "institutions of eminence" [IoEs]will be charted using the "renowned" QS and THE rankings. Apparently, "an incentive mechanism will be developed for those institutes which are performing well." That is definitely not a good idea: it will reward behaviour that leads to improved ranking performance not to improved output or quality.

Recently, some of the leading Indian Institutes of Technology (IITs), four of which are on the list of IoEs, announced that they would be boycotting the THE rankings.  I am not sure whether this means that there is now a split within the higher education sector in India or whether the IITs are rethinking their opposition to the rankings.

There is nothing wrong with evaluating and comparing universities, research centers, researchers, or departments. Indeed it would seem very helpful if a country is going to maintain an effective higher education system.  But it is questionable whether these rankings are the best way or even a good way of doing it. Research might be evaluated by panels of peer researchers, provided these are unbiased and fair, by international experts, surveys, or by bibliometric and scientometric indicators. The quality of teaching and learning is more problematic but national rankings around the world have used several measures that, although not very satisfactory, might provide a rough and imperfect assessment.

There is now a broad range of international rankings covering publications, citations, innovation, web presence, and other metrics. The IREG Inventory of International Rankings identified 17 global rankings in addition to regional and specialist ones and there are now more. If the Indian government wanted to use a ranking to measure research output and quality then it would probably be better to refer to the Leiden Ranking produced by the CWTS at Leiden University, or other straightforward research-based rankings such as URAP, published by the Middle East Technical University in Ankara,  the Shanghai Rankings, or the National Taiwan University Rankings, which have a generally stable and transparent methodology. Another possibility is the Scimago Institution Rankings which include indicators measuring web activity and altmetrics. Round University Rankings uses several metrics that might have a relationship to teaching quality.

It is, however, debatable whether the THE rankings are useful for evaluating Indian universities. There are several serious problems, which I have been talking about since 2011. I will discuss just three of them.  

The THE world rankings lack transparency. Eleven of its 13 indicators are bundled in three super-indicators so that it is impossible to figure out exactly what is doing what. If a university, for example, gets an improved score for Teaching: The Learning Environment this could be because of an improved score for teaching reputation, an increase in the number of staff, a reduction in the number of students, an increase in the number of doctorates awarded, a reduction in the number of bachelor degrees, a decrease in the number of academic staff, an increase in institutional income, or a combination of two or more of these.

THE does make disaggregated data available to universities but that is of little use for students or other stakeholders.

Another problem is the face validity of the two stand-alone indicators. Take a look at the citations indicator results for 2020-2021, which are supposed to measure research impact or research quality These are the top universities for 2020-2021: Anglia Ruskin University, Babol Noshirvani University of Technology, Brighton and Sussex Medical School,  Cankaya University, the Indian Institute of Technology Ropar, Kurdistan University of Medical Sciences. 

Similarly with the industry income indicator, which is presented as a measure of innovation.. At the top of this indicator are Anadolu University, Asia University Taiwan, University of Freiburg, Istanbul Technical University, Khalifa University, LMU Munich, the Korea Advanced Institute of Science and Technology, and Makerere University. The German and Korean universities seem plausible but one wonders about the others.

THE has not discovered a brilliant new method of finding diamonds in the rough. It is just using a flawed and eccentric methodology, one that it has repeatedly claimed that it will reform but somehow has never quite got round to doing so.

Third, the THE rankings include indicators that dramatically favour high status western universities with money, prestige and large numbers of postgraduate students. There are three measure of income, two reputation surveys accounting for  a third of the total weighting, and two measures counting doctoral students.

The QS rankings are somewhat better but there are issues here as well. There is only one indicator for research, citations per faculty, and only one that is directly related to teaching quality, that is the employer reputation indicator with a ten per cent weighting.

The QS rankings are heavily overweight on research reputation, which has a 40% weighting and is hugely biased to certain countries. There are more respondents from Malaysia than from China  and more from Kazakhstan than from India.

Using either of these rankings opens the way to attempts to manipulate the system. It is possible to get a high score in the THE rankings by recruiting somebody involved in the Global Burden of Disease Study. Doing well in the QS rankings might be influenced by signing up for a reputation management program.

It seems, however, that there is a new mood of scepticism about rankings in academia. One sign is the Rating the Rankings project by the Research Evaluation Working group of the International Network of Research Management Systems. This is a rating, definitely not a ranking, of six rankings by an international  team of expert reviewers who did an evaluation according to four criteria: good governance, transparency, measure what matters, and rigour.

The results are interesting. No ranking is perfect but it seem that the famous brands are more likely to fall short of the criteria. 

The Indian government and others would be wise to take note of the analysis and criticism that is available before committing themselves to using rankings for the assessment of research or higher education.

How will the COVID-19 crisis affect the global rankings?

My article has just been published in University World News. 

You can read there and comment here.

The Great Acceleration

A major impact of the current virus crisis is that trends that were developing gradually have suddenly gathered new momentum. In many places around the world we can see the introduction of something like a universal basic income, the curtailment of carbon consuming transport, moves to abolish prisons and the police, and the continued rise of online education, online shopping, online almost anything.

Western universities have been facing an economic crisis for some time. Costs have soared and the supply of competent students has been drying up. For a while, the gap was filled with international, mainly Chinese, students but even before the virus that supply was dwindling. It seems likely that as universities open up, wholly or partly, there will be a lot fewer Chinese students and not many from other places. Meanwhile, local students and their parents will wonder whether there's any point in going deep into debt to pay for online or hybrid courses that will lead nowhere.

Organisations, like organisms, struggle to grow or to survive and universities need students to bring in revenue. If capable students are going to stay away then universities will have to recruit less capable students, close shop, merge, or become some other kind of institution. 

Another factor is the desperate need to calibrate the demographic composition of the student body, administration and faculty with that of the region or country or the world. Standardised testing has long been a problem here. Tests like the SAT, ACT, GRE, LSAT, and GMAT  are good predictors of academic ability and cognitive skills but they invariably give better scores to Whites and East Asians than to African Americans, Hispanics and Native Americans.

For many years there have been repeated demands that American universities abandon objective testing for admission and placement. One element in these demands was the observation that there was a correlation between family income and test scores. This was attributed to the ability of rich white parents to provide expensive test preparation courses for their  children.

There was an element of hypocrisy in these claims. If test prep courses were the cause of racial differences why not just make them a required part of the high school curriculum or pay the test centre fees and travel costs of low-income students? The failure to propose such measures suggests that everyone knows that this is not really the cause of racial or social achievements. 

The University of California (UC), once the exemplar of public tertiary education, has now decided to remove the SAT and ACT from the admissions process. The implications of this are large. Almost certainly many other public and perhaps some private universities will follow suit. The  UC Board of Regents has announced a plan to phase out standardised testing for undergraduate students by 2025.

In 2021 and 2022 UC will be test-optional. Students can submit test scores if they wish and UC campuses may use them as they see fit.

In 2023 and 2024 UC will be test-blind. Test scores will not be used for admissions although they might be used for course placement and scholarships. If I understand it correctly test scores could still be used for the admission of out of state and international students.

In 2025 the SAT/ACT will be phased out altogether and supposedly replaced by a new test "that more closely aligns with what we expect incoming students to know to demonstrate their preparedness for UC."

I would like to emulate Wayne Rooney and declare that I will never make a prediction but I cannot resist this one: there will never be a standardised test that will reconcile the need to predict academic ability with the demand for zero or minimal disparate racial impact.

UC will most probably end up with a complicated and holistic system of admission that will try to combine a semblance of selectivity with a mix of metrics that relate to subjective and marginal traits like grit, response to adversity, social awareness and so on and which produced an acceptable mix of groups.

It is very likely that the academic competence and cognitive skills of undergraduates and postgraduates at UC campuses will go into sharp decline. No doubt there will be compensations. Students will have be grittier, more diverse, more aware. Whether that is sufficient to balance the decline in cognitive ability remains to be seen

Meanwhile in  China and India, growing political authoritarianism and centralisation may lead to some decline in academic rigour but comparison to the US and Europe that still seems fairly limited.

Observations on the Indian Ranking Boycott

Seven Indian Institutes of Technology (IITs) -- Delhi, Bombay, Guwahati, Kanpur, Kharagpur, Madras, and Roorkee -- have announced that they will be boycotting this year's Times Higher Education (THE) World University Rankings. The move has been coming for some time. Indian universities have not performed well in most rankings but they have done especially badly in THE's.

Take a look at the at the latest THE world rankings and the performance of three elite institutions. IIT Delhi (IITD) and IIT Bombay (IITB) are in the 401-500 band, and the Indian Institute of Science (IISc) is 301-350.

It is noticeable that these three all do much better in the QS world rankings where IIT Delhi is 182nd, IIT Bombay 152nd, and IISc  184th. That no doubt explains why these Institutes are boycotting THE but still engaging with QS. 

It should be pointed out that with regard to research THE probably  treats the Institutes better than they deserve. The Shanghai rankings, which are concerned only with research, have IITD and IITB in the 701-800 band, and IISc at 401-500. In the US News Best Global Universities IITD is 654th, ITB 513th, and IISc 530th.

The dissatisfaction with THE is understandable. Indeed it might be surprising that the IITs have taken so long to take action. They complain about transparency and the parameters. They have a point, in fact several points. The THE rankings are uniquely opaque: they combine eleven indicators into three clusters so it is impossible for a reader to figure out exactly why a university is doing so well or so badly for teaching or research. THE's income and international metrics, three of each, also work against Indian universities.

It is, however, noticeable that a few Indian universities have done surprisingly well in the THE world rankings: IIT Ropar and IIT Indore are in the top 400 and IIT Gandhinagar in the top 600 thanks to high scores for citations. IIT Ropar is credited with a score of 100, making it fourth in the world behind those giants of research impact: Aswan University, Brandeis University, and Brighton and Sussex Medical School. 

Regular readers of this blog will know what is coming next. IIT Ropar has contributed to 15 papers related to the multi-author and hugely cited Global Burden of Disease Study (GBDS), which is slightly less than 1.5% of its total papers over the relevant period but well over 40% of citations. 

It would be relatively simple for the mutinous seven to recruit one or two researchers involved in the GBDS and in a few years -- assuming the current methodology or something like it continues -- they too would be getting near "perfect" scores for citations and heading for top three hundred spots.

They may, however, have judged that the THE methodology is going to be changed sooner or later -- now looking like a little bit later -- or that aiming for the QS reputation surveys is more cost effective. Or perhaps they were simply unaware of exactly how to get a good score in the THE rankings.

It is sad that the Indian debate over ranking has largely been limited to comparisons between THE and QS. There are other rankings that are technically better in some ways and are certainly better suited to Indian circumstances. The Round University Ranking which has 20 indicators and a balanced weighting has IISC in 62nd place with extremely good scores for financial sustainability and doctoral students.

The boycott is long overdue. If it leads to a more critical and sceptical approach to ranking then it may do everybody a lot of good.

THE's WUR 3.0 is on the way

Alert to readers. Some of this post covers ground I have been over before. See here, here and here. I plead guilty to self-plagiarism.

Times Higher Education (THE) is talking about a 3.0 version of its World University Rankings to be announced at this year's academic summit in Toronto and implemented in 2021, a timetable that may not survive the current virus crisis. I will discuss what is wrong with the rankings, what THE could do, and what it might do.

The magazine has achieved an enviable position in the university rankings industry. Global rankings produced by reliable university researchers with sensible methodologies, such as the CWTS Leiden Ranking, University Ranking by Academic Performance (Middle East Technical University) and the National Taiwan University Rankings are largely ignored by the media, celebrities and university administrators. In contrast, THE is almost always one of the Big Four rankings (the others are QS, US News, and Shanghai Ranking), the Big Three or the Big Two and sometimes the only global ranking that is discussed. 

The exalted status of THE is remarkable considering that it has many defects. It seems that the prestigious name -- there are still people who think that is the Times newspaper or part of it -- and  skillful public relations campaigns replete with events, workshops. gala dinners and networking lunches have eroded the common sense and critical capacity of the education media and the administrators of the Ivy League, the Russell Group and their imitators.

There are few things more indicative of the inadequacy of the current leadership of Western higher education than their toleration of a ranking that puts Aswan University top of the world for  research impact by virtue of its participation in the Gates funded Global Burden of Disease Study and Anadolu University top for innovation because it reported its income from private online courses as research income from industry. Would they really accept that sort of thing from a master's thesis candidate? It is true that the "Sokal squared" hoax has shown that that the capacity for critical thought has been seriously attenuated in the humanities and social sciences but one would expect better from philosophers, physicists and engineers.    

The THE world and regional rankings are distinctively flawed in several ways. First, a substantial amount of their data comes directly from institutions. Even if universities are 100% honest and transparent the probability that data will flow smoothly and accurately from branch campuses, research centres and far flung campuses through the committees tasked with data submission and on to the THE team is not very high.

THE has implemented an audit by PricewaterhouseCooper (PwC) but that seems to be about "testing the key controls to capture and handle data, and a full reperformance of the calculation of the rankings" and does not extend to checking the validity of the data before it enters the mysterious machinery of the rankings. PwC state that this is a "limited assurance engagement."

Second, THE is unique among the well-known rankings in bundling eleven of its 13 indicators in three groups with composite scores. That drastically reduces the utility of the rankings since it is impossible to figure out whether, for example, an improvement for research results from an increase in the number of published papers, an increase in research income, a decline in the number of research and academic staff, a better score for research reputation, or some combination of these. Individual universities can gain access to more detailed information but that is not necessarily helpful to students or other stakeholders.

Third, the THE rankings give a substantial weighting to various input metrics. One of these is income which is measured by three separate indicators, total institutional income, research income, and research income from industry. Of the  other world rankings only the Russian Round University Rankings do this. 

There is of course some relationship between funding and productivity but it is far from absolute and universal. The Universitas 21 system rankings, for example, show that countries like Malaysia and Saudi Arabia have substantial resources but so far have achieved only a  modest scientific output while Ireland has done very well in maintaining output despite a limited and declining resource base.    

The established universities of the world seem to be quite happy with these income indicators which, whatever happens, are greatly to their advantage. If their overall score goes down this can be plausibly attributed to a decline in funding that can be used to demand money from national resources. At a time when austerity has threatened the well being of many vulnerable groups, with more suffering to come in the next few months, it is arguable that universities are not those most deserving of state funding. 

Fourth, another problem arises from THE counting doctoral students in two indicators. It is difficult to see how the number of doctoral students or degrees can in itself add to the quality of undergraduate or master's teaching and this could act to the detriment of liberal arts colleges like Williams or Harvey Mudd which have an impressive record of produced employable graduates.

These indicators may also have the perverse consequence of  forcing people who would benefit from a master's or post graduate diploma course into doctoral programs with high rates of non-completion. 

Fifthly, the two stand alone indicators are very problematic. The industry income indicator purports to represent universities' contributions to innovation. An article by Alex Usher found that the indicator appeared to be based on very dubious data. See here for a reply by Phil Baty that is almost entirely tangential to the criticism. Even if the data were accurate it is a big stretch to claim that this is a valid measure of a university's contribution to innovation.

The citations indicator which is supposed to measure research impact, influence or quality is a disaster. Or it should be: the defects of this metric seem to have passed unnoticed everywhere it matters.

The original sin of the citations indicator goes back to the early days of the THE rankings after that unpleasant divorce from QS. THE used data from the ISI database, as it was then known, and in return agreed to give prominence to an indicator that was almost the same as the InCites platform that was a big-selling product.

The indicator is assigned a weighting of 30% which is much higher than that given to publications and higher than given to citations by QS, Shanghai US News or RUR. In fact this understates the weighting. THE has a regional modification or country bonus that divides the impact score of a university by the square root of the impact score of the country where it is located. The effect of this is that the scores of  universities in the top country will remain unchanged but everybody else will get an increase, a big one for low scoring countries, a smaller one for those scoring higher. Previously the bonus applied to the whole of the indicator but now it is 50%. Basically this means that universities are rewarded for being in a low scoring country.

The reason originally given for this was that some countries lack the networking and funds to nurture citation rich research. Apparently, such a problem has no relevance to international indicators. This was in fact probably an ad hoc way of getting round the massive gap between the world's elite and other universities with regard to citations, much bigger than most other metrics. 

The effect of this was to give a big advantage to mediocre universities surrounded by low achieving peers. Combined with other defects it has produced big distortions in the indicator.

This indicator is overnormalised. Citation scores are based not on a simple count of citations but rather on a comparison with the world average of citations according to year of publication, type of publication, and academic field, over three hundred of them. A few years ago someone told THE that absolute counting of citations was  a mortal sin and that seems to have become holy scripture. There is clearly a need to take account of disciplinary variations, such as the relative scarcity of citations in literary studies and philosophy and their proliferation in medical research and physics  but the finer the analysis gets the more chance there is that outliers  will exert a disproportionate effect on the impact score.

Perhaps the biggest problem with the THE rankings is the failure to use fractional counting of citations. There is an increasing problem with papers with scores, hundreds, occasionally thousands of "authors", in particle physics, medicine and genetics. Such papers often attract thousands of citations partly because of their scientific importance, partly because many of their authors will find opportunities to cite themselves.

The result is that until 2014-15  a university with  a modest contribution to a project like the Large Hadron Collider Project could get a massive score for citations especially if its overall output of papers was not high and especially if it was located in a country were citations were generally low.

The 2014-15 THE world rankings included among the world's leaders for citations Tokyo Metropolitan University, Federico Santa Maria Technical University, Florida Institute of Technology and Bogazici University.

Then THE introduced some reforms. Papers with over a thousand authors were excluded from the citation count, the country bonus was halved, and the source of bibliometric data was switched from  ISI to Scopus. This was disastrous for those universities that had over-invested in physics especially in Turkey, South Korea and France. 

The next year THE started counting the mega-papers again but introduced a modified form of fractional counting. Papers with a thousand plus papers were counted according to their contribution to the paper with a minimum of five per cent.

The effect of these changes was to replace physics privilege with medicine privilege. Fractional counting did not apply to papers with hundreds of authors but less than a thousand and so a new batch of improbable universities started getting near perfect scores for citations and began to break into the top five hundred or thousand in the world. Last year these included Aswan University, the Indian University of Technology Ropar, the University of Peradeniya, Anglia Ruskin University, the University of Reykjavik, and the University of Environmental and Occupational Health Japan.

They did so because of participation in the Global  Burden of Disease Study combined with  a modest overall output of papers and/or the good fortunate to be located in a country with a low impact score.

There is something else about the indicator that should be noted. THE includes self-citations and on a couple of occasions has said that this does not make any significant difference. Perhaps not in the aggregate, but there have been occasions when self-citers have in fact made a large difference to the scores of specific universities. In 2009 Alexandria University broke into the top 200 world universities by virtue of a self-citer and a few friends. In 2017 Veltech University was the third best university in India and the best in Asia for citations all because of exactly one self-citing author, In 2018 the university had for some reason completely disappeared from the Asian rankings.

So here are some fairly obvious things that THE ought to do:

• change the structure of the rankings to give more prominence to publications and less to citations
• remove the income indicators or reduce their weighting
• replace the income from industry indicator with a count of patents preferably those accepted rather than filed
• in general, where possible replace self-submitted with third party data
• if postgraduate students are to be counted then count master's as well as doctoral students
• get rid of the country bonus which exaggerates the scores of mediocre or sub-mediocre institutions because they are in the poorly performing countries
• adopt a moderate form of normalisation with a dozen or a score of fields rather than the present 300+ 
• use full-scale fractional counting 
• do not count self citations, even better do not count intra-institutional citations
• do not count secondary affiliations, although that is something that is more the responsibility of publishers
• introduce two or more measures of citations.

 But what will THE actually do?

Duncan Ross, THE data director, has published a few articles setting out some talking points (here, here, here,  here).
    He suggests that in the citation indicator THE should take the 75th percentile as the benchmark rather than the mean when calculating  field impact scores. If I understand it correctly this would reduce the extreme salience of outliers in this metric.

It seems that a number of new citations measures are being considered with the proportion of most cited publications apparently  getting the most favourable consideration. Unfortunately it seems that they are not going any further with fractional counting, supposedly because it will discourage collaboration. 

Ross  mentions changing the weighting of the indicators but does not seem enthusiastic about this. He also discusses the importance of measuring cross-disciplinary research.

THE is also considering the doctoral student  measures with the proportion of doctoral students who eventually graduate. They are  thinking about replacing institutional income with "a more precise measure," perhaps spending on teaching and teaching related activities. That would probably not be a good idea. I can think of all sorts of ways in which institutions could massage the data so that in the end it would be as questionable as the current industry income indicator.

It seems likely that patents will replace income from industry as the proxy for innovation.

So it appears that there will be some progress in reforming the THE world rankings. Whether it will be enough remains to be seen.

Who's doing research on Covid-19?

This is crude and simple. I typed in Covid-19 with fields Article title, Abstract, Keywords.

The first and oldest item of 1,500 to appear was "The New Coronavirus, the Current King of China." by S A Plotkin in the Journal of the Pediatric Infectious Diseases Society. I wonder if there will be a sequel, "The Current King of the World."

The top universities for number of publications are:

1.   Tongji Medical College
2.   Huazhong University of Science and Technology
3.   Chinese Academy of Medical Sciences and Peking Union Medical College
4.   "School of Medicine"
5.   London School of Hygiene and Tropical Medicine
6.   Wuhan University
7.   Chinese Academy of Sciences
8.   Capital Medical University
9.   Fudan University
10. National University of Singapore.

The top funding agencies are:

1.   National Natural Sciences Foundation of China
2.   National Basic Research Program of China
3.   National Institutes of Health
4.   Fundamental Research Funds for the Central Universities 
5.   Wellcome Trust
6.   Chinese Academy of Sciences
7.   Canadian Institutes of Health Research
8.   National Science Foundation
9.   Agence Nationale de la Recherche
10.  Chinese Academy of Medical Research