Indian University Performance to be Judged by Rankings

I have commented on Indian responses to the rankings before and many times on problems with the better known rankings so I apologize for repeating myself. 

The influence of global university rankings continues to expand. There seem to be few areas of higher education or research where they are not consulted or used for appointments, promotion, admissions, project evaluation, grant approval, assessment, publicity and so on.

The latest example is from India.  The Indian Express reports that the Minister of Education has announced that the progress of "institutions of eminence" [IoEs]will be charted using the "renowned" QS and THE rankings. Apparently, "an incentive mechanism will be developed for those institutes which are performing well." That is definitely not a good idea: it will reward behaviour that leads to improved ranking performance not to improved output or quality.

Recently, some of the leading Indian Institutes of Technology (IITs), four of which are on the list of IoEs, announced that they would be boycotting the THE rankings.  I am not sure whether this means that there is now a split within the higher education sector in India or whether the IITs are rethinking their opposition to the rankings.

There is nothing wrong with evaluating and comparing universities, research centers, researchers, or departments. Indeed it would seem very helpful if a country is going to maintain an effective higher education system.  But it is questionable whether these rankings are the best way or even a good way of doing it. Research might be evaluated by panels of peer researchers, provided these are unbiased and fair, by international experts, surveys, or by bibliometric and scientometric indicators. The quality of teaching and learning is more problematic but national rankings around the world have used several measures that, although not very satisfactory, might provide a rough and imperfect assessment.

There is now a broad range of international rankings covering publications, citations, innovation, web presence, and other metrics. The IREG Inventory of International Rankings identified 17 global rankings in addition to regional and specialist ones and there are now more. If the Indian government wanted to use a ranking to measure research output and quality then it would probably be better to refer to the Leiden Ranking produced by the CWTS at Leiden University, or other straightforward research-based rankings such as URAP, published by the Middle East Technical University in Ankara,  the Shanghai Rankings, or the National Taiwan University Rankings, which have a generally stable and transparent methodology. Another possibility is the Scimago Institution Rankings which include indicators measuring web activity and altmetrics. Round University Rankings uses several metrics that might have a relationship to teaching quality.

It is, however, debatable whether the THE rankings are useful for evaluating Indian universities. There are several serious problems, which I have been talking about since 2011. I will discuss just three of them.  

The THE world rankings lack transparency. Eleven of its 13 indicators are bundled in three super-indicators so that it is impossible to figure out exactly what is doing what. If a university, for example, gets an improved score for Teaching: The Learning Environment this could be because of an improved score for teaching reputation, an increase in the number of staff, a reduction in the number of students, an increase in the number of doctorates awarded, a reduction in the number of bachelor degrees, a decrease in the number of academic staff, an increase in institutional income, or a combination of two or more of these.

THE does make disaggregated data available to universities but that is of little use for students or other stakeholders.

Another problem is the face validity of the two stand-alone indicators. Take a look at the citations indicator results for 2020-2021, which are supposed to measure research impact or research quality These are the top universities for 2020-2021: Anglia Ruskin University, Babol Noshirvani University of Technology, Brighton and Sussex Medical School,  Cankaya University, the Indian Institute of Technology Ropar, Kurdistan University of Medical Sciences. 

Similarly with the industry income indicator, which is presented as a measure of innovation.. At the top of this indicator are Anadolu University, Asia University Taiwan, University of Freiburg, Istanbul Technical University, Khalifa University, LMU Munich, the Korea Advanced Institute of Science and Technology, and Makerere University. The German and Korean universities seem plausible but one wonders about the others.

THE has not discovered a brilliant new method of finding diamonds in the rough. It is just using a flawed and eccentric methodology, one that it has repeatedly claimed that it will reform but somehow has never quite got round to doing so.

Third, the THE rankings include indicators that dramatically favour high status western universities with money, prestige and large numbers of postgraduate students. There are three measure of income, two reputation surveys accounting for  a third of the total weighting, and two measures counting doctoral students.

The QS rankings are somewhat better but there are issues here as well. There is only one indicator for research, citations per faculty, and only one that is directly related to teaching quality, that is the employer reputation indicator with a ten per cent weighting.

The QS rankings are heavily overweight on research reputation, which has a 40% weighting and is hugely biased to certain countries. There are more respondents from Malaysia than from China  and more from Kazakhstan than from India.

Using either of these rankings opens the way to attempts to manipulate the system. It is possible to get a high score in the THE rankings by recruiting somebody involved in the Global Burden of Disease Study. Doing well in the QS rankings might be influenced by signing up for a reputation management program.

It seems, however, that there is a new mood of scepticism about rankings in academia. One sign is the Rating the Rankings project by the Research Evaluation Working group of the International Network of Research Management Systems. This is a rating, definitely not a ranking, of six rankings by an international  team of expert reviewers who did an evaluation according to four criteria: good governance, transparency, measure what matters, and rigour.

The results are interesting. No ranking is perfect but it seem that the famous brands are more likely to fall short of the criteria. 

The Indian government and others would be wise to take note of the analysis and criticism that is available before committing themselves to using rankings for the assessment of research or higher education.

How will the COVID-19 crisis affect the global rankings?

My article has just been published in University World News. 

You can read there and comment here.

The Great Acceleration

A major impact of the current virus crisis is that trends that were developing gradually have suddenly gathered new momentum. In many places around the world we can see the introduction of something like a universal basic income, the curtailment of carbon consuming transport, moves to abolish prisons and the police, and the continued rise of online education, online shopping, online almost anything.

Western universities have been facing an economic crisis for some time. Costs have soared and the supply of competent students has been drying up. For a while, the gap was filled with international, mainly Chinese, students but even before the virus that supply was dwindling. It seems likely that as universities open up, wholly or partly, there will be a lot fewer Chinese students and not many from other places. Meanwhile, local students and their parents will wonder whether there's any point in going deep into debt to pay for online or hybrid courses that will lead nowhere.

Organisations, like organisms, struggle to grow or to survive and universities need students to bring in revenue. If capable students are going to stay away then universities will have to recruit less capable students, close shop, merge, or become some other kind of institution. 

Another factor is the desperate need to calibrate the demographic composition of the student body, administration and faculty with that of the region or country or the world. Standardised testing has long been a problem here. Tests like the SAT, ACT, GRE, LSAT, and GMAT  are good predictors of academic ability and cognitive skills but they invariably give better scores to Whites and East Asians than to African Americans, Hispanics and Native Americans.

For many years there have been repeated demands that American universities abandon objective testing for admission and placement. One element in these demands was the observation that there was a correlation between family income and test scores. This was attributed to the ability of rich white parents to provide expensive test preparation courses for their  children.

There was an element of hypocrisy in these claims. If test prep courses were the cause of racial differences why not just make them a required part of the high school curriculum or pay the test centre fees and travel costs of low-income students? The failure to propose such measures suggests that everyone knows that this is not really the cause of racial or social achievements. 

The University of California (UC), once the exemplar of public tertiary education, has now decided to remove the SAT and ACT from the admissions process. The implications of this are large. Almost certainly many other public and perhaps some private universities will follow suit. The  UC Board of Regents has announced a plan to phase out standardised testing for undergraduate students by 2025.

In 2021 and 2022 UC will be test-optional. Students can submit test scores if they wish and UC campuses may use them as they see fit.

In 2023 and 2024 UC will be test-blind. Test scores will not be used for admissions although they might be used for course placement and scholarships. If I understand it correctly test scores could still be used for the admission of out of state and international students.

In 2025 the SAT/ACT will be phased out altogether and supposedly replaced by a new test "that more closely aligns with what we expect incoming students to know to demonstrate their preparedness for UC."

I would like to emulate Wayne Rooney and declare that I will never make a prediction but I cannot resist this one: there will never be a standardised test that will reconcile the need to predict academic ability with the demand for zero or minimal disparate racial impact.

UC will most probably end up with a complicated and holistic system of admission that will try to combine a semblance of selectivity with a mix of metrics that relate to subjective and marginal traits like grit, response to adversity, social awareness and so on and which produced an acceptable mix of groups.

It is very likely that the academic competence and cognitive skills of undergraduates and postgraduates at UC campuses will go into sharp decline. No doubt there will be compensations. Students will have be grittier, more diverse, more aware. Whether that is sufficient to balance the decline in cognitive ability remains to be seen

Meanwhile in  China and India, growing political authoritarianism and centralisation may lead to some decline in academic rigour but comparison to the US and Europe that still seems fairly limited.

Observations on the Indian Ranking Boycott

Seven Indian Institutes of Technology (IITs) -- Delhi, Bombay, Guwahati, Kanpur, Kharagpur, Madras, and Roorkee -- have announced that they will be boycotting this year's Times Higher Education (THE) World University Rankings. The move has been coming for some time. Indian universities have not performed well in most rankings but they have done especially badly in THE's.

Take a look at the at the latest THE world rankings and the performance of three elite institutions. IIT Delhi (IITD) and IIT Bombay (IITB) are in the 401-500 band, and the Indian Institute of Science (IISc) is 301-350.

It is noticeable that these three all do much better in the QS world rankings where IIT Delhi is 182nd, IIT Bombay 152nd, and IISc  184th. That no doubt explains why these Institutes are boycotting THE but still engaging with QS. 

It should be pointed out that with regard to research THE probably  treats the Institutes better than they deserve. The Shanghai rankings, which are concerned only with research, have IITD and IITB in the 701-800 band, and IISc at 401-500. In the US News Best Global Universities IITD is 654th, ITB 513th, and IISc 530th.

The dissatisfaction with THE is understandable. Indeed it might be surprising that the IITs have taken so long to take action. They complain about transparency and the parameters. They have a point, in fact several points. The THE rankings are uniquely opaque: they combine eleven indicators into three clusters so it is impossible for a reader to figure out exactly why a university is doing so well or so badly for teaching or research. THE's income and international metrics, three of each, also work against Indian universities.

It is, however, noticeable that a few Indian universities have done surprisingly well in the THE world rankings: IIT Ropar and IIT Indore are in the top 400 and IIT Gandhinagar in the top 600 thanks to high scores for citations. IIT Ropar is credited with a score of 100, making it fourth in the world behind those giants of research impact: Aswan University, Brandeis University, and Brighton and Sussex Medical School. 

Regular readers of this blog will know what is coming next. IIT Ropar has contributed to 15 papers related to the multi-author and hugely cited Global Burden of Disease Study (GBDS), which is slightly less than 1.5% of its total papers over the relevant period but well over 40% of citations. 

It would be relatively simple for the mutinous seven to recruit one or two researchers involved in the GBDS and in a few years -- assuming the current methodology or something like it continues -- they too would be getting near "perfect" scores for citations and heading for top three hundred spots.

They may, however, have judged that the THE methodology is going to be changed sooner or later -- now looking like a little bit later -- or that aiming for the QS reputation surveys is more cost effective. Or perhaps they were simply unaware of exactly how to get a good score in the THE rankings.

It is sad that the Indian debate over ranking has largely been limited to comparisons between THE and QS. There are other rankings that are technically better in some ways and are certainly better suited to Indian circumstances. The Round University Ranking which has 20 indicators and a balanced weighting has IISC in 62nd place with extremely good scores for financial sustainability and doctoral students.

The boycott is long overdue. If it leads to a more critical and sceptical approach to ranking then it may do everybody a lot of good.