Thursday, November 20, 2014

More on the Ranking Status Wars


The Economist thinks there are two international university rankings worth talking about. Will the prestigious THE rankings continue to be prestigious now they are no longer powered by Thomson Reuters but have to share their data partner with QS?


"But most universities still have far to go. Only two Chinese institutions number in the top 100 in the Times Higher Education World University Rankings. Shanghai’s Jiao Tong University includes only 32 institutions from mainland China among the world’s 500 best. The government frets about the failure of a Chinese scholar ever to win a Nobel prize in science (although the country has a laureate for literature and an—unwelcome—winner in 2010 of the Nobel peace prize, Liu Xiaobo, an imprisoned dissident)".

The Times [Higher Education rankings] they are a-changing

Maybe I'll get my five minutes of fame for being first with a Dylan quotation. I was a bit slow because, unlike Jonah Lehrer, I wanted to check that the quotation actually exists.

Times Higher Education (THE) have announced that they will be introducing reforms to their World University Rankings and ending their partnership with media and data giant, Thomson Reuters (TR).

Exactly why is not stated. It could be rooted in financial disagreement. Maybe THE feels betrayed because TR let  US News use the reputation survey for their new Best Global Universities rankings. Perhaps THE got fed up with explaining why places like Bogazici University, Federico Santa Maria Technical University and Royal Holloway were world beaters for research impact, outshining Yale, Oxford and Cambridge,

The reputation survey will now be administered by THE itself in cooperation with Elsevier and will make use of the Scopus database. Institutional data will be collected  from universities, the Scopus database and the Scival analysis tool by a new THE team.

The coming months will reveal what THE have in store but for now this is a list of recommendations. No doubt there will be many more from all sorts of people.

Display each indicator separately instead of lumping them together into Teaching, Research and International Outlook. It is impossible to work out exactly what is causing a rise or fall in the rankings unless they are separated.

Try to find some why of reducing the volatility of the reputation survey. US News do this by using a five year average and  QS by rolling over unchanged responses for a further two years.

Consider including questions about undergraduate teaching or doing another survey to assess student satisfaction.

Reduce the weighting of the citations indicator and use more than one measure of citations to assess research quality (citations per paper), faculty quality (citations per faculty) and  research impact (total citations). Use field normalisation but sparingly and sensibly and forget about that regional modification.

Drop the Industry Income: Innovation indicator. It is unfair to liberal arts colleges and private universities and too dependent on input from institutions. Think about using patents instead.

Income is an input. Do not use unless it is to assess the efficiency of universities in producing research or graduates.

Considering dropping the international students indicator or at least reducing its weighting. It is too dependent on geography and encourages all sorts of immigration scams.

Benchmark scores against the means of a constant number of institutions. If you do not, the mean indicator scores will fluctuate from year to year causing all sorts of distortions.






Thursday, November 06, 2014

The US News Arab Region Rankings

Hardly a week passes without the publication of yet more international university rankings. This week it was the Best Arab Region Universities from the US News, famous for producing America's Best Colleges for over three decades.

These rankings are research based. There are nine indicators, one of which measures the number of publications and has a weighting of 30 per cent. The other eight relate to citations in some way. There are no indicators measuring faculty student ratio, teaching quality, graduate employment,  income or reputation.

Inclusion in the rankings required 400 papers in the Scopus database over a five year period,2009 to 2013. It is a serious indictment of Arab universities that only 91 institutions could  reach this  modest target.

There is an interesting section in the methodology:

"Papers published by Arab region institutions in the subject area of physics and astronomy were excluded based on input from Elsevier's bibliometric experts, who determined that their citation characteristics would distort the results of the overall rankings. There is, however, a separate subject ranking for physics and astronomy that is based on papers published exclusively in those fields."
This presumably means that US News is aware of the distorting effect of physics publications with a large number of contributing authors, which has helped propel institutions such as Panjab University and Federico Santa Maris Technical University into high spots in the THE world rankings.

The rankings show that research in the Arab world is dominated by a few countries. Just over half of the universities in the rankings come from three countries, Egypt, Saudi Arabia and Algeria. However, at the very top the rankings are dominated by Saudi Arabia, which holds the first three places.

The Top Ten are:

1.  King Saud University, Saudi Arabia
2.  King Abdualaziz University, Saudi Arabia
3.  King Abdullah University of Science and Technology, Saudi Arabia
4.  Cairo University, Egypt
5.  American University of Beirut, Lebanon
6.  Mansoura University, Egypt
7.  Ain Shams University, Egypt
8   King Fahd University of Petroleum and Minerals, Saudi Arabia
9.  Alexandria University, Egypt
10. United Arab Emirates University

There are also 16 subject rankings. Every one of these is topped by a Saudi university except for the Social Sciences which is headed by the American University of Beirut. In first place in the Physics rankings is King Abdulaziz University which has benefited from those multi-contributor publications which feature at least one of its adjunct faculty with a double affiliation.

Universite Cadi Ayyad Marrakech, Morocco, which was declared by THE to be the best Arab university and best in Africa north of the Kalahari, is in thirtieth place here. I wonder why.












Tuesday, November 04, 2014

The Taiwan (NTU) Rankings

These rankings are entirely research based and make no attempt to measure teaching quality. They also have a very strong bias against the humanities and social sciences: the London School of Economics and the Stockholm School of Economics do not appear at all.

They tend to reward size rather than quality so that Johns Hopkins is in 2nd place and Caltech 36th. The emphasis on citations gives a boost to medical schools like the University of California San Francisco and Rockefeller University.

The results, with these limitations, are quite reasonable.


Publisher

National Taiwan University



Scope

Global. Data provided for 500 universities. 903 ranked.selected from Essential Science Indicators and other rankings.

Indicators

Number of articles 2003-13: 10%
Number of articles 2013: 15%
Number of citations 2003-13: 15%
Number of citations 2012-13:10%
Average number of citations 2003-13: 10%
h-index 2012-13: 10%
Number of highly cited papers 2003-13: 15%
Number of articles in high-impact journals 2012: 15%


Top Ten (total score)


PlaceUniversity
1Harvard University
2Johns Hopkins University
3Stanford University
4University of Toronto
5University of Washington Seattle
6University of California Los Angeles                   
7University of Michigan Ann Arbor   
8=University of California Berkeley
8=Massachusetts Institute of Technology
8=University of Oxford



Countries with Universities in the Top Hundred


Country      Number of Universities  
USA45
UK8
Netherlands   7
Germany                                     5
Canada5
Australia4
China4
Sweden3
Japan3
France2
Belgium2
Denmark2
Switzerland2
Singapore1
Spain1
Italy1
Finland1
Brazil1
Taiwan1
Norway1
South Korea1



Top Ranked in Region (Total Score)


North America 
Harvard
AfricaUniversity of Cape Town
EuropeOxford University
Latin AmericaUniversidade de Sao Paulo                                    
AsiaUniversity of Tokyo                                 
Central and Eastern Europe  Charles University in Prague                                 
Arab WorldKing Abdulaziz University                                 
Middle EastTel Aviv University                              
OceaniaUniversity of Melbourne                              



Noise Index

In the top 20, the NTU rankings are more volatile than the THE world rankings but less so than QS, with the average university moving up or down 1.2 places since last year.

RankingAverage Place Change
 of Universities in the top 20 
NTU rankings 2013-141.20
THE World rankings 2013-140.70
QS World Rankings 2013-20141.45
ARWU 2013 -2014 0.65
Webometrics 2013-20144.25
Center for World University Ranking (Jeddah)
2013-2014 
0.90


Looking at the top 100 universities, the  NTU rankings are very volatile, since several of the indicators cover a one or two year period. The average university has changed 7.3 places over the year.

RankingAverage Place Change
 of Universities in the top 100 
NTU rankings 2013-147.30
THE World Rankings 2013-20144.34
QS World Rankings 2013-143.94
QS World Rankings 2013-143.94
ARWU 2013 -2014 4.92
Webometrics 2013-201412.08
Center for World University Ranking (Jeddah)
2013-2014 
10.59


Note: universities falling out of the top 100 are treated as though they fell to 101st position.


Methodology

See here