Saturday, March 12, 2016
I'm glad somebody's noticed
Uwe Brandenburg, Managing Director of CHE Consult in Times Higher Education:
"Frankly, I think the success of certain institutions in rankings is more to do with the rankings’ methodology than anything else. They inevitably favour factors that are statistically more likely to be found among certain universities than others."
Ranking Rankings 1: Stability
Updated 13/03/16, 15/03/16
Making a start on ranking global university rankings, here is the average change in position of the top 20 twenty universities in 2014 in seven global university rankings between 2014 and 2015.
Note: both QS and THE introduced major methodological changes in 2015.
The table refers only to the top 20. Things might (or might not) be different if the top 100 or 500 were considered.
The Shanghai ARWU and URAP are so far well ahead of the others for stability.
| Rankings | Mean position change of top 20 universities 2014-15 |
|---|---|
| Shanghai Academic Ranking of World Universities (ARWU) | 0.30 |
| Center for World University Rankings (CWUR) (Jeddah) | 0.40 |
| University Ranking By Academic Performance (URAP) (Middle East Technical University) |
0.55 |
| THE World University Rankings | 2.10 |
| Round University Rankings (Russia) | 2.35 |
| QS World University Rankings | 3.50 |
| Webometrics | 8.85 |
News Lite from Times Higher
The mainstream media in the UK and parts of Europe is getting very excited about the latest ranking from Times Higher Education, the top 200 European universities.
There is nothing new here. THE have just pulled the European universities out of last year's world rankings.
Here are some of the headlines that have greeted the remarkable revelations.
Leicester Mercury
Google News
Quarter of top 200 European universities are in UK - league table
Wessex Scene
World News Online
Myinforms
Dundee University 'on a global scale' after European rankings accolade
Bailiwick Express
A quarter of Europe's top 200 universities are in the UK
Flanders Today
In-Cyprus
UCy among the top 200 Unis, according to Times
The Tab
UCL ranked 5th best university in Europe
Plymouth Herald
Westcountry universities carve out a place in elite list
Shanghai Daily
Oxford, Cambridge top rankings of European universities
There is nothing new here. THE have just pulled the European universities out of last year's world rankings.
Here are some of the headlines that have greeted the remarkable revelations.
Leicester Mercury
|
Quarter of top 200 European universities are in UK - league table
Wessex Scene
World News Online
Myinforms
Dundee University 'on a global scale' after European rankings accolade
Bailiwick Express
A quarter of Europe's top 200 universities are in the UK
Flanders Today
UCy among the top 200 Unis, according to Times
The Tab
UCL ranked 5th best university in Europe
Plymouth Herald
Westcountry universities carve out a place in elite list
Shanghai Daily
Oxford, Cambridge top rankings of European universities
Monday, March 07, 2016
Is it possible to rank rankings?
At a seminar recently at Ural Federal University in Ekaterinburg the question was raised whether we could evaluate and rank rankings.
That's a tough one. Different rankings have different approaches, audiences and methodologies. The Shanghai rankings embody the concerns of the Chinese ruling class, convinced that salvation lies in science and technology and disdainful -- not entirely without reason -- of the social sciences and humanities. The Times Higher Education world rankings have claimed to be looking for quality, recently tempered by a somewhat unconvincing concern for inclusiveness.
But it is possible that there are some simple metrics that could be used to compare global rankings. here are some suggestions.
Stability
Universities are big places typically with thousands of students and hundreds of staff. In the absence of administrative reorganisation or methodological changes we should not expect dramatic change from one year to another. Apparently a change of four places over a year is normal for the US News America's Best Colleges so nobody should get excited about going up or down a couple of places.
It would seem reasonable then that rankings could be ordered according to the average change in position over a year. I have already done some calculations with previous years' rankings (see posts 09/07/14, 17/07/14, 04/11/14).
So we could rank these international rankings according to the mean number of position changes in the top 100 between 2013 and 2014. The fewer the more stable the rankings.
1. Quacquarelli Symonds World University Rankings 3.94
2. Times Higher Education World University Rankings 4.34
3. Shanghai ARWU 4.92
4. National Taiwan University Rankings 7.30
5. CWUR (Jeddah) 10.59
6. Webometrics 12.08
Consistency and Redundancy
It is reasonable that if the various ranking indicators are measuring quality or highly valued attributes there should be at least a modest correlation between them. Good students will attract good teachers who might also have the attributes, such as an interest in their field or reading comprehension skills, required to do research. Talented researchers will be drawn to places that are generously funded or highly reputed.
On the other hand, if there is a very high correlation between two indicators, perhaps above .850, then this probably means that they are measuring the same thing. One of them could be discarded.
Transparency
Some rankings have adopted the practice of putting universities into bands rather than giving them individual scores. This is, I suppose, a sensible way of discouraging people from getting excited about insignificant fluctuation but it might also suggest a lack of confidence in the rankers' data or the intention of selling the data in some way, perhaps in the guise of benchmarking. Since 2010 THE have bundled indicator scores into clusters making it very difficult to figure out exactly what is causing universities to rise or fall. Rankings could be ranked according to the number of universities for which overall scores and indicator scores are provided.
Inclusiveness
It would be very easy to rank rankings according to the number of universities that they include. This is something where they vary considerably. The Shanghai ARWU ranks 500 universities while Webometrics ranks close to 24,000.
Comprehensiveness
Some rankings such as ARWU, the NTU rankings (Taiwan), URAP (Middle East Technical University) measure only research output and impact. The THE and QS rankings attempt to include metrics related, perhaps distantly, to teaching quality and innovation. QS has an indicator that purports
Balance
Some rankings award a disproportionate weighting to a single indicator, QS's academic survey (40%), THE's citation indicator (30%). Also, if a university or universities are getting disproportionately high scores for a specific indicator this might mean that the rankings are being manipulated or are seriously flawed in some way.
External Validation
How do we know that rankings measure what they are supposed to measure? It might be possible to measure the correlation between international rankings and national rankings which often include more data and embody local knowledge about the merits of universities.
Replicability
How long would it take to check whether the rankings have given your university the correct indicator score? Try it for yourself with the Shanghai highly cited researchers indicator. Go here and find the number of highly cited researchers with Harvard as their primary affiliation and the number with your university. Find the square root of both numbers. Then give Harvard a score of 100 and adjust your university's score accordingly.
Now for the THE citations impact indicator. This is normalised by field and by year of citation so that the what matters is not the number of citations that a publication gets but the number of publications compared to the world average in 334 fields and in the first, second, third, fourth, fifth or sixth year of publication.
Dead simple isn't it?
And don't forget the regional modification.
I hope the next post will be a ranking of rankings according to stability.
That's a tough one. Different rankings have different approaches, audiences and methodologies. The Shanghai rankings embody the concerns of the Chinese ruling class, convinced that salvation lies in science and technology and disdainful -- not entirely without reason -- of the social sciences and humanities. The Times Higher Education world rankings have claimed to be looking for quality, recently tempered by a somewhat unconvincing concern for inclusiveness.
But it is possible that there are some simple metrics that could be used to compare global rankings. here are some suggestions.
Stability
Universities are big places typically with thousands of students and hundreds of staff. In the absence of administrative reorganisation or methodological changes we should not expect dramatic change from one year to another. Apparently a change of four places over a year is normal for the US News America's Best Colleges so nobody should get excited about going up or down a couple of places.
It would seem reasonable then that rankings could be ordered according to the average change in position over a year. I have already done some calculations with previous years' rankings (see posts 09/07/14, 17/07/14, 04/11/14).
So we could rank these international rankings according to the mean number of position changes in the top 100 between 2013 and 2014. The fewer the more stable the rankings.
1. Quacquarelli Symonds World University Rankings 3.94
2. Times Higher Education World University Rankings 4.34
3. Shanghai ARWU 4.92
4. National Taiwan University Rankings 7.30
5. CWUR (Jeddah) 10.59
6. Webometrics 12.08
Consistency and Redundancy
It is reasonable that if the various ranking indicators are measuring quality or highly valued attributes there should be at least a modest correlation between them. Good students will attract good teachers who might also have the attributes, such as an interest in their field or reading comprehension skills, required to do research. Talented researchers will be drawn to places that are generously funded or highly reputed.
On the other hand, if there is a very high correlation between two indicators, perhaps above .850, then this probably means that they are measuring the same thing. One of them could be discarded.
Transparency
Some rankings have adopted the practice of putting universities into bands rather than giving them individual scores. This is, I suppose, a sensible way of discouraging people from getting excited about insignificant fluctuation but it might also suggest a lack of confidence in the rankers' data or the intention of selling the data in some way, perhaps in the guise of benchmarking. Since 2010 THE have bundled indicator scores into clusters making it very difficult to figure out exactly what is causing universities to rise or fall. Rankings could be ranked according to the number of universities for which overall scores and indicator scores are provided.
Inclusiveness
It would be very easy to rank rankings according to the number of universities that they include. This is something where they vary considerably. The Shanghai ARWU ranks 500 universities while Webometrics ranks close to 24,000.
Comprehensiveness
Some rankings such as ARWU, the NTU rankings (Taiwan), URAP (Middle East Technical University) measure only research output and impact. The THE and QS rankings attempt to include metrics related, perhaps distantly, to teaching quality and innovation. QS has an indicator that purports
Balance
Some rankings award a disproportionate weighting to a single indicator, QS's academic survey (40%), THE's citation indicator (30%). Also, if a university or universities are getting disproportionately high scores for a specific indicator this might mean that the rankings are being manipulated or are seriously flawed in some way.
External Validation
How do we know that rankings measure what they are supposed to measure? It might be possible to measure the correlation between international rankings and national rankings which often include more data and embody local knowledge about the merits of universities.
Replicability
How long would it take to check whether the rankings have given your university the correct indicator score? Try it for yourself with the Shanghai highly cited researchers indicator. Go here and find the number of highly cited researchers with Harvard as their primary affiliation and the number with your university. Find the square root of both numbers. Then give Harvard a score of 100 and adjust your university's score accordingly.
Now for the THE citations impact indicator. This is normalised by field and by year of citation so that the what matters is not the number of citations that a publication gets but the number of publications compared to the world average in 334 fields and in the first, second, third, fourth, fifth or sixth year of publication.
Dead simple isn't it?
And don't forget the regional modification.
I hope the next post will be a ranking of rankings according to stability.
Wednesday, March 02, 2016
The Decline of Free Speech in American Universities
The Foundation for Individual Rights in Education has just released its list of the ten worse colleges for free speech in the US. Here they are along with the incidents that put them on the list.
Mount St Mary's University, Maryland
Two faculty were sacked for criticising the President's plan to get rid of low performing students, "drowning the bunnies" as he so charmingly put it. They were later reinstated.
Northwestern University
Laura Kipnis was investigated for sexual harassment for writing an essay criticising the sexual harassment mania sweeping US colleges. She was cleared only after writing an account of her persecution in the Chronicle of Higher Education.
Louisiana State University
Theresa Buchanan was fired for using profanity in the classroom for a pedagogical reason.
University of California San Diego
The administration attempted to defund a student newspaper for making fun of "safe spaces".
St Mary's University of Minnesota
An adjunct classics professor was fired for sexual harassment which have have something to do with an authentic production of Seneca's Medea. He was also fired from his other job as a janitor (!).
University of Oklahoma
Two fraternity members for leading a racist chant. The supreme court has ruled that offensive speech is protected by the first amendment
Marquette University
John Mcadams was for criticising an instructor for suppressing a student's negative comments about same sex marriage.
Colorado College
A student was suspended for unchivalrous remarks about African American women on Yik Yak.
University of Tulsa
A student was removed from class because of Facebook posts written by his fiance criticising a professor.
Wesleyan University
The student government voted to remove funding from a student newspaper that was mildly critical of Black Lives Matter.
Wednesday, February 24, 2016
Britain leads in sniffing edge research
There must be some formula that can predict the scientific research that will go viral over the social media or reach the pages of the popular press or even Times Higher Education (THE).
The number of papers in the natural and social sciences is getting close to uncountable. So why out of all of them has THE showcased a study of the disgust reported by students when sniffing sweaty T-shirts from other universities?
Anyway, here is a suggestion to the authors for a follow-up study. Have the students read the latest QS, THE or Shanghai world rankings before having a sniff and see if that makes any difference to the disgust experienced.
Subscribe to:
Posts (Atom)