Wednesday, August 16, 2017

Problems with global rankings

There is a problem with any sort of standardised testing. A test that is useful when a score has no financial or social significance becomes less valid when coaching industries workout how to squeeze a few points out of docile candidates and motivation becomes as important as aptitude.

Similarly, a metric used to rank universities may be valid and reliable when nobody cares about the rankings. But once they are used to determine bureaucrats' bonuses, regulate immigration, guide student applications and distribute research funding then they become less accurate. Universities will learn how to apply resources in exactly the right place, submit data in exactly the right way and engage productively with the rankers. The Trinity College Dublin data scandal, for example, has indicated how much a given reported income can affect ranks in the THE world rankings.

All of the current "big three" of global rankings have indicators that have become the source of volatility and that are given a disproportionate weighting. These are the normalised citations indicator in the THE rankings, the QS academic survey and the highly cited researchers list in the Shanghai ARWU.

Examples in the next post.


3 comments:

Tony Sheil said...

Richard,

Unfortunately the ARWU Hi-Ci indicator is slowly succumbing to Goodhart's Law - economics context but the same principle applies:

“Once a social or economic indicator is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play such a role.”

Charles Goodhart, Chief Economic Adviser to the Bank of England, 1975

Tony Sheil said...

Unfortunately the ARWU Hi-Ci indicator is over time succumbing to Goodhart's Law - economics context but same principle applies:

“Once a social or economic indicator is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play such a role.”

Charles Goodhart, Chief Economic Adviser to the Bank of England, 1975

Anonymous said...

Hello, you used to write wonderful, but the last several posts have been kinda boring?

I miss your super writings. Past few posts
are just a little bit out of track! come on!