Discussion in the Straits Times

Two weeks ago the Singapore Straits Times included an article by Phil Baty of Times Higher education in which he explained why THE had "torn up" its annual rankings.

"Perhaps the most embarrassing aspect of the old rankings was the so-called 'peer review' score. Some 40 per cent of a university's overall ranking score was based on this 'peer review' - in effect, a simple opinion survey, asking university staff which institutions they rated most highly.

Our former data provider, QS, achieved only a very small number of responses to this survey. Last year, around 3,500 people responded. Figures for individual countries were shocking. In 2008, just 563 responses from Britain were received, and just 180 responses from Malaysia. Most shockingly, only 116 responses were collected from China's many, many thousands of scholars in 2008. "

He then refers to the extreme volatility of the rankings:

"The University of Malaya in Malaysia, for instance, plummeted from 89th place in 2004 to joint 169th in 2005, before dropping out of the top 200 altogether later. Between 2008 and last year, Keio University in Japan moved up an amazing 72 places to 142. Pohang University of Science and Technology in South Korea jumped 54 places to 134."

Pohang did rise because of a marked improvement in the peer review but Keio's rise was because it did better on the student faculty ratio indicator. The fall of Universiti Malaya was because ethnic minorities were counted as international students and faculty in 2004 but not, after a "clarification of data" in 2005.

He then describes some features of the new THE reputational survey and concludes:

"So much rests on the results of our rankings: individual university reputations, student recruitment, vice-chancellors' and presidents' jobs sometimes, and major government investment decisions. We have a duty to overhaul the rankings to make them fit for such purposes. "

Two days ago there was a reply by Nunzio Quacquarelli of QS. He argued:

"The numbers of respondents to the QS academic peer review, quoted by Mr Baty, are misleading.

Our 2009 rankings were based upon 9,386 respondents, not 3,500 as quoted. QS received statistically significant numbers of academic respondents from all major Organisation for Economic Cooperation and Development (OECD) countries, and 222 in Singapore. "

Some clarification is in order. Quacquarelli is correct in noting that the 2009 world rankings were based on 9,386 respondents. But it should be pointed out that about two thirds of those were respondents who had filled out the survey forms in 2008 and 2007 and had been given the opportunity to update their responses. If there was no updating then the old responses were included. Thus it is not impossible that some respondents had by 2009 retired, lost interest, moved or even died.

He then quotes from noted statistician Paul Thurman to claim that the academic opinion survey was valid.

Quacquarelli has a point in that it is not numbers alone that contribute to validity. However, it is hard to accept that a survey with more respondents from Ireland than from Russia and more from Hong Kong than from Japan can be regarded as a valid representation of global academic opinion.

Then there is an unconvincing assertion that THE had consistently endorsed the THE-QS rankings. This in fact refers to the time when John O'Leary and Martin Ince, now with QS, were editor and deputy editor of THE. Well, they would, wouldn't they?

It still remains to be seen whether the new THE survey will be better than QS's. It asks more questions and more detailed ones and has been distributed in several languages but the question of representativeness still remains. Thomson Reuters appear to be satisfied with the number of responses they have received but have not said how many there were or how they were distributed. I have a subjective impression that Southeast Asia and other regions may be underrepresented this time. I would, however, welcome detailed information from Thomson Reuters that would prove me wrong.