Comment by Nunzio Quacquarelli

Several items about or by leading figures in the ranking business have appeared recently. Nunzio Quacquarelli, director of QS, says:

"QS research has become highly respected and every year is referenced in roughly 1000 different newspapers, journals and web sites – a Who’s Who of the best media around the world"
There has also been an enormous amount of criticism, especially by academic experts such as Anthony van Raan, Simon Marginson, Eric Beerkens and so on. In comparison, favourable comments by Alan Kantrow, former editor of the McKinsey Quarterly, and Martin Ince do not really count for much. We also have that old quotation from Richard Sykes, "it takes smart people to know smart people." This begs the question of whether the ability to sign up for a an academic mailing list is sufficient to be a certified smart person. Sykes, incidentally, is described as the Rector of Imperial College London, which he has not actually been for nearly two years.

Quacquarelli then refers to Paul Thurman, a specialist in public health statistics, who says that a 2% response rate yielding 9,500 responses is as good as or better than most political opinion polls. Perhaps, although political polls are far from perfect. But this comment misses an important point. QS may have achieved a representative sample of subscribers to World Scientific, an academic publishing company based in Singapore with links to Imperial College (no wonder Richard Sykes is talking about smart people) but are those subscribers representative of expert academic opinion?


The article continues:

"Over 8000 academics have attended seminars specifically debating the QS World University Rankings methodology. Amongst attendees there has been almost universal acceptance of the QS ranking criteria."

Yes, but what matters is those who did not attend attend the seminars.

While we are on the subject of meetings with academics, it might be worth remembering a meeting in Kuala Lumpur in November 2005 at which Quacquarelli is reported to have said that QS was "not aware of Malaysia's racial composition" (the Star 18/11/2005). This is forgivable. After all, the QS office nearest to Malaysia is in Singapore which is very far away. Another report indicated that the "talk attracted comments from the floor some of who disagreed with his ranking methodology." (Sun 22/11/2005).

It seems that Nunzio Quacquarelli sees nothing wrong in the THE-QS rankings. Improvements have been made but there is certainly room for more. Failing to recognise the flaws in the rankings will not help anyone.

Graduate School Rankings



US News has released its annual ranking of American graduate schools. These are subject rankings rather than holistic.



The top schools in selected categories are:



Business: Harvard, Stanford

Education: Vanderbilt

Engineering: MIT

Law: Yale

Medical Research: Harvard

Medical Primary Care: University of Washington, Seattle

Biological Sciences: Stanford

Chemistry: Caltech

Computer Science: Carnegie-Mellon, MIT, Stanford

Earth Sciences: Caltech, MIT

Mathematics: MIT

Physics: Caltech, MIT, Berkeley

Statistics:Stanford

Economics: Harvard, Princeton, Chicago, Stanford

Library and Information Sciences: University of illinois: Urbana-Champaign, University of North Carolina: Chapel Hill

English: Berkeley

Psychology: Stanford, Berkeley

History: Princeton, Stanford, Berkeley, Yale

Public Affairs: Syracuse

Fine Arts: Rhode Island School of Design

The Survey Season

A few weeks ago there was a flurry of activity in the rankosphere as the leading rankers announced that they would be starting their surveys. It seems that we are now in a quiet period.

• QS announced that their employer survey would be open on March 23rd but to date there is no sign of life on their site.
• Academic survey forms from Thomson Reuters on behalf of Times Higher Education have been sent out to some countries. I am wondering though about how complete the coverage is. I have not received one although I admit that my record of publication in ISI indexed journals is not particularly brilliant.
• I have heard that several people have signed up for the World Scientific mailing lists, which provide the bulk of the respondents for the QS academic survey, but have not received any newsletters or other material. If World Scientific has frozen its lists or there is some sort of technical problem there might be a problem for QS.

Something from THE

An editorial by Phil Baty in Times Higher Education refers to a comment by Janez Potocnik, former European Commissioner for science and research, that university rankings are now used to assess national economic strength.

He then goes on to indicate other uses of rankings and to provide some more information about the forthcoming THE rankings. It seems that universities will be ranked in six subject areas, one more than in the THE-QS rankings, with life sciences and biomedicine being divided into life sciences and "clinical, pre-clinical and health".

Also "(w)e will also judge subject strength on the full range of measures used in the overall table. We believe this will represent another great improvement."

There might be a problem here. If THE are going to publish subject rankings based on the indicators used in the overall rankings it would not make sense to use the proportions of international students and international faculty, student faculty ratios or citations per faculty for the whole university to determine standing in specific disciplinary areas. So, Thomson Reuters would have to collect specific data about the numbers of international students in the social sciences and so on. Getting accurate information about numbers of students and faculty is diffcult enough for a university as a whole but for each disciplinary area it would be close to impossible. In any case, in many universities the boundaries between disciplinary areas may not correspond to those used by Thomson Reuters.

The reference to ranking the top 200 universities is disappointing. There is an enormous unmet demand for valid information about universities around the world, not just the top 200. Thomson Reuters say they are collecting information about a thousand universities (down from "thousands" a few weeks ago). It would be a pity to waste it.

Proposal for a Student League Table

The conference of the National Union of Students in the UK will condsider a proposal to start a student-run league table that will address "bread and butter" issues.

Another proposal for the conference is that university staff protesting against government cuts should "adopt a boycott on publishing their research".

The problem with that is that they would have to do some research in the first place.

Comments on the Economist article

The Economist has published a number of comments on its article on international university rankings.

The second, as we should have expected, is a complaint about the failure of the rankings to acknowledge the brilliance of LSE. The author, who has a master's from LSE, complains that "the THES and the Times usually score Oxford higher than the LSE- partly due to expenditure and, in the case of the Times, the higher number of 'good marks' " Unfortunately, the THE (no S now) - QS rankings never included expenditure as an indicator: Oxford outscored LSE mainly because of its performance on the academic survey component. I wonder if this says something about the excellent and rigorous research training provided by LSE.

There are also several complaints that teaching quality is not reflected in the current rankings and may even be inversely related to the research productivity of universities. However, there is a sensible observation from Rojr that

"There seems to be a popular but silly assumption that clever Oxbridge graduates are made clever by their experience at Oxbridge. Why? It's well known that Oxbridge picks the cleverest students up front--who's surprised that they're still the cleverest after graduating? Correlation does not imply causation! I wish the league tables would catch on to this.

If I were an employer, an acceptance letter from a top ranking university would be exactly as impressive as a graduation certificate (with any grade on it) from the same university. "

I suspect that global rankings would be more improved by a measure of student quality based on performance in national standardised tests such as the SAT than by efforts to assess the vague and culture-bound concept of teaching quality. I wonder also whether Oxbridge is still picking the cleverest students now that the entrance exams have been scrapped and A levels have become largely meaningless, leaving colleges reliant on admission interviews, probably the least valid of any selection procedure.

Finally, there are comments from Phil Baty of THE and Ben Sowter of QS.

Article in the Australian

An article by Andrew Trounson in the Australian discusses the developng rivalry between the Times Higher Education and the QS rankings and its implications for Australian universities.

Both ranking groups are attempting to increase the number of responses to the academic surveys that will be in both rankings. This might have serious consequences for Australian universities which always did better in the survey-based components of the THE_QS rankings than in the other indicators and which now face the prospect of losing ground in a bigger and more diverse survey.