The Politics of Rankings: The Case of Macquarie

The Sydney Morning Herald has an article by Steven Schwartz, Vice-Chancellor of Macquarie University. He begins by arguing that university rankings cannot capture the full complexity of a large modern university. A good point, although it would have been more convincing had it been made before rather than after Macquarie's spectacular fall in the THES-QS rankings.

Schwartz goes on to say that:

Although those who work in universities know the pace of change is glacial, university rankings can change dramatically. For example, the Times Higher Education Supplement in Britain dropped Macquarie more than 80 places down the ranks in one year - front-page news in this newspaper. Was the previous ranking incorrect? Is the present one more accurate? The answer in both cases is no.

The changed ranking resulted from a decision by the publication to reduce the weight given to international students, so that many universities with large international enrolments dropped down the rankings. The prestigious London School of Economics dropped from 17 to 59. By omitting mention of this change in method, the Herald's report on November 9 produced more heat than light.

This is an extraordinary claim. There has been no change whatsoever in the weighting given to international students in the THES-QS rankngs. It is five per cent this year just as it has been since 2004.

Macquarie has fallen in the rankings for two reasons. First it fell from 93rd position in the survey of academic opinion to 142nd (among the overall top 400 universities). This could be because QS, the consultants who collect the data for the rankings, did not allow respondents to vote for their own institutions this year or because the number of respondents from Australia was lower.

Second, in 2006 Macquarie was in first place for international faculty, meaning that QS, must have thought that at least half of Macquarie's faculty were international. This year the rankings have Macquarie in 55th place for international faculty. This represents, according to the QS website (registration required), a figure of 25 % for international faculty.

Dr Schwartz would be well advised to find out how QS received incorrect information about international faculty in 2006.

The international students section had nothing at all to do with Macquarie's fall.

Dr Schwartz would probably claim that he has better things to do than read about the methodology of the rankings. I would entirely sympathise with him although perhaps he should be more careful when writing about them or hire an assistant that would read them carefully.

Reactions to the Rankings

There is a very interesting article by Moshidi Sirat at GlobalHigherEd. He notes that reactions to the latest THES-QS rankings in the UK have varied widely. There is a lot of scepticism there but many universities are developing explicit strategies to boost their performance, with the aim of recruiting more international students. In Australia, there has been much debate, especially among universities that did not do so well. Brazilian universities do not seem to have shown much interest.

Sirat also notes:

A colleague in France noted that the manner Malaysia, especially the Malaysian Cabinet of Ministers and the Parliament, reacted to Times Higher rankings is relatively harsh. It appears that, in the specific case of Malaysia, the ranking outcome is being used by politicians to ‘flog’ senior officials governing higher education systems and/or universities. And yet critiques of such ranking schemes and their methodologies (e.g., via numerous discussions in Malaysia, or via the OECD or University Ranking Watch) go unnoticed. Malaysia better watch out, as the world is indeed watching us.

In a little while I hope to comment on the relative performance of Malaysian universities over the last few years. Reality is very different from the alleged ongoing decline presented by THES-QS.

Something about the QS "Peer Review"

QS's topuniversities site (registration required) has some information about the 2007 survey of academic opinion, which they insist on calling a peer review and which carries a 40% weighting in the THES-QS world university rankings.


First, there is list of the subject areas and the number of respondents in 2007:

All areas 43
Arts and Humanities 312
Engineering and IT 810
Life Sciences and Biomedicine 339
Natural Sciences 776
Social Sciences 715

There is also a section that cross-references the respondents' chosen geographical region of expertise and their subject area.

An interesting item is the current location of the respondents:


United States
307
Italy
174
United Kingdom
171
New Zealand
125
Canada
123
Australia
108
India
106
Malaysia
99
Germany
92
Belgium
79
Singapore
76
France
74
Spain
70
Japan
58
Hong Kong
57
Philippines
56
Sweden
52
China
50
Ireland
47
Switzerland
46
Austria
41
Denmark
37
Indonesia
36
Brazil
33
Turkey
33
Portugal
27
Mexico
26
Poland
26
South Korea
25
Argentina
23
South Africa
23
Greece
22
Iran
22
Russia
21
Taiwan
21
Netherlands
19
Thailand
19
Finland
16
Other
586

Is it possible to keep a straight face while maintaining that this a representative sample of international academic opinion? More respondents from the UK and Italy combined than from the US. More from New Zealand than from Germany. Almost the same number from Hong Kong and Japan. More from Ireland than from Russia. More from Belgium than from France.

Manipulating the Rankings?

There is a very interesting post at Wouter on the Web. I thought it worth pasting all of it.

From the university newspaper of Groningen we get some interesting insights in the way Groningen University has optimized their data for submission to the THES rankings. Deemed not to be important, the rector nevertheless wanted Groningen University to score better in the THES-QS rankings. For the rector, the first notation in the top 200 of the THES rankings, 173 to be exactly, was a good reason to celebrate with his subordinates.

What did they do? They concentrated on the questions of the most favourable number of students. The number of PhD students was a number they could play with. In the Netherlands PhD students are most often employed as faculty, albeit they are students as well to international standards. They contemplated on the position of the researchers in the University hospital. This would increase the number of staff considerably and thus lower the student/faculty ratio, but on the other hand this could have an important effect on the number of citations per research staff as well. Increases in staff number will lower the citations per staff. Which is detrimental to the overall performance. However, if they only could guarantee that citations to hospital staff were included in the citation counts as well?

So in Groningen they have exercised through some scenarios of number of students, number of staff, student/staff ratio and citations/staff ratio to arrive at the best combination to enhance their performance. I really do wonder if the contact between Groningen and QS -the consultants establishing the rankings- did also lead to the improvement of the search for citations by including the University Hospital for the university results. It is known from research by CWTS that searches for papers from all parts of the university are notoriously difficult. Especially to include the papers produced by staff from the teaching hospitals. In Groningen they have the feeling that it helped what they did in their contacts with QS. Well, at least it resulted in a nice picture on their university profile page.

Optimization or manipulation? It is only a thin line. If you only could make sure that all staff of your university would use the proper name of the institution in the authors affiliation. The university would gain a lot.

Chris Patten, Oxford and the Rankings

On 21st November, the London Spectator had an article by Chris Patten, former governor of Hong Kong, that referred to the THES-QS international university rankings and the high place that they give to Oxford and Cambridge.

In the last month, another respected international survey
placed Oxford and Cambridge joint second to Harvard in the league table of
world-class universities. This confirms what others have suggested in recent
years. Moreover, other British universities — most notably London’s Imperial
College and University College — came out high on the list. There are, alas, too
few areas of our national life — the armed forces, the City of London, our
diplomatic service — where we do as well in global comparisons. And it
matters.

Patten suggests that the strength of Oxford and Cambridge lies in the balance between the colleges and the universities and that this is reflected in their performance in the rankings.
Patten is not the first to comment on the apparently excellent performance of British universities, Oxford and Cambridge especially, compared to other national institutions, not least the increasingly pathetic football team. There is a slight touch of desperation here. Even if we can’t beat Croatia or Macedonia, at least Cambridge and Oxford can still run rings around the Universities of Zagreb or Skopje or even Berkeley or Johns Hopkins. Nor is he the first to refer to the THES-QS rankings when commenting on the question of reorganising major British universities. The rankings have, for example, been used to bolster Imperial College London’s claim to become fully independent of the University of London.

But there are some very dubious claims here. I am not sure what Patten is referring to when he talks about another respected international survey. It is certainly not the Shanghai rankings which have Oxford in tenth place and Cambridge in second overall by virtue of long dead Nobel laureates and much lower down by more contemporary criteria.

As for being respected, while the THES-QS rankings are avidly followed in Australia and Southeast and East Asia and routinely used in advertising by British universities, they are usually ignored or politely dismissed by American schools. Washington University in St Louis has apparently not even noticed that QS think that they have done almost no research at all over the last few years.

In fact, even QS does not provide much evidence that Oxford is a world-beater. In 2006 it did extremely well on the peer review and very well on the recruiter review but posted a mediocre performance on everything else, especially research as measured by citations per faculty, 63rd , behind the Hebrew University of Jerusalem, the Tokyo Institute of Technology and the University of Naples 2.

In fact, it isn’t that Oxford is that bad at research but that QS apparently inflated the numbers of faculty to get a good faculty student ratio at the cost of an unrealistically bad score for research. Still, it looks as though for research Oxford is now trailing around the middle to bottom of the Ivy League.

And this year? QS has introduced a new scoring system that in effect compresses scores at the top. So, Oxford did well for nearly everything with 100 for peer review, recruiter review, and student faculty ratio, 97 for international faculty and 96 for international students. This does not mean very much. The better universities now get high marks for just about everything. So does Oxford but again, according to QS, it lags behind on citations per faculty in 85th place behind Colorado State University, Showa University and Georgia Institute of Technology
Again, the problem probably is not that Oxford researchers are doing little research or not getting cited enough but that QS is using an inflated faculty figure.

Still it seems clear that Oxford’s position in the rankings is derived from a dubious “peer review” and from a scoring system that blurs differences at the top of the scale. It is not a result of measured research excellence. The THES-QS rankings are simply covering up the relative decline of Oxford and Cambridge.

Aston Business School

And what is Aston Business School doing in the THES-QS rankings at 266th place?

The THES-QS Rankings: Citations per Faculty

This year there have been two main changes in this section of the World University Rankings. First, a new database has been used. Second, as in the other sections, scores have been converted into Z scores.

The use of the Scopus database, which is run by the Dutch-based publishing company, Elsevier, is questionable. QS correctly state that, with over 15,000 journals and many other sources, it is generally more inclusive than the ESI database, which was used in previous years. A more comprehensive database is, however, not necessarily a better one if the objective is to evaluate quality as well as quantity of research. The Scopus database includes 785 conference proceedings and 703 trade journals out of 25, 483 titles. Such items are likely to be subject to a much less rigorous process of review or perhaps to none at all. Furthermore, 7,972 of the titles are listed as inactive.

It is possible therefore that the shift to Scopus means that a lot of mediocre or inferior research is being counted. Whether this is desirable in a measure of quality is debatable.

The most obvious feature of the Scopus database is its geographical bias. Here are the number of titles from selected countries:

US 8,090
UK 4,968
Netherlands 2,184
Germany 1,878
Japan 1,174
France 748
Australia 667
Canada 576
Switzerland 491
Russia 429
Korea 208
Belgium 39
Singapore 121
Taiwan 119
Hong Kong 59

In relation to population, number of universities, output of research, quality of research or almost anything else the UK appears overrepresented in relation to the USA. There are also, perhaps not surprisingly, many more journals from the Netherlands than from Belgium or countries with a similar population.

The citations per faculty section is now as biased towards the UK as the “peer review”. With a forty % weighting given to the "peer review", in which in 2006 UK respondents alone were 71% of those from the US and 20 % towards a citations count in , which UK items alone are 61 % of those from the USA, it is difficult to avoid the conclusion that this is a blatant exercise in academic gerrymandering.

What is more, this measure seems to have little validity. Looking at the rankings in relation to the other criteria, we find that the correlations are very low and usually insignificant.

“Peer review” .260
Employer review - .008
Faculty student ratio .088
International faculty .018
International students .039

The only significant correlation, a slight one, is with the "peer review". There is then no association between the university’s performance on this criterion and four of the five others. In 2006, when the ESI database was used the correlations were much stronger:

"Peer review” .480
Employer review . 348
Faculty student ratio .135
International faculty –045
International students .094

There is also a very modest correlation of .467 between the citations per faculty in 2006 and in 2007 (among the 174 universities that were in the top 200 in both years). It seems that Alejandro Pisanty is quite correct when he says that these look like two completely different sets of data.

“The canvassing of publications and citations seems to bring results which are so different, using Scopus instead of the Thomson/ISI products of the last three years, and the changes are in such way non-uniform among institutions, that it seems appropriate to consider the new version really a new ranking. There will hardly be any comparability with the previous years.”

Furthermore, there are some entries here that look a bit strange. Is the University of Alabama really the fifth best university in the world on this measure, Pohang University of Science the 12th, Renssalaer Polytechnic Institute the 36th? And do leading British universities really deserve to be so low, with Cambridge in 80 th place and Imperial College in 86th? Certainly, they are grossly overrated in the “peer review” but are they as bad at research as this data suggests?

There are also dramatic and suspicious changes from 2006. Cambridge is down from 47th place to 80th, National University of Singapore up from 160th to 74th, Kuopio (Finland) down from 14th to 70th, Tokyo Institute of Technology up from 58th to29th.

So, we now have a database that emphasises quantity rather than quality, which has an even more pronounced pro-British and anti-American bias and which is noticeably lacking in validity.
I will conclude by returning to the extraordinarily poor performance of Cambridge, Oxford and Imperial on this criterion, below previous years and well below their performance on the more contemporary parts of the Shanghai Jiao Tong rankings.

I wonder whether at least a part of the answer can be found in the faculty part of the equation. Is it possible that faculty numbers in British universities have been inflated to give a high score for student faculty ratio at the price, probably very acceptable, of driving down the score for citations per faculty?

What has Really Happened to Malaysian universities?

Three years ago the administration at Universiti Malaya (UM) was celebrating getting into the world's top 100 universities according to the THES-QS rankings. A year later it turned out that it was just the result of one of many errors by QS Quacquarelli Symonds, the consultants who produced the rankings.

Now it looks as though the same thing is happening all over again, but in the opposite direction.

This year four Malaysian universities have fallen in the rankings. UM and Universiti Kebangsaan Malaysia (UKM) went right out of the top 200.

Commentators have listed several factors responsible for the apparent slide and proposed remedies. Tony Pua at Education in Malaysia says that

our universities are not competitive, are not rigourous in nature, do not promote and encourage merit and the total lack of transparency in admission and recruitment exercises served the perfect recipe for continual decline in global recognition and quality

According to the Malaysian opposition leader, Lim Kit Siang

JUST as Vice Chancellors must be held responsible for the poor rankings of their universities, the Higher Education Minister, Datuk Mustapha Mohamad must bear personal responsibility for the dismal international ranking of Malaysian universities - particularly for Malaysia falling completely out of the list of the world’s Top 200 Universities this year in the 2007 Times Higher Education Supplement (THES)-Quacquarelli Symonds (QS) World University Rankings.

An article in the Singapore Straits Times reported that

eminent academician Khoo Kay Kim felt there was too much emphasis on increasing the number of PhD holders, instead of producing quality doctorate graduates. 'If this goes on, next year I expect the rankings to slip further,' he said.

Everyone seems to assume that the decline in the rankings reflects a real decline or at least a lack of quality that the rankings have finally exposed.

But is this in fact the case ?

To understand what really happened it is necessary to look at the methodological changes that have been introduced this year.

QS`have done four things. They have stopped respondents to their "peer review" selecting their own institutions. They are using full time equivalent (FTE) numbers for staff and students instead of counting heads. They now use the Scopus data base instead of ISI. They use Z scores which means that the mean of all scores is subtracted from the raw score. The result is divided by the standard deviation. Then the resulting figures are normalised with the mean score converted to 50.

The prohibition on voting for one's own university would seem like a good idea if was also extended to voting for one's alma mater. I suspect that Oxford and Cambridge are getting and will continue to get many votes from their graduates in Asia and Australia, which would seem just as bad as picking one's current employer.

Using FTEs is not a bad idea in principle. But note that QS` are still apparently counting research staff as faculty, giving a large and undeserved boost to places like Imperial College London.

I am sceptical about the shift to Scopus . This database includes a lot of conference papers, reviews and so on and therefore not all of the items included would have been subject to rigorous peer review It therefore might include research that is a a lower quality than that in the ISI database. There are some also strange things the citations section this year . The fifth best university for citations is the University of Alabama. (You have to look at the details for the top 400 to see this because it is not in the overall top 200.) According to QS's data, theEcole Normale Superieure in Paris is better than Harvard. Oxford is down at number 85, which seems a bit too low. It could be that the database is measuring quantity more than quality or perhaps there have been s number of errors.

Using Z scores is a standard practice among other rankers but it does cause huge fluctuations when introduced for the first time. What Z scores do is, in effect, to compress scores at the top and stretch them out lower down the rankings. They make it easier to distinguish among the mediocre universities at the price of blurring differences among the better ones.

So how did Malaysian universities do in 2007?

There is no point in looking at the scores for the various criteria. The introduction of Z scores means that scores in 2006 and 2007 cannot be compared. What we have to do is to work out the relative position of the universities on each component.

[Two days ago the scores for the six components of the rankings were available (registration required) at the QS topuniversities site for the top 400 universities. They could not be accessed today, which might mean that the details for the 400-500 univerities are being prepared or, just possibly, that errors are being corrected.]

Peer review
In 2006 UM was 90th for peer review among the top 400 in that year. In 2007 it fell to 131st position among the top 400.

Recruiter rating
In 2006 it was 238th for recruiter rating . In 2007 it rose to159th place.

Student faculty ratio
In 2006 it was in 274th place for student faculty ratio. In 2007 it rose to 261st place.

International faculty
In 2006 it was 245th for international faculty. In 2007 it rose to 146th place.

International Staff
In 2006 it was 308th for international students. In 2007 it rose to 241st place.

International students
In 2007 it was 342nd for citations per faculty . In 2007 it fell to 377th place.


This means that UM did much better compared to other universities on the following measures:

• Recruiter rating

• Student faculty ratio

• International students

• International faculty

It did somewhat worse on two items , peer review and citations. But notice that the number of places by which it fell are much less than than the number of places by which it rose, except for student faculty ratio.

The peer review was given a weighting of forty per cent and this meant that the modest fall here cancelled out the greater rises on the other sections.

It was, however, the citations part that scuppered UM this year. Without this, it would have remained roughly where it was. Basically falling from position 342 to 377 meant losing a bit more than 30 points on this section or about six points on the total score, sufficient to eject UM from the top two hundred.

Why should such a modest fall have such dire consequences?

Basically what happened is that Z scores , as noted earlier, compress scores at the top and stretch them out over the middle . Since the mean for the whole group is normalised at fifty and since the maximum score is hundred, an institution like Caltech will never get more than twice as many points as a university scoring around the mean, even if it were, as in fact it does, to produce ten times as much research.

So, in 2006 Caltech scored 100, Harvard 55 , the National University of Singapore (NUS) 8, Peking University 2 and UM 1.

Now in 2007, Caltech gets 100, Harvard 96, NUS 84, Peking 53 and UM 14.

The scores have been stretched out at the bottom and compressed at the top. But there has almost certainly been no change in the underlying reality.

So what is the real position? UM, it seems, has, relative to other universities, recruited more international staff and admitted more international students. Its faculty student ratio has improved very slightly relative to other universities. The employers contacted by QS think more highly of its graduates this year.

This was all cancelled out by the fall in the "peer review", which may in part have been caused by the prohibition on voting for the respondent's own institution.

The real killer for UM, however, was the introduction of Z scores I'll leave it to readers to decide whether a procedure that represents Caltech as only slightly better than Pohong University of Science and the University of Helsinki is superior to one that gives Peking only twice as many points as UM.

The pattern for the other Malaysian universities was similar, although less pronounced. It is also unfortunately noticeable that UKM got a very high score for international faculty, suggesting that an error similar to that of 2004 has occurred.

What is the real situation with regard to Malaysian universities? Frankly, I consider the peer review a dubious exercise and the components relating to student faculty ratio and internationalisation little better.

Counts of research and citations produced by third parties are, however, fairly reliable. Looking at the Scopus database (don't trust me -- get a 30 day free trial)l I found that 1,226 research papers (defined broadly to include things like reviews and conference papers) by researchers affiliated to Malaysian universities and other institutions were published in 2001 and 3,372 in 2006. This is an increase of 175% over 5 years.

For Singapore the figures are 5, 274 and 9,630, an increase of 83%.

For Indonesia they are 511 and 958, an increase of 87%.

For Australia they are 25,939 and 38, 852, an increase of 56%.

For Japan they are 89, 264 and 103, 428, an increase of 16%.

It is of course easy to grow rapidly when you start from a low base and these statistics say nothing about the quality of the research. Nor do they distinguish between a conference paper with a graduate student as sixth author and an article in a leading indexed journal.

Still the picture is clear. The amount of research done in Malaysia has increased rapidly over the last few years and has increased more rapidly than in Singapore, Japan and Australia. Maybe Malaysia is not improving fast enough but it is clear that there has been no decline, either relative or absolute, and that the THES-QS rankings have, once again, given a false impression of what has happened.






I

Congratulations to Malaysian Universities

Note: corrections have been made to an earlier draft. Some of the figures for 2006 have been changed.

No, I am not being sarcastic. Information just released by QS, the consultants who prepare the data for the THES-QS rankings, shows that Malaysian universities have improved quite significantly in some respects over the last year.

QS have now published detailed information on the top 400 universities in the 2007 rankings. This confirms what I had suspected, namely that there has been no real decline in the quality of any Malaysian university and that the apparent fall in the positions of Universiti Malaya (UM), Universiti Kebangsaan Malaysia (UKM), Universiti Sains Malaysia (USM) and Universiti Putra Malaysia (UPM) is largely the result of nothing more than a change in methodology.

There is no point in comparing the scores in 2006 and 207 for the various components because of changes in methods this year. Basically, the introduction of Z scores means that any such comparison has no meaning. Last year, for example, UM was given a score of 1 for citations and 14 this year. That does not mean anything since the mean score among the top 400 universities for citations was 9 in 2006 and 66 in 2007. To measure genuine change it is necessary to look at the relative positions of the universities.

I have calculated the position of the Malaysian universities on the various criteria in 2006 and 2007 . In both years I have looked only at the top 400 since information for the universities below the 400th place in 2007 is not currently available. .

In 2006, University Malay was 90th for the "peer review", 238th for recruiter rating, 274th for student faculty ratio , 245th for international faculty, 308th for international students and 342th for citations per faculty. UM managed to get into the top 200 because the score for "peer review" was given a much larger weighting than any other criterion.

This year , UM was 131st for the "peer review", 159th for recruiter rating, 261st for student faculty ratio , 146th for international faculty, 241st for international students and 377th for citations per faculty.

Thus, if QS are to be believed, UM has improved its standing with local employers, recruited more teaching staff, and increased the numbers of international faculty and students, all relative to other universities . It did slightly worse on citations per faculty.

The only serious blemish was that UM did rather worse on the "peer review", perhaps because as QS has suggested, respondents were no longer allowed to vote for their own institutions.

So, how could UM suffer such a catastrophic fall?

The answer lies in the in the use of Z scores. To summarise a Z score is constructed when the population mean is subtracted from the raw score, divided by the standard deviation and then normalised,.

The effect of this is to squash scores together at the top and not at the bottom as was previously the case. To go back to the scores for citations, in 2006 UM got a score of 1 for citations, which was quite a bit below average. In 2007, because in the introduction of Z scores, the average was much higher. So UM got 1 for citations in 2006 and Peking ("Beijing" then) got 2. This year UM got 14 and Peking got 53. So Peking got an extra 39 points instead of one.

The switch from ISI to Scopus may also had had some effect but probably not all that much.

Similarly, we find that UPM improved its position for two criteria and USM and UKM for 3 each. All suffered a decline on the "peer review".

UM's fall in the "peer review" section did not make a dramatic difference. Had UM remained in 90th place it would have made a difference of only ten 10 points, 76 instead of 66, for that section.

UM 's supposed tumble happened solely because universities that are doing a bit more research are now getting a lot more points than before.

There has been no decline. Maybe Malaysian universities are not improving fast enough but that is quite a different thing from what the rankings appear to show and what is causing so much anxiety among Malaysian commentators.

A bit more on Imperial College London

• Imperial College Press is a joint venture of Imperial College and World Scientific.
• World Scientific is a Singapore-based publishing company whose subscription list is used by QS to construct their "peer review".
• Imperial gets a perfect score of 100 (rounded) for the "peer review" in the 2007 THES-QS rankings.
• Until last year, citation data were collected for QS by Evidence Ltd, a company headed by a former Imperial faculty member.
• QS gave Imperial a much better student faculty ratio than even the college itself claimed.
• Imperial is, according to the THES-QS rankings, the fifth best university in the world.
• Richard Sykes, Vice-chancellor of Imperial, is the second highest paid in the UK.
• Richard Sykes wants a massive increase in fees.
• Imperial is now the most popular UK destination for Singapore students.
• Richard Sykes is on QS's questionnaire telling respondents that it takes smart people to recognise smart people.

Is it conceivable that some of these might just possibly have something to do with one another?

The Imperial Ascendency

One of the most remarkable things about the THES-QS rankings is the steady rise of Imperial College London. It has now reached 5th place, just behind Oxford, Cambridge and Yale and ahead of Princeton, MIT, Stanford and Tokyo.

How did this happen? Imperial's research performance is rather lacklustre compared with many American universities. The Shanghai Jiao Tong index puts it at 23rd overall, 33rd for highly cited researchers , 28th for publications in Science and Nature, and 29th for citations in the Science Citation Index.

Google Scholar also indicates that Imperial does much worse than many other places. A quick search comes up with 22,500 items for research published since 2002, compared to 22, 700 for Seoul National University, 25,800 for McGill, 44,00 for Tokyo and 151,00 for Princeton.

Imperial does well on the THES QS rankings partly because of outstanding scores on the peer review (99 out of 100) , employer review (99) and international students (100).

It also comes first (along with 15 others with scores of 100) for student faculty ratio. Is this justified?

On its web site QS indicates that Imperial has 2,963 full time equivalent (FTE) faculty and 12,025 FTE students, a ratio of 4.06.

However, if we look at Imperial's site we find that the college claims 12,129 FTE students and 1,114 academic and 1,856 research staff.

It appears that QS has counted both academic and research staff when calculating Imperial's ratio. Looking at other universities, it appears that it is QS's standard practice to count research staff who do not teach as part of the faculty total. In contrast, Imperial itself calculates the ratio by dividing students by academic staff to produce a ratio of 11.2. If that ratio had applied Imperial would have been many places lower.


If QS has been counting research staff in the total faculty score it would lead to the truly bizarre result that universities could hire a large number of researchers and get a substantial boost for the student faculty score.

So far it looks as though this is s general procedure and not a special privilege granted to Imperial alone but it would introduce a definite bias in favour of those universities that, like Imperial, employ large numbers of non-teaching researchers.

Update on Macquarie

The QS topuniversities site does not provide information about ranking components for universities outside the top 200. It does , however, provide links to pages with the raw data and their sources, for which they are to be highly commended.

It seems that this year Macquarie is recorded as having 1,018 full time equivalent faculty (865 headcount) and 255 full time equivalent international faculty (267 headcount). So, 25 % of Macquarie's faculty are international.

The information on total faculty was submitted by Baerbel Eckelmann on 8/10/07 and on international faculty by "Director" on 15/6/07.

The figure for 2006 was presumably much higher and incorrect. It would be interesting if Macquarie or QS could indicate how it was derived.

It would seem that one reason for the apparent decline of Macquarie was simply the correction of a previous error.

More From QS

The QS site topuniversities has lists of the top 400 and top 401 - 50 universities in the 2007 THES-QS rankings.

What happened to Macquarie?

There have been quite different reactions to the latest THES-QS rankings in the USA and Australia. It seems that nobody has noticed that Washington University in St. Louis fell from 48th place in 2006 to 161st this year. But there has been a great deal of discussion about why Macquarie University fell from 82nd to 168th.

It is difficult to figure out exactly what happened since the introduction of a new scoring method makes it difficult to compare 2006 and 2007 but it is possible to compare the relative positions in these years of a university for the components of the rankings.

In 2006 Macquarie was 93rd in the "peer review", 46th for recruiter rating, 198th for student faculty rating, 159th for citations per faculty, 1st for international faculty and 13th for international students.

In 2007 Macquarie was 142nd for the "peer review", 62nd for recruiter rating, 189th for student faculty ratio, 190th for citations per faculty, 55th for international faculty and 11th for international students.

It seems that the decline of Macquarie is due primarily to a poorer score for the "peer review", possibly because of a change in QS's methodology that meant that respondents could not select their own institutions, and to a dramatic fall from first place for numbers of international faculty.

It is impossible that the latter represents a real change over the year unless Macquarie has been expelling hundreds of international lecturers. Either QS used the wrong figure for 2006 and the correct one this year or they got it right in 2006 but made a mistake this year.

The administration at Macquarie ought to be able to answer these questions:

Did Macquarie provide QS with any information about international faculty in 2006 and in 2007? If so, what information was given and was it correct?

Malaysian Universities

Since THES has suggested that Malaysian and Singaporean universities suffered in this year's rankings because respondents were not allowed to vote for their own institutions and since this is obviously not true of the Singaporean universities, who got scores of 100 and 84 for the "peer review", I think it would be a good idea to wait a bit before making assumptions about the cause of the apparent decline of Malaysian universities. It is not totally impossible that QS has made another error or errors.

Another Kenan-Flagler: The Case of Washington University in St. Louis

Earlier this year Fortune published a ranking of business schools from which Kenan-Flagler at the University of North Carolina had been omitted. It turned out that QS Quacquarelli Symonds Ltd, who collected the data for the rankings, had confused the school with one at North Carolina State University.

Kenan-Flagler was justifiably outraged and protested, Fortune promptly removed the rankings from their site and Nunzio Quacquarelli, director of QS, was appropriately apologetic, saying that it was the fault of junior staff and that such a thing had never happened before and would never happen agian.

It appears that such a thing has happened again in the THES-QS 2007 World University Rankings. According to QS, Washington University in St Louis (WUSL) fell from 48th place to 161st. This happened, not because of Z-scores or anything like that, but because it was awarded a score of precisely 1 for citations per faculty, well below any other university in the top 200.

This is obviously completely wrong. WUSL is a major research university and is ranked 12th in the US by the US News and World Report for 2007.

A clue to what happened is that at the same time the University of Washington rose from 84th to 55th place in the rankings with a score for research of 92 , much higher than its score on any other component.

It looks as though QS got the two universities mixed up and attributed all the citations that should have gone to WUSL to the University of Washington.

It will be interesting to see what happens. Will WUSL protest or even notice? Will THES do the decent thing like Fortune and withdraw the 2007 rankings from their web site? Will Nunzio Quacquarelli blame his junior staff and promise that such a thing will never happen again?

The Power of Z

This year QS has introduced several "methodological enhancements" into the THES-QS rankings. One is the use of Z-scores. Basically, this means that the mean for all universities is deducted from the raw score and the result is then divided by the standard deviation. In effect, the score represents not an absolute number but how far each university is from the average. One consequence of using Z-scores is that differences at the very top are reduced.

In principle this is not a bad idea and other rankers do it but it has produced some odd results in this case.

In the survey of academic opinion, for example, the following universities all get a maximum score of 100: Harvard, Cambridge, Oxford, Yale, Caltech, MIT, Columbia, McGill, Australian National University, Stanford, Cornell, Berkeley, Melbourne, British Columbia, National University of Singapore, Peking and Toronto.

Do THES and QS really expect us to believe that Melbourne, British Columbia and Peking are just as good at research as Harvard ? Especially since Harvard is far ahead on every one of the subject rankings?

THES has a headline about fine tuning revealing distinctions. Really?

What happened to the National University of Singapore?

The National University of Singapore is among the best in Asia and has always been ranked highly by THES-QS. This year, however, it has fallen from 19th to 33rd.

THES suggest that Malaysian and Singaporean universities have suffered because the "peer review" no longer allows respondents to pick their own institutions. This would, not however, seem to apply to NUS -- and I wonder whether it applies to Malaysian universities either -- which got the maximum score of 100 (along with Oxford, Harvard and Caltech) on the survey. What happened was that NUS scored very poorly on the faculty student section.

It got 100 for "peer review", international faculty and international students, 93 for recruiter review, 84 for citations per faculty and 34 for faculty student ratio.

NUS has a self-reported ratio of about 17 students per faculty. Peking reports about 10 but QS gives it a score of 98, almost the same as Caltech at 100 with a well known ratio of about three.

There is something about this that needs some explanation.

THES-QS`Rankings

Full details of the top 200 are available here (subscription required but a free trial is available).

The top 100 subject rankings are available here (registration required)

Malaysian Universities in Free Fall?

The Kuala Lumpur New Straits Times has a report on the performance of Malaysian universities on the latest THES-QS rankings

Malaysian universities are on a slippery slope. None of them made it to the top 200 placing in the Times Higher Education Supplement (THES)-Quacquarelli Symonds (QS) World University Rankings this year.

This poor showing comes on the back of a recent government survey of local public universities which found that none deserved a place in the outstanding category. Last year, Universiti Kebangsaan Malaysia and Universiti Malaya made it to the top 200 in the THES-QS rankings. UKM ranked 185th, up from the 289th spot in 2005, beating well-known universities like University of Minnesota in the United States and University of Reading, Britain. This year, it has fallen to 309th. Similarly UM, which was ranked among the world's top 100 universities three years ago, was in 169th position in 2005 and tied with University of Reading in the 192nd spot last
year. It has dropped to the 246th spot. Universiti Sains Malaysia has fallen
to 307 from 277 last year.

UKM and UM vice-chancellors attributed their fall to the new methodology used to calculate rankings this year."Even the National University of Singapore (NUS) has dropped to the 33rd spot when it was always within the top 10," Universiti Malaya vice-chancellor Datuk Rafiah Salim said."The way I look at it, smaller countries like Malaysia are bound to lose out as THES has introduced new criteria which is peer review and has changed the citation and list of publications."Rafiah said with more
than 3,000 universities getting ranked by THES annually, Malaysian
universities had to improve if they wanted to remain on top of the list."If
we want to compete with some of the top universities in the world, first we
have to be in the same league. "Right now, we are not. One way to overcome
that is through adequate funding."She said NUS received an annual funding of
S$1.2 billion (RM2.7 billion) a year compared to UM's RM400 million annual
budget.

There is no mention of Universiti Putra Malaysia or Universiti Teknologi Malaysia both of which were on the list of universities sent out by QS this year.

It is impossible to be sure until the full data is released but I suspect that the "decline" of Malaysian universities has nothing to do with any real change but with QS preventing survey respondents from voting for their own institutions this year.

THES-QS Top 200 List Available

The THES-QS Top 200 universities list is available here.

There is a press release here.

Some Highlights

The two Malaysian universities, UM and USM, are out of the top 200. Most probably this is because of new procedures for the "peer review".

Berkeley, National University of Singapore, Peking (well done QS for getting the name right), and LSE have fallen dramatically.

The IITs and IIMs are out of the top 200, maybe out of the rankings altogether.

Two Brazilian universities have risen dramatically.

Changes such as these could not possibly result from real changes but are most likely the consequence of "methodological enhancements", errors or the correction of errors.

Dramatic Fall by Macquarie

The Sydney Morning Herald Herald reports on the THES-QS`rankings. Macquarie has fallen from 82 to 168.

AUSTRALIAN universities have slipped in one of the most respected world rankings. The most dramatic drop was suffered by Macquarie University - jeopardising a $100,000 bonus for its vice-chancellor, Steven Schwartz.

His bonus depends on improving Macquarie University's ranking in the Australian sector, but it has plummeted from 82 to 168 in the Times Higher Education Supplement's annual survey, released in Britain overnight. It has dropped from seventh to ninth among local universities.

Professor Schwartz, an American academic who had previously been head of Brunel University in Britain, replaced Di Yerbury in a messy coup last year. There was a bitter dispute between Macquarie and Professor Yerbury over ownership of paintings and other material she had accumulated over 19 years.

We will have to wait until the online results are available but the fall of Macquarie and perhaps of Steven Schwartz may have something to do with a reported change in the percentage of international faculty or possibly the introduction of z scores in the rankings. Last year Macquarie held top place for international faculty but QS did not reveal how they got the information and Macquarie did not confirm what the correct number was. Given the money at stake, it would not be totally astonishing if the 2006 figure for international faculty had been massaged a little bit.

Report from India

The Economic Times of India has a report on the THES-QS`rankings:

Three Latin American universities make it to the world’s top 200, while even Africa makes a debut, with Cape Town ranked at 200. IIMs and IITs are not universities.

According to Martin Ince, who compiles and edits the survey, “The 2007 THES-QS World University Rankings are the most rigorous and complete so far. They show that the US and the UK model of independent universities supported with significant state funding produces great results.”

UK universities are closing in on their American counterparts, with University College, London, making it into the top 10 for the first time, and Imperial College, London, moving up from 9th to 5th this year. Chicago too, is a first time entrant into the top 10.

While the top 10 list is still restricted to the US and the UK universities, in the top 50, the addition to the Netherlands, 12 countries are featured in the top 50 compared to 11 in 2006.

Universities of Tokyo, Hong Kong, Kyoto, National University of Singapore, Peking, Chinese University of Hong Kong, Tsinghua and Osaka lead Asian higher education, all featuring in the top 50. The top 100 sees the number of Asian universities increase to 13 (12 in 2006), while the number of European Universities has dropped to 35 (41 in 2006).

North America strengthened its tally to 43 Universities (37 in 2006). McGill tops in Canada, and a number of universities from New Zealand and Australia have also joined the top 50 list.

The increasing trend in internationalisation is also borne out by the fact that 143 of the top 200 universities reported an increase in their percentage of international faculty to total faculty, while 137 of the top 200 universities reported an increase in their percentage of international students to total students.

The last comment is rather interesting. Is this genuine internationalisation or simply a manipulation of data provided by universities?

Reports on the 2007 THES-QS World University Rankings

Education Guardian

BBC

Chronicle of Higher Education

Preview of the Top One Hundred

Beerkens Blog has the top 100. Here are the top 20.

Rank	Name	Country
1	HARVARD University	United States
2=	University of CAMBRIDGE	United Kingdom
2=	YALE University	United States
2=	University of OXFORD	United Kingdom
5	Imperial College LONDON	United Kingdom
6	PRINCETON University	United States
7=	CALIFORNIA Institute of Technology (Caltech)	United States
7=	University of CHICAGO	United States
9	UCL (University College LONDON)	United Kingdom
10	MASSACHUSETTS Institute of Technology (MIT)	United States
11	COLUMBIA University	United States
12	MCGILL University	Canada
13	DUKE University	United States
14	University of PENNSYLVANIA	United States
15	JOHNS HOPKINS University	United States
16	AUSTRALIAN National University	Australia
17	University of TOKYO	Japan
18	University of HONG KONG	Hong Kong
19	STANFORD University	United States
20=	CORNELL University	United States
20=	CARNEGIE MELLON University	United States