Oxford and Cambridge and THES

George Best used to tell a story about being asked by a waiter in a five star hotel "where did it all go wrong?" Best, who was signing a bill for champagne, with his winnings from the casino scattered around the room and Miss World waiting for him, remarked "he must have seen something that I'd missed". It looks like The Times Higher Education Supplement (THES) has seen something about Oxford and Cambridge that everybody else has missed.

The THES world university rankings have proved to be extraordinarily influential. One example is criticism of the president of Yonsei University in Korea for his institution's poor performance on the rankings.

Another is the belief of Terence Kealey, Vice-Chancellor of the University of Buckingham, that since Oxford and Cambridge are the best universities in the world apart from Harvard, according to THES, they are in no need of reform. He argues that Oxford should reject proposals for administrative change since Oxford and Cambridge are the best run universities in the world.

Oxford's corporate madness

by Terence Kealey
THIS YEAR'S rankings of world
universities reveal that Oxford is one of the three best in the world. The other
two are Cambridge and Harvard.

It is obvious that Oxford and Cambridge are the best
managed universities in the world when you consider that Harvard has endowments
of $25 billion (many times more than Oxford or Cambridge's); that Princeton,
Yale and Stanford also have vast endowments; and that US universities can charge
huge fees which British universities are forbidden to do by law.

Kealey evidently has complete confidence in the reliability of the THES rankings and if they were indeed reliable then he would have a very good point. But if they are not then the rankings would have done an immense disservice to British higher education by promoting a false sense of superiority leading to a rejection of attempts that might reverse a steady decline.

Let's have a look at the THES rankings. On most components the record of Oxford and Cambridge is undistinguished. For international faculty, international students, and faculty-student ratio they have scores of 54 and 58, 39 and 43, 61 and 64 respectively, compared to top scores of 100, although these scores are perhaps not very significant and are easily manipulated. More telling is the score for citations per faculty, a measure of the significance of the institutions' research output. Here, the record is rather miserable with Oxford and Cambridge coming behind many institutions including the Indian Institutes of Technology, Helsinki and the University of Massachusetts at Amherst.

I would be the first to admit that the latter measure has to be taken with a little bit of salt. Science and technology are more citation-heavy than the humanities and social sciences, which would help to explain why the Indian Institutes of Technology apparently do so well, but they are suggestive.

Of course, this measure also depends on the number of faculty as well as the number of citations. If there has been an error in counting the number of faculty then the citations per faculty score would also be affected. I am wondering whether something like that happened to the Indian Institutes. THES refers to institutes but their consultants, QS, refer to institute and provide a link to the institute in Delhi. Can we be confident that QS did not count the faculty for Delhi but citations for all the IITs?

When we look at the data provided by THES for citations per paper, a measure of research quality, we find that the record of Oxford and Cambridge is equally unremarkable. For Science, Oxford is 20th and Cambridge 19th. For technology, Oxford is 11th and Cambridge 29th. For biomedicine, Oxford is seventh and Cambridge ninth. For Social Sciences, Oxford is 19th and Cambridge is 22nd.

The comparative performance of Oxford and Cambridge is just as unimpressive when we look at the data provided by Shanghai Jiao Tong University. Cambridge is second on alumni and awards, getting credit for Nobel prizes awarded early in the last century but 15th for highly cited researchers, 6th for publications in Nature and Science and 12th for citations in the Science Citation Index and Social Science Citation Index. Oxford is ninth for awards, 20th for highly cited researchers , seventh for papers in Nature and Science and 13th for citations in the SCI and SSCI.

So how did Oxford and Cambridge do so well on the overall THES rankings? It was solely because of the peer review. Even on the recruiter ratings they were only 8th and 6th. On the peer review, Cambridge was first and Oxford second. How is this possible? How can reviewers give such a high rating to universities that produce research that in most fields is inferior in quality to that of a dozen or more US universities, that now produce relatively few Nobel prize winners or citations or papers in leading journals.

Perhaps like the waiter in the George Best the THES reviewers have seen something that everybody else has missed.

Or is it simply a product of poor research design? I suspect that QS sent out a disproportionate number of surveys to European researchers and also to those in East Asia and Australia. We know that respondents were invited to pick universities in geographical areas with which they were familiar. This in itself is enough to render the peer review invalid as a survey of international academic opinion even if we could be sure that an appropriate selection procedure was used.

It is surely time for THES to provide more information about how the peer review was conducted.

Is Korea University's Rise the Result of a THES Error?

It looks as though the Times Higher Education Supplement (THES) world university rankings will soon claim another victim. The president of Yonsei University, Republic of Korea, Jung Chang-young, has been criticised by his faculty. According to the Korea Times:

The school has been downgraded on recent college evaluation charts, and Jung has
been held responsible for the downturn.

Associations of professors and alumni, as well as many students, are questioning the president’s leadership. Jung’s poor results on the survey were highlighted by the school’s rival, Korea University, steadily climbing the global education ranks.
When Jung took the position in 2004, he stressed the importance of the international competitiveness of the university. “Domestic college ranking is meaningless, and
I will foster the school as a world-renowned university,” he said during his
inauguration speech.

However, the school has moved in the other direction. Yonsei University ranked behind Korea University in this year’s JoongAng Daily September college evaluation for the first time since 1994. While its rival university saw its global rank jump from 184th last year to 150th this year, Yonsei University failed to have its name listed among the world’s top 200 universities in the ranking by the London based The Times.

It is rather interesting that Yonsei university has been ahead of Korea University (KU) between 1995 and 2005 on a local evaluation while lagging far behind on the THES rankings in 2005 and 2006. In 2005 Korea was 184th in the THES rankings while Yonsei was 467th. This year Korea University was 150th and Yonsei 486th.

It is also strange that Yonsei does quite a bit a better than KU on most parts of the Shanghai Jiao Tong rankings. Both get zero for alumni and awards but Yonsei does better on highly cited researchers (7.7 and 0), articles in Nature and Science (8.7 and 1.5), and Science Citation Index (46.4 and 42.6) , while being slightly behind on size (16 and 16.6) Overall, Yonsei is in the 201-300 band and KU in the 301-400.

So why has KU done so well on the THES rankings while Yonsei is languishing almost at the bottom? It is not because of research. KU gets a score of precisely 1 in both 2005 and 2006 and, anyway, Yonsei does better for research on the Shanghai index. One obvious contribution to KU’s outstanding performance is the faculty- student ratio. KU had a score of 15 on this measure in 2005 and of 55 in 2006, when the top scoring university is supposedly Duke with a ratio of 3.48 .

According to QS Quacquarelli Symonds, the consultants who prepared the data for THES, Korea University has 4,407 faculty and 28,042 students, giving a ratio of 6.36.

There is something very odd about this. Just last month the president of KU said that his university had 28 students per faculty and was trying had to get the ratio down to 12 students per faculty. Didn’t he know that, according to THES and QS, KU had done that already?

President Euh also noted that in order for Korean universities to provide better
education and stand higher among the world universities' ranking, the
faculty-student ratio should improve from the current 1: 28 (in the case of
Korea University) to 1: 12, the level of other OECD member nations. He insisted
that in order for this to be realized, government support for overall higher
education should be increased from the current level of 0.5% of GNP to 1% of GNP
to be in line with other OECD nations.

It is very unlikely that the president of KU has made a mistake. The World of Learning 2003 indicates that KU had 21, 685 students and 627 full time teachers. That gives us a ratio of 1: 35. suggesting that KU has been making steady progress in this respect over the last few years.
How then did QS arrive at the remarkable ratio of 6.36? I could not find any data on the KU web site. The number of students on the QS site, however, seems reasonable, suggesting a substantial but plausible increase over the last few years but 4,407 faculty seems quite unrealistic. Where did this figure come from? Whatever the answer, it is quite clear that KU‘s faculty-student score is grossly inflated and so therefore is it’s total score. If Duke had a score of 100 and a ratio of 3.48 (see archives) then KU’s score for faculty-student ratio should have been, by my calculation, 12 and not 55. Therefore its overall score, after calibrating against top scoring Harvard’s, would have been 20.3 and not 32.2. This would have left KU well outside the top 200.

Incidentally, Yonsei’s faculty student ratio, according to QS and its own web site, is 34.25, quite close to KU's self-admitted ratio.

It appears that the criticism directed at Yonsei’s president is perhaps misplaced since KU’s position in the rankings is the result of a QS error. Without that error, Yonsei might have been ahead of KU or at least not too far behind.

The Best Universities for Biomedicine?

THES has published a list of the world's 100 best universities for biomedicine. This is based, like the other subject rankings, on peer review . Here are the top twenty according to the THES reviewers.

1. Cambridge
2. Harvard
3. Oxford
4. Imperial College London
5. Stanford
6. Johns Hopkins
7. Melbourne
8. Beijing (Peking)
9. National University of Singapore
10. Berkeley
11. Yale
12. Tokyo
13. MIT
14. University of California at San Diego
15. Edinburgh
16. University College London
17. Kyoto
18. Toronto
19. Monash
20. Sydney

Here are the top twenty according to citations per paper, a measure of the quality of research.


1. MIT
2. Caltech
3. Princeton
4. Berkeley
5. Stanford
6. Harvard
7. Oxford
8. University of California at San Diego
9. Cambridge
10. Yale
11. Washington (St Louis)
12. Johns Hopkins

13. ETH Zurich

14. Duke
15. Dundee
16. University of Washington
17. Chicago
18. Vanderbilt
19. Columbia
20. UCLA

The two lists are quite different. Here are the positions according to citations per paper of some of the universities that were in the top twenty for the peer review;

University College London -- 24
Edinburgh -- 25
Imperial College London -- 28
Tokyo -- 34
Toronto -- 35
Kyoto -- 36
Monash -- 52
Melbourne -- 58
Sydney -- 67
National University of Singapore -- 74
Beijing -- 78=

Again, there is a consistent pattern of British, Australian and East Asia universities doing dramatically better in the peer review than in citations per paper. How did they acquire such a remarkable reputation if their research was of such undistinguished quality? Did they acquire a reputation for producing a large quantity of mediocre research?

Notice that Cambridge with the top score for peer review produces research of a quality inferior to, according to QS's data, eight universities, seven of which are in the US and four in California.

There are also 23 universities that produced insufficient papers to be counted by the consultants. Thirteen are in Asia, 5 in Australia and New Zealand, 4 in Europe and one in the US. How did they acquire such a remarkable reputation while producing so little research? Was the little research they did of a high quality?

More on Methodology

Christopher Pandit has asked QS why the Universities of Essex and East Anglia and Royal Holloway are not included in the THES-QS top 500 universities. Check here and see if you are impressed by the answer.

I would like to ask if the State University of Stony Brook is at number 165 in the rankings how come the other three SUNY university centers at Albany, Binghamton and Buffalo cannot even get into the top 520.

THES and QS: Some Remarks on Methodology

The Times Higher Education Supplement (THES) has come out with their third international ranking of universities. The most important part is a peer review, with academics responding to a survey in which they asked to nominate the top universities in subject areas and geographical regions.

QS Quacquarelli Symonds, THES's consultants have published a brief description of how they did the peer review.

Here is what they have to say:

Peer Review: Over 190,000 academics were emailed a request to
complete our online survey this year. Over 1600 responded - contributing to our
response universe of 3,703 unique responses in the last three years. Previous
respondents are given the opportunity to update their response.

Respondents are asked to identify both their subject area of expertise
and their regional knowledge. They are then asked to select up to 30
institutions from their region(s) that they consider to be the best in their
area(s) of expertise. There are at present approximately 540 institutions in the
initial list. Responses are weighted by region to generate a peer review score
for each of our principal subject areas which are:
Arts &
Humanities
Engineering & It
Life Sciences & Biomedicine
Natural Sciences
Social Sciences
The five scores by subject area are
compiled into a single overall peer review score with an equal emphasis placed
on each of the five areas.

The claim that QS sent e-mails to 190,000 academics is unbelievable and the correct number is surely 1,900. I have queried QS about this but so far there has been no response.

If these numbers are correct then it means that QS have probably achieved the lowest response rate in survey research history. If they sent e-mails to 1,900 academics and added a couple of zeros by mistake, we have to ask how any more mistakes they have made. Anyway, it will be interesting to see how QS responds to my question, if indeed they ever do.

Combined with other snippets of information we can get some sort of picture of how QS proceeded with the peer review.

In 2004 they sent emails to academics containing a list of 300 universities, divided into subject and regional areas and 1,300 replied. Respondents were asked to pick up to 30 universities in the subjects and the geographical areas in which they felt they had expertise. They were allowed to add names to the lists.

in 2005, the 2004 reviewers were asked if they wanted to add to or subtract from their previous responses. Additional reviewers were sent e-mails so that the total was now 2,375.

In 2006 the 2004 and 2005 reviewers were asked whether they wanted to make changes. A further 1,900 (surely?) academics were sent forms and 1,600 returned them making a total (after presumably some of the old reviewers did not reply) of 3,703 reviewers. With additions made in previous years, QS now has a list of 520 institutions.

I would like to make three points. Firstly, it seems that a lot depends on getting on to the original list of 300 universities in 2004. Once on, it seems that universities are not removed. If not included, it is possible that a university might be very good but never quite good enough to get a "write-in vote". So how was the original list chosen?

Secondly, the subject areas in three case are different from those indicated by THES . QS has Natural Sciences, Engineering and IT, and Life Sciences and Biomedicine, THES has Science, Technology and Biomedicine. This is a bit sloppy and maybe indicative of communication problems between THES and QS.

Thirdly, it is obvious that the review is of research quality -- QS explicitly says so -- and not of other things as some people have assumed.