More on the Duke and Beijing Scandals

Sorry, there's nothing here about lacrosse players or exotic dancers. This is about how Duke supposedly has the best faculty-student ratio of any university in the world and how Beijing (Peking) university is supposedly the top university in Asia.

In previous posts I reported how Duke, Beijing and Ecole Polytechnique in Paris (see srchives) had apparently been overrated in the Times Higher Educational Supplement (THES) world university rankings because of errors in counting the number of faculty and students.

QS Quacquarelli Symonds, the consultants who conducted the collection of data for THES, have now provided links to data for each of the universities in the top 200 in the latest THES ranking.
Although some errors have been corrected, it seems that new ones have been committed.

First of all, this year Duke was supposed to be top for faculty-student ratio. The QS site gives a figure of 3,192 faculty and 11,106 students, that is a ratio of 3.48, which is roughly what I suspected it might be for this year. Second placed Yale, with 3,063 faculty and 11,441 students according to QS, had a ratio of 3.74 and Beijing (Peking University -- congratulations to QS for getting the name right this year even if THES did not), with 5,381 faculty and 26,912 students, a ratio of 5.01.

But are QS's figures accurate? First of all, looking at the Duke site, there are 13,088 students. So how did QS manage to reduce the number by nearly 2,000? No doubt, the site needs updating but universities do not lose nearly a sixth of their students in a year.

Next, the Duke site lists 1,595 tenure and tenure track faculty and 925 non-teaching faculty. Even counting the latter we are still far short of QS's 3,192.

If we count only teaching faculty the Duke faculty-student ration would be 8.21 students per faculty. Counting non-teaching faculty would produce a ratio of 5.20, still a long way behind Yale.
It is clear then from data provided by QS themselves that Duke should not be in first place in this part of the rankings. This means that all the data for this component are wrong since all universities are benchmarked against the top scorer in each category and, therefore, that all the overall scores are wrong. Probably not by very much, but QS does claim to be the best.

Where did the incorrect figures come from? Perhaps Duke gave QS a different set of figures from those on its web site. If so, this surely is deliberate deception. But I doubt if that is what happened for the Duke administration seems to have been as surprised as anyone by the THES rankings.

I am wondering if this has something to do with Duke in 2005 being just below Ecole Polytechnique Paris in the overall ranking, The Ecole was top scorer for the faculty-student component in 2005. Is it possible that the data for 2006 was entered into a form that also included the 2005 data and that the Ecole's 100 for 2005 was typed in for Duke for 2006? Is it possible then that the data for numbers of students and faculty were constructed to fit the score of 100 for Duke?

As for Beijing (Peking), QS this year provides a drastically reduced number of faculty and students, 5,381 and 26,912 respectively. But even these figures seem to be wrong. The Peking University site indicates 4,574 faculty. So where did the other 800 plus come from?

The number of students provided by QS is roughly equally to the number of undergraduates, master's and doctoral students listed on Peking University's site. It presumably excludes night school and correspondence students and international students. It could perhaps be argued that the first two groups should not be counted but this would be a valid argument only if the the university itself did not count them in the total number of students and if their teachers were not counted in the number of faculty. It still seems that the most accurate ratio would be about 10 students per faculty and that Beijing's overall position is much too high.

Finally, QS has now produced much more realistic data for the number of faculty at the Ecole Polytechnique Paris, Ecole Normale Superieure Paris and Ecole Polytechnique Federale Lausanne. Presumably, this year part-time staff were not counted.

The Best Universities for Technology?

The Times Higher Education Supplement (THES) have published a list of the supposed top 100 universities in the world in the field of technology. The list purports to be based on opinion of experts in the field. However, like the ranking for science, it cannot be considered valid. First, let us compare the top 20 universities according to peer review and then the top 20 according to the data provided by THES for citations per paper, a reasonable measure of the quality of research.

First, the peer review:

1. MIT
2. Berkeley
3. Indian Institutes of Technology (all of them)
4. Imperial College London
5. Stanford
6. Cambridge
7. Tokyo
8. National University of Singapore
9. Caltech
10. Carnegie-Mellon
11. Oxford
12. ETH Zurich
13. Delft University of Technology
14. Tsing Hua
15. Nanyang Technological University
16. Melbourne
17. Hong Kong University of science and Technology
18. Tokyo Institute of Technology
19. New South Wales
20. Beijing (Peking University)

Now, the top twenty ranked according to citations per paper:

1. Caltech
2. Harvard
3. Yale
4. Stanford
5. Berkeley
6. University of California at Santa Barbara
7. Princeton
8. Technical University of Denmark
9. University of California at San Diego
10. MIT
11. Oxford
12. University of Pennsylvania
13. Pennsylvania State University
14. Cornell
15. Johns Hopkins
16. Boston
17. Northwestern
18. Columbia
19. Washington (St. Louis)
20. Technion (Israel)

Notice that the Indian Institutes of Technology, Tokyo, National University of Singapore, Nanyang Technological University, Tsing Hua, Melbourne, New South Wales and Beijing are not ranked in the top 20 according to quality of published research. Admittedly, it is possible that in this field a substantial amount of research consists of unpublished reports for state organizations or private companies but this would surely be more likely to affect American rather than Asian or Australian universities.

Looking a bit more closely at some of the universities in the top twenty for technology according to the peer review, we find that, when ranked for citations per paper, Tokyo is in 59th place, National University of Singapore 70th, Tsing Hua 86th, Indian Institutes of Technology 88th, Melbourne 35th, New South Wales 71st, and Beijing 76th. Even Cambridge, sixth in the peer review, falls to 29th.

Again, there are a large number of institutions that did not even produce enough papers to be worth counting, raising the question of how they could be sufficiently well known for there to be peers to vote for them. This is the list:

Indian Institutes of Technology
Korean Advanced Institute of Science and Technology
Tokyo Institute of Technology
Auckland
Royal Institute of Technology Sweden
Indian Institutes of Management
Queensland University of Technology
Adelaide
Sydney Technological University
Chulalongkorn
RMIT
Fudan
Nanjing

Once again there is a very clear pattern of the peer review massively favoring Asian and Australasian universities. Once again, I can see no other explanation than an overrepresentation of these regions, and a somewhat less glaring one of Europe, in the survey of peers combined with questions that allow or encourage respondents to nominate universities from their own regions or countries.

It is also rather disturbing that once again Cambridge does so much better on the peer review than on citations. Is it possible that THES and QS are manipulating the peer review to create an artificial race for supremacy – “Best of British Closing in on Uncle Sam’s finest”. Would it be cynical to suspect that next year Cambridge and Harvard will be in a circulation-boosting race for the number one position?

According to citations per faculty Harvard was 4th for science, second for technology and 6th for biomedicine while Cambridge was 19th, 29th and 9th.

For the peer review, Cambridge was first for science, 6th for technology and first for biomedicine. Harvard was 4th, 23rd and second.

Overall, there is no significant relationship between the peer review and research quality as measured by citations per paper. The correlation between the two is .169, which is statistically insignificant. For the few Asian universities that produced enough research to be counted, the correlation is .009, effectively no better than chance.

At the risk of being boringly repetitive, it is becoming clearer and clearer that that the THES rankings, especially the peer review component, are devoid of validity.

The World’s Best Science Universities?

The Times Higher Education Supplement (THES) has now started to publish lists of the world’s top 100 universities in five disciplinary areas. The first to appear were those for science and technology.

THES publishes scores for its peer review by people described variously as “research-active academics” or just as “smart people” of the disciplinary areas along with the number of citations per paper. The ranking is, however, based solely on the peer review, although a careless reader might conclude that the citations were considered as well.

We should ask for a moment what a peer review, essentially a measure of a university’s reputation, can accomplish that an analysis of citations cannot. A citation is basically an indication that another researcher has found something of interest in a paper. The number of citations of a paper indicates how much interest a paper has aroused among the community of researchers. It coincides closely with the overall quality of research, although occasionally a paper may attract attention because there is something very wrong with it.

Citations then are a good measure of a university’s reputation for research. For one thing, votes are weighted. A researcher who publishes a great deal has more votes and his or her opinion will have more weight than someone who publishes nothing. There are abuses of course. Some researchers are rather too fond of citing themselves and journals have been known to ask authors to cite papers by other researchers whose work they have published but such practices do not make a substantial difference.

In providing the number of citations per paper as well as the score for peer review, THES and their consultants, QS Quacquarelli Symonds, have really blown their feet off. If the scores for peer review and the citations are radically different it almost certainly means that there is something wrong with the review. The scores are in fact very different and there is something very wrong with the review.

This post will review the THES rankings for science.

Here are the top twenty universities for the peer review in science:

1. Cambridge
2. Oxford
3. Berkeley
4. Harvard
5. MIT
6. Princeton
7. Stanford
8. Caltech
9. Imperial College, London
10. Tokyo
11. ETH Zurich
12. Beijing (Peking University)
13. Kyoto
14. Yale
15. Cornell
16. Australian National University
17. Ecole Normale Superieure, Paris
18. Chicago
19. Lomonosov Moscow State University
20. Toronto


And here are the top 20 universities ranked by citations per paper:


1. Caltech
2. Princeton
3. Chicago
4. Harvard
5. John Hopkins
6. Carnegie-Mellon
7. MIT
8. Berkeley
9. Stanford
10. Yale
11. University of California at Santa Barbara
12. University of Pennsylvania
13. Washington (Saint Louis?)
14. Columbia
15. Brown
16. University of California at San Diego
17. UCLA
18. Edinburgh
19. Cambridge
20. Oxford


The most obvious thing about the second list is that it is overwhelmingly dominated by American universities with the top 17 places going to the US. Cambridge and Oxford, first and second in the peer review, are 19th and 20th by this measure. Imperial College London. Beijing, Tokyo, Kyoto and the Australian National University are in the top 20 for peer review but not for citations.

Some of the differences are truly extraordinary. Beijing is 12th for peer review and 77th for citations, Kyoto13th and 57th, the Australian National University 16th and 35th Ecole Normale Superieure, Paris 17th and 37th, Lomsonov State University, Moscow 18th and 82nd National University of Singapore, 25th and 75th, Sydney 35th and 70th , Toronto 20th and 38th. Bear in mind that there are almost certainly several universities that were not in the peer review top 100 but have more citations per paper than some of these institutions.

It is no use saying that citations are biased against researchers who do not publish in English. For better or worse, English is the lingua franca of the natural sciences and technology and researchers and universities that do not publish extensively in English will simply not be noticed by other academics. Also, a bias towards English does not explain the comparatively poor performance by Sydney, ANU and the National University of Singapore and their high ranking on the peer review.

Furthermore, there are some places for which no citation score is given. Presumably, they did not produce enough papers to be even considered. But if they produce so few papers, how could they become so widely known that their peers would place them in the world’s top 100? These universities are:

Indian Institutes of Technology (all of them)
Monash
Auckland
Universiti Kebangsaan Malaysia
Fudan
Warwick
Tokyo Institute of Technology
Hong Kong University of Science and Technology
Hong Kong
St. Petersburg
Adelaide
Korean Advanced Institute of Science and Technology
New York University
King’s College London
Nanyang Technological University
Vienna Technical University
Trinity College Dublin
Universiti Malaya
Waterloo

These universities are overwhelmingly East Asian, Australian and European. None of them appear to be small, specialized universities that might produce a small amount of high quality research.

The peer review and citations per paper thus give a totally different picture. The first suggests that Asian and European universities are challenging those of the United States and that Oxford and Cambridge are the best in the world. The second indicates that the quality of research of American universities is still unchallenged, that the record of Oxford and Cambridge is undistinguished and that East Asian and Australian universities have a long way to go before being considered world class in any meaningful sense of the word.

A further indication of how different the two lists are can be found by calculating their correlation. Overall, the correlation is, as expected, weak (.390). For Asia-Pacific (.217) and for Europe (.341) it is even weaker and statistically insignificant. If we exclude Australia from the list of Asia-Pacific universities and just consider the remaining 25, there is almost no association at all between the two measures. The correlation is .099, for practical purposes no better than chance. Whatever criteria the peer reviewers used to pick Asian universities, quality of research could not have been among them.

So has the THES peer review found out something that is not apparent from other measures? Is it possible that academics around the world are aware of research programmes that have yet to produce large numbers of citations? This, frankly, is quite implausible since it would require that nascent research projects have an uncanny tendency to concentrate in Europe, East Asia and Australia.

There seems to be no other explanation for the overrepresentation of Europe, East Asia and Australia in the science top 100 than a combination of a sampling procedure that included a disproportionate number of respondents from these regions, allowing or encouraging respondents to nominate universities in their own regions or even countries and a disproportionate distribution of forms to certain countries within regions.

I am not sure whether this is the result of extreme methodological naivety, with THES and QS thinking that they are performing some sort of global affirmative action by rigging the vote in favour of East Asia and Europe or whether it is a cynical attempt to curry favour with those regions that are involved in the management education business or are in the forefront of globalization.

Whatever is going on, the peer review gives a very false picture of current research performance in science. If people are to apply for universities or accept jobs or award grants in the belief that Beijing is better at scientific research than Yale, ANU than Chicago, Lomonosov than UCLA, Tsinghua than Johns Hopkins then they are going to make bad decisions.

If this is unfair then there is no reason why THES or QS should not indicate the following:

The universities and institutions to which the peer review forms were sent.
The precise questions that were asked.
The number of nominations received by universities from outside their own regions and countries.
The response rate.
The criteria by which respondents were chosen.

Until THES and /or QS do this, we can only assume that the rankings are an example of how almost any result can be produced with the appropriate, or inappropriate, research design.

Why Beijing University is not the Best in Asia

According to the Times Higher Education Supplement's (THES) recent ranking of world universities, Beijing University ( the correct name is actually Peking University, but never mind) is the best university in Asia and 14th in the world.

Unfortunately, it is not. It is just another mistake by QS Quacquarelli Symonds, THES's consultants. Unless, of course, they have information that has been kept secret from everybody else.

In 2005, Beijing University was, according to THES, ranked 15th in the world. This was partly due to remarkably high scores for the peer review and the recruiter ratings. It also did quite well on the faculty/student section with a score of 26. In that year the top score on that part of the ranking was Ecole Polytechnique in Paris whose 100 score appears to represent a ratio of 1.3 students per faculty. It seems that QS derived this ratio from their datafile for the Ecole, although they also give other figures in another part of their page for this institution. Comparing the Ecole's score to others confirms that this was the data used by QS. It is also clear that for this measure QS was counting all students, not just undergraduates, although there is perhaps some inconsistency about the inclusion of non-teaching faculty. It seems then that, according to QS, Beijing University had a ratio of five students per faculty.

Here is the page from QS's web site with the 2005 data for Beijing University.

Datafile
Demographic
No. of faculty:
15,558
No. of international
faculty:
617
No. of students:
76,572
No. of international
students:
2,015
No. of undergraduates:
15,182
No. of
international
undergraduates:
1,025
No. of postgraduates:
13,763
No. of
international postgrads:
308
Financial
Average
undergrad course
fees:
USD$ 3,700
Average postgrad course fees:
USD$
4,700


Annual library spend:
USD$ 72,000
Source:
World University Research (QS & Times Higher Education Supplement)

Postgraduate Course List
For information on undergraduate
courses, please look out for thelaunch of TopUniversities.com in March
2006

Notice that it indicates that there are 76,572 students and 15, 558 faculty, which would give a ratio of 4.92, very close to 5. We can therefore safely assume that this is where QS got the faculty/student ratio.

But there is something wrong with the data. QS gives a total of 76,572 students but there are only 15,182 undergraduates and 13,763 postgraduates, a total of 28, 945. So where did the 46,000 plus students come from? When there is such a glaring discrepancy in a text it usually means that two different sources were used and were imperfectly synthesised. If we look at Beijing University's web site (it calls itself Peking University), we find this data.

Faculty
At present, Peking University has over 4,574 teachers, 2,691 of whom
are full or associate professors. Among the teachers are not only a number
of senior professors of high academic standing and world fame, but also a host
of creative young and middleaged experts who have been working at the forefront
of teaching and research

And this.

At present, Peking University has 46,074 students.
15,001
undergraduates8,119 master candidates3,956 doctoral candidates18,998 candidates
for a correspondence courses or study at the night school1,776 international
students from 62 countries and regions

QS's data were used for the 2005 ranking exercise. The information on Peking University's web site has no doubt been updated since then. However, it looks like QS obtained the numbers of undergraduates and post graduates from Peking University's site although they left out the 18,998 correspondence and night school students that the university counted.

According to the university's definition of students and teachers, the faculty student ration would be 10.07. Excluding correspondence and night school students but counting international students gives us a ratio of 6.31. The former ratio would probably be the correct one to use. THES's definition of a student is someone "studying towards degrees or substantial qualifications" and there is no indication that these students are studying for anything less. Therefore, it seems that the correct ratio for ratio for Beijing University should be around 10 students per faculty.

Looking at the reference work The World of Learning 2003 (2002) we find that Beijing University had 55,000 students and 4,537 teachers. Probably the data reported to this reference included several thousand students from research institutes or branch campuses or was simply an overstatement. The number of teachers is however, almost identical. But whatever the exact numbers, it is clear that QS made a serious mistake and this meant that the score for faculty/student ratio in 2005 was incorrect. Since it appears that a similar or identical ratio was used for this year's ranking as well, the ratio for 2006 is also wrong.

We still have the problem of where QS came up with the figure of 76,572 students and 15, 558 faculty on its web site. It did not come from Peking University.

Or maybe it did. This is from a brief history of Peking University on its site.

After the readjustment, Peking University became a university comprising
departments of both liberal Arts and Sciences and emphasizing the teaching and
research of basic sciences. By 1962, the total enrollment grew to 10,671
undergraduate students and 280 graduate students. Since 1949, Peking University
has trained for the country 73,000 undergraduates and specialty students,
10,000 postgraduates and 20,000 adult-education students, and many of them have become the backbones on all fronts in China.

There has evidently been a massive expansion in the number of postgraduate students recently. The figure of 73,000 undergraduates who ever completed studies at Peking University is close enough to QS's total of students to arouse suspicion that somebody may have interpreted the data for degrees awarded as that for current enrollment.

There is another possible source. There are several specialist universities in the Beijing area, which is one reason why it is rather silly of THES and QS to refer to Peking University as Beijing University. These include the Beijing Foreign Studies University, the Beijing University of Aeronautics and Astronautics, the Beijing University of Business and Technology and so on.

The sum total of students at these institutions, according to the World of Learning is 75,746 students and 12, 826 teachers. The first is very close to QS's figure and the latter somewhat so. A bit of double counting somewhere might have brought the number of teachers closer to that given by QS. I am inclined to suspect that the figures resulted from an enquiry that was interpreted as a request for information about the specialist Beijing universities.

So what about 2006? Wherever the numbers came from this much is clear. Using Yale as a benchmark for 2006 ( there are problems discussed already with top scoring Duke) it would appear that the ratio of 5 students per faculty was used in the latter year as well as in 2005. But according to the data on the university web site, the ratio should be around 10.

What this means is that Beijing University should have got a score for faculty/student ratio of 31 and not 69. I calculate that Beijing University's overall score, applying THES's weighting, dividing by Harvard's total score and then multiplying by 100, should be 57.3. This would put Beijing University in 28th position and not 14th. It would also mean that Beijing University is not the best in the Asia-Pacific region. That honour belongs to the Australian National University. Nor is it the best in Asia. That would be the National University of Singapore. Also Tokyo and Melbourne are ahead of Beijing University.

If there is a mistake in these calculations please tell me and I will correct it.

This is of course assuming that the data for these universities is correct. We have already noted that the score for Duke is too high but if there are no further errors (a very big assumption I admit) then Beijing should have a much lower position than the one assigned by QS. If QS have information from Beijing University that has not been divulged to the public then they have a duty to let us know.


In a little while I shall write to THES and see what happens.

No Cover-up at Duke

The Duke administration has commented on the university's performance in the latest THES university rankings. While welcoming Duke's continued high placing, senior administrator John Burness, has expressed surprise about Duke's 100 score for faculty-student ratio. He notes that several universities are recorded as doing better on this measure.

The full story is here.

I wonder what THES and QS are going to say.

The Names of Universities

This year the THES online edition referred to "University of Kebangsaan Malaysia" although QS used the correct form, "Universiti Kebangsaan Malaysia" . In 2005, however, QS had "University Putra Malaysia" and "University Sains Malaysia". In 2004 THES had "Sains Malaysia University".

If they can't get things like this right what else have they got wrong?

What happened to Macquarie?

In 2005 Macquarie University in Sydney, Australia, was ranked 67th on the THES university rankings. In 2006 it slipped a bit to 82nd place, even though its score rose on every section except for citations per faculty. This is not in itself a problem since it is possible that changes at the top could cause nearly everybody to go up if 100 represented a smaller number.

What is surprising is that Macquarie got a score of 100 for the international faculty component, compared with 53 in 2005.

We should point out that in 2005 Australian universities received the same or almost the same score for this component. Thirteen were given a score of 53, two a score of 54, one a score of 52 and one 33. I would guess that the four different scores are probably data entry errors since they all differ from the majority by a single digit. This makes it more plausible that some of the more surprising changes in 2006 may have resulted from similar errors.

THES indicated that in 2005 in some cases they had to make an estimate for some data so presumably the 2005 figures represent an estimate of the proportion of international faculty for the whole of Australia.

The doubling of Macquarie's score may not then be so implausible. Perhaps the 2005 figure was actually far too low. Also, it is possible Macquarie's 100 may represent a lower figure than the 100 that was given to the City University of Hong Kong in 2005.

Even so, it does look like a surprisningly high score for Macquarie. In 2005 City University of Hong Kong had 55.47% international faculty, which was converted into 100. In 2006 they had a score of 75. So if their score had remained the same, Macquarie's would have been 73.96 %. In 2006, second-ranking LSE had, according to its web site, 44% international faculty, which would equal a score of 89.

This means that Macquarie would have an international faculty of 48 to 74 %. The first number might be plausible but the second seems too high. But does Macquarie have such a score? I have spent a few hours trawling the internet to find anything about the proportion of international faculty at Macquarie or other Australian universities without success.

Is there anybody out there who knows anything about the proportion of international faculty at Macquarie?

Is it possible that we have another data entry error here that has affected all the scores in that section?

Congratulations!

Congratulations to THES on their discovery of a new city, Kebangsaan, in Malaysia. The online edition refers to "University of Kebangsaan Malayia".

Letter sent to THES

The Editor
Times Higher Education Supplement

Dear Sir
I would like to draw your attention to an apparent error in the latest world university rankings, which may have rendered them invalid.

The faculty/student score indicates that Duke University was the highest scorer in this category. Therefore, its score was converted to 100 and the other scores adjusted accordingly.It is, in fact, difficult to see how Duke could really be the top scorer in this category. According to the US News and World Report's America's Best Colleges (2007 edition) Duke has eight students per faculty, a ratio that is confirmed by Duke' own data, which refers to a total of 13,088 students, of whom 6,244 were undergraduates, and 1,595 tenure and tenure track faculty, producing a ratio of 8.2. However, the USNWR lists 13 US national universities with better ratios than this, Caltech three to one, Princeton five to one and so on. There are also probably a few non-US institutions that do better.

It is, of course, possible that QS, your consultants, and USNWR adapted different conventions with regard to including or excluding adjunct staff, researchers without teaching responsibilities, medical school staff and so on. This still does not get over the problem. QS appears to have constructed this component of the rankings largely from data entered into files that are available on its web site. These are linked to the 2005 rankings and some no doubt have been revised this year but in general one would expect them to be similar to whatever data was used for 2006.
Thus, QS refers to 2,172 students and 441 staff at Caltech, a ratio of 4.93 students per faculty and 4,633 students and 664 faculty at Rice, a ratio of 6.98. If QS were using these data -- and a quick survey of this category suggests that they were -- then Caltech's ratio of 4.93 and score of 67 would turn Duke's score of 100 into a ratio of 3.3. Rice's ratio of 6.98 and score of 50 would produce a ratio of 3.5 for Duke. In general it appears that QS gave Duke a student to faculty ratio of somewhere between 3 and 3.5.

So Duke, according to QS, would have a ratio of somewhere between 3 and 3.5 students per faculty, which is far lower than any figure that can be derived from the university's current data.
The problem is aggravated by QS's data on Duke which records a total of 12,223 students and 6,244 faculty. The latter figure is obviously far too high and is most probably a data entry error that occurred when someone transferred the figure of 6,244 undergraduates indicated on Duke’s web site to the faculty section in QS's files. This results in a ratio of 1.96 students per faculty, which was probably the figure used by QS in the 2005 rankings. It is probably though not the number used in 2006. If it were, then the scores for other universities would have been very much lower. Rice, for example, would have had a score of around 30 rather than 50 if that had been the case.

It is hard to see how QS came up with the ratio of 3.0 - 3.5 for Duke. It certainly does not come from any information that the university itself has provided. Perhaps it was just another data entry error that nobody noticed.

Anyway, there is no way in which the data can be manipulated, stretched or compressed to put Duke at the top of the faculty- student ratio component. That position probably belongs to Yale, which, according to Yale itself, QS and Wikipedia, has about three students per faculty.Therefore Yale's score, whatever the exact ratio that it represents, should be 100 instead of 93 and the score of everybody except Duke would have to go up accordingly. All the scores for this part need to be corrected and so therefore do the total scores.You might argue that the changes are so small that they are not worth bothering about. At the top, maybe this is true but it might make quite a bit of difference further down. In any case, surely an attempt to rank world-class universities ought to be held to the highest methodological l standards.

If you can provide a reasonable explanation for Duke's high score, such as the use of information withheld by Duke from the general public, I am sure that everybody would be glad to hear it. Otherwise, it might be a good idea to withdrew the rankings and then republish them after they have been thoroughly checked.


Richard Holmes
Shah Alam
Malaysia

Institute or Institutes Revisited

QS says Indian Institute of Management and of Technology (one of each). THES says Institutes (five of each).

So which is it?

Institute or Institutes?

In 2005, THES lumped all the Indian Institutes of Technology and Management (five of each according to the World of Learning) as single institutions. Now, according to the QS site, only one of each has been included in the rankings. In both cases the -s has been omitted

This could be a simple error or maybe the error was in 2005. Consultants who can change the name of China's most venerable university or confuse undergraduate students with faculty are probably capable of anything.

But if there is really only one Institute of Technology and one Institute of Management this year, which one is it? And did QS ensure that they collected data from only one and not from all the institutes?

I wonder when we will know?

Comments on the THES Top 200 Universities

Looking at the list of 200 universities, the most striking thing is the remarkable changes in the position of many universities. The Sorbonne has risen from 305 to 200, Wollongong from 308 to 196, Aberdeen from 267 to 195, Tubingen from 260 to 170, Ulm in Germany from 240 to 158. Meanwhile, Queensland University of Technology has fallen from 192 to 118, Purdue from 61 to 127, Helsinki from 62 to 116, and the Korean Advanced Institute of Science and Technology from 143 to 198.

Altogether, 41 universities went up or down 50 places or more.

This is, to say the least, very strange. If the scores for 2005 and 2006 were both valid, then there would be very little change. A change in policy on the admission of international students or recruitment of international faculty would take a few years to produce a change in overall numbers. A massive increase in research funding would take even longer to produce more research papers, let alone an increase in the citations of those papers.

Some changes may have occurred because of research errors or corrections of errors, "clarification of data" as THES likes to put it. In other cases, universities may have have done a bit of rearranging of data about numbers of faculty and students.

However, most of the changes are probably the result of changes in the score on the peer review. Unless THES and QS are more forthcoming about how they conducted this survey we can only assume that rises and falls on the peer review reflect nothing more than QS's distribution of their survey, with a lot of forms this year being sent to Britain, continental Europe and the US. This in turn is probably influenced by the international ebb and flow of MBA recruitment. However, we will have to wait until the thirteenth to be certain about this.

Here is something interesting from the Cambridge Evening News

CAMBRIDGE is the best university in the world, say academics.
It comes second
in the Times Higher Educational Supplement's (THES) world ranking of
universities - but made it into first place when given a score by
academics.
Last year Cambridge came third in the overall rankings, 14
percentage points behind Harvard, but this year it is only narrowly beaten by
the US university.
Oxford made third place and Yale fourth.
But academics
said Cambridge was the best university, followed by Oxford and then
Harvard.
John O'Leary, editor of The THES, said:
"These results show
academics think Cambridge is the world's best university, with Oxford close
behind. On this measure they both come ahead of Harvard.
"In addition,
Cambridge is popular with employers. Its score on quantitative criteria such as
international appeal and staff/student ratio provides numerical corroboration of
its excellence."

So, according to the academic peer review, which is 40 % of the total score, Cambridge and Oxford both beat Harvard this year. In 2004, however, Cambridge was 19 % behind Harvard and in 2005 4 %. Oxford was 16 % behind in 2004 and 7 % behind in 2005. A rise of this magnitude in the popularity of Oxford and Cambridge over two years is simply not possible if the same reviewers were on the panel in these three years or if comparable, objectively selected respondents were polled. It would be possible though if QS sent a larger number of forms this year to UK universities, particularly to Oxbridge.

Breaking News: THES Top 200 Revealed

The THES top 200 universities is now available on the QS website. http://www.topgraduate.com/universityrankings/thes_qs_world_university_rankings_2006/

More in a little while.

Preliminary observations are that there are some massive changes up and down that could only come from fluctuations in the peer review and the employers' ratings.

Meanwhile, Universiti Malaya has fallen a bit from 169 to 192 while Universiti Kebangsaan Malaysia has gone up from.289 to 185, something that certainly needs explaining

Preview of the 2006 THES World University Rankings

THES has just released a preview of its 2006 university ranking exercise. See The Times Online. This includes the top 100 universities with their positions in 2005. Here are some initial observations.

• There are, like last year, some dramatic changes. I counted over twenty universities that went up or down twenty places. That does not include any university that slipped out of the top 100 altogether. This is a bad sign. If the rankings were accurate in both 2005 and in 2006 there would be very little change from one year to another. Changes like this are only likely to occur if there has been a change in the scoring method, data collection or entry errors, corrections of errors or variations in survey bias.
• The Ecole Polytechnique in Paris has fallen from tenth place to 37th. Very probably, this is because QS has corrected an error in the counting of faculty in 2005, but we will have to wait until October 13th to be sure.
• Two Indian institutions or groups of institutions are in the top 100. The Times Online list refers to "Indian Inst of Tech" and "Indian Inst of Management", obscuring whether this refers to Institutes as in 2005 or Institute as in 2004. Whether singular or plural, a bit of digging needs to be done.
• A number of Eastern US universities have done dramatically well. For example, Vanderbilt has risen from 114th to 53rd, Emory from 141st to 56th, Pittsburgh from 193rd to 86th and Dartmouth from 117th to 61st. Does this represent a genuine improvement or is it an artifact of the distribution of the peer review survey, with QS trying to rectify an earlier bias to the West coast?
• There are a couple of cases of pairs of universities in the same city where one goes up and one goes down. See Munich University and the Technical University of Munich and the University of Lausanne and EPF Lausanne. Is it possible that in previous years these institutions got mixed up and have now been disentangled?
• There are dramatic rises by a couple of New Zealand universities, Auckland and Otago, and some Asian universities, Tsing Hua, Osaka and Seoul National. Again, this is probably a result of the way the peer review was conducted.
• Duke falls a little bit, suggesting that the errors in its 2005 score have not been corrected.
• Overall, it looks like the rankings have changed largely because of the peer review. It is hard to see how an increase in the number of international students or faculty or citations of research papers that were started several years ago could produce such remarkable changes in a matter of months. This time the review, and perhaps the recruiter ratings also, seems to have worked to the advantage of British and Eastern US universities and a few select East Asian, Swiss and New Zealand institutions.
  

Another Duke Scandal?

Part 2

It looks as though Duke University achieved its outstanding score on the Times Higher Education Supplement (THES) world university rankings in 2005 largely through another blunder by THES's consultants, QS. What is scandalous about this is that none of the statisticians, mathematicians and educational management experts at Duke noticed this or, if they did notice it, that they kept quiet about it.

Between 2004 and 2005 Duke went zooming up the THES rankings from 57th to 11th place. This would be truly remarkable if the THES rankings were at all accurate. It would mean that the university had in twelve months recruited hundreds of new faculty, multiplied the number of its international students, doubled its research output, convinced hundreds of academics and employers around the world of its superlative qualities, or some combination of the above. If this did not happen, it means that there must have been something wrong with THES's figures.

So how did Duke achieve its extraordinary rise? First, we had better point out that when you are constructing rankings based on thousand of students, hundreds of faculty, tens of thousands of research papers or the views of hundreds or thousands of reviewers you are not going to get very much change from year to year if the ranking is reliable. This is why we can be fairly confident that the Shanghai Jiao Tong University index, limited though it is in many respects, is accurate. This, unfortunately, also makes it rather boring. Notice the near total lack of interest in the latest edition of the index which came out a few weeks ago. It is hard to get excited about Tokyo inching up one place to number 19. Wait another four decades and it will be challenging Harvard! Compare that with the catastrophic fall of Universiti Malaya between 2004 and 2005 on the THES index and the local media uproar that ensued. That was much more interesting. What a pity that it was all the result of a research error, or a "clarification of data".

Or look at what happened to Duke. Last September it crept up from number 32 to 31 on the Shanghai Jiao Tong ranking, reversing its fall to 31 in 2004 from 32 in 2003. Who is going to get excited about that?

Anyway, let's have a look at the THES rankings in detail. A brief summary, which could be passed over by those familiar with the rankings, is that in 2005 it gave a weighting of 40 % to a peer review by "research-active academics", 10 % to a rating by employers of "internationally mobile graduates", 20 % to faculty-student ratio, 10 % to proportion of international faculty and international students and 20% to the citations of research papers per faculty member.

I will just mention the peer review section and then go on to the faculty-student ratio where the real scandal can be found.

Peer Review
Duke got a score of 61 on the peer review compared to a top score (Berkeley) of 665 in 2004. This is equivalent to a score of 9.17 out of 100. In 2005 it got a score of 36 compared with the top score of 100 (Harvard), effectively almost quadrupling its score. To some extent, this is explained by the fact that everybody except Berkeley went up on this measure between 2004 and 2005. But Duke did much better than most. In 2004 Duke was noticeably below the mean score of the 164 universities that appeared in the top 200 in both years, but in 2005 it was slightly above the mean of these universities. Its position on this measure rose from 95th to 64th.

How is this possible? How could there be such a dramatic rise in the academic peers' opinions of Duke university? Remember that the reviewers of 2005 included those of 2004 so there must have been a very dramatic shift among the additional reviewers towards Duke in 2005.

A genuine change in opinion is thoroughly implausible. Two other explanations are more likely. One is that QS polled many more academics from the east coast or the south of the United States in 2005, perhaps because of a perceived bias to California in 2004. The other is that the Duke academics invited to serve on the panel passed the survey to others who, in a spontananeous or organised fashion, returned a large number of responses to QS.

Faculty-Student Ratio
Next, take a look at the scores for Faculty-Student ratio. Duke did well on this category in 2004 with a score of 66. In that year the top scorer was Ecole Normal Superieure (ENS) in Paris which, with 1800 students and 900 faculty according to QS, would have had 0.50 faculty per student, Duke therefore would have 0.33 faculty per students. This would be a very good ratio if true.

In 2005, the top scorer was Ecole Polytechnique in Paris, which supposedly had 2,468 students and 1,900 faculty, or 0.77 faculty per student. ENS's score went down to 65 which is exactly what you what expect if the ratio remained unchanged at 0.5. Duke's score in 2006 was 56, which works out at 0.43 faculty per student.

Duke's faculty-student ratio currently according to QS's web site is 0.51 or 6,244 faculty divided by 12,223 students. This is fairly close to the 2005 figure indicated in the THES rankings. Here is the information provided by QS

Datafile
Demographic
No. of faculty:
6,244
No. of international
faculty:
825
No. of students:
12,223

However, Duke's web site currently gives a figure of 13,088 students and 1,595 tenure and tenure track faculty and 923 others (professors of the practice, lecturers, research professors and medical associates). This makes a total of 2,518 tenure, tenure track and other regular faculty.

So where on Earth did QS find another 3,700 plus faculty?

Take a look at this data provided by Duke. Notice anything?

STUDENTS Enrollment
(full-time) Fall 2005
Undergraduate
6,244
African-AmericanAsian-American

So what happened is that someone at QS confused the number of undergraduate students with the number of faculty and nobody at QS noticed and nobody at THES noticed. Perhaps nobody at Duke noticed either, although I find that hard to believe. This resulted in a grossly inflated score for Duke on the faculty-student ratio component and contributed to its undeserved ascent in the rankings.

Anyway, it's time to stop and post this, and then work out what Duke's score really should have been.