Comments on the THES Top 200 Universities

Looking at the list of 200 universities, the most striking thing is the remarkable changes in the position of many universities. The Sorbonne has risen from 305 to 200, Wollongong from 308 to 196, Aberdeen from 267 to 195, Tubingen from 260 to 170, Ulm in Germany from 240 to 158. Meanwhile, Queensland University of Technology has fallen from 192 to 118, Purdue from 61 to 127, Helsinki from 62 to 116, and the Korean Advanced Institute of Science and Technology from 143 to 198.

Altogether, 41 universities went up or down 50 places or more.

This is, to say the least, very strange. If the scores for 2005 and 2006 were both valid, then there would be very little change. A change in policy on the admission of international students or recruitment of international faculty would take a few years to produce a change in overall numbers. A massive increase in research funding would take even longer to produce more research papers, let alone an increase in the citations of those papers.

Some changes may have occurred because of research errors or corrections of errors, "clarification of data" as THES likes to put it. In other cases, universities may have have done a bit of rearranging of data about numbers of faculty and students.

However, most of the changes are probably the result of changes in the score on the peer review. Unless THES and QS are more forthcoming about how they conducted this survey we can only assume that rises and falls on the peer review reflect nothing more than QS's distribution of their survey, with a lot of forms this year being sent to Britain, continental Europe and the US. This in turn is probably influenced by the international ebb and flow of MBA recruitment. However, we will have to wait until the thirteenth to be certain about this.

Here is something interesting from the Cambridge Evening News

CAMBRIDGE is the best university in the world, say academics.
It comes second
in the Times Higher Educational Supplement's (THES) world ranking of
universities - but made it into first place when given a score by
academics.
Last year Cambridge came third in the overall rankings, 14
percentage points behind Harvard, but this year it is only narrowly beaten by
the US university.
Oxford made third place and Yale fourth.
But academics
said Cambridge was the best university, followed by Oxford and then
Harvard.
John O'Leary, editor of The THES, said:
"These results show
academics think Cambridge is the world's best university, with Oxford close
behind. On this measure they both come ahead of Harvard.
"In addition,
Cambridge is popular with employers. Its score on quantitative criteria such as
international appeal and staff/student ratio provides numerical corroboration of
its excellence."

So, according to the academic peer review, which is 40 % of the total score, Cambridge and Oxford both beat Harvard this year. In 2004, however, Cambridge was 19 % behind Harvard and in 2005 4 %. Oxford was 16 % behind in 2004 and 7 % behind in 2005. A rise of this magnitude in the popularity of Oxford and Cambridge over two years is simply not possible if the same reviewers were on the panel in these three years or if comparable, objectively selected respondents were polled. It would be possible though if QS sent a larger number of forms this year to UK universities, particularly to Oxbridge.

Ranking Insights from Russia

The ranking industry is expanding and new rankings appear all the time. Most global rankings measure research publications and citations. Others try to add to the mix indicators that might have something to do with teaching and learning. There is now a  ranking that tries to capture various third missions.

The Round University Rankings published in Russia are in the tradition  of  holistic rankings. They give a 40 % weighting to research, 40 % to teaching, 10% to international diversity and 10% to financial sustainability. Each group contains five equally weighted indicators. The data is derived from Clarivate Analytics who also contribute to the US News Best Global Universities Rankings.

These rankings are similar to the THE rankings in that they attempt to assess quality rather than quantity but they have 20 indicators instead of 13 and assign sensible weightings. Unfortunately, they receive only a fraction of the attention given to the THE rankings.

They are, however, very  valuable since they dig deeper into the data than  other global rankings. They also show that there is a downside to measures of quality and that data submitted directly by institutions should be  treated with caution and perhaps scepticism.

Here are the top universities for each of the RUR indicators.

Teaching
Academic staff per students: VIB (Flemish Institute of Biotechnology), Belgium
Academic staff per bachelor degrees awarded: University of Valladolid, Spain
Doctoral degrees per academic staff: Kurdistan University of Medical Science, Iran
Doctoral degrees per bachelor degrees awarded:  Jawaharlal Nehru University, India
World teaching reputation  Harvard University, USA.

Research
Citations per academic and research staff: Harvard
Doctoral degrees per admitted PhD: Al Farabi Kazakh National University
Normalised citation impact: Rockefeller University, USA
Share of international co-authored papers: Free University of Berlin
World research reputation: Harvard.

International diversity
Share of international academic staff: American University of Sharjah, UAE
Share of international students: American University of Sharjah
Share of international co-authored papers: Innopolis University, Russia
Reputation outside region: Voronezh State Technical University, Russai
International Level: EPF Lausanne, Switzerland.

Financial sustainability:
Institutional income per academic staff: Universidade Federal Do Ceara, Brazil
Institutional income per student: Rockefeller University
Papers per research income:  Novosibersk State University of Economics and Management, Russia
Research income per academic and research staff: Istanbul Technical University, Turkey
Research income per institutional income: A C Camargo Cancer Center, Brazil.

There are some surprising results here. The most obvious is Voronezh State Technical University which is first for reputation outside its region (Asia, Europe and so on), even though its overall scores for reputation and for international diversity are very low. The other top universities for this metric are just what you would expect, Harvard, MIT, Stanford, Oxford and so on. I wonder whether there is some sort of bug in the survey procedure, perhaps something like the university's supporters being assigned to Asia and therefore out of region. The university is also in second place in the world for papers per research income despite very low scores for the other research indicators.

There are other oddities such as Novosibersk State University of Economics and Management placed first for papers per research income and Universidade Federal Do Ceara for institutional income per academic staff. These may result from anomalies in the procedures for reporting and analysing data, possibly including problems in collecting data on income and staff.

It also seems that medical schools and specialist or predominantly postgraduate institutions such as Rockefeller University, the Kurdistan University of Medical Science, Jawarhalal Nehru University and VIB have a big advantage with these indicators since they tend to have favourable faculty student ratios, sometimes boosted by large numbers of clinical and research only staff, and a large proportion of doctoral students.

Jawaharlal Nehru University is a mainly postgraduate university so a high placing for academic staff per bachelor degrees awarded is not unexpected although I am surprised that it is ahead of Yale and Princeton. I must admit that the third place here for the University of Baghdad needs some explanation.

The indicator doctoral degrees per admitted PhD might identify universities that do a good job of selection and training and get large numbers of doctoral candidates through the system. Or perhaps it identifies universities where doctoral programmes are so lacking in rigour that nearly everybody can get their degree once admitted. The top ten of this indicator includes De Montfort University, Shakarim University, Kingston University, and the University of Westminster, none of which are famous for research excellence across the range of disciplines.

Measures of international diversity have become a staple of global rankings since they are fairly easy to collect. The problem is that international orientation may have something do with quality but it may also simply be a necessary attribute of being in a small country next to larger countries with the same or similar language and culture. The top ten for the international student indicators includes the Central European University and the American university of Sharjah. For international faculty it includes the University of Macau and Qatar University.

To conclude, these indicators suggest that self submitted institutional data should be used sparingly and that data from third party sources may be preferable. Also, while ranking by quality instead of quantity is sometimes advisable it also means that anomalies and outliers are more likely to appear.

The New Webometrics Rankings

The latest Webometrics rankings are out.

In the overall rankings the top five are:

1.  Harvard
2.  MIT
3.  Stanford
4.  Cornell
5.  Columbia.

Looking at the indicators one by one, the top five for presence (number of webpages in the main webdomain) are:

1.  Karolinska Institute
2.  National Taiwan University
3.  Harvard
4.  University of California San Francisco
5.  PRES Universite de Bordeaux.

The top five for impact (number of external inlinks received from third parties) are:

1.  University of California Berkeley
2.  MIT
3.  Harvard
4.  Stanford
5.  Cornell.

The top five for openness (number of rich files published in dedicated websites) are:

1.  University of California San Francisco
2.  Cornell
3.  Pennsylvania State University
4.  University of Kentucky
5.  University of Hong Kong.

The top five for excellence (number of papers in the 10% most cited category) are:

1.  Harvard
2.  Johns Hopkins
3.  Stanford
4.  UCLA
5.  Michigan

Three British universities in the THES-QS Top Five

Bits and pieces about the THES-QS 2007 rankings are appearing in online newspapers. Here is a quotation from the London Times

Cambridge and Oxford are the second best universities in the world according to the latest rankings, and British universities are closing the gap with those in the United States.

Oxford and Cambridge share the number two spot with Yale, with Harvard ranked number one in the latest league tables from The Times Higher Education Supplement.

The findings will bring cheer to the higher education sector in Britain at a time of growing concern among vice-chancellors and employers that British universities will lose students to better-financed institutions abroad and that business will then follow them with jobs and investment.

The commercial implications of the rankings are made very clear:

Professor Rick Trainor, the president of Universities UK, representing vice-chancellors, added: “Our competitors are increasingly marketing themselves more aggressively so it is vital that the UK remains among the foremost destinations for international students and staff.”

Harvard, whose endowment of $35 billion (£16.6 billion) is roughly equal to the combined annual funding for all English universities, tops the table, but its lead over its closest rivals has fallen from 3.2 to 2.4 points. Nunzio Quacquarelli, the managing director of QS, the careers and education group that compiled the rankings, said: “In an environment of increasing student mobility, the UK is putting itself forward as a top choice for students worldwide.

“They are taking a closer look at the quality of faculty, international diversity and, of course, to the education they will receive.”

A detailed analysis will have to wait until the component scores are available but the continued closing of the gap between Oxbridge and Harvard and the rise of University College London from 25th to 9th and Imperial College London from 8th to 5th are rather suspicious.

The top ten are

1 Harvard University US
2 University of Cambridge UK
2 University of Oxford UK
2 Yale University US
5 Imperial College, London UK
6 Princeton University US
7 California Institute of Technology (Caltech) US
7 University of Chicago US
9 University College London (UCL) UK
10 Massachusetts Institute of Technology (MIT) US

Measuring graduate employability; two rankings

Global university rankings are now well into their second decade. Since 2003, when the first Shanghai rankings appeared, there has been a steady growth of global and regional rankings. At the moment most global rankings are of two kinds, those that focus entirely or almost entirely on research and those such as the Russian Round Rankings, Times Higher Education (THE) and Quacquarelli Symonds (QS) that claim to also measure teaching, learning or graduate quality in some way, although even those are biased towards research when you scratch the surface a little.

The ranking industry has become adept at measuring research productivity and quality in various ways. But the assessment of undergraduate teaching and learning is another matter.

Several ranking organisations use faculty student ratio as a proxy for quality of teaching which in turn is assumed to have some connection with something that happens to students during their programmes. THE also count institutional income, research income and income from industry, again assuming that there is a significant association with academic excellence. Indicators like this are usually based on those supplied by institutions. For examples of problems here see an article by Alex Usher and a reply by Phil Baty.

An attempt to get at student quality is provided by the CWUR rankings now based in UAE, counting alumni who win international awards or who are CEOs of major companies. But obviously this is relevant only for a very small number of universities. A new pilot ranking from Moscow also counts international awards.

The only attempt to measure student quality  by the well known rankers that is relevant to most institutions is the survey of employers in the QS world and regional rankings. There are some obvious difficulties here. QS gets respondents from a variety of channels and this may allow some universities to influence the survey. In recent years some Latin American universities have done much better on this indicator than on any other.

THE now publish a global employability ranking which is conducted by two European firms, Trendence and Emerging. This is based on two surveys of recruiters in Argentina, Australia, Austria, Brazil, Canada, China, Germany, France, India, Israel, Italy, Japan, Mexico, Netherlands, Singapore, Spain, South Africa, South Korea, Turkey, UAE, UK, and USA. There were two panels with a total of over 6,000 respondents.

A global survey that does not include Chile, Sweden, Egypt, Nigeria, Saudi Arabia, Russia, Pakistan, Indonesia, Bangladesh, Poland, Malaysia or Taiwan can hardly claim to be representative of international employers. This limited representation may explain some oddities of the rankings such as the high places of the American University of Dubai and and the National Autonomous University of Mexico.

The first five places in these rankings are quite similar to the THE world rankings: Caltech, Harvard, Columbia, MIT, Cambridge.  But there some significant differences after that and some substantial changes since last year. Here Columbia, 14th in the world rankings, is in third place, up from 12th last year. Boston University is 6th here but 70th in the world rankings. Tokyo Institute of Technology in 19th place is in the 251-300 band in the world rankings. CentraleSupelec, is 41st,  but in the world  401-500 group.

These rankings are useful only for a small minority of universities, stakeholders and students. Only 150 schools are ranked and only a small proportion of the world's employers consulted.

QS have also released their global employability rankings with 500 universities. These combine the employer reputation survey, used in their world rankings with  other indicators: alumni outcomes, based on lists of high achievers, partnership with employers, that is research collaboration noted in the Scopus database, employer-student connections, that is employers actively present on campus, and graduate employment rate. There seems to be a close association, at least at the top, between overall scores, employer reputation and alumni outcomes. Overall the top three are Stanford, UCLA, Harvard. For employer reputation they are Cambridge, Oxford, Harvard and for alumni outcomes Harvard, Stanford Oxford.

The other  indicators are a different matter. For employer-student connections the top three are Huazhong University of Science and Technology, Arizona State University, and New York University. In fact seven out of the top ten on this measure  are Chinese. For graduate employment rate they are Politecnico di Torino, Moscow State Institute of International Relations, and Sungkyunkwan University and for partnership with employers Stanford, Surrey and Politecnico Milano. When the front runners in indicators are so different one has to wonder about their validity.

There are some very substantial differences in the ranks given to various universities in these rankings. Caltech is first in the Emerging-Trendence rankings and 73rd in QS. Hong Kong University of Science and Technology is 12th in Emerging-Trendence but not ranked at all by QS. The University of Sydney is 4th in QS and 48th in Emerging-Trendence. The American University of Dubai is in QS's 301-500 band but 138th for Emerging-Trendence

The  rankings published by THE could be some value to those students contemplating careers with the leading companies in the richest countries.

The QS rankings may be more helpful for those students or stakeholders looking at universities outside the very top of the global elite. Even so QS have ranked only a fraction of the world's universities.

It still seems that the way forward in the assessment of graduate outcomes and employability is through standardised testing along the lines of AHELO or the Collegiate Learning Assessment.

The Triple Package

I have just finished reading The Triple Package by Amy Chua and Jed Rubenfeld, a heavily anecdotal book that tells us, as every reader of the New York Times now knows, what really determines success.

An irritating thing is the presentation of urban legends -- no dogs, no Cubans and so on -- and generalizations to support the authors' thesis.

Here is one example: "men like Alfred Kazin, Norman Mailer, Delmore Schwatz, Saul Bellow, Celement Greenberg, Norman Podhoretz, and so many of the New York intellectuals who grew up excluded from anti-Semitic bastions of education and culture but went on to become famous writers and critics".

Alfred Kazin went to City College of New York when it was a selective institution. Norman Mailer went to Harvard at the age of 16 and, after serving in the army, to the Sorbonne. Delmore Schwartz attended Columbia, the University of Wisconsin and New York University and did postgraduate work at Harvard with Alfred North Whitehead. Saul Bellow was at the University of Chicago and then Northwestern. He was also also a postgraduate student at the University of Wisconsin. Clement Greenberg studied at Syracuse University. Norman Podhoretz was accepted by Harvard and NYU but went to Columbia which offered him a full scholarship. He went to Cambridge on a Fulbright and was offered a fellowship at Harvard which he turned down

Bellow famously endured several anti Semitic slights and sneers and no doubt did the others. But can we really say that were excluded from bastions of education?

i

Round University Rankings

The latest Round University Rankings have been released by the Russian company, RUR Rankings Agency. These are essentially holistic rankings that attempt to go beyond the measurement of research output and quality. There are twenty indicators, although some of them such as Teaching Reputation, International Teaching Reputation and Research Reputation and International Students and International Bachelors are so similar that the information they provide is limited.

Basically these rankings cover much the same ground as the Times Higher Education (THE) World University Rankings. The income from industry indicator is not included but there are an additional eight indicators. The data is taken from Thomson Reuters' Global Institutional Profiles Project (GIPP) which was used by THE for their rankings from 2010 to 2014.

Unlike THE, which lumps its indicators together into groups,  the scores in the RUR are listed separately in the profiles. In addition, the rankings provide data for seven continuous years from 2010 to 2016. This provides an unusual opportunity to examine in detail the development of universities over a period of seven years, measured by 20 indicators. This is not the case with other rankings which have fewer indicators or which have changed their methodology.

It should be noted that participation in the GIPP is voluntary and therefore the universities in each edition could be different. For example, in 2015 100 universities dropped out of the project and 62 joined.

It is, however,  possible to examine a number of claims that have been made about changes in university quality over the last few years. I will  take a look at these in the next few posts.

For the moment, here are the top five in the overall rankings and the dimension rankings.

Overall
1.   Caltech
2.   Harvard
3.   Stanford
4.   MIT
5.   Chicago


Teaching
1.   Caltech
2.   Harvard
3.   Stanford
4.   MIT
5.   Duke

Research
1.   Caltech
2.   Harvard
3.   Stanford
4.   Northwestern University
5.   Erasmus University Rotterdam

International Diversity
1.   EPF Lausanne
2.   Imperial College London
3.   National University of Singapore
4.   University College London
5.   Oxford

Financial Sustainability
1.   Caltech
2.   Harvard
3.   Scuola Normale Superiore Pisa
4.   Pohang University of Science and Technology
5.   Karolinska Institute

Unfortunately these rankings have received little or no recognition outside Russia. Here are some examples.

MIPT entered the top four universities in Russia according to the Round University Ranking

Russian Universities in the lead in terms of growth in the international ranking of Round University Ranking

TSU [Tomsk State University]  has entered the 100 best universities for the quality of teaching

• ITMO UNIVERSITY IN TOP-200 FOR TEACHING QUALITY

[St Petersburg]

Russian universities to top intl rankings by 2020 – Education Minister Livanov to RT

NUST MISIS has shown the best dynamic of growth among Russian universities in the Round University Ranking [Moscow]

UNN [Lobachevsky] Enters Top 500 World Universities in 2016 Round University Ranking

SCImago Innovation Rankings: The East-West Gap Gets Wider

The decline of western academic research becomes more apparent every time a ranking with a stable and moderately accurate methodology is published. This will not be obvious if one just looks at the top ten, or even the top fifty, of the better known rankings. Harvard, Stanford, and MIT are usually still there at the top and Oxford and Cambridge are cruising along in the top twenty or the top thirty.

But take away the metrics that measure inherited intellectual capital such as the Nobel and Fields laureates in the Shanghai rankings or the reputation surveys in the QS, THE, and US world rankings, and the dominance of the West appears ever more precarious. This is confirmed if we turn from overall rankings to subject and field tables.

Take a look at the most recent edition of the CWTS Leiden Ranking, which is highly reputed among researchers although much less so among the media. For sheer number of publications overall, Harvard still holds the lead although Zhejiang, Shanghai Jiao Tong and Tsinghua are closing in and there are more Chinese schools in the top 30.  Chinese dominance is reduced if we move to the top 10% of journals but it may be just a matter of time before China takes the lead there as well. 

But click to physical sciences and engineering. The top 19 places are held by Mainland Chinese universities with the University of Tokyo coming in at 20.  MIT is there at 33, Texas A & M at 55 and Purdue 62. Again the Chinese presence is diluted, probably just for the moment, if we switch to the top 10% or 1% of journals.  

Turning to developments in applied research, the shift to China and away from the West, appears even greater.

The SCImago Institutions rankings are rather distinctive. In addition to the standard measures of research activity, there are also metrics for innovation and societal impact. Also, they include the performance of government agencies, hospitals, research centres and companies.

The innovation rankings combine three measures of patent activity. Patents are problematic for comparing universities but they can establish broad long-term trends. 

Here are the top 10 for Innovation in 2009:

1.   Centre Nationale de la Recherche Scientifique

2.   Harvard University 

3.   National Institutes of Health, USA

4.   Stanford University 

5.   Massachusetts Institute of Technology

6.   Institute National de las Sante et de la Recherche Medicale

7.   Johns Hopkins University 

8.   University of California Los Angeles

9.   Howard Hughes Medical Institute 

10.  University of Tokyo.

And here they are for 2023:

1.   Chinese Academy of Sciences 

2.   State Grid Corporation of China  

3.   Ministry of Education PRC

4.   DeepMind

5.   Ionis Pharmaceuticals

6.   Google Inc, USA

7.   Alphabet Inc 

8.  Tsinghua University

9.   Huawei Technologies Co Ltd

10.  Google International LLC.

What happened to the high flying universities of 2009?  Harvard is in 57th place, MIT in 60th, Stanford 127th, Johns Hopkins 365th, and Tokyo in 485th. 

it seems that the torch of innovation has left the hand of American, European, and Japanese universities and research centres and has been passed to Multinational, Chinese, and American companies and research bodies, plus a few Chinese universities. I am not sure where the loyalties of the multinational institutions lie, if indeed they have any at all.

Dumbing Down Watch

There is a "cheating" scandal at Harvard. Apparently, students were given an open-book, open-anything take-home exam for 'Introduction to Congress' and were not expected to consult each other in any way.
 

Harvard College’s disciplinary board is investigating nearly half of the 279 students who enrolled in Government 1310: “Introduction to Congress” last spring for allegedly plagiarizing answers or inappropriately collaborating on the class’ final take-home exam.
Dean of Undergraduate Education Jay M. Harris said the magnitude of the case was “unprecedented in anyone’s living memory.”
Harris declined to name the course, but several students familiar with the investigation confirmed that Professor Matthew B. Platt's spring government lecture course was the class in question.
The professor of the course brought the case to the Administrative Board in May after noticing similarities in 10 to 20 exams, Harris said. During the summer, the Ad Board conducted a review of all final exams submitted for the course and found about 125 of them to be suspicious.

Presumably this is not the only take-home exam in Harvard and presumably not the first for this course. So why now has half the class felt compelled to plagiarise or to collaborate inappropriately?

Has the course become more difficult than it used to be? Or are the students less capable? Or have admission standards become less rigorous?

Maybe QS and Times Higher were on to something after all when they dethroned Harvard from the number one spot in their world rankings.

Best Grad Schools

The US News Graduate School Rankings were published on March 15th. Here are the top universities in various subject areas.

Business: Stanford

Education: Vanderbilt

Engineering:  MIT

Law:  Yale

Medical: Harvard

Biology: Stanford

Chemistry: Caltech, MIT, UC Berkeley

Computer Science; Carnegie-Mellon, MIT, Stanford, UC Berkeley

Earth Sciences: Caltech, MIT

Mathematics: MIT

Physics: Caltech, Harvard, MIT, Stanford

Statistics: Stanford

Library and Information Studies: Illinois at Urbana-Champagne

Criminology: Maryland -- College Park

Economics: Harvard, MIT, Princeton, Chicago

English: UC Berkeley

History: Princeton

Political Science: Harvard, Princeton, Stanford

Psychology: Stanford, UC Berkeley

Sociology: UC Berkeley

Public Affairs: Syracuse

Fine Arts: Rhode Island School of Design

The Best Universities for Technology?

The Times Higher Education Supplement (THES) have published a list of the supposed top 100 universities in the world in the field of technology. The list purports to be based on opinion of experts in the field. However, like the ranking for science, it cannot be considered valid. First, let us compare the top 20 universities according to peer review and then the top 20 according to the data provided by THES for citations per paper, a reasonable measure of the quality of research.

First, the peer review:

1. MIT
2. Berkeley
3. Indian Institutes of Technology (all of them)
4. Imperial College London
5. Stanford
6. Cambridge
7. Tokyo
8. National University of Singapore
9. Caltech
10. Carnegie-Mellon
11. Oxford
12. ETH Zurich
13. Delft University of Technology
14. Tsing Hua
15. Nanyang Technological University
16. Melbourne
17. Hong Kong University of science and Technology
18. Tokyo Institute of Technology
19. New South Wales
20. Beijing (Peking University)

Now, the top twenty ranked according to citations per paper:

1. Caltech
2. Harvard
3. Yale
4. Stanford
5. Berkeley
6. University of California at Santa Barbara
7. Princeton
8. Technical University of Denmark
9. University of California at San Diego
10. MIT
11. Oxford
12. University of Pennsylvania
13. Pennsylvania State University
14. Cornell
15. Johns Hopkins
16. Boston
17. Northwestern
18. Columbia
19. Washington (St. Louis)
20. Technion (Israel)

Notice that the Indian Institutes of Technology, Tokyo, National University of Singapore, Nanyang Technological University, Tsing Hua, Melbourne, New South Wales and Beijing are not ranked in the top 20 according to quality of published research. Admittedly, it is possible that in this field a substantial amount of research consists of unpublished reports for state organizations or private companies but this would surely be more likely to affect American rather than Asian or Australian universities.

Looking a bit more closely at some of the universities in the top twenty for technology according to the peer review, we find that, when ranked for citations per paper, Tokyo is in 59th place, National University of Singapore 70th, Tsing Hua 86th, Indian Institutes of Technology 88th, Melbourne 35th, New South Wales 71st, and Beijing 76th. Even Cambridge, sixth in the peer review, falls to 29th.

Again, there are a large number of institutions that did not even produce enough papers to be worth counting, raising the question of how they could be sufficiently well known for there to be peers to vote for them. This is the list:

Indian Institutes of Technology
Korean Advanced Institute of Science and Technology
Tokyo Institute of Technology
Auckland
Royal Institute of Technology Sweden
Indian Institutes of Management
Queensland University of Technology
Adelaide
Sydney Technological University
Chulalongkorn
RMIT
Fudan
Nanjing

Once again there is a very clear pattern of the peer review massively favoring Asian and Australasian universities. Once again, I can see no other explanation than an overrepresentation of these regions, and a somewhat less glaring one of Europe, in the survey of peers combined with questions that allow or encourage respondents to nominate universities from their own regions or countries.

It is also rather disturbing that once again Cambridge does so much better on the peer review than on citations. Is it possible that THES and QS are manipulating the peer review to create an artificial race for supremacy – “Best of British Closing in on Uncle Sam’s finest”. Would it be cynical to suspect that next year Cambridge and Harvard will be in a circulation-boosting race for the number one position?

According to citations per faculty Harvard was 4th for science, second for technology and 6th for biomedicine while Cambridge was 19th, 29th and 9th.

For the peer review, Cambridge was first for science, 6th for technology and first for biomedicine. Harvard was 4th, 23rd and second.

Overall, there is no significant relationship between the peer review and research quality as measured by citations per paper. The correlation between the two is .169, which is statistically insignificant. For the few Asian universities that produced enough research to be counted, the correlation is .009, effectively no better than chance.

At the risk of being boringly repetitive, it is becoming clearer and clearer that that the THES rankings, especially the peer review component, are devoid of validity.

Sarah Palin and the University of Idaho

One of the more distasteful aspects of the US presidential election is the obsession in some quarters with Sara Palin's IQ. See here for example. Some have suggested that she is not particularly bright because she graduated from the University of Idaho and not from Columbia or Harvard.

May I point out that the University of Idaho, although it is not Harvard or Columbia, is ranked in the 400s in the Shanghai Jiao Tong rankings, which puts it way above most institutions in the world and in the United States.

May I also point out that Palin, unlike Joe Biden, has not been accused of plagiarism. Also, if we are going to be snobby about universities, one also wonders why Biden, if he were so clever, would plagiarise from Neil Kinnock who had to take his final exams twice at a university that also wasn't Harvard or Columbia

Very Interesting Rankings from Times Higher Education

The latest edition of the Times Higher Education (THE) World University Rankings has just been published, along with a big dose of self-flattery and congratulations to the winners of what is beginning to look more like a lottery than an objective exercise in comparative assessment.

The background to the story is that at the end of last year THE broke with their data suppliers Thomson Reuters (TR) and announced the dawn of a new era of transparency and accountability 

There were quite a few things wrong with the THE rankings, especially with the citations indicator which supposedly measured research impact and was given nearly a third of the total weighting. This meant that THE was faced with a serious dilemma. Keeping the old methodology would be a problem but radical reform would raise the question of why THE would want to change what they claimed was a uniquely trusted and sophisticated methodology with carefully calibrated indicators.

It seems that THE have decided to make a limited number of changes but to postpone making a decision about other issues.

They have broadened the academic reputation survey, sending out forms in more languages and getting more responses from outside the USA. Respondents are now drawn from those with publications in the Scopus database, much larger than the Web of Science, as was information about publications and citations. In addition, THE have excluded 649 “freakish” multi – author papers from their calculations and diluted the effect of the regional modification that boosted the scores in the citations indicator of low performing countries.

These changes have led to implausible fluctuations with some institutions rising or falling dozens or hundreds of places. Fortunately for THE, the latest winners are happy to trumpet their success and the losers so far seem to have lapsed into an embarrassed silence.

When they were published on the 30th of September the rankings provided lots of headline fodder about who was up or down.

The Irish Times announced that the rankings showed  Trinity College Dublin had fallen while University College Dublin was rising.

In the Netherlands the University of Twente bragged about its “sensationally higher scores”.

Study International asserted that “Asia Falters” and that Britain and the US were still dominant in higher education.

The London Daily Telegraph claimed  that European universities were matching the US.

The Hindu found something to boast about by noting that India was at last the equal of co-BRICS member Brazil.

Russian media celebrated the remarkable achievement of Lomonosov Moscow State University in rising 35 places.

And, of course, the standard THE narrative was trotted out again. British universities are wonderful but they will only go on being wonderful if they are given as much money as they want and are allowed to admit as many overseas students as they want.

The latest rankings support this narrative of British excellence by showing Oxford and Cambridge overtaking Harvard, which was pushed into sixth place. But is such a claim believable? Has anything happened in the labs or lecture halls at any of those places between 2014 and 2015 to cause such a shift?

In reality, what probably happened was that the Oxbridge duo were not actually doing anything better this year but that Harvard’s eclipse came from a large drop from 92.9 to 83.6 points for THE’s composite teaching indicator. Did Harvard’s teaching really deteriorate over twelve months? It is more likely that there were relatively fewer American respondents in the THE survey but one cannot be sure because there are four other statistics bundled into the indicator.

While British universities appeared to do well, French ones appeared to perform disastrously. The École Normale Supérieure recorded a substantial gain going from 78^th to 54^th place but every other French institution in the rankings fell, sometimes by dozens of places. École Polytechnique went from 61^st place to 101^st, Université Paris-Sud from 120th  to 188^th , the University of Strasbourg from the 201-225 band to 301-350, in every case because of a substantial fall in the citations indicator. If switching to Scopus was intended to help non-English speaking countries it did not do France any good.

Meanwhile, the advance of Asia has apparently come to an end or gone into screeching reverse. Many Asian universities  slipped down the ladder although the top Chinese schools held their ground. Some Japanese and Korean universities fell dozens of places. The University of Tokyo went from 23^rd to 43^rd place, largely because of a fall in the citations indicator from 74.7 points to 60.9 and the University of Kyoto from 59th to 88th with another drop in the score for citations. Among the casualties was Tokyo Metropolitan University which used to advertise its perfect score of 100 for citations on the THE website. This year, stripped of the citations for mega-papers in physics, its citation score dropped to a rather tepid 72.2.

The Korean flagships have also foundered. Seoul National University fell 35 places and the Korean Advanced Institute of Technology 66, largely because of a decline in the scores for teaching and research. Pohang University of Science and Technology (POSTECH) fell 50 places, losing points in all indicators except income from industry

The most catastrophic fall was in Turkey. There were four Turkish universities in the top 200 last year. All of them have dropped out. Several Turkish universities contributed to the Large Hadron Collider project with its multiple authors and multiple citations and they also benefited from producing comparatively few research papers and from the regional modification, which gave them artificially high scores for the citations indicator in 2014 but not this year.

The worst case was Middle East Technical University which had the 85^th place in 2014, helped by an outstanding score of 92 for citations and reasonable scores for the other indicators. This year it was in the 501-600 band with reduced scores for everything except Industry Income and a very low score of 28.8 for citations.

The new rankings appear to have restored the privilege given to medical research. In the upper reaches we find St George’s, University of London, a medical school, which according to THE is the world's leading university for research impact,  Charité - Universitätsmedizin Berlin,  a teaching hospital affiliated to Humboldt University and the Free University of Berlin, and Oregon Health and Science University.

It also appears that THE's methodology continues to gives an undeserved advantage to small or specialized institutions such as Scuola Superiore Sant’Anna in Pisa, which does not appear to be a truly independent university,  the Copenhagen Business School, and Rush University in Chicago, the academic branch of a private hospital.

These rankings appear so far to have got a good reception in the mainstream press, although it is likely that that before long we will hear some negative reactions from independent experts and from Japan, Korea, France, Italy and the Middle East.

THE, however, have just postponed the hard decisions that they will eventually have to make.

Towards a transparent university ranking system

For the last few years global university rankings have been getting more complicated and more "sophisticated".

Data makes it way from branch campuses, research institutes and far flung faculties and departments and is analysed, decomposed, recomposed, scrutinised for anomalies and outliers and then enters the files of the rankers where it is normalised, standardised, square rooted, weighted and/or subjected to regional modification. Sometimes what comes out the other end makes sense: Harvard in first place, Chinese high fliers flying higher. Sometimes it stretches academic credulity: Alexandria University in fourth place in the world for research impact, King Abdulaziz University in the world's top ten for mathematics.

The transparency of the various indicators in the global rankings varies. Checking the scores for Nature and Science papers and indexed publications in the Shanghai rankings is easy if you have access to the Web of Knowledge. It is also not difficult to check the numbers of faculty and students on the QS, Times Higher Education (THE)and US News web sites.

On the other hand, getting into the data behind the THE citations is close to impossible. Citations are normalised by field, year of publication and year of citation. Then, until last year the score for each university was adjusted by division by the square root of the citation impact score of the country in which it was located. Now this applies to half the score for the indicator. Reproducing the THE citations score is impossible for almost everybody since it requires calculating the world average citation score for 250 or 300 fields and then the total citation score for every country.

It is now possible to access third party data from sources such as Google, World Intellectual Property Organisation and various social media such as LinkedIn. One promising development is the creation of public citation profiles by Google Scholar.

The Cybermetrics Lab in Spain, publishers of the Webometrics Ranking Web of Universities, has announced the beta version of a ranking based on nearly one million individual profiles in the Google Scholar Citations database. The object is to see whether this data can be included in future editions of the Ranking Web of Universities

It uses data from the institutional profiles and counts the citations in the top ten public profiles for each institution, excluding the first profile.

The ranking is incomplete since many researchers and institutions have not participated fully. There are, for example, no Russian institutions in the top 600. In addition, there are technical issues such as the duplication of profiles.

The leading university is Harvard which is well ahead of its closest rival, the University of Chicago. English speaking universities are dominant with 17 of the top 20 places going to US institutions and three, Oxford, Cambridge and University College London, going to the UK.

Overall the top twenty are:

1.   Harvard University
2.   University of Chicago
3.   Stanford University
4.   University of California Berkeley
5.   Massachusetts Institute of Technology (MIT)
6.   University of Oxford
7.   University College London
8.   University of Cambridge
9.   Johns Hopkins University
10.   University of Michigan
11.   Michigan State University
12.   Yale University
13.   University of California San Diego
14.   UCLA
15.   Columbia University
16.   Duke University
17.   University of Washington
18.   Princeton University
19.   Carnegie Mellon University
20.   Washington University St Louis.

The top universities in selected countries and regions are:

Africa: University of Cape Town, South Africa 244^th

Arab Region: King Abdullah University of Science and Technology, Saudi Arabia 148^th

Asia and Southeast Asia: National University of Singapore 40^th

Australia and Oceana: Australian National University 57^th

Canada: University of Toronto 22^nd

China: Zhejiang University 85^th

France: Université Paris 6 Pierre and Marie Curie 133^rd

Germany: Ludwig Maximilians Universität München 194^th

Japan: Kyoto University 100^th

Latin America: Universidade de São Paulo 164^th

Middle East: Hebrew University of Jerusalem 110^th

South Asia: Indian Institute of Science Bangalore 420^th.

This seems plausible and sensible so it is likely that the method could be extended and improved.

Are the US and the UK really making a comeback?

The latest QS World University Rankings and the THE World Reputation Rankings have just been published. The latter will feed into the forthcoming world rankings where the two reputation indicators, research and postgraduate teaching, will account for 33 per cent of the total weighting, 

The THE reputation rankings include only 100 universities. QS is now ranking close to 1,000 universities and provides scores for 500 of them including academic reputation and employer reputation.

The publication of these rankings has led to claims that British and American universities are performing well again after  a period of stress and difficulty. In recent years we have heard a great deal about the rise of Asia and the decline of the West. Now it seems that THE and QS are telling us that things are beginning to change.

The rise of Asia has perhaps been overblown but if Asia is narrowly as Northeast Asia and Greater China then there is definitely something going on. Take a look at the record of Zhejiang University in the Leiden Ranking publications indicator. In 2006-9 Harvard produced a total of 27,422 papers and Zhejiang University 11,173. In the period 2013-16 the numbers were 33,045 for Harvard and 20,876 for Zhejiang. In seven years Zhejiang has gone from 42% of Harvard's score to 63%. It is not impossible that Zhejiang will reach parity within two decades.

We are talking about quantity here. Reaching parity for research of the highest quality and the greatest impact will take longer but here too it seems likely that within a generation universities like Peking, Zhejiang, Fudan, KAIST and the National University of Singapore will catch up with and perhaps surpass the Ivy League, the Russell Group and the Group of Eight.

The scientific advance of China and its neighbours is confirmed by data from a variety of sources, including the deployment of supercomputers,  the use of robots, and, just recently,  the Chinese Academy of Science holding its place at the top of the Nature Index.

There are caveats. Plagiarism is a serious problem and the efficiency of Chinese research culture is undermined by  cronyism and political conformity. But these are problems that are endemic, and perhaps worse, in Western universities.

So it might seem surprising that the two recent world rankings show that American and British universities are rising again. 

But perhaps it should not be too surprising. QS and THE emphasise reputation surveys, which have a weighting of 50% in the QS world rankings and 33% in THE's. There are signs that British and American universities and others in the Anglosphere are learning the reputation management game while universities in Asia are not so interested.

Take a look at the the top fifty universities in the QS academic reputation indicator, which is supposed to be about the best universities for research. The countries represented are:
US 20
UK 7
Australia 5
Canada 3
Japan 2
Singapore 2
China 2
Germany 2.

There is one each for Switzerland, Hong Kong, South Korea, Mexico, Taiwan, France and Brazil.

The top fifty universities in the QS citations per faculty indicator, a measure of research excellence, are located in:
USA 20
China 4
Switzerland 4
Netherlands 3
India  2
Korea 2
Israel 2
Hong Kong 2
Australia 2.

There is one each from Saudi Arabia, Italy, Germany, UK, Sweden, Taiwan, Singapore and Belgium.

Measuring citations is a notoriously tricky business and probably some of the high flyers in the reputation charts are genuine local heroes little known to the rest of the world. There is also now a lot of professional advice available about reputation management for those with cash to spare. Even so it is striking that British, Australian, and Canadian universities do relatively well on reputation in the QS rankings while China, Switzerland, the Netherlands, India and Israel do relatively well for citations.

For leading British universities the mismatch is very substantial. According to the 2018-19 QS world rankings, Cambridge is 2nd for academic reputation, 71st for citations, Manchester is 33rd and 221st, King's College London 47th and 159th, Edinburgh 24th and 181st. It is not surprising that British universities should perform well in rankings where there is a 40 % weighting for reputation.

The THE reputation rankings have produced some good results for several US universities.
UCLA has risen from 13th to 9th    
Cornell from 23rd to 18th                      
University of Washington from 34th to 28th                
University of Illinois Urbana-Champaign from 36th to 32nd            Carnegie Mellon from 37th to 30th                    
Georgia Institute of Technology from 48th to 44th.                             
Some of this is probably the result of a change in the distribution of survey responses. I have already pointed out that the fate of Oxford in the THE survey rankings is tied to the percentages of responses from the arts and humanities. THE have reported that their survey this year had an increased number of responses from computer science and engineering and a reduced number from the social sciences and the humanities. Sure enough, Oxford has slipped slightly while LSE has fallen five places. 

The shift to computer science and engineering in the THE survey might explain the improved reputation of Georgia Tech and Carnegie Mellon. There is, I suspect, something else going on and that is the growing obsession of some American universities with  reputation management, public relations and rankings, including the hiring of professional consultants.

In contrast, Asian universities have not done so well in the THE reputation rankings.

University of  Tokyo has fallen from 11th to 13th place    
University of Kyoto from 25th to 27th      
Osaka University from 51st to 81st         
Tsinghua University is unchanged in 14th  
Peking University 17 unchanged in 17th   
Zhejiang University has fallen from the 51-60 band to 71-80          University of Hong Kong has fallen from 39th to 40th.        

All but one of the US universities have fallen in the latest Nature Index, UCLA by 3.1%, University of Washington 1.7%, University of Illinois Urbana-Champaign 12%, Carnegie Mellon 4.8%, Georgia Tech 0.9%.

All but one of the Asian universities have risen in the Nature Index, Tokyo by 9.2%, Kyoto 15.1%, Tsinghua 9.5%, Peking 0.9%, Zhejiang 9.8%, Hong kong 25.3%.

It looks like that Western and Asian universities are diverging. The former are focussed on branding, reputation, relaxing admission criteria, searching for diversity. They are increasingly engaged with, or even obsessed with, the rankings.

Asian universities, especially in Greater China and Korea, are less concerned with rankings and public relations and more with academic excellence and research output and impact. 

As the university systems diverge it seems that two different sets of rankings are emerging to cater for the academic aspirations of different countries.

The Power of Z

This year QS has introduced several "methodological enhancements" into the THES-QS rankings. One is the use of Z-scores. Basically, this means that the mean for all universities is deducted from the raw score and the result is then divided by the standard deviation. In effect, the score represents not an absolute number but how far each university is from the average. One consequence of using Z-scores is that differences at the very top are reduced.

In principle this is not a bad idea and other rankers do it but it has produced some odd results in this case.

In the survey of academic opinion, for example, the following universities all get a maximum score of 100: Harvard, Cambridge, Oxford, Yale, Caltech, MIT, Columbia, McGill, Australian National University, Stanford, Cornell, Berkeley, Melbourne, British Columbia, National University of Singapore, Peking and Toronto.

Do THES and QS really expect us to believe that Melbourne, British Columbia and Peking are just as good at research as Harvard ? Especially since Harvard is far ahead on every one of the subject rankings?

THES has a headline about fine tuning revealing distinctions. Really?

Graduate School Rankings



US News has released its annual ranking of American graduate schools. These are subject rankings rather than holistic.



The top schools in selected categories are:



Business: Harvard, Stanford

Education: Vanderbilt

Engineering: MIT

Law: Yale

Medical Research: Harvard

Medical Primary Care: University of Washington, Seattle

Biological Sciences: Stanford

Chemistry: Caltech

Computer Science: Carnegie-Mellon, MIT, Stanford

Earth Sciences: Caltech, MIT

Mathematics: MIT

Physics: Caltech, MIT, Berkeley

Statistics:Stanford

Economics: Harvard, Princeton, Chicago, Stanford

Library and Information Sciences: University of illinois: Urbana-Champaign, University of North Carolina: Chapel Hill

English: Berkeley

Psychology: Stanford, Berkeley

History: Princeton, Stanford, Berkeley, Yale

Public Affairs: Syracuse

Fine Arts: Rhode Island School of Design

The THE Subject Rankings

The ranking seasons has drawn to a close, or at least it will when we have digested the feasibility report from the European Commission's U-Multirank project. Meanwhile, to tie up some loose ends, here are the top 3 from each of THE's subject group rankings.

Engineering and Technology

1.  Caltech
2.  MIT
3.  Princeton

Arts and Humanities

1.  Stanford
2.  Harvard
3.  Chicago

Clinical, Pre-Clinical and Health

1.  Oxford
2.  Harvard
3.  Imperial College London

Life Sciences

1.  Harvard
2.  MIT
3.  Cambridge

Physical Sciences

1.  Caltech
2.  Princeton
3.  UC Berkeley

Social Sciences

To be posted on the 17th of November.

The Shanghai Rankings 2008



Shanghai Jiao Tong University (SJTU) has just released their rankings for 2008. Compared to the THE-QS rankings, public response, especially in Asia and Australia, has been slight. This is largely because ascent and descent within the Shanghai index is minimal, a tribute to their reliability. In contrast, the THE-QS rankings, with their changes in methodology and frequent errors, arouse almost as much interest as a country's performance in the Olympics.



Still, it is instructive to check how well various universities do on the different components of the Shanghai rankings.



The current top ten are as follows:

1. Harvard
2. Stanford
3. Berkeley
4. Cambridge
5. MIT
6. Caltech
7. Columbia
8. Princeton
9. Chicago
10. Oxford

The Shanghai index includes two categories based on Nobel prizes and Fields medals. These measure the quality of research that might have been produced decades ago. Looking at the other criteria gives a rather different picture of current research.

It is interesting to see what happens to these ten if we rank them according to SJTU's PUB category, the total number of articles indexed in the Science Citation Index-Expanded (SCIE) and Social Science Citation Index (SSCI) in 2007. The SSCI gets a double weighting.

Harvard remains at number 1

Stanford goes down to number 8

Berkeley goes down to 11

Cambridge goes down to 23

MIT is down at 34

Caltech tumbles to 86

Columbia is down just a bit at 10

Princeton crashes to 120

Chicago falls to 72

Oxford goes down to 18

If this category represents current research output then it looks as though some American universities and Oxbridge have entered a period of decline. Of course, Caltech and MIT may suffer from the PUB category including social science research but would that explain why Princeton and Chicago are now apparently producing a relatively small amount of research?

The top ten for PUB is

1. Harvard

2. Tokyo

3. Toronto

4. University of Michigan

5. UCLA

6. University of Washington

7. Stanford

8. Kyoto

9. Columbia

10. Berkeley

New QS Rankings

QS have just released their Life Sciences rankings based on their employer and academic surveys and citations per paper.

Here are the top five for medicine, biology and psychology.

Medicine

1.  Harvard
2.  Cambridge
3.  MIT
4.  Oxford
5.  Stanford

Biological Sciences

1.  Harvard
2.  MIT
3.  Cambridge
4.  Oxford
5.  Stanford

Psychology

1.  Harvard
2.  Cambridge
3.  Stanford
4.  Oxford
5.  UC Berkeley