The War of the THE-QS Succession

There is a comment by Nunzio Quacquarelli on the QS topuniversities rankings blog.

Here is an extract:

"In October 2009, QS and THE ended their collaboration under which THE was licensed to publish the QS results known as “Times Higher Education (THE) – QS World University Rankings”. Since then, THE have announced they intend to produce their own rankings and have been systematically critical of QS’ methodology as part of their explanation for the split. This is surprising; THE consistently praised the QS methodology throughout the six-year publishing collaboration. Indeed, their former publishing director described it as one of the best partnerships in the history of THE.

Similarly, Ann Mroz, Editor of THE wrote in October 2008: "These rankings use an unprecedented amount of data to deliver the most accurate measure available of the world’s best universities, and of the strength of different nations’ university systems. They are important for governments wanting to gauge the progress of their education systems, and are used in planning by universities across the world."

Phil Baty, Associate Editor of THE wrote only on October 10 2009: “Congratulations on a highly successful campaign on the rankings again this year. The internet is buzzing.” Yet it seems our objectives and methodological principles have subsequently diverged. QS will continue to produce our rankings using citation data from the Scopus database of Elsevier. THE have decided to align themselves with Thomson Reuters’ academic citation database."

New Webometrics Ranking

The new Webometrics ranking is out.

Some interesting points

The top 20 are all in the USA.

The best non-US university is Cambridge at 27.

British universities do not do very well. Oxford is at 37, University College London at 57 and Imperial at 157 while Webometrics joins the anti-LSE conspiracy by putting it at 234.

The top European universities seem to be in the North -- Edinburgh, Oslo, Helsinki. Something about the cold weather?

Regional Rankings

Best in Latin America: Sao Paulo
Best in Europe: Cambridge
Best in Central and Eastern Europe: Charles University
Best in Asia: Tokyo
Best in South East Asia: National University of Singapore
Best in South Asia: Indian Institute of Technology Bombay
Best in the Arab World: King Saud University
Best in Oceania: Australian National University
Best in Africa: Cape Town

Finally Israeli universities should get a special award for mobility. They manage to be in Asia and Europe at the same time.

More Rising Stars



The rise of China to scientific superpower status has been well documented. See here for a report by Jonathon Adams, Christopher King and Nan Ma.


This can be confirmed by a simple search of the Scopus database which reveals 38,360 scientific publications from China in 1999 compared to 250,452 in 2009. For the United States the corresponding figures were 311,879 and 367,641.


The UK, France and Germany recorded modest increases over the decade while research output in Russia actually fell.

A certain amount of caution is in order. These figures refer to the quantity of research, not to its quality and China does have a large, although stable, population. Still, the West has cause to be concerned.


Some other countries have improved quite considerably over the decade. Korea, India, Australia and Hong Kong have doubled or nearly doubled and Thailand has more than tripled its research output.


It is especially noticeable that Malaysia is catching with Singapore. The former had 1,235 publications in 1999 and the latter 4,538 . In 2009 the figures were 7,834 and 10,993.


However, the prize f0r rapid growth goes to Iran which had 1,351 publications in 1999 and 19,088 in 2009. Compare this with Israel: 11,918 and 16,335.


If research in Iran goes on advancing at this rate and if other countries in the region also develop their scientific capabilities and if the ultra -orthodox extend their assault on reason and science into Israeli schools and universities, it looks as though Hamas and Hezbollah are going to the least of Israel's problems.

Who is Biased?

There has been a lot of discussion about university rankings recently. In Times Higher Education, Phil Baty refers to a comment in the satirical magazine Private Eye about the forthcoming European Union rankings. Why spend public money on the ranking of universities when there are already two recognised rankings? Perhaps, it has something to do with the striking absence of continental European universities from the upper reaches of the THE-QS and Shanghai rankings.

Baty claims to be less cynical than Private Eye. He says that:


"While I am sure CHERPA will strive to be fully independent, it is a group made up exclusively of European universities, and was set up in direct response to Europe's poor showing in the current rankings, so some suspicion is inevitable.

More serious, and entertaining, questions have been asked over other rankings. Russia's RatER raised eyebrows for putting Moscow State University in fifth place, ahead of Harvard and Cambridge, and a ranking from France's Mines ParisTech has been ridiculed for putting five French universities into the top 20."

However, one should not assume that the forthcoming THE rankings will be biased because

"these concerns give THE great confidence - as an independent magazine we are free from the influence of any institution or authority.

We are accountable only to our readers - an increasingly international community of thousands of academics and university administrators. "


But this raises certain questions. Is THE not accountable to the company that owns it? Another question is that "increasingly international" community. "Increasingly" from what to what? And who are those administrators responsible to?


The national bias of the Paris Mines ranking is indisputable. There the top French institution is in sixth place. In the most recent THE-QS rankings the top French institution was 38th, in the Russian RaTER rankinigs 36th, in the Shanghai Aacademic Ranking of World Universities 40th, in the Taiwan rankings 88th and in Webometrics 129th.



The bias of the Russian rankings is even more glaring. They put Moscow State University in 5th place. In no other ranking did they even get intio the top fifty.


I am not suggesting that there is anything dishonest about the Paris and Russian rankings. The Paris rankings is as transparent as it is possible to be. It simply counts the number of CEOs of top 500 companies who attended particular schools. Everything is in the public record. The Russian rankings are not so transparent. The problem here is that its questionnaire contains many references to indicators specific to Russia and the CIS. It is also written in a style that many people would find close to incomprehensible.


The bias in the Paris and Russian rankings stems not from dishonesty but from the choice of criteria that are likely to give an advantage to universities in their countries while downplaying or ignoring those in which their countries are not so strong.


In contrast, the Shanghai, Taiwan, Webometrics, and Scimargo rankings appear to have no home country bias at all.

What about THE? The old THE- QS rankings were pretty obviously biased in favour of British universities. Last year it had Cambridge in second place. The Shanghai rankings put it in 4th place, although that will not be sustained as the impact of old Nobel winners fades. In the Paris Mines ranking it was 7th, in the Russian rankings 8th, in the Taiwan rankings 15th, in Webometrics 22nd , in Scimargo 34th and in the Leiden green index (the size-independent, field-normalized average impact) 37th.


We will see if Cambridge and Imperial College maintain their suspiciously high places in the new THE rankings. If they start slipping a little I will be inclined to agree that THE has in fact overcome its anglocentric bias.

Article in the Chronicle

The Chronicle of Higher Education has a substantial article on world rankings by Aisha Labi. She describes a number of recent developments

• The European Union "began moving ahead in the development of a nuanced and more complex rankings system". No doubt it will soon start moving as fast as Concorde.
• A Russian ranking was met with derision, even in Russia.
• THE and QS "had an acrimonious split, with each now promising to produce a superior product."

There are some comments from Phil Baty of THE who describes the old rankings as "no longer fit for purpose". He indicates that the new THE rankings will see two improvements. One is a new academic survey that will be larger, better targeted and more representative. The other is some sort of extra weighting for the social science citations.

Meanwhile Ben Sowter of QS defends

"its [QS] continuing emphasis on a peer-review component, adding that it seeks increased input from academics and aims to increase response numbers through measures such as translated surveys for academics in non-English-speaking institutions.

"Of all the measures that different rankings are using at a global level, from my perspective peer review is the one that is fairest to universities with different disciplines," he says. The use of peer reviews "enables institutions with great strengths in the arts and humanities to shine in a way that they are not able to in other measures." "

So it looks like there will be a survey war with THE flaunting the size of its sample and QS stressing the diversity of theirs.

Of course, the last word goes to Nian Cai Liu of Shanghai Jiao Tong University: "We think that more and diversified rankings are good for the higher-education community and the general public in general,"

New Report

A comprehensive and interesting report on university rankings from the Swedish National Agency for Higher Education is available here.

Thanks to Beerkens' Blog

Something from QS

Finally something has appeared on the QS Intelligence Unit Blog. Ben Sowter writes:


"The QS World University Rankings will continue to be published in 2010, albeit through a number of new channels which we are working on. At present, there are no plans to alter the methodology, in fact it seems important to maintain some comparability in a time when a number of new and different interpretations are going to emerge. So in 2010, we are focused on improving our engagement with institutions, redesigning some of our data collection systems to be more user-friendly and intuitive, and our work in specific regional and discipline oriented contexts."

I am not sure that keeping the methodology is a good idea but it is understandable. However, even with the same basic methods there are a couple of minor changes that might help QS find a niche in the "holistic" ranking market as Times Higher appears to focus on making fine distinctions among leading research institutions. One would be to use the academic survey to ask about general excellence or activities other than research. The other would be to remove non-teaching faculty from the faculty totals when calculating faculty student ratio. As it is, the QS rankings are heavily weighted towards research, with an academic survey asking about research, an indicator based on citations and a teaching resources measure that includes researchers who never teach.

Now that QS have done an Asian ranking and are apparently preparing Arab and Latin American ones, they could also also outflank THE by preparing survey forms in additional languages. They offered a Spanish option last year. They ought to have the resources to produce forms in Chinese, French, German. Arabic and Japanese.

Opinion Surveys in University Rankings

In this week's Times Higher Education, Phil Baty discusses the role of reputational surveys in university ranking. It was a distinctive feature of the THE-QS rankings that they devoted 40 % of the weighting to a survey of academic opinion about the research excellence of universities. Baty points out that "The reputation survey used in the now-defunct Times Higher Education-QS World University Rankings was one of its most controversial elements: a survey of a tiny number of academics should not determine 40 per cent of a university's score".

It was not so much that a tiny number of academics was surveyed but that a tiny number responded and that this (relatively) tiny number was heavily biased towards particular countries and regions. A very obvious effect of the survey was to boost the position of Oxford and Cambridge well beyond anything they would have attained on indicators based on other more objective factors.

Whether THE can produce a better survey remains to be seen. But at least they have at last stopped calling it a peer review.

The Rise of China

An article in the Financial Times describes the impressive growth of scientific research in China


"China has experienced the strongest growth in scientific research over the past three decades of any country, according to figures compiled for the Financial Times, and the pace shows no sign of slowing.

Jonathan Adams, research evaluation director at Thomson Reuters, said China’s “awe-inspiring” growth had put it in second place to the US – and if it continues on its trajectory it will be the largest producer of scientific knowledge by 2020.

Thomson Reuters, which indexes scientific papers from 10,500 journals worldwide, analysed the performance of four emerging markets countries: Brazil, Russia, India and China, over the past 30 years."

In contrast, the performance of Indian universities and institutes has been rather limp:

" A symptom of this is the poor performance of India in international comparisons of university standards. The 2009 Asian University Rankings, prepared by the higher education consultancy QS, shows the top Indian institution to be IIT Bombay at number 30; 10 universities in China and Hong Kong are higher in the table.

Part of India’s academic problem may be the way red tape ties up its universities, says Ben Sowter, head of the QS intelligence unit. Another issue is that the best institutions are so overwhelmed with applications from would-be students and faculty within India that they do not cultivate the international outlook essential for world-class universities. This looks set to change as India’s human resource minister has stepped up efforts to build links with US and UK institutions. "

A couple of observations. China's research output might not be so impressive if population were taken into account. I also wonder if India's relatively poor performance is the result of a failure to cultivate an international outlook. Is China really so much more international than India? Is it possible that other factors are more important?

Batting for Britain

A notorious feature of the THE-QS rankings was its over-valuation of British and Australian universities. It would seem that Times Higher and Thomson Reuters are not really bothered by this. Indeed it looks like they are set on course to add to this bias in their new rankings, at least as far as British universities are concerned. An opinion piece by Jonathon Adams, the Director of Research Evaluation at Thomson Reuters, echoes previous comments in THE by lamenting the maltreatment of the London School of Economics in the old league table.

"The London School of Economics is generally agreed to be an outstanding institution globally. But how can we judge that? A lot of people would like to study there. If you wanted an informed opinion, you would consult the people who work there. A lot of people who have been there have gone on to great things. These are good indicators that the place is intellectually vibrant and delivers excellent teaching, and those values are endorsed internationally.


Good, but not perfect. Three major problems spring to mind. First, that quick summary tells us there are many ways in which we may value what a university does. It is a knowledge business and a source for teaching, research and dissemination to users. Second, the LSE is a specialist. Its astronomy is weak, so we need to consider subject portfolio. And, third, what will we measure? I need an informed expert to confirm my judgment, but as I can't send my expert to every institution, I need a proxy indicator (not a "metric": an indicator).

Our view of the LSE does not translate readily into anything useful unless we are careful and we make sure our information is appropriate. The LSE stood at only 67th in the last Times Higher Education-QS World University Rankings - some mistake surely? Yes, and quite a big one. LSE academics publish papers in social and economic sciences, which have lower citation rates than the natural sciences; so on the simple "citations per paper" used by QS in analysing the Scopus publications data, it slipped way down the list. Not a good way of comparing it with nearby King's College London, which has a huge medical school.

We need a lot more information than has typically been gathered before we can build an even halfway sensible picture of what a university is doing."

The problem with this is that there are many institutions that scored lower than LSE in the rankings that are agreed by some people somewhere to be outstanding. The “good indicators” raise more questions. A lot of people want to study at LSE. Is that because of its intrinsic merits or shrewd marketing? And who is the "you" who would consult the LSE? A lot of its alumni and alumnae have done great things? No doubt many have become MPs, civil servants, university administrators and CEOs but given the current moral condition of British politics and the performance of the British and European economies that might not be something to be proud of.

It is difficult to concur with the claim that LSE has been treated unfairly in previous rankings. In 2009 they were number five for social sciences and 32nd for arts and humanities. They got top marks for international faculty and international students and in the employer review. They did somewhat less well in the academic survey, which had a disproportionate number of respondents from Britain and Commonwealth countries with large numbers of British alumni and alumae, but that is surely to be expected when LSE excels in a very limited range of disciplines.

LSE also did badly in the citations per faculty indicator (not citations per paper – QS used that for their Asian rankings, not the world rankings) partly because it is a specialist social science institution and it is conventional in the social sciences to produce fewer papers and to cite them less frequently but also because LSE actually does not produce as much social science research, as measured by Scopus and ISi publications, as general institutions such as the Universities of Manchester, Birmingham, Harvard, Yale, Chicago, Toronto, Melbourne and Sao Paulo.

It is difficult to think of changes in the structure or content of the rankings that would benefit LSE but not a host of others. Giving extra weighting to social science publications is an excellent idea and would boost LSE relative to King’s College or Imperial College (I wonder if THE is prepared to let Imperial slip a few places) but it would probably help US state universities and European universities even more. Counting “contributions to society”, such as sitting on committees and commissions and boards of directors would help LSE a bit but might well help Japanese universities and the French grandes ecoles a lot more.

LSE is a narrowly based specialist institution and QS gave it as much as or more than it deserved by ranking it highly in the social science and arts and humanities categories and putting it in the top 100 in the general world rankings. It is good at what it does but it does not do all that much. It would be a shame if the rankings are going to be restructured to promote it beyond its real merits.

The other item is a Rick Trainor’s review of Robert Zemsky’s book Making Reform Work. In the course of his review Trainor, who is president of King’s College London, says that:

"Most fundamentally, while the US debate is premised on a clear and widespread belief in the great, if imperilled, merits of the US system, British opinion often pays too little attention to the successes of UK universities, even in comparison with their US counterparts. For example, British commentators often overlook UK universities' superior completion rates, the greater rigour concerning undergraduate assessment inherent in the existence of an external examiner system, their greater ability (allowing for the much greater size of the US population and its university system) to attract overseas students and, as suggested by the Sainsbury report, their arguably superior record in commercialisation.

Of course, this is not to suggest that the UK higher education system is perfect, any more than US universities are. Nonetheless, there has been too little recognition in the UK of its high international research standing (aided by rises in public investment in recent years), despite persisting American strength and rapidly rising competition from countries such as China and India. Likewise, the UK system receives too little credit domestically for its success in protecting standards despite the huge increase in UK student numbers during the past 25 years. Similarly too few observers on this side of the Atlantic have learned one of the basic lessons propounded by Zemsky: that outstanding achievement in higher education depends on adequate resources - for teaching (which was substantially underfunded, even before the UK's public expenditure crisis began) as well as for research. "

This is a rather odd set of claims. Superior completion rate? I wonder how that happened. Greater rigour because of the external examiner system? Really? Do British universities still have a high international research standing? Just look at their performance on the Shanghai rankings, after removing the cushion of the thirty percent weighting for Nobel and Fields laureates. Have standards really been protected? Would more money make any difference?

It is beginning to look as though an implicit consensus is developing in the British higher educational establishment that the rankings should reflect its self-serving view of the merits of British higher education and that they have an important role to play in fending off the economic crisis. It appears that THE and Thomson Reuters are only too happy to oblige.

Is anything happening at QS?

Although there has been a lot of activity, so far mainly rhetorical, at Times Higher Education and Thomson Reuters about their forthcoming rankings, nothing has been heard from QS apart from an advert for a manager of a university ranking for Latin America and Iberia.

Nothing has been added to the 2010 ranking news page since December and Ben Sowter’s blog has been silent for a month.

Are they preparing a response to THE or are they just fading away?

The Kiplinger Rankings are Out


Kiplinger has produced its 2009-2010 ranking of US universities. This is very much a student consumer ranking that measures the value for money delivered by each institution. It is based on information about student debt, tuition costs, financial aid, gender ratio, class size and average SAT scores, among others.

There is no doubt a lot of room for argument about the validity of the data and how the indicators were weighted but this sort of index does seem very useful.

I am wondering if something like this can be incorporated into existing international rankings. A lot of Kiplingers's data would be difficult or impossible to obtain outside the US but information about things like tuition fees, gender ratio, class size, and number of books in the library is widely available.

The top five private universities are:

1. Caltech
2. Princeton
3. Yale
4. Rice
5. Harvard

The top five public schools (for out-of-state students) are:

1. SUNY Binghamton
2. SUNY College at Geneseo
3. University of North Carolina at Chapel Hill
4. University of Florida
5. College of New Jersey

The European Union Rankings

The European Union is trying to develop a new ranking system to rival the existing ones. The motivation is fairly transparent. The object, as reported in the EUObserver is "to improve the ranking of European universities and improve Europe's economic power".

The EUObserver provides an excellent and succinct summary of the forces underlying the universities ranking boom.

"This means the rankings are increasingly receiving more attention for different specific purposes: Students use them to short-list their choice of university; public and private institutions use them to decide on funding allocations; universities use them to promote themselves; while some politicians use them as a measure of national economic achievements or aspirations. "

It seems that planning for the new rankings took place in the second half of 2009 and that in the first half of 2010 it will be tested on 150 institutions around the world, but only for engineering and business studies.

At that rate, THE, QS and Shanghai Jiao Tong University have nothing to worry about.

Global Institutional Profiles Project

Thomson Reuters have set up a new site here. It contains information, although not much so far, about the new Times Higher ranking system.

They will "address industry concerns over current profile systems... The 21st century research institution has many fluid layers, and Thomson Reuters is committed to developing an equally robust and dynamic dataset".

Notice that they are talking about research institutions as though universities do nothing but research and that they refer to higher education as an industry.

The page provides some hints about what might be included in the forthcoming rankings: peer review, scholarly outputs, citation patterns, funding levels and faculty characteristics.

I do not know whether there is any significance in the absence of internationalisation and faculty student ratio from the list.

The page could have done with some editing. There are too many barely meaningful adjectives -- robust, dynamic, flexible, data-driven, globally significant. And exactly what is a "fluid layer"?

Webometrics Out Soon

The next edition of the Webometrics rankings will be published at the end of January. Watch this space.

www.webometrics.info

The THE Reputational Survey

Thomson Reuters, acting on behalf of Times Higher Education, have published an open letter to university administrators announcing the development of a new ranking system. They promise much. The new ranking is the only one that "seeks to fundamentally change the way data is collected and analyzed". They believe "this development underscores a major breakthrough within the rankings dialogue".

There is some good news. Finally, the inaccurate term "peer review" is being dropped to be replaced by "reputational survey". Also, according to a comment on a previous post from Phil Baty, Deputy Editor at THE, "we will be looking to focus the survey more on non-research elements. It allows us to get at the less tangible elements of university activity that can not be measured through numbers." This is very sensible.

The two points above are welcome but I still do not see anything very revolutionary about the forthcoming survey.

There is another question. Thomson Reuters are asking university administrators to encourage their researchers and colleagues to take part. This would seem to introduce an element of bias into the survey from the very start. How many university administrators will read the open letter? How many will act on it? Will there be as many in Japan as in England?

What does this say about Nigerian Universities?

According to University World News, Nigerian banks prefer to recruit holders of polytechnic diplomas rather than university graduates. One bank manager said that diploma holders could perform most of the tasks normally done by graduates for less pay and did not require extensive computer literacy training.

I wonder if this would show up in any of the current university rankings.

Something about Ipsos MORI

Anyone interested in Ipsos MORI, the company appointed to conduct a survey of academic opinion for Times Higher Education can go here or have a look at the column to the left.

It seems that they have a number of junior staff outside the UK, or at least a lot of telephone interviewers, so that does to some extent allay one of my concerns about the company.

However the biodata for the senior staff is rather disconcerting. Some snippets:

"after graduating from Oxford University"
"has worked closely with both Conservative and Labour ministers ... as well as a wide range of local authorities and NHS trusts"
"served as Finance Director of BMRB"
"started her career at the BBC"
"has been a User Fellow at the centre for the Analysis of Social Exclusion at LSE and spent time working in the Prime Minister's Stategy Unit"
"a member of the MRQSA council"
"a full member of the Chartered Institute of Marketing"
"member of the council of Roehamton University"
"has chaired a number of round-table discussions with senior peers"
"has a BSc, MSc and MBA from Imperial College"

Very British (and just a little bit cosmopolitan -- "speaks five European languages", "always busy cookng up the next plan to explore to a far flung destination"), very establishment, rather politically correct and perhaps a little inward looking -- in much of the world, working with British government ministers, peers and the NHS is not something you would want to boast about.

Will a survey carried out by such a group reveal that in most respects places like Oxford, LSE and Imperial College are performing increasingly less well than leading American and Japanese universities?

In case you thought ranking was just a bit of fun


NTU ACCUSES SPANISH RANKING INSTITUTE OF LIBEL

The National University of Taiwan is protesting about a statement on the webometrics site that some universities had resorted to 'bad practices."

The practices consisted of hosting papers written by authors at other institutions. As well as NTU, webometrics referred to the University of Sao Paulo.

Other universities are listed as having more than one webdomain. These include the University of Maryland, the University of Manchester, Yonsei University, Korea University, Chiang Mai University, The Indian Institutes of Technology at Delhi and Kharagpur, Kuwait University and the University of Bahrein

Does Size Really Matter?
Times Higher Education (THE) are keeping the "peer review" but possibly with new questions. According to a recent article they will be using the British pollsters Ipsos MORI to collect data.

"So we are delighted to confirm that for the 2010 Times Higher Education World University Rankings, our new rankings partner Thomson Reuters has commissioned one of the world's leading polling companies, Ipsos Mori, to carry out research to support the peer-review element of the tables. Using a professional polling company means that we can inject proper targeting and transparency into the process while ensuring that we get a much larger response rate than in the past - the aim is for at least 25,000 responses in 2010. It also means that the questions in the opinion survey can be carefully crafted to elicit meaningful and consistent responses while ensuring that every respondent knows what is being asked of them. "

THE seems to be overly concerned with the number of respondents, claiming that the 9,000 plus of the 2009 THE-QS rankings was an inadequate number to represent the millions of academics of one sort or another around the world. They are right to be concerned but the number of respondents is not the main determinant of the validity of any survey. What matters more is the extent to which the sample is representative of the population about which data is sought. If THE and if Ipsos MORI are going to do no more than get a lot of people to fill out online forms then their new survey will be little better than the old one.

If the rankings industry is going to descend into a squabble about who's got the biggest survey then QS might be able to trump THE. They could revive their retired respondents from 2004-06, purchase a large stash of email addresses from Mardev, make the survey more user-friendly (tick boxes instead of typing names) and they might well be able to get above the 25,000 mark.

The choice of Ipsos MORI, whose offices are in London, Harrow, Manchester, Edinburgh, Belfast and Dublin might be an indicator of a narrowing of vision. THE's editorial board, which seems to have become more active of late, is predominantly British with a heavy bias towards officialdom. Discussion about rankings in THES seems rather anglocentric. A subtle slip was Phil Baty's recent reference to "overseas" universities. They may be overseas to you but you are overseas to them and everybody else.

Whither the QS Rankings?

While Times Higher Education is looking around for a new methodology, QS, judging from a recent conversation with Ben Sowter and Tony Martin and comments on its website, appears set on continuing with the old system perhaps with a bit of tweaking.

The need to maintain some sort of continuity is understandable, especially after the yo-yoing of some universities in recent editions of the THE-QS rankings. However, criticism of the rankings is such that it would seem a good idea to seize the opportunity to make some simple changes.

The least liked element of the THE-QS rankings of 2004-09 was the "peer review". It had, being based on the mailing lists of a Singapore-based publishing company with links to Imperial College London, an obvious geographical bias. The declared response rate was too low to meet conventional standards of face validity. Its weighting was too high. As a survey of research expertise it was quite redundant since citations are a far better measure of research impact and quality.

Furthermore, the "peer review" added to the overemphasis on research. The THE-QS rankings gave a 20 % weighting to citations, the faculty student ratio gave a big and obvious boost to universities with large numbers of non-teaching research-only faculty and then there was 40% for a research-based survey.

I would like to suggest a simple change. Keep the survey of academic opinion (and stop calling it a peer review because it is nothing of the sort) but use it to assess the general excellence or reputation, perhaps including teaching and student satisfaction, of universities. It is not credible that someone with a functioning mouse can sign up for the World Scientific list and became competent to assess the research performance of universities but he or she might have some idea of the general reputation of institutions. This would require minimal changes to the current procedure: all that is needed is to change the questions.

A couple of other refinements might be in order. The division of the academic world into three super-regions for weighting purposes is too crude. Latin America, Africa, Southwest Asia and Southeast Asia deserve to be treated as separate regions.

Telling everybody that you have sent 180,000 e-mails is asking for trouble if you are going to get a negligible response. It would be better to use the World Scientific lists to accumulate a list of people willing to participate in the survey, combine it with names collected from various events and then send out the survey. If nothing else, the response rate would be a little more respectable.

A Ranking from SCImago

Tekmillinen Korkeakoulu-Tekniska Hogskolen in the top 400
Ollscoil Luimnigh just misses top 1000
Good showing by Debreceni Egyetem

SCImago, a research group based on Spanish universities has published SIR, SCImago Institutions Rankings, has published its 2009 report which includes a ranking of 2124 institutions, including research centres as well as universities.

There are five indicators, one of which, the number of publications in Scopus-indexed journals, is used for ranking.

There are some positive things about this ranking. It uses Scopus data: anything which reduces the emerging Thomson Reuters monopoly is welcome. It ranks more than two thousand places. It is quite transparent: I have checked a few institutions and the figures seem accurate.

The most striking thing about this index is that it shows that a vast amount of research is being done outside universities. The top three places for research output go to government research centres in France, China and Russia, lending support to French claims that current ranking systems fail to take account of their distinctive system of higher education and research.

One irritating thing about these rankings is the eccentric naming policy. Japanese universities are referred to by their Japanese names but Korean and Chinese ones are in English. Some New Zealand universities are listed with English and Maori names but the Universities of Auckland and Waikato are only in English. Dublin Institute of Technology is in Irish but Trinity College Dublin is in English. Some Saudi institutions are in English and some in Arabic. Three Israeli universities are in Catalan (or German without the umlaut!)

An Ancient Dinosaur Reborn?

Times Higher Education and some of its readers seem to be concerned about what they think is the low position of the London School of Economics (LSE) in previous rankings. It is true that institutions that specialise in the social sciences and humanities suffer from any ranking based on citations and publications since they produce longer and fewer papers with fewer authors and more books and use citations more sparingly than do those in the natural sciences and medicine. However, this seems to affect universities like Yale and Princeton as much as LSE. It would be quite simple for rankers to use some sort of weighting to reduce the disadvantage of such places and it would be an improvement if THE were to do this in any future ranking system.

But the concern with LSE is rather suspicious. Should specialist institutions be regarded as the equal of universities that excel in all disciplines? Perhaps THE should also think about the overrating of Oxford and Cambridge (take away the peer review from the THE-QS rankings of 2004-09 or the alumni and awards indicators from the Shanghai rankings and see where they are) as they discuss their new system.

It might be worth recalling a comment made by a THE reader back in October.


"It is always quite interesting to see that British institutions are still regarded as the top of the world. (I just compare it with the FT MBA rankings as well, where UK institutions dominate all rankings). As someone from the continent I only can say "Long live the British Empire!" It seems to me that the stereotype of British domination is still very alive in UK. A closer look at the British economy, engineering and scientifc achievements, however, reveals the the mental fraud. Travelling across UK, I often realize that UK is frozen in time. Sometimes the technology, housing and machines are like from a 3rd world. London Metro is like from 1899. Trains across the country are like in the 30s. Communication technology is like mid of last century. I would have reasoned that with all the best universities, as you have figured out yourself, only bright scientist and engineers evolve. It's an illusion. Travel across Europe, marvel at French TGV trains, drive German cars and have a look at Spanish solar power plants and you will see that others, with officially inferior schooling systems, have achieved far more. Your university ranking is an illusion, buried in century long self-perception of world dominance. I am sorry to write that, but it is true. The British dominance is long gone, same with academic instituions. Your ranking list is an ancient dinosaur."

Whither the Times Higher Rankings?

Times Higher Education has announced that it will be producing a new ranking system to replace the THE-QS World University Rankings.

THE does not seem to have much idea about where it is going. Its advisory committe (it would be interesting to find out who they are) is reported to have complained that the number of respondents in the peer review is too small and that the citations indicator is biased against the social sciences and the humanities.

Neither of these is very helpful. The small number of respondents is not for lack of trying by QS. They have been sending out nearly 200,000 e-mails a year. I doubt if there is very much anyone can do get many more respondents. What could be done and should be done is to improve the validity of the survey by clearly identifying the group whose opinion is being sought or using databases that are less obviously biased. The second problem could be dealt with quite easily by assigning appropriate weighting to the various dsicipline clusters.

THE has also published comments from readers about future directions for its rankings. Some of these seem unaware of the basic methods of the THE-QS rankings. One, for example wants to see an "increased number of academics interviewed" -- QS never interviewed anyone for its survey. Others want the rankings to include criteria that are of limited global comparabilty such as starting salaries or graduate job prospects.

Several readers are unhappy with what they feel is the unfairly low position of LSE. This would seem misplaced. The rankings are supposed to be of universities not of research institutes and offering a full range of courses ought to be a significant element in the assessment of a university.

Other readers are sceptical about the significance of internationalisation and there appears to be division about whether citations are an adaequate nmeasure of research quality.

The response so far appears to be predominantly British. If THE are going to listen to their readers it is likely that the obvious pro-British and even pro-Oxbridge bias of the old rankings will continue.

Anyone interested in taking part in a survey by Thomson Reuters and THE can do so by going here.

Announcing GRAPE: Global Ranking of Academic Performance

I am surprised that nobody has thought of doing this before.

There are now six international university ranking systems and five of these, World University Rankings (THE-QS London), Academic Ranking of World Universities (Shanghai Jiao Tong University), International Professional Ranking of Higher Education Institutions (Ecole des Mines de Paris), Performance Ranking of Scientific Papers for World Universities (Taiwan) and Global Ranking of Universities (Russia), provide a numerical score. I have simply added the scores for all universities that were in the top 30 on any one of these, converting the top score for The Paris and Taiwan rankings into 100. The top scorer in the composite ranking was of course Harvard which was awarded a composite score of 100. The other scores were then adjusted accordingly. Yale, Imperial College London, Northewestern and Waseda were not included in the Russian rankings so they were ranked according to their total score for the other four.

There are some interesting results. The University of Tokyo comes in second, with a good record for recent research and for CEOs of big companies. University College London and Imperial College perform poorly. Oxford and Cambridge are slipping a bit and Australian universities do badly.

Here then are the top 30 with the combined scores.

1. Harvard 100

2. University of Tokyo 79.91

3. MIT 74.05

4. Stanford 71.21

5. Columbia 62.61

6. Cambridge 61.87

7. Caltech 59.81

8. Oxford 59.29

9. University of Pennsylvania 57.65

10. Yale 57.00

11. Johns Hopkins 56.7

12. University of California Berkeley 55.22

13. Chicago 54.87

14. Cornell 53.42

15. Kyoto 53.42

16 . UCLA 53.07

17. Duke 52.66

18. Princeton 51.49

19. University College London 50.46

20. Michigan 49.19

21. Imperial College London 47.74

22. University of Washington Seattle 47.08

23. University of California San Diego 45.60

24. Toronto 45.46

25. Northwestern 46.09

26. University of Wisconsin Madison 42.98

27. Manchester 42.49

28. Edinburgh 42.46

29. McGill 42.41

30. University of Illinois Urbana Champagne 41.69

It is also intersting to look at the correlations between the specific rankings and the combined scores. The correlations (top 30 institutions only) are as follows.

Paris .818
Shanghai .815
Taiwan .773
Russia .652
THE-QS .491