New Rankings on the Way

Times Higher Education have announced that their new rankings will be published on September 16th and have revealed the outline of their methodology.

The rankings will include five groups of indicators as follows:

A new broad category, called "Teaching - the learning environment", will be
given a weighting of 30 per cent.

Using five separate indicators, this category will use data on an institution's income, staff-student ratios and undergraduate-postgraduate mix, as well as the results of the first-ever global academic reputation survey examining the quality of teaching.

A further 30 per cent of the final rankings score will be based on another new indicator, "Research - volume, income and reputation".

This category will use four separate indicators, including data on research income, research output (measured by publications in leading peer-reviewed journals) and the results of the academic reputation survey relating to research.

The highest-weighted category is "Citations - research influence".

This category will examine a university's research influence, measured by the number of times its published work is cited in other academics' papers.

Based on the 12,000 journals indexed by Thomson Reuters' Web of Science, and taken over a five-year period, the citations data will be normalised to take account
of different volumes of citations between disciplines.

Reflecting the high levels of correlation between citations data and research excellence, this category will be given a weighting of 32.5 per cent.

A fourth category, "International mix - staff and students", will use data on the proportion of international staff and students on campus. This indicator will be given a 5 per cent weighting.

Knowledge transfer activities will be reflected in "Industry income - innovation", a new category worth 2.5 per cent of the total rankings score. This will be based on just one measure in 2010 - research income from industry.

There is still a lot apparently left undecided such as the distribution of indicators within the groups and exactly what faculty will count for scaling. In general, though, the broad outlines of the new ranking look promising with the exception of the large weighting -- nearly one third -- assigned to a single indication, citations. Certainly citations are a good measure of research impact and more difficult to manipulate than some others but putting so much emphasis on just one indicator will be a problem for face validity and will also amplify any data entry errors should they occur.

Finally, I wonder if it is a good idea to refer to the "seventh annual survey". Wouldn't it better to start all over again with the First THE Rankings?

From THE

I am reproducing Phil Baty's column from Times Higher Education in its entirety

One of the things that I have been keen to do as editor of the Times Higher
Education World University Rankings is to engage as much as possible with our
harshest critics.

Our editorial board was trenchant in its criticism of our old rankings. In particular, Ian Diamond, principal of the University of Aberdeen and former chief executive of the Economic and Social Research Council, was scathing about our use of research citations.

The old system failed to normalise data to take account of the dramatically different citation volumes between different disciplines, he said - unfairly hitting strong work in fields with lower average figures. We listened, learned and have corrected this
weakness for the 2010 rankings.

Another strong critic is blogger Richard Holmes, an academic at the Universiti Teknologi MARA in Malaysia. Through his University Ranking Watch blog, he has perhaps done more than anyone to highlight the weaknesses in existing systems: indeed, he highlighted many of the problems that helped convince us to develop a new methodology with a new data provider, Thomson Reuters.

He has given us many helpful suggestions as we develop our improved methodology. For example, he advised that we should reduce the weighting given to the proportion of international students on campus, and we agreed. He added that we should increase the weighting given to our new teaching indicators, and again we concurred.

Of course, there are many elements that he and others will continue to disagree with us on, and we welcome that. We are not seeking anyone's endorsement. We simply ask for open engagement - including criticism - and we expect that process will continue long after the new tables are published.

There are still issues to be resolved but it does appear that the new THE rankings are making progress on several fronts. There is a group of indicators that attempts to measure teaching effectiveness. The weighting given to international students, an indicator that is easily manipulable and that has had very negative backwash effects, has been reduced. The inclusion of funding as a criterion, while obviously favouring wealthy regions, does measure an important input. The weighting assigned to the subjective academic survey has been reduced and it is now drawn from a clearly defined and at least moderately qualified set of respondents.



There are still areas where questions remain. I am not sure that citations per paper is the only way to measure impact. At the very least, the h-index could be added, which would add another ingredient to the mix.



Also, there are details that need to be sorted out. Exactly what sort of faculty will be counted in the various scalings? Is self-citation be counted? I also suspect that not everybody will be enthusiastic about using statistics from UNESCO for weighting the results of the reputational survey. That is not exactly the most efficient organization in the world. There is also a need for a lot more information about the workings of the reputational survey. What was the response rate and exactly how many responses were there from individual countries?

Something that may well cause problems in the future is the proposed indicator of the ratio of doctoral degrees to undergraduate degrees. if this is retained it is easy to predict that universities everywhere will be encouraging or coercing applicants to master's programs to switch to doctoral programs.

Still, it does seem that THE is being more open and honest about the creation of the new rankings than other ranking organizations and that the final result will be a significant improvement.

America' Best Colleges 2011

US News and World Report's Ameica's Best Colleges 2011 is now out.

The top ten National Universities are:

1. Harvard
2. Princeton
3. Yale
4. Columbia
5. Stanford
6. University of Pennsylvania
7 = Caltech
7 = MIT
9 = Dartmouth
9 = Duke
9 = Chicago

Shanghai Rankings: Shifting Research Landscape

My article in University World News on the 2010 Academic Ranking of World Universisities can be viewed here.

More from THE

Phil Baty in Times Higher Education gives us some clues about what the forthcoming THE World University Rankings will contain.

"While all the self-reported material bears the imprimatur of the supplying
institutions (and our tables include only those that have cooperated with our exercise) and it has been vetted for quality, the consultation had some concerns
about its consistency and robustness - especially in this inaugural year. For example, not all institutions could provide a clear or internationally comparable figure for their research income from industry.

For maximum robustness, we plan to give extra weighting to data that have been sourced independently of the institutions themselves and are globally consistent.

Citations data, for example, which are widely accepted as a strong proxy for research quality, will have a high weighting - perhaps about 30 per cent of the total ranking score.

We also have high confidence in the validity and independence of the results of our reputation survey. Although we may yet adjust its weighting, this subjective measure will not be weighted as highly as it was in our old methodology (2004-09), where reputation was worth 40 per cent."

It looks as though citations per paper, a measure of its influence throughout a research community, will count for a lot in the forthcoming rankings. It is questionable whether such a high weighting for a single component is justified. At the very least it could be combined with other measures of quality such as the h-index which is, in effect, a measure of both productivity and impact.

The reluctance to place too much emphasis on research income and perhaps other types of income, is understandable but perhaps unfortunate. This indicator would give the new rankings a distinctive feature and might also allow us to see whether institutions are giving value for money.

It is inevitable that the reputational survey would never be given the same weight that its predecessors received in the THE-QS rankings. Whether its results are really valid -- we still do not know the response rate -- remains to be seen.

Why international students are not a good indicator of quality

Times Higher Education describes a dispute between Coventry University and a recruiting agent in Chennai. According to the article, Ram Beegala was hired as a recruiting consultant and would only be paid if he succeeded in getting the number of Indian students above 450.

There is a comment by "To John" which might be slightly exaggerated:

"It is no secret that the Indian students who cannot get into any of their universities and colleges are the ones that are willing to come to the UK. Their intention is the 20 hour/week work allowed and assume rightly once they use the university route to get into UK they can stay in the country to work. In my university which recruits these students, the drop out rates for such students is high as they work more than 20 weeks to meet their expenses. Their attendance drops down after a few months. I have yet to come across a single non-EU student who comes with enough funds to complete a 3 UG degree. They are told by agents that they can work in the UK to meet part of their fees and all the living expenses. The students coming in to do MSc are poorly equipped and struggle to pass their modules and write project proposals."

Big Names and Unsung Heroes

In Times Higher Education, Phil Baty hints that the reduction in the weighting for subjective indicators in the forthcoming THE rankings will mean that those dominant in the past will suffer a decline and that there may be some new schools at the top.

"We can expect some big-name institutions to take a hit in the new World
University Rankings.

Why? Because the rankings we will publish this autumn will be based less on subjective opinion and more on objective evidence".

..........................................................................................................

"Under the initial proposals for our methodology, currently being refined in line with responses from the global academy, reputational measures are worth no more than 20 per cent of overall scores.

I have also set a cap to ensure that subjective elements are never again anywhere near the 50 per cent used in our previous methodology. This means that big names with big reputations that lack world-class research output and influence to match will suffer in comparison with previous exercises. Conversely, unsung heroes have a better chance of recognition".

Another Ranking

The ic4u ranking of 200 top universities is based on web popularity.

The top five are:

1. Stanford
2. MIT
3. National Autonomous University of Mexico
4. Berkeley
5. Peking

The Forbes Ranking

The 2010 edition of the Forbes College Rankings is now out. These are basically an evaluation from the students' viewpoint. The criteria are the number of alumni in Who's Who in America, ratings in RatemyProfessor, graduation rates, number of students and faculty winning national awards and accumulated student debt.

There are some surprises. Top place goes to Williams College a private liberal arts college that does not even get into Shanghai's top 500. The service academies do very well. On the other hand, Harvard is 8th, Yale 10th and Chicago 2oth.

Long Term Trends in the Shanghai Rankings

The Shanghai Rankings are noted for their methodological stability. Whereas frequent changes combined with the insertion and removal of errors produced wild fluctuations in the THE-QS rankings, the ARWU have remained essentially the same since they started. The Shanghai index never aroused as much public interest as the now defunct THE-QS league table but over the long run it is more likely to reveal real and significant trends.

If we compare the 2004 rankings with those just announced there are some noticeable changes over six years. Cambridge and Oxford have each slipped a couple of places while Imperial College and University College London have moved up a bit, although not as high as their implausible position in THE-QS. Tokyo has slipped from 14th to 20th and Kyoto from 21st to 24th. The leading Australian university has also fallen.

Russia has stagnated with only two institutions in the top 500 in 2004 and 2010. India has fallen back with the University of Calcutta dropping out of the rankings. The rising stars for scientific research are Mainland China (8 in 2004 and 22 in 2010), South Korea (7 in 2004 and 10 in 2010), Brazil (4 in 2004 and 6 in 2010) and the Middle East (none in 2004 and 4 from Saudi Arabia, Turkey and Iran in 2010).

The Shanghai Rankings

The 2010 Academic Ranking of World Universities by Shanghai Jiao Tong University is now out. See here.

From QS

A media advisory has been sent by Martin Ince, Chair of the Advisory Board of QS World University Rankings. See here.

The document describes the structure of the current rankings. Something interesting is that apparently the number of responses has increased to over 13,000, although about half of those would be from people who filled out the form in 2009 and 2008 and did not update their forms this year. The number of respondents is now about the same as that reported by Times Higher for their survey, although THE will no doubt point out that they can be fairly confident that their respondents are still alive and working in academia.

The number of respondents is less important than the response rate and so far neither QS or THE have said how many forms were distributed.

An Epic?

Times Higher Education (THE) has announced the completion of the collection of data for its forthcoming World University Rankings:

An epic effort by our world university rankings data supplier, Thomson Reuters, to collect information from hundreds of universities around the world concluded successfully last week.

I am not sure whether "epic" is the right word. The number of universities in the database does not seem much higher than that for which QS has collected information. The data does apparently include some information that QS has ignored such as institutional income and research income but has not included items counted by QS such as total student numbers or the number of postgraduate students other than doctoral candidates. Meanwhile, the number of respondents to the opinion survey has fallen far short of the original target of 25,000, even with a bit of topping up, like QS, from the Mardev mailing lists.

A proposal to rank universities by disciplines as specific as Agriculture has been dropped. Now, THE will rank universities in six disciplinary clusters, up from five in the THE-QS and QS rankings.

THE also give some idea of errors will be detected. That might be an improvement although I suspect that in many countries third party sources may not be as reliable as THE thinks.

One thing that is not mentioned is whether any universities have refused to participate in the data collection and what THE will do if there are any abstentions.

Discrimination In Top US Colleges

Russell K. Nieli in Minding the Campus discusses a study by Thomas Espenshade and Alexandria Radford that details the extent and depth of the racial and social discrimination practiced by America's top colleges.

"Consistent with other studies, though in much greater detail, Espenshade and Radford show the substantial admissions boost, particularly at the private colleges in their study, which Hispanic students get over whites, and the enormous advantage over whites given to blacks. They also show how Asians must do substantially better than whites in order to reap the same probabilities of acceptance to these same highly competitive private colleges. On an "other things equal basis," where adjustments are made for a variety of background factors, being Hispanic conferred an admissions boost over being white (for those who applied in 1997) equivalent to 130 SAT points (out of 1600), while being black rather than white conferred a 310 SAT point advantage. Asians, however, suffered an admissions penalty compared to whites equivalent to 140 SAT points.

The box students checked off on the racial question on their application was thus shown to have an extraordinary effect on a student's chances of gaining admission to the highly competitive private schools in the NSCE database. To have the same chances of gaining admission as a black student with an SAT score of 1100, an Hispanic student otherwise equally matched in background characteristics would have to have a 1230, a white student a 1410, and an Asian student a 1550. Here the Espenshade/Radford results are consistent with other studies, including those of William Bowen and Derek Bok in their book The Shape of the River, though they go beyond this influential study in showing both the substantial Hispanic admissions advantage and the huge admissions penalty suffered by Asian applicants. Although all highly competitive colleges and universities will deny that they have racial quotas -- either minimum quotas or ceiling quotas -- the huge boosts they give to the lower-achieving black and Hispanic applicants, and the admissions penalties they extract from their higher-achieving Asian applicants, clearly suggest otherwise."

The advantage accorded to Non-Asian minority students, even those whose claim to moral reparation for generations of slavery or dispossession is questionable, is well known. What is surprising about Espenshade and Radford's study is the extent of the discrimination against poor, rural and working class whites.

In part, this is a consequence of the indicators used by American ranking organizations. Selective colleges are apparently reluctant to offer places to students who might not take up an offer for financial reasons since this would push down their acceptance rates and yield scores.

But there is more. Espenshade and Radford found that less affluent whites were dramatically less likely to be offered a place in a competitive private college even when SAT scores, a reasonable proxy for general intelligence, and high school grades were controlled for. In addition, they found evidence of serious discrimination against students who were involved in incorrect activities such as ROTC and Future Farmers of America, especially those holding leadership positions. Apparently "feeding the homeless" will boost one's chances of getting into a top private college if it means doling out soup in between starring in the school play and AP English classes but not if means showing an interest in growing the stuff that the homeless eat.

As cognitive skills become increasingly irrelevant to admission into America's best schools, it seems almost certain that US higher education will be less and less able to compete with those countries that continue to recruit those students most capable of demanding college-level work.

New Webometrics Rankings

The new Webometrics rankings are out.
There are few surprises. Here are the top universities in various categories.

World: Harvard
North America: Harvard
Latin America: Universidad Nacional Autonoma de Mexico
Europe: Cambridge
Central and Eastern Europe: Charles University, Prague
Asia: Tokyo
South East Asia: National University of Singapore
South Asia: Indian University of Technology, Bombay
Arab World: King Saud University
Oceania: Australian National University
Africa: Cape Town

One interesting feature of the Arab World rankings is that universites in the Palestinian territories do very well in comparison with many in more affluent countries. Would anyone like to suggest an explanation?

Global-Rankings Ping Pong

Ben Wildavsky has an article in the Chronicle of Higher Education about the competition between Times Higher Education and QS over this year's university rankings. It is actually called Global-Rankings Smackdown! but the smackdown bit is rather exaggerated and the exclamation mark is unnecessary. There are some well informed and incisive comments on recent developments in international university ranking, including the divorce between THE and QS.

He concludes:

Will a redemption narrative help Times Higher earn credibility for its new rankings? Perhaps. It should certainly be applauded for its openness to criticism, and for all it is doing to inform the public about its next moves in what its editor characterizes, with appropriate caution, as “a decent first step” at improvement. But ultimately, debating tactics notwithstanding, the global league tables will be judged on their merits. As the wars over league tables continue, the next rankings season should be well worth watching.

I am not entirely sure about how much THE, or more accurately their new partners, Thomson Reuters are doing to inform the public about what they are doing. At the moment there are some things we know about the QS survey that we do not know about Thomson Reuters' -- number of forms sent out, response rate, number of responses from individual countries. Still, all that could change within a few weeks and it did take QS a couple of years before they gave out anything beyond the bare minimum about their survey.

The Avalanche

A short article in the Chronicle of Higher Education ,by Mark Bauerlein, Mohamed Gad-el-Hak, Wayne Grody, Bill McKelvey, and Stanley W. Trimble, 'We Must Stop the Avalanche of Low-Quality Research', calls for a halt to the seemingly inexorable rise in the production of uncited and unread scholarly and scientific papers.

While brilliant and progressive research continues apace here and there, the amount of redundant, inconsequential, and outright poor research has swelled in recent decades, filling countless pages in journals and monographs. Consider this tally from Science two decades ago: Only 45 percent of the articles published in the 4,500 top scientific journals were cited within the first five years after publication. In recent years, the figure seems to have dropped further. In a 2009 article in Online Information Review, Péter Jacsó found that 40.6 percent of the articles published in the top science and social-science journals (the figures do not include the humanities) were cited in the period 2002 to 2006.

As a result, instead of contributing to knowledge in various disciplines, the increasing number of low-cited publications only adds to the bulk of words and numbers to be reviewed. Even if read, many articles that are not cited by anyone would seem to contain little useful information. The avalanche of ignored research has a profoundly damaging effect on the enterprise as a whole. Not only does the uncited work itself require years of field and library or laboratory research. It also requires colleagues to read it and provide feedback, as well as reviewers to evaluate it formally for publication. Then, once it is published, it joins the multitudes of other, related publications that researchers must read and evaluate for relevance to their own work. Reviewer time and energy requirements multiply by the year. The impact strikes at the heart of academe.

Unfortunately, now that authorship of an ISI-indexed article has become the qualification for participation n the reputational survey section of the THE World University Rankings I suspect that universities will go on encouraging their staff to produce more and more articles of questionable quality. Or perhaps we should say more and more email addresses in the ISI database.

An Unwelcome Message

Phil Baty, who is in charge of world university rankings at Times Higher Education, writes about an email that he has received.

I was disturbed by an email that dropped into my in-box late last
month.

No, it was not another offer of cheap Viagra, or an announcement that I
had won an overseas lottery. It was more unsettling than that.

"Dear academic," it began. The greeting alone was a surprise, given
that I am a journalist with little more than a bachelor's degree by way of
academic credentials.

But my unease grew with each line of the message. The email was from a
major education information company inviting me to take part in an online survey
that would be used to create a university ranking.

It said that my role as a leading educationalist combined with my
subject focus made my opinion very important. It even offered to enter me into a
prize draw if I passed on my great wisdom and spent 10 minutes filling in the
form.

It would be amusing if the implications were not so serious. As the
email claimed, the audience for the company's annual exercise is in the
millions, and it is clear that university league tables in various forms have
become a very big business with wide influence.

Any organisation, such as Times Higher Education, that seeks to create
rankings must accept its responsibility to conduct thorough research and to
employ sound data.

There is a responsibility on companies doing such surveys that
academics are selected carefully by discipline, and by country and continent if
appropriate. If compilers want universities and students to see their league
table as robust the onus is on them to take a rigorous approach. When rankings can make or break a university's reputation, or influence multimillion-pound strategic decisions, anything less will simply not do.

I am sure that anyone reading this blog has received the message by now and knows that the mysterious sender is not Voldemort but QS, who are now producing their own university rankings independently of THE.

The sending of the message and form to Phil Baty actually represents an improvement for the QS survey. Even without a doctorate, he is probably better qualified to evaluate universities than most subscribers to the World Scientific mailing list, of whom nearly 200,000 receive the form every year. Subscription requires nothing more than the ability to click a mouse a few times.

I wonder though whether those who completed the THE survey form sent out by Thomson Reuters to authors who have published in ISI indexed journals are significantly better qualified. I have heard that there are many parts of the world where the granting of co-authorship of research papers is simply a perquisite of seniority within a department and nomination as corresponding author, the one who gets to go to conferences and do a bit of shopping, is decided partly or largely by political pressures.

It may be that the time has come for a greater variety of reputational surveys to be conducted. There is certainly room for a QS - style survey, essentially open to anyone who, for whatever reason, is interested. After all, that is a constituency that deserves some consideration . But equally, perhaps more so, we need as survey of research excellence that targets demonstrably competent researchers. The ability to be nominated as corresponding author -- I assume that is the one whose email addresses is entered in the ISI archives -- of a paper once in an academic career mught not be sufficient evidence of competence to evaluate university research and teaching. There is a case for a survey based on a more rigorous working definition of research competence, such as inclusion in the ISI list of highly cited researchers. Another possiblty might be to survey editors of academic journals. Response rates could be boosted by publishing the journals who took part. There is also an obvious niche for a student based survey of teaching.

Anyway, Phil, you might as well do the survey. There are many people less knowledgable than you filling out the form and, for that matter, the one for THE . You might even be the one who wins the BlackBerry.

Presentation by Phil Baty

A presentation by Phil Baty of Times Higher Education at the ISTIC meeting in Beijing reviewed the background of the now defunct THES-QS World University Rankings and the rationale for the development of a new ranking system.

There are some quotations that highlight familiar complaints about the THE-QS rankings:


“Results have been highly volatile. There have been many sharp rises and falls… Fudan in China has oscillated between 72 and 195…” Simon Marginson, University of Melbourne.


“Most people think that the main problem with the rankings is the opaque way it constructs its sample for its reputational rankings”. Alex Usher, vice president of Educational Policy Institute, US.


“The logic behind the selection of the indicators appears obscure”. Christopher Hood, Oxford University.



Baty also indicates several problems with the "peer review", citations, faculty student ratio and internationalisation indicators.



All of this is very sound. But it is not yet certain how much of an improvement the new THE rankings will be.



THE will now obtain citations and publication data from Thomson Reuters rather than Scopus. The Thomson Reuters data is based on the ISI indexes, which are somewhat more selective than the Scopus database. There is, however, a great deal of overlap and simply using ISI data rather than Scopus will not in itself make very much difference except perhaps that there will be a somewhat greater bias towards English using researchers and the research output that is measured may be of a somewhat higher quality. We should also remember that from 2004 and 2006, the THE-QS citations data were collected by the very same Jonathon Adams who is now overseeing the development of the new THE rankings.



Some of the "confirmed improvements" noted by Baty are certainly that. Normalising citation scores between various disciplinary groups to take account of varying patterns of publication and citation is something overdue. The presentation of information about various types of income will, if the raw data is publicly available, make it possible to evaluate universities in terms of value for money.


In some ways the reputational survey may be better then the QS "peer review" but exactly how much better is not yet clear. Baty says that only published researchers were asked to take part but this apparently could mean no more than being listed as the corresponding author for an article once in a lifetime. No doubt this yields a better qualified group of respondents than that made up those with the energy to sign up with World Scientific but is it really significantly better?


Also, there is much that we have not been told about the reputational survey. We know the total number of respondents, which was much lower than the original target, but not the response rate. Nor has there been indication of the number of responses from individual countries. This is particularly irksome since rumour and subjective impression suggest that many countries have been neglected by the recently closed THE survey.


The methodology still appears in need of refinement. Research income of various kinds appears four times as an indicator or part of an indicator: research income from industry as the sole indicator in the Economic Activity/Innovation category; as part of overall research income and as part of research income from industry and public sources in Research Indicators; and as part of total institutional income in Institutional Indicators. This is a bit messy.

There is still time for THE to produce an improved ranking system. Let's hope they can do it.

Presentation by Jonathon Adams

A summary of progress so far on Thomson Reuters' Global Institutional Profiles Project an be found here.

A Bit More About the THE Survey

Thomson Reuters have released a bit more information about the reputational survey they recently conducted for the 2010 Times Higher Education World University Rankings.

They managed to get 13,388 responses. This is quite a lot less than the original target of 25,000 although it is higher than the 9,000 plus respondents to the 2009 THE-QS rankings. This means that QS, who are preparing their own rankings, now have an opportunity to boost the numbers of their respondents by using the usual devices -- reminders, extended deadlines, a chance to win an iPad instead of a Blackberry and so on . Thomson Reuters may have made a mistake by closing their survey so early.

Still, numbers are not everything. Thomson Reuters can claim that their survey, which uses the ISI database of authors published in reputable academic journals, targets people who know something about research. The QS survey, on the other hand, consists merely of those who have managed to get on the mailing list of World Scientific.

Thomson Reuters have also provided some information about the regional and disciplinary distributions of their respondents. The largest group is from the Americas. While most disciplinary clusters are well represented, there is a very small number from the arts and humanities. Respondents spend slightly more than half their time doing research and slightly less than a third teaching.

Is this really enough? It would be interesting to know how many forms were sent out and what the response rate was. Also, how far back in time did Thomson Reuters go in collecting respondents? If they went back five or ten years many respondents might have retired or lost interest in research since publishing.

It also would be helpful if more information were given about the geographical distribution of the survey. One notable absurdity of the THE-QS surveys of 2004-2009 was the marked bias in favor of particular countries – more respondents from Indonesia than from Germany, more from the UK plus Australia than from the US, more from Ireland (just the Republic?) than from Russia. Thomson Reuters have probably overcome these biases but have new ones emerged? Has there been an adequate response from Southeast Asia outside Singapore? Have Russia and Central Asia and the Middle East outside Israel been affected by the omission of Russian and Arabic from the list of languages in which the forms can be completed?

It is good that Thomson Reuters have released some information but if they are to fulfill their promise of greater transparency more is needed.

The New THE Ranking Methodology

Times Higher Education has given some information about the proposed structure and methodology of their forthcoming World University Rankings. At first sight, the new rankings look as thought they might be an improvement on the THE-QS rankings of 2004-2009 but there are still unanswered questions and it is possible that the new rankings might have some defects of their own.

The proposed methodology will feature 13 indicators, possibly rising to 16 next year. Here we have the first problem. Frequent changes of method bedevilled the THE-QS rankings, producing, along with a series of errors, implausible rises and falls. If the new rankings are going to see further changes not just in the fine detail of data collection but in the actual indicators themselves then we going to see more spurious celebration or lamentation as universities bounce up down the rankings. Still, if THE are going to standardise the indicator scores from the beginning it is unlikely that their rankings will ever be as interesting as the THE-QS used to be.

The largest component of the proposed ranking is "research indicators" which accounts for 55% of the weighting. These include academic papers, citation impact, research income, research income from public sources and industry and a reputational survey of research.

Another category is "institutional indicators", which together get 25%: number of undergraduate entrants, number of PhDs awarded, a reputation survey of teaching and institutional income.

Ten per cent will go to "international diversity", divided equally, as in the THE-QS rankings, into international students and international faculty.

Another ten per cent goes to economic activity/ innovation. At the moment this consists entirely of research income from industry although there are apparently plans to add two other measures next year.

There are some obvious rough edges in the proposals. The economic activity/innovation income consists entirely of research income from industry but research income from public sources and industry appears under research indicators. In the institutional indicators, universities will get credit for admitting undergraduate students and for PhD students but nothing for anyone in between. I doubt if this will go unchanged. If undergraduates and PhD students are to be institutional indicators then we will see seriously negative backwash effects with masters programs being phased out and marginal students being herded into doctoral programs.

The new methodology is less diverse than appears from a simple count of the number of indicators. It is heavily research orientated. As noted, more than half of the weighting goes to a bundle of research indicators. However, economic activity/innovation is for this year nothing more than research income.

Adding to the emphasis on research, the institutional indicators include the number of doctorates awarded and the the ratio of doctorate to bachelor degrees awarded. Under institutional indicators there is a survey of teaching but the respondents are largely selected on the basis of their being authors of academic articles published in ISI indexed journals. There seems to be no evidence that the respondents do very much teaching and if Thomson Reuters include researchers with a non-university affiliations, of whom there are many in medicine and engineering ,then it is likely that many of those called upon to evaluate teaching have never done any teaching at all. Meanwhile student faculty ratio, a crude measure of teaching quality, has been removed.

It is regrettable that QS has apparently decided to keep the international students indicator. This has caused demonstrable harm to universities in several countries by encouraging the recruitment of students with inadequate linguistic and cognitive skills. One modification that THE should consider if they want to keep this measure, is declaring the EU a single entity. That was supposed to be the point of the Bologna process.

The proposed rankings include several indicators related to university income including research income. This is not a bad idea. After all, the provision of adequate funds is a necessary although far from a sufficient condition for the attainment of a reasonable level of quality. The inclusion of research income will, however, be detrimental to the interests of institutions like LSE that focus on the humanities and social science.

There are still unanswered questions. Some of these indicators will be scaled by dividing by the number of faculty. There will be many raised eyebrows if universities are required to include teaching staff who do no research in the measures of research output or research only staff in the other indicators. Whatever decision is made there is bound to be acrimonious wrangling.

Also unstated is the period from which the data for publications and citations are drawn. The further back the data collectors go the better for traditional elite universities. It is also not stated whether they will count self citations or publications in conference proceedings that are not rigorously reviewed.

So, if you want rankings that emphasise research and funding then THE and Thomson Reuters may be heading, somewhat uncertainly, in the right direction but perhaps at the price of neglecting other aspects of university quality.

From Brazil

There is an interview in Portuguese with Phil Baty of the THE World University Rankings here.

Reactions to the Asian Rankings

Last Sunday the Malaysian New Straits Times published an article by Dzulkifli Abdul Razak, Vice Chancellor of Universiti Sains Malaysia (USM). This particular university has generated considerable publicity by refusing to cooperate with QS, the organisers of the now defunct THE-QS rankings and the Asian University Rankings. Prof. Dzulkifli said of the latest edition of the 2010 rankings:


"FIVE Malaysian universities were ranked among the top institutions in the 2010 QS Asian University Rankings.

While some cheered and reckoned this is deservedly so -- considering the hard work -- others felt it is about time.

At Universiti Sains Malaysia (USM), however, most were amused by the news for the simple reason that they have neither agreed to the invitation to participate nor have they submitted any data or information.

Nevertheless, USM has been assigned a meaningless "number".

Early in the year, the university took a firm stand on not participating in the newly minted QS, and officially notified the company of its intention.

As they made no response, it was assumed that the message was well understood. The silence adds to the conviction that the rankings exercise is suspect at best. "

USM's decision is especially significant since it has been awarded APEX status, with various privileges and enhanced autonomy, with the express purpose of becoming a world class university as measured by a high place in the rankings.

The refusal to participate is entirely understandable. The errors and shortcomings of QS products have been catalogued extensively in this blog and other sources. However, a boycott of the rankings does not mean very much. Most of the information used by QS is easily available from third party sources. In any case, USM's stance did not do it any harm. It ended up in the same position in the current Asian rankings as it did last year although with a larger number of points (average scores for all universities were much higher this year largely because of all those international exchange students rushing backwards and forwards.)

It is also surprising that a Malaysian university should find the QS rankings objectionable. After all, Malaysian and Southeast Asian universities in general have always done much better in QS rankings, thanks to a survey based on the mailing lists of a Singapore based company than in the the Shanghai, Scimargo or Webometrics rankings. It is not clear whether USM's stance is against rankings in general or just those produced by QS.

Discussion in the Straits Times

Two weeks ago the Singapore Straits Times included an article by Phil Baty of Times Higher education in which he explained why THE had "torn up" its annual rankings.

"Perhaps the most embarrassing aspect of the old rankings was the so-called 'peer review' score. Some 40 per cent of a university's overall ranking score was based on this 'peer review' - in effect, a simple opinion survey, asking university staff which institutions they rated most highly.

Our former data provider, QS, achieved only a very small number of responses to this survey. Last year, around 3,500 people responded. Figures for individual countries were shocking. In 2008, just 563 responses from Britain were received, and just 180 responses from Malaysia. Most shockingly, only 116 responses were collected from China's many, many thousands of scholars in 2008. "

He then refers to the extreme volatility of the rankings:

"The University of Malaya in Malaysia, for instance, plummeted from 89th place in 2004 to joint 169th in 2005, before dropping out of the top 200 altogether later. Between 2008 and last year, Keio University in Japan moved up an amazing 72 places to 142. Pohang University of Science and Technology in South Korea jumped 54 places to 134."

Pohang did rise because of a marked improvement in the peer review but Keio's rise was because it did better on the student faculty ratio indicator. The fall of Universiti Malaya was because ethnic minorities were counted as international students and faculty in 2004 but not, after a "clarification of data" in 2005.

He then describes some features of the new THE reputational survey and concludes:

"So much rests on the results of our rankings: individual university reputations, student recruitment, vice-chancellors' and presidents' jobs sometimes, and major government investment decisions. We have a duty to overhaul the rankings to make them fit for such purposes. "

Two days ago there was a reply by Nunzio Quacquarelli of QS. He argued:

"The numbers of respondents to the QS academic peer review, quoted by Mr Baty, are misleading.

Our 2009 rankings were based upon 9,386 respondents, not 3,500 as quoted. QS received statistically significant numbers of academic respondents from all major Organisation for Economic Cooperation and Development (OECD) countries, and 222 in Singapore. "

Some clarification is in order. Quacquarelli is correct in noting that the 2009 world rankings were based on 9,386 respondents. But it should be pointed out that about two thirds of those were respondents who had filled out the survey forms in 2008 and 2007 and had been given the opportunity to update their responses. If there was no updating then the old responses were included. Thus it is not impossible that some respondents had by 2009 retired, lost interest, moved or even died.

He then quotes from noted statistician Paul Thurman to claim that the academic opinion survey was valid.

Quacquarelli has a point in that it is not numbers alone that contribute to validity. However, it is hard to accept that a survey with more respondents from Ireland than from Russia and more from Hong Kong than from Japan can be regarded as a valid representation of global academic opinion.

Then there is an unconvincing assertion that THE had consistently endorsed the THE-QS rankings. This in fact refers to the time when John O'Leary and Martin Ince, now with QS, were editor and deputy editor of THE. Well, they would, wouldn't they?

It still remains to be seen whether the new THE survey will be better than QS's. It asks more questions and more detailed ones and has been distributed in several languages but the question of representativeness still remains. Thomson Reuters appear to be satisfied with the number of responses they have received but have not said how many there were or how they were distributed. I have a subjective impression that Southeast Asia and other regions may be underrepresented this time. I would, however, welcome detailed information from Thomson Reuters that would prove me wrong.