Malaysia and the Rankings

An article by Phil Baty of Times Higher Education in the Malaysia Star repeats earlier comments on the flaws of the old THE-QS World University Rankings:

"I must make a confession: the rankings of the world’s top universities that my magazine has been publishing for the past six years, and which have attracted enormous global attention, are not fit for purpose."

He then continues:

"Those who have used our rankings to cast judgment on the state of Malaysian higher education (and many, in very senior positions have done so) must be told that the annual tables had some serious flaws — flaws which I have a responsibility to put right."

He is absolutely right about the flawed rankings of 2004 - 2009 and about the use of ranking data for political purposes. It is particularly noticeable that any fall by Malaysian universities in the rankings is treated by some writers as the consequence of serious problems in the Malaysian education system. I remember at the end of 2007 receiving a request from a Singaporean newspaper to comment on the latest rankings in which Universiti Malaya (UM) had suffered a serious decline. I replied in detail that it was highly likely that the apparent fall in UM's position was due to changes in methodology and nothing else. This was confirmed a few days later when detailed indicator scores were published showing that UM's fall between 2006 and 2007 was almost entirely the result of the introduction of Z scores which boosted the scores for research for moderately productive research universities like Peking while slightly lifting those for relatively less productive ones like UM. The newspaper article, however, simply asserted that the decline was the result of deficiencies in UM and Malaysian higher education in general.

I do not dispute that Malaysian universities have problems. It is also obvious that in many years they tumbled down the QS rankings. The two just did not have anything to do with each other.

Equally it is true that the quantity of research in Malaysian universities has expanded greatly in recent years and that in some years some Malaysian universities rose. But again these two things were quite unrelated.

In the critique of the old rankings the focus is on the survey of academic opinion, which accounted for 40% of the rankings. Baty points out that a relatively small number of responses were collected from world academics, 563 from the UK, 180 from Malaysia, 201 from the Philippines.

It is true that the old THE-QS rankings collected a small number of responses but size alone is not the crux of the matter. What matters is whether the the sample is an adequate repesentation of the population from which it is drawn. It is arguable that subscribers to World Scientific (THE-QS) are less representative of international academic opinion than published researchers in peer reviewed journals (THE and Thomson Reuters . The actual number is less important. The new rankings will be vindicated not so much by the number of responses received but by how representative and qualified they are.

The article suggests that Malaysian scholars will be able to participate in the ranking than before. I am wondering about that. I know a few people in Malaysia who took part in the 2008 survey but have not received a form this year (perhaps they do not deserve to). It will be interesting to see the exact number of voters from Malaysia and elsewhere when the polls close.

The Emerging THE Rankings


Times Higher Education has announced some information about the data that will be collected for their forthcoming rankings.


It is now possible to speculate about what might be included or excluded. First, there has been no mention of an employer review. This is a pity since this was the only input from outside universities in the old THE-QS rankings.


Possible new indicators are number of bachelor's degrees, number of doctoral students admitted, number of doctorates awarded "including those funded by competitive research scholarships", total institutional income, research grant income and research contract income.


It is likely that if any of these are indicators in this year's and subsequent rankings there will be negative backwash effects. It is easy to foresee that diploma and certificate courses will be "upgraded", more and more marginal candidates admitted to doctoral programs and more and more dubious doctorates awarded. It is also likely that every award to a postgraduate student will somehow turn into a competitive research scholarship.


On the other hand data about income and source of income sounds promising since this is something that universities will find difficult to manipulate.


It looks as though student faculty ratio might be maintained as will the proportion of international students and international faculty. The reference to research only staff suggests that only teachers will be counted in the student faculty ratio, a very sensible idea.


As for international students and international faculty, there is a big difference between the two, namely that the former are paid to come to universities but the latter are not. There is surely enough evidence from the UK and Australia that artificial incentives to mass importation of unqualified students does nobody any good.

If THE decides to keep the internationalisation indicators it might be time to stop calling people who more a few miles within the EU international. Similarly, the special dispensation whereby Mainland Chinese students in Hong Kong are designated international ought to be abolished.

The new rankings will probably include a research indicator based on citations rather than publications. This makes sense. The impact of research, indicated by the number of times it is cited, is more important than the simple fact of publication. There is, however, a risk that this will allow a further element of gaming into the rankings. Researchers will not only divide papers into the smallest possible unit of publication but will also start doing things like citing themselves copiously and unnecessarily or citing colleagues with whom reciprocal citing arrangements can be established.

The reputational survey (congratulations to THE for not calling it a peer review) appears to be under way. THE and their associates seem determined to avoid the Anglo-Saxon bias of the THE-QS rankings (and perhaps the bias among Anglo-Saxons towards the UK and Australia). It is possible though that new biases may be emerging. The distribution of forms will be based on UN data that could be several years out of date by the time they emerge from national bureaucracies. Taking a sample of opinion from ISI publications, which will reflect research projects that began several years earlier, may create a bias in favour of the traditional elite and against newcomers in the research world such as Iran, Southeast Asia, the Middle East and South America.

Comment by Nunzio Quacquarelli

Several items about or by leading figures in the ranking business have appeared recently. Nunzio Quacquarelli, director of QS, says:

"QS research has become highly respected and every year is referenced in roughly 1000 different newspapers, journals and web sites – a Who’s Who of the best media around the world"
There has also been an enormous amount of criticism, especially by academic experts such as Anthony van Raan, Simon Marginson, Eric Beerkens and so on. In comparison, favourable comments by Alan Kantrow, former editor of the McKinsey Quarterly, and Martin Ince do not really count for much. We also have that old quotation from Richard Sykes, "it takes smart people to know smart people." This begs the question of whether the ability to sign up for a an academic mailing list is sufficient to be a certified smart person. Sykes, incidentally, is described as the Rector of Imperial College London, which he has not actually been for nearly two years.

Quacquarelli then refers to Paul Thurman, a specialist in public health statistics, who says that a 2% response rate yielding 9,500 responses is as good as or better than most political opinion polls. Perhaps, although political polls are far from perfect. But this comment misses an important point. QS may have achieved a representative sample of subscribers to World Scientific, an academic publishing company based in Singapore with links to Imperial College (no wonder Richard Sykes is talking about smart people) but are those subscribers representative of expert academic opinion?


The article continues:

"Over 8000 academics have attended seminars specifically debating the QS World University Rankings methodology. Amongst attendees there has been almost universal acceptance of the QS ranking criteria."

Yes, but what matters is those who did not attend attend the seminars.

While we are on the subject of meetings with academics, it might be worth remembering a meeting in Kuala Lumpur in November 2005 at which Quacquarelli is reported to have said that QS was "not aware of Malaysia's racial composition" (the Star 18/11/2005). This is forgivable. After all, the QS office nearest to Malaysia is in Singapore which is very far away. Another report indicated that the "talk attracted comments from the floor some of who disagreed with his ranking methodology." (Sun 22/11/2005).

It seems that Nunzio Quacquarelli sees nothing wrong in the THE-QS rankings. Improvements have been made but there is certainly room for more. Failing to recognise the flaws in the rankings will not help anyone.

Graduate School Rankings



US News has released its annual ranking of American graduate schools. These are subject rankings rather than holistic.



The top schools in selected categories are:



Business: Harvard, Stanford

Education: Vanderbilt

Engineering: MIT

Law: Yale

Medical Research: Harvard

Medical Primary Care: University of Washington, Seattle

Biological Sciences: Stanford

Chemistry: Caltech

Computer Science: Carnegie-Mellon, MIT, Stanford

Earth Sciences: Caltech, MIT

Mathematics: MIT

Physics: Caltech, MIT, Berkeley

Statistics:Stanford

Economics: Harvard, Princeton, Chicago, Stanford

Library and Information Sciences: University of illinois: Urbana-Champaign, University of North Carolina: Chapel Hill

English: Berkeley

Psychology: Stanford, Berkeley

History: Princeton, Stanford, Berkeley, Yale

Public Affairs: Syracuse

Fine Arts: Rhode Island School of Design

The Survey Season

A few weeks ago there was a flurry of activity in the rankosphere as the leading rankers announced that they would be starting their surveys. It seems that we are now in a quiet period.

• QS announced that their employer survey would be open on March 23rd but to date there is no sign of life on their site.
• Academic survey forms from Thomson Reuters on behalf of Times Higher Education have been sent out to some countries. I am wondering though about how complete the coverage is. I have not received one although I admit that my record of publication in ISI indexed journals is not particularly brilliant.
• I have heard that several people have signed up for the World Scientific mailing lists, which provide the bulk of the respondents for the QS academic survey, but have not received any newsletters or other material. If World Scientific has frozen its lists or there is some sort of technical problem there might be a problem for QS.

Something from THE

An editorial by Phil Baty in Times Higher Education refers to a comment by Janez Potocnik, former European Commissioner for science and research, that university rankings are now used to assess national economic strength.

He then goes on to indicate other uses of rankings and to provide some more information about the forthcoming THE rankings. It seems that universities will be ranked in six subject areas, one more than in the THE-QS rankings, with life sciences and biomedicine being divided into life sciences and "clinical, pre-clinical and health".

Also "(w)e will also judge subject strength on the full range of measures used in the overall table. We believe this will represent another great improvement."

There might be a problem here. If THE are going to publish subject rankings based on the indicators used in the overall rankings it would not make sense to use the proportions of international students and international faculty, student faculty ratios or citations per faculty for the whole university to determine standing in specific disciplinary areas. So, Thomson Reuters would have to collect specific data about the numbers of international students in the social sciences and so on. Getting accurate information about numbers of students and faculty is diffcult enough for a university as a whole but for each disciplinary area it would be close to impossible. In any case, in many universities the boundaries between disciplinary areas may not correspond to those used by Thomson Reuters.

The reference to ranking the top 200 universities is disappointing. There is an enormous unmet demand for valid information about universities around the world, not just the top 200. Thomson Reuters say they are collecting information about a thousand universities (down from "thousands" a few weeks ago). It would be a pity to waste it.

Proposal for a Student League Table

The conference of the National Union of Students in the UK will condsider a proposal to start a student-run league table that will address "bread and butter" issues.

Another proposal for the conference is that university staff protesting against government cuts should "adopt a boycott on publishing their research".

The problem with that is that they would have to do some research in the first place.

Comments on the Economist article

The Economist has published a number of comments on its article on international university rankings.

The second, as we should have expected, is a complaint about the failure of the rankings to acknowledge the brilliance of LSE. The author, who has a master's from LSE, complains that "the THES and the Times usually score Oxford higher than the LSE- partly due to expenditure and, in the case of the Times, the higher number of 'good marks' " Unfortunately, the THE (no S now) - QS rankings never included expenditure as an indicator: Oxford outscored LSE mainly because of its performance on the academic survey component. I wonder if this says something about the excellent and rigorous research training provided by LSE.

There are also several complaints that teaching quality is not reflected in the current rankings and may even be inversely related to the research productivity of universities. However, there is a sensible observation from Rojr that

"There seems to be a popular but silly assumption that clever Oxbridge graduates are made clever by their experience at Oxbridge. Why? It's well known that Oxbridge picks the cleverest students up front--who's surprised that they're still the cleverest after graduating? Correlation does not imply causation! I wish the league tables would catch on to this.

If I were an employer, an acceptance letter from a top ranking university would be exactly as impressive as a graduation certificate (with any grade on it) from the same university. "

I suspect that global rankings would be more improved by a measure of student quality based on performance in national standardised tests such as the SAT than by efforts to assess the vague and culture-bound concept of teaching quality. I wonder also whether Oxbridge is still picking the cleverest students now that the entrance exams have been scrapped and A levels have become largely meaningless, leaving colleges reliant on admission interviews, probably the least valid of any selection procedure.

Finally, there are comments from Phil Baty of THE and Ben Sowter of QS.

Article in the Australian

An article by Andrew Trounson in the Australian discusses the developng rivalry between the Times Higher Education and the QS rankings and its implications for Australian universities.

Both ranking groups are attempting to increase the number of responses to the academic surveys that will be in both rankings. This might have serious consequences for Australian universities which always did better in the survey-based components of the THE_QS rankings than in the other indicators and which now face the prospect of losing ground in a bigger and more diverse survey.

News about ranking

There has been a lot of ranking-related activity over the last few days.

• Phil Baty of Times Higher Education and the QS team of John O'Leary, Martin Ince, Nunzio Quacquarelli and Ben Sowter have given presentations at the British Council Going Global 4 conference in London. Phil Baty was apologetic over the flaws of the old THE-QS rankings while the QS team saw no reason change.

• The Economist has an article "Leagues Apart" that briefly reviews the development of international university rankings. Observations include the volatility of the rankings. Perhaps inevitably the example chosen is the fall of LSE after QS introduced standardised scores which helped universities that produced more citations.

• Phil Baty in THE comments on the problems of assessing the quality of teaching in unversities.

FAQs at QS

QS have answered some questions over at QS TOPUNIVERSITIES. Here are some of the questions and ATFAQS (Answers to ...) or extracts and some comments.


"1) How do you plan to address the perceived bias towards English-speaking (and particularly UK) universities?

...The reality is, however, that in many areas of university competitiveness, operating in English is an advantage. English language journals are more widely read and cited, the top four destinations for international students (and I suspect also faculty) are the US, Canada, UK and Australia – all English speaking. Many universities in non-English speaking Asia, recognising this are operating more programs in English and all global rankings currently carry this bias, not just ours. Our objective is to minimise the bias, but it is far from clear whether eliminating it entirely would be appropriate."

Fair enough. But the bias within the English speaking world (the high scores for Oxbridge, the London schools and colleges, Edinburgh and Australian universities compared to the US and Canada) in the THE-QS rankings was probably more significant.

"3) Following the launch of the government-funded Assessment of Higher Education Learning Objectives (AHELO) pilot scheme, how do you respond to the suggestion that an insufficient emphasis is given to teaching standards and student skills within the more research-oriented established methodologies?


QS absolutely concurs that teaching and learning is inadequately embraced in any of the existing global rankings, including our own and is watching the AHELO exercise with great interest to see if lessons can be drawn and applied to the much broader geographical scope of our rankings. QS is also assessing whether student and alumni inputs can help draw a clearer picture of comparative performance in teaching and learning. On the student skills side of things, QS is currently the only global ranking taking this aspect seriously – via the Employer Review indicator."

Assessing the quality of teaching has so many pitfalls that it may never be possible to do it objectively on an international scale. A global version of RateMyProfessor might be feasible but there is obvious potential for rigging. It also has to be said that for more proficient students -- and that would include many or most of those in universities that will be in the top 200 0r 300 in any sort of ranking -- teaching is largely irrelevant. I doubt if any high fliers from the Ivy League or the grandes ecoles were ever quizzed by interviewers about the staff-student ratio in their classes or whether their instructors explained desired learning outcomes or whether they felt safe in their lecture halls. If teaching is to be assessed an opinion survey is probably no worse than anything else that might be proposed.

"4) Do you think that the low ranking of LSE in the 2009 rankings (67th) is reflective of an inherent bias toward scientific subjects within citations-based methodologies, and if so how do you plan to address this in 2010?

The QS World University Rankings™ are designed to assess the all-round quality of universities across all disciplines and levels, in teaching, research, employability and internationalisation. LSE is a fantastic institution, as is reflected by their persistent high position in the Social Sciences – the faculty area in which they are focused. In fact, it is so strong with its narrower focus that it manages to compete with world leading institutions with a much broader range. Even if we only take the proportion of world universities recognised by UNESCO a Top 100 placing represents the top 1% - a prolific achievement for an institution that focuses on only a small part of the academic spectrum. To put things in perspective, LSE fails to break the top 200 in the Shanghai Ranking."

It seems that the position of LSE in the forthcoming rankings will be closely watched. Yes, there has been a bias against institutions with strengths in the social sciences and this may be corrected in the THE rankings but anything that benefits LSE will also benefit general universities as much or more.

"5) How can the shift in position of some universities in the THE -QS World University Rankings 2004-2009 be explained?"

QS essentially answers this questions, or rather avoids answering it, by pointing out that later editions of the THE-QS rankings showed more stability and that national rankings of British universities were even more volatile.

One reason why the THE-QS rankings were so unstable is simply the large number of errors that were made. These include counting ethnic minorities in Malaysia as international faculty and students, giving 1 out of 100 for citations to Washington University in St. Louis, the Indian Institutes of technology and Technion Israel and then boosting their scores in the following year, overcounting the number of faculty at Duke University and overcounting the number of citations or undercounting the number of faculty at the University of Alabama.

Such errors do, however, appear to have been eliminated from the most recent rankings.

Another problem arose from the the frequent changes in methods and sources of data. Here there is a real and serious dilemma . Methodological improvements are necessary to maintain validity but at the same time they can undermine credibility by causing noticeable fluctuations.

One solution to this might simply be to publish two sets of rankings every year, one with an unchanged methodology called the QS Classic or the Shanghai Classic ranking and another incorporating the latest methodological changes called the New, Alpha.. Mega.. or whatever ranking.

Job Advertisement

QS are recruiting a Research Assistant for their London office, presumably to work on the data collection for the 2010 rankings.

"You will have analytical insight and familiarity with working with large data sets as well as you will be an effective communicator both personally and in writing. You will be results-oriented and dedicated to contributing to the success and development of our business unit and its research outputs. Responsibilities include Data collection gathering correct information from universities directly via email website telephone or third party sources Data entry accurate data entry into existing online database Correspondence dealing with university representatives or third party clients handling enquiries promoting the products Research research the web or other applicable sources for useful information Research Outputs contributing to high quality and insightful research outputs Gathering the correct information from universities can be a challenging task and often requires a surprising level of skill tenacity and diplomacy as well as a healthy appetite for problem solving. Therefore Skills attributes required Ability to stay focused and high attention to detail -Tenacity diplomacy and reliability Healthy appetite for problem solving Inquisitive mind and genuine interest Good communication Effective time management Commitment and Enthusiasm Excellent knowledge and experience of office software applications Additional languages desirable. "

It sounds like they are getting serious. The additional languages might be significant. But this bit at the end is surprising.


"This is a full time position requiring a minimum of 35 hours per week and a maximum of 40."

A maximum work week of 40 hours! I wonder if there are any universities anywhere in the world who have that. And I bet they don't in Shanghai.

Article on Rankings

There is a detailed and thorough review of recent developments by David Jobbins in University World News:

First shots fired in ranking war

Ranking News

There has been a lot news from the rankosphere over the last week.

On the 8th of March QS World University Rankings announced that they were launching their 2010 Research.


"largest review of international universities ever conducted

* Over 2000 participating universities from more than 130 countries

* Over 200,000 university selections by academics, for excellence in research quality*

* 5000 participating employers"

If the response is similar to last year when the average academic reviewer listed 12 universities, then 200,000 university selections would mean about 17,000 respondents, quite a big jump. Two thousand participating universities would mean more than doubling the number of universities assessed, a very good idea in principle, although there could be logistical problems and, of course, the chances of embarassing errors will increase.

Meanwhile, QS have started a new newsletter QS Rankings & Global Higher Education Trends and also started a question and answer page.

On the 11th of March, Times Higher Education announced:


"The biggest and most ambitious project to measure universities' academic reputation for the Times Higher Education World University Rankings was launched this week.

Thomson Reuters, the exclusive data supplier and analyst for the THE rankings in 2010 and beyond, unveiled its Academic Reputation Survey in Philadelphia on 11 March.

Over the coming weeks, thousands of academics around the world, who have been carefully selected as being statistically representative of the global academic workforce, will be asked to complete a short, invitation-only survey to state which in their opinion are the strongest universities in their fields of expertise.

In a major new development, the survey will gather opinions on the standards of both research and teaching, raising the prospect of the first worldwide reputation-based measure of teaching quality in higher education. "

It sounds like THE are going to draw much of their survey sample from the database of Thomson Reuters. In other words they will survey only or mainly published researchers, which is highly appropriate if research quality is the only thing that is being assessed. Now that THE are going to ask about teaching quality, it might be worth thinking about also surveying teaching-only university staff and undergraduate and postgraduate students.

So we are going to have the largest review ever conducted versus the biggest and most ambitious project. Whatever happened to that British gift for understatement?

Comparing Two Surveys

Over the last few years university rankings have acquired a large audience. Each year since 2003 , when the first Shanghai index came out, the ups and downs of universities, especially in East and Southeast Asia, have commanded almost as much attention as the fortunes of national football teams.

This year it seems that competition between the rankers, Times Higher Education and their former partners, QS, will be get as much attention as that between universities and a lot of that attention will go to the merits or flaws of the surveys that are now under way.

Times Higher have just announced the launching of the new reputational survey while QS have started a sign -up facility. If THE are going to start the survey now then they could create a problem for QS since after one e-mail message plus a few follow-ups (I expect Ipsos MORI will tell them about this) and, for some people, a form from the EU rankings, severe ranking fatigue will set in and the later survey forms will go unanswered.

Here are some points of comparison of the two main surveys that will be filling academic e-mail boxes in the next few weeks or months.


Indicator Weighting

QS have stated that their survey will continue to have a weighting of 40 percent. Times Higher say that theirs will have a smaller weighting but have not said exactly how small. Probably the reduction will not be too great if the expense and effort of conducting a survey is to be justified.

Participants

The bulk of QS's survey respondents have come from the mailing lists of World Scientific, a Singapore based publishing company that is linked with Imperial College London and has had a close relationship with Peking University. Others, mainly in the humanities and social sciences, have come from Mardev, a company that collects academic addresses. Some no doubt have been identified during QS's various seminars and tours. This year QS have added a sign up facility that will screen those who wish to take part.

THE will get most of their respondents from the Thomson Reuters internal database by which they presumably mean authors of papers in ISI-indexed journals and conference proceedings, supplemented by so far unidentified third party sources.

The basic qualification then for participating in the QS survey is therefore to subscribe to a newsletter from World Scientific. For the THE survey it will be to to have published a paper in a reputable academic journal or conference proceedings. The THE respondents should then be better qualified to comment on research quality, although one might note that the assigning of the role of first or corresponding author is sometimes a political decision rather than a recognition of actual contributions to a research project.

Numbers

THE have said that they are aiming at a target of 25,000 participants. QS appear to be aiming at close to 17,000 this year.

Regional and Disciplinary Balance

QS have stated that they weight by discipline and subject when selecting potential respondents from the World Scientific and Mardev databases. After data collection they balance responses between three super- egions, the Americas, Asia and the Pacific, and Africa, Europe and the Middle east, but not apparently within those regions. THE have stated they will distribute the survey forms to reflect the world distribution of academic researchers geographically and in terms of discipline.

Questions

THE have stated that they will be asking questions about teaching and research and that the questions about research will be more focused than in the past. QS will continue to ask only about research, which is a little odd since their respondents probably include many who teach but do not do research.

Languages

Last year the THE- QS forms could be answered in English or Spanish. QS may be including other language options this year. So far, it looks as though the THE forms will be entirely in English.

General

It appears that THE may produce a valid survey of the opinion of recently published researchers that reflects the current global distribution of academic research activity. The main problem may well be that there will be a serious conflict between quantity and quality. Academic e-mail addresses are highly degradable and THE may find that many of their published researchers have retired, been downsized, moved, died, forgotten their password or just got fed up with filling out online survey forms. If, in pursuit of the targeted 25,000, they are forced to start trying to contact scientists who published an article (or just put their names on the work of graduate students) several years ago the validity of the survey may become questionable.


On the other hand, it would seem an error for QS to insist on continuing to ask only about research. The THE-QS survey was a dubious measure of research performance but it might have more credibility if it also measured teaching quality or social and economic contributions.

On balance, it would seem that THE, if it can get the the number of respondents it needs, will produce a more accurate and credible survey of opinion about research, although QS might claim that by reaching out to university teachers and non-English speakers they are providing a platform for those whose views ought to be considered in any opinion survey.

Hyper-bureaucracy

Times Higher has a leader by Ann Mroz on the rising tide of academic bureaucracy, to which, we might add, rankings, ratings and assessment have made no small contribution.

"But banal and mind-numbing though it is, bureaucracy isn't neutral. It is insidious, changing the nature of both teaching and research; it also, of course, has been used to push academics in uncomfortable directions.

A scary new word to emerge in our cover story is "hyper-bureaucracy", which describes "an out-of-control system" that emerges in the search for optimum efficiency and takes no account of the costs in time, energy and money that are needed to achieve it. It is a bureaucratic nightmare in which there is no end to the extra information that can be acquired. The monitoring of contact hours and how academics spend their time are examples of the type of bureaucracy that "eats up people and resources", according to Andrew Oswald, professor of economics at the University of Warwick. "

Comment from Tines Higher

Phil Baty in today's Times Higher Education explains why the rankings need an overhaul despite their growing influence.

"So if the rankings have become an accepted reference point, why are we making such dramatic changes, switching our data provider and revamping our methodology? We are doing so precisely because the rankings have become such a respected reference point. If they are starting to influence strategic thinking and even government policy, we have a responsibility to make them as rigorous as possible."

Article on Rankings in Nature

A substantial article on developments in international university rankings by Declan Butler has appeared here.

Some extracts:

"Several approaches to university rankings now being developed are switching the emphasis away from crude league tables and towards more nuanced assessments that could provide better guidance for policy-makers, funding bodies, researchers and students alike. They promise to rank universities on a much wider range of criteria, and assess more intangible qualities, such as educational excellence. And the THE ranking list is trying to remake itself in the face of the criticism."

On the new Times Higher Education rankings

"Thomson Reuters plans to continue reputational surveys, but aims to have at least 25,000 reviewers, compared with the 4,000 used by QS for the THE 2009 rankings. It has partnered with UK pollster Ipsos MORI to try to ensure the survey is representative. "We are not doing this randomly, but putting a lot of thought behind it," says Simon Pratt, project manager for institutional research at Thomson Reuters. "We want a more balanced view across all subject areas." The THE will continue to rank all universities in the form of a league table, which critics say offers a false precision that exaggerates differences between institutions. But the new rankings will be more nuanced and detailed, according to Pratt, including data that enable institutions to compare themselves on various indicators with peers having similar institutional profiles."

On the European Union rankings

"U-Multirank also hopes to overcome one of the major criticisms of many existing ranking systems: that they focus excessively on research output, neglecting the many other crucial roles that universities have, not least teaching. Indeed, the Academic Ranking of World Universities, compiled by Shanghai Jiao Tong University in China and generally known as the Shanghai index, focuses exclusively on research output and citation impact, including variables such as numbers of Nobel prizewinners and publications in Nature and Science(see 'Top marks')."

On the QS rankings

"QS intends to continue developing its university ranking despite losing its link to the THE. "We will continue improving the methodology and response levels to the surveys," says Sowter, adding that he welcomes the new competition. Other experts say that having more rankings will be beneficial, as it will reduce the undue influence of any one ranking."

I would like to add one point. I have been as critical of QS as anyone but it is rather unfair to talk about 4,000+ respondents to their academic survey. They had 4,000 new respondents last year, making a total of 9,000+.

How Much Instability?

Phil Baty in Times Higher Education writes about fluctuations in the old THE-QS rankings

"Magazines that compile league tables have an interest in instability - playing around with their methodologies to ensure rankings remain newsworthy.

This was the argument made by Alice Gast, president of Lehigh University, Pennsylvania, at the Lord Dearing memorial conference at the University of Nottingham this month.

She has a point. Dramatic movements in the league tables make the news and generate interest - helpful for the circulation figures.

But too much movement raises questions about credibility: everyone knows that it takes more than 12 months for an 800-year-old university to lose its status, or for a young pretender to ascend the heights. "

The THE-QS rankings were famous for their yearly fluctuations. This of course helped to make them much more popular than the reliable but boring Shanghai rankings (unless you were prepared to spend a few hours cutting and pasting the indicator scores of universities in the 300s and 400s into an Excel file and then they could be interesting). The rises and falls resulted from changes in methodology, errors, correction of errors and inconsistent application of guidelines.

Still, there are cases when universities undergo serious restructuring or pour massive funds into research or recruit administrators of the highest calibre and these developments should be reflected in any valid index. Rankings that do not show some upward movement by, say, the Hong Kong University of Science and Technology or King Abdullah University of Science and Technology ought to be considered suspect

Equally, it is striking that the major rankings contain elements, the THE-QS academic opinion survey, the Nobel laureates in the Shanghai rankings, even eleven year old publications and citations in the Taiwan rankings, that disguise the steady relative decline of Oxford and Cambridge over the last two decades.

We shall have to wait until 2011 to see if the new THE ranking will avoid the suspicious fluctuations of the THE-QS rankings and also be sensitive to genuine changes in international higher education.

RateMyProfessors and Amy Bishop

Richard Vedder of the Center for College Affordability and Productivity produces an interesting US ranking based on value for money for students. One key element is data provided from the famous or notorious site, RateMyProfessors. I used to think that it would a step forward in international university rankings to try to do something like this on a global scale. Now I am not so sure.

I assume that everbody has heard of the tragic shooting at the University of Alabama at Huntsville. My first suspicion was that the alleged murderer, Amy Bishop, was a talented but socially awkward academic who had snapped after being denied tenure on flimsy grounds of collegiality or for being politically incorrect.

That does not look like being the case. A blog, Shepherds and Black Sheep has analysed her research output and found that it seems inflated, with one article having her children as c0-authors and another published in an "online vanity press' and several more being co-authored with her husband

Bishop's pages at RateMyProfessors are also interesting. At first sight they look quite impressive with a total score of 3.6 out of 5, fifth best in her faculty, and 3.4 for clarity and 3.7 for helpfulness. But there are some oddities.

The user comments start with three excellent reviews, one on 26 May, 2009 and two on 19 May. A little odd. Back in June 2004 there were also two rave reviews again posted on the same day.

It is also noticeable that the good reviews tend to cluster together with three consecutve good reviews in January and February 2006 and another three, one after the other, in November, 2004 and January, 2005.

Another odd thing is that for the helpfulness and clarity indicators there is a very distinctive distribution curve. For helpfulness, Bishop had 6 ones (the worse), 2 twos , 5 threes, 3 fours, and then 17 fives (the best) For clarity, it was 7 ones, 5 twos, 4 threes, 3 fours and 15 fives. Note the dramatic jump from four to five in both categories.

Compare this with a low scoring teacher who gets 4 fours and 4 fives for helpfulness and 5 fours and 6 fives for clarity.

Compare also a high scoring faculty member with 13 fours and 25 fives for helpfulness and 15 fours and 28 fives for clarity. A big jump from four to five but proportionately much less than Bishop's.

Is it possible to rig RateMyProfessors?

According to Tenured Radical it is very easy.

"To test my theory that ratings could be posted by people who had never been my students, I went to the dreaded site, and registered myself, under my own name, as a Zenith student. Easy-peasy. The only false information I provided was a birth date that made me 19 years old (I wish!) and the box I checked that affirmed my status as a Zenith sophomore. I then successfully added a rating about myself. You can see it here: it's the anxious looking green emoticon that has the comment "interesting." I thought it only fair to add something right down the middle, neither good nor bad. Inflammatory perhaps, but arrogant never, that's my motto. "

So, I have a strong suspicion that someone had been going to RateMyProfessors and posting effusive commuents about Bishop (and nasty ones about other faculty members.?)

Perhaps RateMyProfessors is not such as good indicator after all.

A Survey from Thomson Reuters

A survey was conducted recently for Thomson Reuters to provide input for the forthcoming Times Higher Education World University Rankings. The results of the survey can be accessed at the Global Institutional Profiles Project set up by Thomson Reuters.

The results of the survey are important since they might provide a clue to what the new ranking will look like.

There were 350 respondents from the “global academic community” This is apparently more than the numbers that answered similar surveys by QS but it does not seem very large especially when THE has raised justified concern about the low and possibly unrepresentative numbers participating in the THE-QS survey of academic opinion.

Of those 350, 107 were from the UK (31%), 90 from the US, 30 from Australia, nine from Canada and seven from New Zealand. Thirty three were from the rest of Europe, 32 from Asia and 42 from the others, i.e. not from North America, Europe, Asia and Australasia. With nearly a third of the respondents coming from the UK and over two thirds from just five English speaking countries this is a distinctly Anglo-Saxon-centric affair.

The first question was the level of familiarity with various rankings. The THE ranking was the one with which the largest number of respondents were familiar. This is a slightly odd result since from 2004 to 2009 THE published rankings under the name THE (S) - QS World University Rankings. The Times Higher Education World University Rankings (minus QS) have yet to appear. No doubt, THE would claim that it was they who published the ranking and can retrospectively rename them if they wish.

Summarizing the responses to the survey, it seems that the respondents believe that university rankings
• are useful
• have methodological problems
• are biased
• encourage the manipulation of data
• encourage a focus on numerical comparisons
• use data that is not transparent or reproducible
• do not not include appropriate metrics
• favour research institutions

Among the information that respondents need or would like to have are

• Publications and citations
• Research awards
• Patents
• Faculty student ratio
• Faculty activity ratios (teaching income/research grants/publications per staff)
• Number of faculty by gender, international, ethnicity or race
• Number of graduate programs and degrees
• Collaboration
• Community engagement
• Perceptions of researchers, employers, alumni and community

Some of this, community engagement for instance, is too vague to be useful. Other items contradict the stated objectives of the developing ranking system: including patents and research collaboration in a general ranking would add more bias in favour of the natural and applied sciences. Others betray the American or European concerns of the respondents: alumni have little significance outside the USA. It is also noticeable that nobody seems interested in student perceptions.

Starting Again

This, in an article by Phil Baty of THE in The Australian, sounds promising.

"So we've started again. For 2010, Thomson Reuters has hired pollsters Ipsos MORI to carry out the reputation survey, and it has committed to obtaining 25,000 responses, from a carefully targeted and properly sampled group that will represent the true demographics of global higher education.

This may mean a slip in the performance of British and Australian institutions but if that is the case then so be it. We are interested in getting closer to the truth. "

Chinese University Rankings

Click here for a survey by Nick Clark in the World Education News and Reviews

Quality Assurance and Ranking of Higher Education in Asia-Pacific and Taiwan

Click here for another presentation from the CHEA International Seminar by Angela Yong-chi Hou of the Higher Education Evaluation and Accereditaion Council of Taiwan.

Presentation by Robert Morse

Click here for a comprehensive and informative presentation by Robert Morse of US News at the CHEA International Seminar in Washington DC on America's best Colleges Rankings: A Brief History.