More Rankings on the Way

Soon it will be springtime in the Northern hemisphere and spring would not be complete without a few more rankings.

The Times Higher Education reputation rankings will be launched in early March at the British Council's Going Global conference in Dubai.

“Almost 50,000 academics have provided their expert insight over just three short annual rounds of the survey, providing a serious worldwide audit of an increasingly important but little-understood aspect of global higher education – a university’s academic brand.”
This year’s reputation rankings will be the based on the 16,639 responses, from 144 countries, to Thomson Reuters’ 2012 Academic Reputation Survey, which was carried out during March and April 2012. The 2011 survey attracted 17,554 responses, and 2010’s survey attracted 13,388 respondents.

The survey is by invitation only and academics are selected to be statistically representative of their geographical region and discipline. All are published scholars, questioned about their experiences in the field in which they work. The average time this year’s respondents spent working in the sector was 17 years. '

Meanwhile, the QS ranking of 30 subjects is coming soon. Until now these have been based on varying combinations of employer opinion, academic opinion and citations. This year they will be adding  an indicator based on the h-index.

Here is a definition from Wikipedia:

"The index is based on the distribution of citations received by a given researchers publications. Hirsch writes:

A scientist has index h if h of his/her N_p papers have at least h citations each, and the other (N_p − h) papers have no more than h citations each.

In other words, a scholar with an index of h has published h papers each of which has been cited in other papers at least h times.^[2] Thus, the h-index reflects both the number of publications and the number of citations per publication. The index is designed to improve upon simpler measures such as the total number of citations or publications. The index works properly only for comparing scientists working in the same field; citation conventions differ widely among different fields.
The h-index serves as an alternative to more traditional journal impact factor metrics in the evaluation of the impact of the work of a particular researcher. Because only the most highly cited articles contribute to the h-index, its determination is a relatively simpler process. Hirsch has demonstrated that h has high predictive value for whether a scientist has won honors like National Academy membership or the Nobel Prize. "

This means that one paper cited once produces an index of 1, 20 papers cited 20 times an index of 20, 100 papers cited 100 times an index of 100 and so on.

The point of this is that it combines productivity and quality as measured by citations and reduces the effect of extreme outliers. This is definitely an improvement for QS.

Freedom Indicator?

It is often argued  that the quality of a a university has something to do with academic freedom. Some Western academic have become noticeably self-righteous about respect for human rights in other countries. There have been criticisms of Yale University's links with Singapore, where gay rights are restricted.

One wonders whether Western campuses should talk so loudly about freedom. A recent incident at Carleton University in Canada suggests that when it comes to human rights some humans are much more equal than others.

Carleton has a freedom wall where students can write thoughts that are forbidden in the rest of the campus, probably even in much or most of Canada. Even this was too much for Arun Smith, a seventh year (yes, that's right) human rights student. From the Macleans On Campus blog:

"Seventh-year Carleton University human rights [apparently human rights and political science with a minor in sexuality studies] student Arun Smith has apparently not been in school long enough to learn that other people have rights to opinions that differ from his. After the “free speech wall” on campus was torn down, he posted a message to his Facebook wall claiming responsibility. “If everyone speaks freely we end up simply reinforcing the hierarchies that are created in our society,” it read. The display had been erected by campus club Carleton Students for Liberty and students were encouraged to write anything they wanted on the paper. Someone wrote “abortion is murder” and “traditional marriage is awesome.” GBLTQ Centre volunteer Riley Evans took offense, telling The Charlatan student newspaper that the wall was attacking those who have had abortions and those in same-sex relationships."

It appears that Arun Smith has been widely condemned and that he will be punished. What seems to have been passed over is that it is apparently necessary to have a wall where mainstream religious opinions can be expressed. Yes, I know that "abortion is murder" is a gross simplification of a complex philosophical issue but whose fault is it that it has to be expressed in three words?

The Justice Centre for Constitutional Freedoms has issued a Campus Freedom Index for Canadian universities. Unsurprisingly, Carleton gets a C and 3 Fs. The best appears to be St Thomas with one A and three Bs

What about an international edition?

The Commission Strikes Back

Jordi Curell from the European Commission's Directorate General for Education and Culture has written in defence of the proposed U-multirank university ranking system. He starts:

"Is Times Higher Education worried about competition to its world university ranking from U-Multirank? It looks like it from the tone of its reporting on the new European ranking initiative launched in Dublin at the end of January. "

He concludes:

"However, the EU should not finance U-Multirank forever; this should be limited to the start-up phase. That is why the contract for delivering the ranking includes the design of a self-sustaining business plan and organising the transition to this model.

These are challenging times for higher education in Europe, and the purpose behind U-Multirank could not be clearer. Our objective is improving the performance of Europe's higher education systems – not just selling newspapers."
 

 By the way, THE is a magazine now, not a newspaper.

Wasting Money

The League of European Research Universities claims to be upset about the 2 million Euros that the Europan Uniion is spending on its proposed multi-dimensional university ranking. What do they or their American counterpartsthink about things like this?

"The president [of the US] will invest $55 million in a new First in the World competition, to support the public and private colleges and non-profit organizations as they work to develop and test the next breakthrough strategy that will boost higher education attainment and student outcomes. The new program will also help scale-up those innovative and effective practices that have been proven to boost productivity and enhance teaching and learning on college campuses."

Update on U-Multirank

Using data supplied by institutions is not a good idea for any international ranking. Apart from questions of reliability and objectivity, there is always the possibility of "conscientious objectors" disrupting the ranking process by refusing to take part.

The League of European Research Universities has just announced that it will not participate in the European Union's proposed multi-dimensional ranking project.

Membership of the League is by invitation only and "is periodically evaluated against a broad set of quantitative and qualitative criteria, such as research volume, impact and funding, strengths in PhD training, size and disciplinary breadth, and peer-recognised academic excellence." At the moment , it includes Oxford, Cambridge, Heidelberg, Geneva and Strasbourg universities.

According to Times Higher Education

'Kurt Deketelaere, secretary-general of Leru, said the organisation, whose members include the universities of Oxford, Cambridge and Edinburgh, believes the project is ill-conceived and poorly designed.

"We consider U-Multirank at best an unjustifiable use of taxpayers' money and at worst a serious threat to a healthy higher education system," he said. "Leru has serious concerns about the lack of reliable, solid and valid data for the chosen indicators in U-Multirank, about the comparability between countries, about the burden put upon universities to collect data and about the lack of 'reality-checks' in the process thus far."'

Considering the sort of thing that European universities spend texpayers' money on, 2 million Euros seem comparatively trivial. There are no doubt genuine concerns about the reliability of data produced by institutions and comparability between countries but if you can swallow the camel of Rice University and Moscow Engineering Physics Institute as the best in the world for research influence according to Times Higher and Thomson Reuters, then why strain at U-Multirank's gnats?

And as for a serious threat to higher education, I think someone should sit down for a few minutes and have a cup of tea before making any more statements.

Another Ranking on the Way

The European Union has just launched its U- Multirankranking system. Data will be collected during 2013 and the results will be out in 2014.

According to the European Commissioner for Education the aim is to to provide a multi-dimensional analysis of institutions rather than one that emphasises research excellence.

It is certainly true that the prominent international rankings focus largely or almost entirely  on research. The Shanghai rankings are all about research except perhaps the 10 percent for Nobel and Field awards given to alumni. The QS rankings have a weighting at least 60 per cent for research (citations per faculty and academic survey) and maybe more since research only faculty are counted in the faculty student ratio. Times Higher Education allocates 30 percent for research influence (citations) and 30 percent for research (volume, income and reputation). Since the scores for the citations indicators are substantially higher than those for the others  it can carry an even greater weight for many universities.  Rankings that measure other significant parts of a university’s mission might therefore fill an obvious gap.

But the new rankings are going to rely on data submitted by universities. What happens if several major institutions, including perhaps many British ones, decline to take part?

 

Article in the Chronicle of Higher Education

The Chronicle of Higher Education has an article by Debra Houry on university rankings. She makes some pertinent comments although her recommendations at the end are either impractical or likely to make things worse.

She points out that several American colleges have been found to have submitted inflated data to the US News and World Report in order to boost their standing in the rankings and notes that "there is an inherent conflict of interest in asking those who are most invested in the rankings to self-report data."

This is true and is even more true of international rankings. One reason why the Shanghai rankings are more credible than those produced by QS and Times Higher Education is that they rely entirely on reasonably accessible public data. Using information provided by institutions is a risky business which, among other things, could lead to universities refusing to cooperate, something which ended the promising Asiaweek rankings in 2001.

She then argues that measures of student quality such as high school class rank and SAT scores should be abandoned because they "discourage colleges from selecting a diverse student body. An institution that begins accepting more African-American students or students from low-income families—two groups that have among the lowest SAT scores, according to the College Board—might see its ranking drop because the average SAT score of its freshmen has gone down."

True, but on the other hand an institution that puts more emphasis on standardized test scores might rise in the rankings and might also increase its intake of Asian students and so become more diverse. Are Asian students less diverse than African- Americans? They are certainly likely to be far more varied in terms of mother tongue, political opinions or religious affiliation.

She also points out that it is now a bit late to count printed books in the law school rankings and wonders about using ratemyprofessor to assess teaching quality.

Then there is a familiar criticism of the QS Stars rating systems.

Professor Houry also makes the common complaint that the rankings do not capture unique features of institutions such as "a program called Living-Learning Communities, which gives upperclassmen at Emory incentives to live on campus and participate in residential learning. But you would never learn about that from the ranking formulas."

The problem is that a lot of people are interested in how smart graduates are or how much  research, if any, faculty are doing or how much money is flowing in. But seriously, what is so interesting about upperlassmen living on campus? In any case if this is unique would you expect  any measure to "capture" it.

Finally she concludes "ranking organizations should develop more-meaningful measures around diversity of students, job placement, acceptance into professional schools, faculty membership in national academies, and student engagement. Instead of being assigned a numerical rank, institutions should be grouped by tiers and categories of programs. The last thing students want is to be seen as a number. Colleges shouldn't want that, either."

But all of these raise more problems than solutions. If we really want diversity of students shouldn't we counting counting conservative students  or evangelical Christians? Job placement raises the possibility, already found in law school rankings, of counting graduates employed in phony temporary jobs or glorified slave labor (internships). Membership in national academies? A bit elitist, perhaps?

The Last Print Issue of Newsweek

At the end of the last print issue of Newsweek (31/12/12) is a special advertising feature about the Best Colleges and Universities in Asia, Korea, Vietnam, Hong Kong, Japan and the USA.

The feature is quite revealing about how the various global rankings are regarded in Asia. There is nothing about the Shanghai rankings, the Taiwan rankings, Scimago, Webometrics, URAP or the Leiden Ranking.

There are five references to the QS rankings one of which calls them "revered" (seriously!) and another that refers to the "SQ" rankings.

There are two to Times Higher Education, two to America's Best Colleges, one to Community College Weekly and one to the Korean Joongang Daily university rankings.

A Bit More on the THE Citations Indicator

I have already posted on the citations (research influence) indicator in the Times Higher Education World University Rankings and how it can allow a few papers to have a disproportionate impact. But there are other features of this indicator that affect its stability and can produce large changes even if there is no change in methodology.

This indicator has a weighting of 30 percent. The next most heavily weighted indicator is the research reputation survey which carries a weighting of 18 percent and is combined with number of publications (6 percent) and research income (6 percent) to produce a weighting of 30 percent for research: volume, income and reputation.

It might  be argued that the citations indicator accounts for only 30 percent of the total weighting so that anomalously high scores given to obscure or mediocre institutions for citations would be balanced or diluted by scores on the other indicators which have a weighting of 70 percent.

The problem with this is that  the scores for the citation indicator are often substantially higher than the scores for other indicators, especially in the 300 - 400 region of the rankings so that the impact of this indicator is correspondingly increased. Full data can be found on the 2012-13 iPhone app.


For example, the University of Ferrara in 378th place with a score of 58.5 for citations has a total score of 31.3 so that nearly 60% of its total score comes from the citations indicator. King Mongkut's Unversity of Technology, Thonburi,  in 389th place has a score of 68.4 for citations but its total score is 30.3 so that two thirds of its total score comes from citations. Southern Methodist University in 375th place gets 67.3 for citations which after weighting comes close to providing two thirds of its overall score of 31.6. For these universities a proportional change in the final processed score for citations would have a greater impact than a similar change in any of the other indicators.

Looking at the bottom 25 universities in the top 400, in eight cases the citation indicator provides half or more of the total score and in 22 cases it provides a third or more. Thus, the indicator could have  more impact on total scores than its weighting of 30 percent would suggest.
 


It is also  noticeable that the mean score for citations of the THE top 400 universities is much higher than that for research, about 65 compared to about 41.This disparity is especially large as we reach the 200s and the 300s.

So we find Southern Methodist University has a score of 67.3 for citations  but 9.0 for research.Then  the University of Ferrara  has a score of 58.5 for citations and 13.0 for research. King Mongkut's University of Technology, has a score of 68.4 for citations and 10.2 for research.


One reason why the scores for the citations indicator are so high is the "regional modification" introduced by Thompson Reuters in 2011. To simplify, this means that the number of citations to a university in a given year and a a given field is divided by the square root of the average number of citations in the field and year for all universities in the country

So if a university in country A receives 100 citations in a certain year of publication and in a certain field and the average impact for that year and field for all universities is 100 then the university will get a score of 10 (100 divided by 10). If a university in country B receives 10 citations in the same year and the same field then but the average impact from all universities in the country is 1 then the citations score for that field year would also be 10 (10 divided by 1).

This drastically reduces the gap for citations between countries that produce many citations and those that produce few. Thompson Reuters justify this by saying that in some countries it is easier to get grants and to travel and  join the networks that lead to citations and international collaborations than in others. The problem with this is that it can produce some rather dubious results.

Let us consider the University of Vigo and the University of Surrey. Vigo has an overall score of 30.7 and is in 383rd place. Surrey is just behind with a score of 30.5 and is 388th place.

But, with the exception of citations Surrey is well  ahead of Vigo for everything: teaching (31.1 to 19.4), international outlook (80.4 to 26.9), industry income (45.4 to 36.6) and research (25.0 to 9.5).

Surrey, however, does comparatively badly for citations with a score of only 21.6. It does have a contribution to the massively cited 2008 review of particle physics but the university has too many publications for this review to have much effect. Vigo however has a score of 63.7 for citations which may be because of a much cited paper containing a new algorithm for genetic analysis but also presumably because it received, along with other Spanish and Portuguese-speaking universities, a boost from the regional modification.

There are several problems with this modification. First, it can contribute another element of instability. If we observe a university's score for citations has declined it could be because its citations have decreased overall or in key fields or because a much cited paper has slipped out of the five year  period of assessment. It could also be that the number of publications has increased without a corresponding increase in citations.

Applying the regional modification could mean that a university's score would be affected by the fluctuations in the impact of the country's universities as a whole. If there was an increase in the number of citations or reduction in publications nationally then this would reduce the citations score of a particular university since the university's score would be divided by the square root of a larger number.

This could lead to the odd situation where stringent austerity measures lead to the emigration of talented researchers and eventually a fall in citations but some universities in the country may improve since they are being compared to a smaller national average.

The second problem is that it can lead to misleading comparisons. It would be a mistake to conclude that Vigo is a better university than Surrey or about the same or even that its research influence is more significant. What has happened is that is that Vigo is more ahead of the Spanish average than Surrey is ahead of the British.

Another problematic feature of the citations indicator is that its relationship with the research indicator is rather modest. Consider that 18 out of 30 points for research are from the reputation survey whose respondents are drawn from those researchers whose publications are in the ISI databases while the citations indicator counts citations in precisely those papers. Then another 6 percent goes to research income which we would expect to have some relationship with the the quality of research.

Yet the correlation between the scores for research and citations for the top 400 universities is modest at .409 which calls into question the validity of one of the indicators or both of them.

A further problem is that this indicator only counts the impact of papers that make it into an ISI database. A university where most of the faculty do not publish in ISI indexed journals would do no worse than one where there was a lot of publications but not many citations or not many citations in low cited fields.

To conclude, the current construction of the citations indicator has the potential to produce anomalous results and to introduce a significant degree of instability into the rankings.

QS Stars Again

The New York Times has an article by Don Guttenplan on the QS Stars ratings which award universities one to five stars according to eight criteria, some of which are already assessed by the QS World University Rankings and some of which are not. The criteria are teaching quality, innovation and knowledge transfer, research quality, specialist subject, graduate employability, third mission, infrastructure and internationalisation. 

The article has comments from rankings experts Ellen Hazelkorn, Simon Marginson and Andrew oswald.

The QS Stars system does raise issues about commercial motivations and conflict of interests. Nonetheless, it has to be admitted that it does fill a gap in the current complex of international rankings. The Shanghai, QS and Times Higher Education rankings may be able to distinguish between Harvard and Cornell or Oxford and Manchester but they rank only a fraction of the world's universities. There are national ranking and rating systems but so far anyone wishing to compare middling universities in different countries has very little information available.

There is, however, the problem that making distinction among little known and mediocre universities, a disproportionate number of which are located in Indonesia,  means a loss of discrimination at the top or near top. The National University of Ireland Galway, Ohio State University and Queensland University of Technology get the same five stars as Cambridge and Kings College London.

The QS Stars has potential to offer a broader assessment of university quality but it would be better for everybody if it was kept completely separate from the QS World University Rankings.

The URAP Ranking


 Another ranking that has been neglected is the University Ranking by Academic Performance [www.urapcenter.org] started by the Middle East Technical University in 2009. This has six indicators: number of articles (21%), citations (21%), total documents (10%), journal impact total (18%), journal citation impact total (15%) and international collaboration (10%).

A distinctive feature is that these rankings provide data for 2000 universities,much more than the current big three.

The top ten are:

1.  Harvard
2.  Toronto
3.  Johns Hopkins
4.  Stanford
5.  UC Berkeley
6.  Michigan Ann Argot
7.  Oxford
8.  Washington Seattle
9.  UCLA
10. Tokyo

 These rankings definitely favour size over quality as shown  by the strong performance of Toronto and Johns Hopkins and the lowly position of Caltech in 51st place and Princeton in 95th. Still, they could be very helpful for countries with few institutions in the major rankings.

The Taiwan Rankings

It is unfortunate that the "big three" of the international ranking scene -- ARWU (Shanghai), THE and QS -- receive a disproportionate amount of public attention while several research-based rankings are largely ignored. Among them is the National Taiwan University Ranking which until this year was run by the Higher Education Evaluation and Acceditation Council of Taiwan.

The rankings, which are based on the ISI databases, assign a weighting of 25%  to research productivity (number of articles over the last 11 years, number of articles in the current year), 35% to research impact (number of citations over the last 11 years, number of citations in the current year, average number of citations over the last 11 years) and 40 % to research excellence (h-index over the last 2 years, number of highly cited papers, number of articles in the current year in highly cited journals).

Rankings by field and subject are also available.

There is no attempt to assess teaching or student quality and publications in the arts and humanities are not counted.

These rankings are a valuable supplement to the Shanghai ARWU. The presentation of data over 11 and 1 year periods allows quick comparisons of changes over a decade.

Here are the top ten.

1. Harvard
2. Johns Hopkins
3. Stanford
4. University of Washington at Seattle
5. UCLA
6. University of Washington Ann Arbor
7. Toronto
8. University of California Berkeley
9. Oxford
10. MIT

High-flyers in other rankings do not do especially well here. Princeton is 52nd, Caltech 34th, Yale 19th, Cambridge 15th most probably because they are relatively small or have strengths in the humanities.

Article in University World News

GLOBAL

Ranking’s research impact indicator is skewed

Richard Holmes15 November 2012 Issue No:247

The 2012 Times Higher Education World University Rankings consisted of 13 indicators grouped into five clusters. One of these clusters consisted of precisely one indicator, research impact, which is measured by normalised citations and which THE considers to be the flagship of the rankings.

It is noticeable that this indicator, prepared for THE by Thomson Reuters, gives some universities implausibly high scores. I have calculated the world's top universities for research impact according to this indicator.

Since it accounts for 30% of the total ranking it clearly can have a significant effect on overall scores. The data are from the profiles, which can be accessed by clicking on the top 400 universities. Here are the top 20:

1. Rice University
1. Moscow (State) Engineering Physics Institute (MEPhI)
3. University of California Santa Cruz
3. MIT
5. Princeton
6. Caltech
7. University of California Santa Barbara
7. Stanford
9. University of California Berkeley
10. Harvard
11. Royal Holloway London
12. Chicago
13. Northwestern
14. Tokyo Metropolitan University
14. University of Colorado Boulder
16. University of Washington Seattle
16. Duke
18. University of California San Diego
18. University of Pennsylvania
18. Cambridge

There are some surprises here, such as Rice University in joint top place, second-tier University of California campuses at Santa Cruz (equal third with MIT) and Santa Barbara (equal seventh with Stanford) placed ahead of Berkeley and Los Angeles, Northwestern almost surpassing Chicago, and Tokyo Metropolitan University ahead of Tokyo and Kyoto universities and everywhere else in Asia.

It is not totally implausible that Duke and the University of Pennsylvania might be overtaking Cambridge and Oxford for research impact, but Royal Holloway and Tokyo Metropolitan University?

These are surprising, but Moscow State Engineering Physics Institute (MEPhI) as joint best in the world is a definite head-scratcher.

Other oddities

Going down a bit in this indicator we find more oddities.

According to Thomson Reuters, the top 200 universities in the world for research impact include Notre Dame, Carleton, William and Mary College, Gottingen, Boston College, University of East Anglia, Iceland, Crete, Koc University, Portsmouth, Florida Institute of Technology and the University of the Andes.

On the other hand, when we get down to the 300s we find that Tel Aviv, National Central University Taiwan, São Paulo, Texas A&M and Lomonosov Moscow State University are assigned surprisingly low places. The latter is actually in 400th place for research impact among the top 400.

It would be interesting to hear what academics in Russia think about an indicator that puts MEPhI in first place in the world for research impact and Lomonosov Moscow State University in 400th place.

I wonder too about the Russian reaction to MEPhI as overall second among Russian and Eastern European universities. See here, here and here for national university rankings and here and here for web-based rankings.

Déjà vu

We have been here or somewhere near here before.

In 2010 the first edition of the THE rankings placed Alexandria University in the world's top 200 and fourth for research impact. This was the result of a flawed methodology combined with diligent self-citation and cross-citation by a writer whose lack of scientific credibility has been confirmed by a British court.

Supposedly the methodology was fixed last year. But now we have an indicator as strange as in 2010, perhaps even more so.

So how did MEPhI end up as world's joint number one for research impact? It should be emphasised that this is something different from the case of Alexandria. MEPhI is, by all accounts, a leading institution in Russian science. It is, however, very specialised and fairly small and its scientific output is relatively modest.

First, let us take a look at another source, the Scimago World Report, which gives MEPhI a rank of 1,722 for total publications between 2006 and 2010, the same period that Thomson Reuters counts.

Admittedly, that includes a few non-university institutions. It has 25.9 % of its publication in the top quartile of journals. It has a score of 8.8 % for excellence – that is, the proportion of publications in the most highly cited 10% of publications. It gets a score of 1.0 for ‘normalised impact’, which means that it gets exactly the world average for citations adjusted by field, publication type and period of publication.

Moving on to Thomson Reuters’ database at the Web of Science, MEPhI has had only 930 publications listed under that name between 2006 and 2010, although there were some more under other name variants that pushed it over the 200 papers per year threshold to be included in the rankings.

It is true that MEPhI can claim three Nobel prize winners, but they received awards in 1958 and 1964 and one of them was for work done in the 1930s.

So how could anyone think that an institution that now has a modest and specialised output of publications and a citation record that, according to Scimago, does not seem significantly different from the international average using the somewhat larger Scopus database – Thomson Reuters uses the more selective ISI Web of Science – could emerge at the top of Thomson Reuters' research impact indicator?

Furthermore, MEPhI has no publications listed in the ISI Social Science Citation Index and exactly one (uncited) paper in the Arts and Humanities Index on oppositional politics in Central Russia in the 1920s.

There are, however, four publications assigned to MEPhI authors in the Web of Science that are listed as being on the literature of the British Isles, none of which seem to have anything to do with literature or the British Isles or any other isles, but which have a healthy handful of citations that would yield much higher values if they were classified under literature rather than physics.

Citations

Briefly, the essence of Thomson Reuters' counting of citations is that a university's citations are compared to the average for a field in a particular period after publication.

So if the average for a field is 10 citations per paper one year after publication, then 300 citations of a single paper one year after publication would count as 30 citations. If the average for the field was one citation it would count as 300 citations.

To get a high score in the Thomson Reuters research impact indicator, it helps to get citations soon after publication, preferably in a field where citations are low or middling, rather than simply getting many citations.

The main cause of MEPhI's research impact supremacy would appear to be a biennial review that summarises research over two years in particle physics and is routinely referred to in the literature review of research papers in the field.

The life span of each review is short since it is superseded after two years by the next review so that the many citations are jammed into a two-year period, which could produce a massive score for ‘journal impact factor’.

It could also produce a massive score on the citations indicator in the THE rankings. In addition, Thomson Reuters then gives a weighting to countries according to the number of citations. If citations are generally low in their countries, then institutions get some more value added.

The 2006 “Review of Particle Physics” published in Journal of Physics G, received a total of 3,662 citations, mostly in 2007 and 2008. The 2008 review published in Physics Letters B had 3,841 citations, mostly in 2009 and 2010, and the 2010 review, also published in Journal of Physics G, had 2,592 citations, nearly all in 2011. Someone from MEPhI was listed as co-author of the 2008 and 2010 reviews.

It is not the total number of citations that matters, but the number of citations that occur soon after publication. So the 2008 review received 1,278 citations in 2009, but the average number of citations in 2009 to other papers published in Physics Letters B for 2008 was 4.4.

So the 2008 review received nearly 300 times as many citations in the year after publication as the mean for that journal. Add the extra weighting for Russia and there is a very large boost to MEPhI's score from just a single publication. Note that these are reviews of research so it is likely that there had already been citations to the research publications that are reviewed. Among the highlights of the 2010 review are 551 new papers and 108 mostly revised or new reviews.

If the publications had a single author or just a few authors from MEPhI then this would perhaps suggest that the institute had produced or made a major contribution to two publications of exceptional merit. The 2008 review in fact had 173 co-authors. The 2010 review listed 176 members of the Particle Data Group who are referred to as contributors.

It seems then that MEPhI was declared the joint best university for research impact largely because of two multi-author (or contributor) publications to which it had made a fractional contribution. Those four papers assigned to literature may also have helped.

As we go through the other anomalies in the indicator, we find that the reviews of particle physics contributed to other high research impact scores. Tokyo Metropolitan University, Royal Holloway London, the University of California at Santa Cruz, Santa Barbara and San Diego, Notre Dame, William and Mary, Carleton and Pisa also made contributions to the reviews.

This was not the whole story. Tokyo Metropolitan University benefited from many citations to a paper about new genetic analysis software and Santa Cruz had contributed to a massively cited multi-author human haplotype map.

Number of authors

This brings us to the total number of publications. There were more than 100 authors of or contributors to the reviews but for some institutions the number of citations had no discernible effect and for others not very much.

Why the difference? Here size really does matter and small really is beautiful.

MEPhI has relatively few publications overall. It only just managed to cross the 200 publications per year threshold to get into the rankings. This means that the massive and early citation of the reviews was averaged out over a small number of publications. For others the citations were absorbed by many thousands of publications.

These anomalies and others could have been avoided by a few simple and obvious measures. After the business with Alexandria in 2010 Thomson Reuters did tweak its system, but evidently this was not enough.

First, it would help if Thomson Reuters scrutinised the criteria by which specialised institutions are included in the rankings. If we are talking about how well universities spread knowledge and ideas, it is questionable whether we should count institutions that do research in one or only a few fields.

There are many methods by which research impact can be evaluated. The full menu can be found on the Leiden Ranking site. Use a variety of methods to calculate research impact, especially those like the h-index that are specifically designed to work around outliers and extreme cases.

It would be sensible to increase the threshold of publications for inclusion in the rankings. The Leiden Ranking excludes universities with fewer than 500 publications a year. If a publication has multiple authors, divide the number of citations by the number of authors. If that is too complex then start dividing citations when the number reaches 10 or a dozen.

Do not count reviews, summaries, compilations or other publications that refer to papers that may have already been cited, or at least put them in a separate publication type. Do not count self-citations. Even better, do not count citations within the same institution or the same journal.

Most importantly, calculate the indicator for the six subject groups and then aggregate them. If you think that a fractional contribution to two publications justifies putting MEPhI at the top for research impact in physics, go ahead and give them 100 for natural sciences or physical sciences. But is it reasonable to give the institution any more than zero for arts and humanities, the social sciences and so on?

So we have a ranking indicator that has again yielded some very odd results.

In 2010 Thomson Reuters asserted that it had a method that was basically robust, transparent and sophisticated but which had a few outliers and statistical anomalies about which they would be happy to debate.

It is beginning to look as though outliers and anomalies are here to stay and there could well be more on the way.

It will be interesting to see if Thomson Reuters will try to defend this indicator again.

* Richard Holmes is a lecturer at Universiti Teknologi MARA in Malaysia and author of the University Ranking Watch blog.

Apology for factual error

In an earlier version of this article in University World News I made a claim that Times Higher Education, in their 2012 World University Rankings, had introduced a methodological change that substantially affected the overall ranking scores. I acknowledge that this claim was without factual foundation. I withdraw the claim and apologise without reservation to Phil Baty and Times Higher Education.

Richard Holmes

 

Apology

In a recent article in University World News I made a claim that Times Higher Education in their recent World University Rankings had introduced a methodological change that substantially affected the overall ranking scores. I acknowledge that this claim was without factual foundation. I withdraw the claim and apologise without reservation to Phil Baty and Times Higher education.

More on MEPhI

Right after putting up the post on Moscow State Engineering Physics Institute and its "achievement" in getting the maximum score for research impact in the latest THE - TR World University Rankings, I found this exchange on Facebook.  See my comments at the end.

• 

  Valery Adzhiev So, the best university in the world in the "citation" (i.e. "research influence") category is Moscow State Engineering Physics Institute with maximum '100' score. This is remarkable achivement by any standards. At the same time it scored in "research" just 10.6 (out of 100) which is very, very low result. How on earth that can be?

  October 3 at 3:29pm

• 

  Times Higher Education World University Rankings Hi Valery,
  
  Regarding MEPHI’s high citation impact, there are two causes: Firstly they have a couple of extremely highly cited papers out of a very low volume of papers.The two extremely highly cited papers are skewing what would ordinarily be a very good normalized citation impact to an even higher level.
  
  We also apply "regional modification" to the Normalized Citation Impact. This is an adjustment that we make to take into account the different citation cultures of each country (because of things like language and research policy). In the case of Russia, because the underlying citation impact of the country is low it means that Russian universities get a bit of a boost for the Normalized Citation Impact.
  
  MEPHI is right on the boundary for meeting the minimum requirement for the THE World University Rankings, and for this reason was excluded from the rankings in previous years. There is still a big concern with the number of papers being so low and I think we may see MEPHI’s citation impact change considerably over time as the effect of the above mentioned 2 papers go out of the system (although there will probably be new ones come in).
  
  Hope this helps to explain things.
  THE

  October 4 at 6:42am

• 

  Valery Adzhiev Thanks for your prompt reply. Unfortunately, the closer look at that case only adds rather awkward questions. "a couple of extremely highly cited papers are actually not "papers": they are biannual volumes titled "The Review of Particle Physics" that ...See More

  October 4 at 7:00am

• 

  Valery Adzhiev I continue. There are more than 200 authors (in fact, they are "editors") from more than 100 organisation from all over the world, who produce those volumes. Look: just one of them happened to be affiliated with MEPhI - and that rather modest fact (tha...See More

  October 4 at 7:16am · 1

• 

  Valery Adzhiev Sorry, another addition: I'd just want to repeat that my point is not concerned only with MEPhI - Am talking about your methodology. Look at the "citation score" of some other universities. Royal Holloway, University of London having justt 27.7 in "res...See More

  October 4 at 7:59am · 1

• 

  Alvin See Great observations, Valery.

  October 4 at 8:14am · Edited

• 

  Times Higher Education World University Rankings Hi Valery,
  
  Thanks again for your thorough analysis. The citation score is one of 13 indicators within what is a balanced and comprehensive system. Everything is put in place to ensure a balanced overall result, and we put our methodology up online for...See More

  October 4 at 8:33am · Edited · 1

• 

  Andrei Rostovtsev This is in fact rather philosofical point. There are also a number of very scandalous papers with definitively negative scientific impact, but making a lot of noise around. Those have also high contribution to the citation score, but negative impact t...See More

  
  It is true that two extremely highly cited publications combined with a low total number of publications skewed the results but what is equally or perhaps more important is that  these citations occur in the year or two years after publication when citations tend to be relatively infrequent compared to later years. The 2010 publication is a biennial review, like the 2008 publication, that will be cited copiously for two years after which it will no doubt be superseded by the 2012 edition.
  
  Also, we should note that in the ISI Web of Science, the 2008 publication is classified as "physics, multidisciplinary". Papers listed as multidisciplinary generally get relatively few citations so if the publication was compared to other multidisciplinary papers it would get an even larger weighting. 

  Valery has an excellent point when he points out that these publications have over 100 authors or contributors each (I am not sure whether they are actual researchers or administrators). Why then did not all the other contributors boost their instutitions' scores to similar heights? Partly because they were not in Russia and therefore did not get the regional weighting but also because they were publishing many more papers overall than MEPhI.  
  
  So basically, A. Romaniouk who contributed 1/173rd of one publication was considered as having more research impact than hundreds of researchers at Harvard, MIT, Caltech etc producing hundreds of papers cited hundreds of times.  Sorry, but is this a ranking of research quality or a lottery?
  
  The worse part of THE's reply is this:
  
  

  Thanks again for your thorough analysis. The citation score is one of 13 indicators within what is a balanced and comprehensive system. Everything is put in place to ensure a balanced overall result, and we put our methodology up online for all to see (and indeed scrutinise, which everyone is entitled to do).
  
  We welcome feedback, are constantly developing our system, and will definitely take your comments on board.

  
  The system is not balanced. Citations have a weighting of 30 %, much more than any other  indicator. Even the research reputation survey has a weighting of only 18%.  And to describe as comprehensive an indicator which allows a fraction of one or two publications to surpass massive amounts of original and influential research is really plumbing the depths of absurdity.
  
  I am just about to finish comparing the scores for research and research impact for the top 400 universities. There is a statistically significant correlation but it is quite modest. When research reputation, volume of publications and research income show such a modestcorrelation with research impact it is time to ask whether there is a serious problem with this indicator.
  
  Here is some advice for THE and TR.
  
  

  • First, and surely very obvious, if you are going to use field normalisation then calculate the score for discipline groups, natural sciences, social sciences and so on and aggregate the scores. So give MEPhI a 100 for physical or natural sciences if you think they deserve it but not for the arts and humanities.

  • Second, and also obvious, introduce fractional counting, that is dividing the number of citations by the number of authors of the cited paper.

  • Do not count citations to summaries, reviews or compilations of research.

  • Do not count citations of commercial material about computer programs. This would reduce the very high and implausible score for Gottingen which is derived from a single publication.

  • Do not assess research impact with only one indicator. See the Leiden ranking for the many ways of rating research.

  • Consider whether it is appropriate to have a regional weighting. This is after all an international ranking.

  • Reduce the weighting for this indicator.

  • Do not count self-citations. Better yet do  not count citations from researchers at the same university.

  • Strictly enforce your rule about  not including single subject institutions in the general rankings.
  • Increase the threshold number of publications for inclusion in the rankings from two hundred to four hundred.