Correction

A previous post, "Debate, Anyone" contained some data about comparative rates of self-citation among various univeristies. The methodology used was not appropriate (calcuting the number of articles that contained self-citations as a percentage of the sum of citations). I am recalculating although the relative level of self-citation among universities is most unlikely to be affected.

Debate, anyone?

Times Higher Education and Thomson Reuters have said that they wish to engage and that they will be happy to debate their new rankings methodology. So far we have not seen much sign of a debate although I will admit that perhaps more things were said at the recent seminars in London and Spain than got into print. In particular, they have been rather reticent about defending the citations indicator which gives the whole ranking a very distinctive cast and which is likely to drag down what could have been a promising development in ranking methodology.

First, let me comment on the few attempts to defend this indicator, which accounts for nearly a third of the total weighting and for more in some of the subject rankings. It has been pointed out that David Willetts, British Minister for Universities and Science has congratulated THE on its new methodology.

“I congratulate THE for reviewing the methodology to produce this new picture of
the best in higher education worldwide. It should prompt all of us who care
about our universities to see how we can improve the range and quality of the
data on offer. Prospective students — in all countries — should have good
information to hand when deciding which course to study, and where. With the
world to choose from, it is in the interests of universities themselves to
publish figures on graduate destinations as well as details of degree
programmes.”

Willetts has praised THE for reviewing its methodology. So have many of us but that is not quite the same as endorsing what has emerged from that review.

Steve Smith, President of Universities UK and Vice-Chancellor of Exeter University is explicit in supporting the new rankings, especially the citations component.


But, as we shall see in a moment, there are serious issues with the robustness of citations as a measure of research impact and, if used inappropriately, they can become indistinguishable from a subjective measure of reputation.

The President of the University of Toronto makes a similar point and praises the new rankings’ reduced emphasis on subjective reputational surveys and refers to the citations (knowledge transfer?) indicator.


It might be argued that this indicator is noteworthy for revealing that some universities possess hitherto unsuspected centres of research excellence. An article by Phil Baty in THE of the 16th of September refers to the most conspicuous case, a remarkably high score for citations by Alexandria University, which according to the THE rankings has had a greater research impact than any university in the world except Caltech, MIT and Princeton. Baty suggests that there is some substance to Alexandria University’s extraordinary score. He refers to Ahmed Zuweil, a Nobel prize winner who left Alexandria with a master’s degree some four decades ago. Then he mentions some frequently cited papers by a single author in one journal.

The author in question is Mohamed El Naschie, who writes on mathematical physics and the journals – there are two that should be given the credit for Alexandria’s performance, not one – are Chaos, Solitons and Fractals and the International Journal of Nonlinear Sciences and Numerical Simulation. The first is published by Elsevier and was until recently edited by El Naschie. It has published a large number of papers by El Naschie and these have been cited many times by himself and by some other writers in CSF and IJNSNS.

The second journal is edited by Ji-Huan He of Donghua University in Shanghai, China with El Naschie as co-editor and is published by the Israeli publishing company, Freund Publishing House Ltd of Tel Aviv.

An amusing digression. In the instructions for authors in the journal the title is given as International Journal of Nonlinear Sciences and Numerical Stimulation. This could perhaps be described as a Freundian slip.

Although El Naschie has written a large number of papers and these have been cited many times, his publication and citation record is far from unique. He is not, for example, found in the ISI list of highly cited researchers. His publications and citations were perhaps necessary to push Alexandria into THE’s top 200 universities but they were not enough by themselves. This required a number of flaws in TR’s methodology.

First, TR assigned a citation impact score that compares actual citations of a paper with a benchmark score based on the expected number of citations for a specific subject in a specific year. Mathematics is a field where citations are relatively infrequent and usually occur a few years after publication. Since El Naschie published in a field in which citations are relatively scarce and published quite recently this boosted the impact score of his papers. The reason for using this approach is clear and sensible, to overcome the distorting effects of varying citation practices in different disciplines when comparing individual researchers or departments. But there are problems if this method is used to compare whole universities. A great deal depends on when the cited and citing articles are published and in which subject they were classified by TR.

A question for TR. How are articles classified? Is it possible to influence the category in which they are placed by the use of key words or the wording of the title?

Next, note that TR were measuring average citation impact. A consequence of this is that the publication of large numbers of papers that are cited less frequently than the high fliers could drag down the score. This explains an apparent oddity of the citation scores in the 2010 THE rankings. El Naschie listed nine universities as his affiliation in varying combinations between 2004 and 2008. yet it was only Alexandria that managed to leave the Ivy League and Oxbridge standing in the research impact dust. Recently, El Naschie’s list of affiliations has consisted of Alexandria, Cairo, Frankfurt University and Shanghai Jiao Tong University.

What happened was quite simply that all the others were producing so many papers that El Naschie’s made little or no difference. For once, it would be quite correct if El Naschie announced that he could not have done it without the support of his colleagues. Alexandria University owes its success not only to El Naschie and his citers but also to all those researchers who refrained from submitting articles to ISI–indexed journals or conference proceedings.

TR have some explaining to do here. If an author lists more than one affiliation, are they all counted? Or are fractions awarded for each paper? Is there any limit on the number of affiliations that an author may have. I think that it is two but would welcome clarification

As for the claim that Alexandria is strong in research, a quick look at the Scimago rankings is enough to dispose of that. It is ranked 1,047th in the 2010 rankings, which admittedly include many non-university organizations, for total publications over a decade. Also, one must ask how much of El Naschie’s writing was actually done in Alexandria, seeing that he had eight other affiliations between 2004 and 2008.

It has to be said that even if El Naschie is, as has been claimed in comments on Phil’s THE article and elsewhere, one of the most original thinkers of our time, it is strange that THE and TR should use a method that totally undermines their claim that the new methodology is based on evidence rather than reputation By giving any sort of credence to the Alexandria score, THE are asking us to believe that Alexandria is strong in research because precisely one writer is highly reputed by himself and a few others. Incidentally, will TR tell us what score Alexandria got in the research reputation survey?

I am not qualified to comment on the scientific merits of El Naschie’s work. At the moment it appears, judging from the comments in various physics blogs, that among physicists and mathematicians there are more detractors than supporters. There are also few documented signs of conventional academic merit in recent years such as permanent full time appointments or research grants. None of his papers between 2004 and 2008 in ISI-indexed journals,for example, apparently received external funding. His affiliations, if documented, turn out to be honorary, advisory or visiting.To be fair, readers might wish to visit El Naschie’s site. I will also publish any comments of a non-libellous nature that support or dispute the scientific merits of his writings.

Incidentally, it is unlikely that Alexandria’s score of 19.3 for internationalisation was faked. TR use a logarithm. If there were zero international staff and students a university would get a score of 1 and a score of 19.3 actually represents a small percentage. On the other hand, I do wonder whether Alexandria counted those students in the branch campuses in Lebanon, Sudan and Chad.

Finally, TR did not take the very simple and obvious step of not counting individual self-citations. Had they done so, they would have saved everybody, including themselves a lot of trouble. It would have been even better if they had excluded intra-institutional and intra-journal citation. See here for the role of citations among the editorial board of IJNSNS in creating an extraordinarily high Journal Impact Factor.

THE and TR have done everyone a great service by highlighting the corrosive effect of self citation on the citations tracking industry. It has become apparent that there are enormous variations in the prevalence of self citation in its various forms and that these have a strong influence on the citation impact score.

Professor Dirk Van Damme is reported to have said at the London seminar that the world’s elite universities were facing a challenge from universities in the bottom half of the top 200. If this were the case then THE could perhaps claim that their innovative methodology had uncovered reserves of talent ignored by previous rankings. But what exactly was the nature of the challenge? It seems that it was the efficiency with which the challengers turned research income into citations. And how did they do that?

I have taken the simple step of dividing the score for citations by the score for the research indicator (which includes research income) and then sorting the resulting values. The top ten are Alexandria, Hong Kong Baptist University, Barcelona, Bilkent, William and Mary, ENS de Lyon, Royal Holloway, Pompeu Fabra, University College Dublin, the University of Adelaide.

Seriously, these are a threat to the world’s elite?

The high scores for citations relative to research were the result of a large number of citations or a small number of total publications or both. It is of interest to note that in some cases the number of citations was the result of assiduous self-citation.

This section of the post contained comments about comparative rates of self citation among various universities. The method used was not correct and I am recalculating.
As noted already, using the THE iPad app to change the importance attached to various indicators can produce very different results. This is a list of universities that rise more than a hundred places when the citations indicator is set to ‘not important’. They have suffered perhaps because of a lack of super-cited papers, perhaps also because they just produced too many papers.

Loughborough
Kyushu
Sung Kyun Kwan
Texas A and M
Surrey
Shanghai Jiao Tong University
Delft University of Technology
National Chiao Tung University (Taiwan)
Royal Institute of Technology Sweden
Tokushima
Hokkaido

Here is a list of universities that fall more than 100 places when the citations indicator is set to ‘not important’. They have benefitted from a few highly cited papers or low publication counts or a combination of the two.

Boston College
University of California Santa Cruz
Royal Holloway, University of London
Pompeu Fabra
Bilkent
Kent State University
Hong Kong Baptist University
Alexandria
Barcelona
Victoria University Wellington
Tokyo Metropolitan University
University of Warsaw

There are many others that rise or fell seventy, eighty, ninety places when citations are taken out of the equation. This is not a case of a few anomalies. The whole indicator is one big anomaly.

Earlier Jonathon Adams in a column that has attracted one comment said:

"Disciplinary diversity is an important factor, as is international diversity. How would you show the emerging excellence of a really good university in a less well known country such as Indonesia? This is where we would be most controversial, and most at risk, in using the logic of field-normalisation to add a small weighting in favour of relatively good institutions in countries with small research communities. Some may feel that we got that one only partially right."

The rankings do not include universities in Indonesia, really good or otherwise. The problem is with good, mediocre and not very good universities in the US, UK, Spain, Turkey, Egypt, New Zealand, Poland etc. . It is a huge weighting, not a small one, the universities concerned range from relatively good to relatively bad, in one case the research community seems to consist of one person and many are convinced that TR got that one totally wrong.


Indonesia may be less well known to TR but it is very well known to itself and neighbouring countries.


I will publish any comments by anyone who wishes to defend the citation indicator of the new rankings. Here are some questions they might wish to consider.


Was it a good idea to give such a heavy weighting to research impact, 32.5% in the oerall rankings , 37.5 in at least 2 subject rankings? Is it possible that commercial considerations, citations data being a lucrative business for TR , had something to do with it?

Are citations such a robust indictor? Is there not enough evidence now to suggest that manipulation of citations, including self-citation, intra-institutional citation and intra-journal citation, is so pervasive that the robustness of this measure is very slight?

Since there are several ways to measure research impact, would it not have been a good idea to have used several methods? After all, Leiden University has several different ways of assessing impact. Why use only one?


Why set the threshold for inclusion so low at 50 papers per year?

Never mind the rankings, THE has a huge database

There has been a debate, or perhaps the beginnings of a debate, about international university rankings following the publication of Bahram Bekhradnia's report to the Higher Education Policy Institute with comments in University World News by Ben Sowter, Phil Baty, Frank Ziegele and Frans van Vought  and Philip Altbach and Ellen Hazelkorn and a guest post by Bekhradnia in this blog.

Bekhradnia argued that global university rankings were damaging and dangerous because they encourage an obsession with research, rely on unreliable or subjective data, and emphasise spurious precision. He suggests that governments, universities and academics should just ignore the rankings.

Times Higher Education (THE) has now published a piece by THE rankings editor Phil Baty that does not really deal with the criticism but basically says that it does not matter very much because the THE database is bigger and better than anyone else's. This he claims is "the true purpose and enduring legacy" of the THE world rankings.

Legacy? Does this mean that THE is getting ready to abandon rankings, or maybe just the world rankings, and go exclusively into the data refining business? 

Whatever Baty is hinting at, if that is what he is doing, it does seem a rather insipid defence of the rankings to say that all the criticism is missing the point because they are the precursor to a big and sophisticated database.

The article begins with a quotation from Lydia Snover, Director of Institutional Research, at MIT:

“There is no world department of education,” says Lydia Snover, director of institutional research at the Massachusetts Institute of Technology. But Times Higher Education, she believes, is helping to fill that gap: “They are doing a real service to universities by developing definitions and data that can be used for comparison and understanding.”

This sounds as though THE is doing something very impressive that nobody else has even thought of doing. But Snover's elaboration of this point in an email gives equal billing to QS and THE as definition developers and suggests the definitions and data that they provide will improve and expand in the future, implying that they are now less than perfect. She says:

"QS and THE both collect data annually from a large number of international universities. For example, understanding who is considered to be “faculty” in the EU, China, Australia, etc.  is quite helpful to us when we want to compare our universities internationally.  Since both QS and THE are relatively new in the rankings business compared to US NEWS, their definitions are still evolving.  As we go forward, I am sure the amount of data they collect and the definitions of that data will expand and improve."

Snover, by the way , is a member of the QS advisory board, as is THE's former rankings  "masterclass" partner, Simon Pratt.

Baty offers a rather perfunctory defence of the THE rankings. He talks about rankings bringing great insights into the shifting fortunes of universities. If we are talking about year to year changes then the fact that THE purports to chart shifting fortunes is a very big bug in their methodology. Unless there has been drastic restructuring universities do not change much in a matter of months and any ranking that claims that it is detecting massive shifts over a year is simply advertising its deficiencies.

The assertion that the THE rankings are the most comprehensive and balanced is difficult to take seriously. If by comprehensive it is meant that the THE rankings have more indicators than QS or Webometrics that is correct. But the number of indicators does not mean very much if they are bundled together and the scores hidden from the public and if some of the indicators, the teaching survey and research survey for example, correlate so closely that they are effectively the same thing. In any case, The Russian Round University Rankings have 20 indicators compared with THE's 13 in the world rankings.

As for being balanced, we have already seen Bekhradnia's analysis showing that even the teaching and international outlook criteria in the THE rankings are really about research. In addition, THE gives almost a third of its weighting to citations. In practice that is often even more because the effect of the regional modification, now applied to half the indicator, is to boost in varying degrees the scores of everybody except those in the best performing country. 

After offering a scaled down celebration of the rankings, Baty then dismisses critics while announcing that THE "is quietly [seriously?] getting on with a hugely ambitious project to build an extraordinary and truly unique global resource." 

Perhaps some elite universities, like MIT, will find the database and its associated definitions helpful but whether there is anything extraordinary or unique about it remains to be seen.

This WUR had such promise

The new Times Higher Education World University Rankings of 2010 promised much, new indicators based on income, a reformed survey that included questions on postgraduate teaching, a reduction in the weighting given to international students.

But the actual rankings that came out in September were less than impressive.  Dividing the year's intake of undergraduate students by the total of academic faculty looked rather odd. Counting the ratio of doctoral students to undergraduates, while omitting masters programs, was an invitation to the herding of marginal students into substandard doctoral degree programmes.

The biggest problem though was the insistence on giving a high weighting – somewhat higher than originally proposed -- to citations. Nearly a third of the total weighting was assigned to the average citations per paper normalized by field and year. The collection of statistics about citations is the bread and butter of Thomson Reuters (TR), THE’s  data collector, and one of their key products is the Incites system, which apparently was the basis for their procedure during the 2010 ranking exercise. This compares the citation records of academics with international scores benchmarked by year and field. Of course, those who want to find out exactly where they stand have to find out what the benchmark scores are and that is something that cannot be easily calculated without Thomson Reuters.

Over the last two or three decades the number of citations received by papers, along with the amount of money attracted from funding agencies, has become an essential sign of scholarly merit. Things have now reached the point where, in many universities, research is simply invisible unless it has been funded by an external agency and then published in a journal noted for being cited frequently by writers who contribute to journals that are frequently cited. The boom in citations has begun to resemble classical share and housing bubbles as citations acquire an inflated value increasingly detached from any objective reality.

It has become clear that citations can be manipulated as much as, perhaps more than, any other indicator used by international rankings. Writers can cite themselves, they can cite co-authors, they can cite those who cite them. Journal editors and reviewers can  make suggestions to submitters about who to cite. And so on.

Nobody, however, realized quite how unrobust citations might become until the unplanned intersection of THE’s indicator and a bit of self citation and mutual citation by two peripheral scientific figures raised questions about the whole business.

One of these two was Mohamed El Naschie who comes from a wealthy Egyptian family. He studied in Germany and took a Ph D in engineering at University College London. Then he taught in Saudi Arabia while writing several papers that appear to have been of an acceptable academic standard although not very remarkable. 

But this was not enough. In 1993 he started a new journal dealing with applied mathematics and theoretical physics called Chaos, Solitons and Fractals (CSF), published by the leading academic publishers, Elsevier. El Naschie’s journal published many papers written by himself. He has, to his credit, avoided exploiting junior researchers or insinuating himself into research projects to which he has contributed little. Most of his papers do not appear to be research but rather theoretical speculations many of which concern the disparity between the mathematics that describes the universe and that which describes subatomic space and suggestions for reconciling the two.

Over the years El Naschie has listed a number of universities as affiliations. The University of Alexandra was among the most recent of them. It was not clear, however, what he did at or for the university and it was only recently, after the publication of the 2010 THE World University Rankings, that there is documentation of any official connection.

El Naschie does not appear to be highly regarded by physicists and mathematicians, as noted earlier in this blog,  and he has been criticized severely in the physics and mathematics blogosphere.  He has, it is true, received some very vocal support but he is not really helped by the extreme enthusiasm and uniformity of style of his admirers. Here is a fairly typical example, from the comments in Times Higher Education: 

“As for Mohamed El Naschie, he is one of the most original thinkers of our time. He mastered science, philosophy, literature and art like very few people. Although he is an engineer, he is self taught in almost everything, including politics. Now I can understand that a man with his charisma and vast knowledge must be the object of envy but what is written here goes beyond that. My comment here will be only about what I found out regarding a major breakthrough in quantum mechanics. This breakthrough was brought about by the work of Prof. Dr. Mohamed El Naschie”

Later, a professor at Donghua University, China, Ji-Huan He, an editor at El Naschie’s  journal, started a similar publication, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS), whose editorial board included El Naschie. This journal was published by the respectable and unpretentious Israeli company, Freund of Tel Aviv. Ji-Huan He’s journal has published 29 of his own papers and 19 by El Naschie. The  two journals have contained articles that cite and are cited by articles in the other. Since they deal with similar topics some degree of cross citation is to be expected but here it seems to be unusually large.

Let us look at how El Naschie worked. An example is his paper, ‘The theory of Cantorian spacetime and high energy particle physics (an informal review)’, published in Chaos, Solitons and Fractals,41/5, 2635-2646, in  September  2009.

There are 58 citations in the bibliography. El Naschie cites himself 24 times, 20 times to papers in Chaos, Solitons and Fractals and 4 in IJNSNS.  Ji-Huan He is cited twice along with four  other authors from CSF. This paper has been cited 11 times, ten times in CSF in issues of the journal published later in the year.

Articles in mathematics and theoretical physics do not get cited very much. Scholars in those fields prefer to spend time thinking about an interesting paper before settling down to comment. Hardly any papers get even a single citation in the same year. Here we have 10 for one paper. That might easily be 100 times the average for that discipline and that year.

The object of this exercise had nothing to do with the THE rankings. What it did do was to push El Naschie’s  journal into the top ranks of scientific journals as measured by the Journal Impact Factor, that is the number of citations per paper within a two year period. It also meant that for a brief period El Naschie was listed by Thomson Reuters’ Science Watch as a rising star of research.

Eventually, Elsevier appointed a new editorial board at CSF that did not include El Naschie. The journal did however continue to refer to him as the founding editor. Since then the number of citations has declined sharply.

Meanwhile, Ji-huan  He was also accumulating a large number of citations, many of them from conference proceedings that he had organized. He was launched into the exalted ranks of the ISI Highly Cited Researchers and his journal topped the citation charts in mathematics. Unfortunately, early this year Freund sold off its journals to the reputed German publishers De Gruyter, who appointed a new editorial board that did not include either him or El Naschie.

El Naschie, He and a few others have been closely scrutinized by Jason Rush, a mathematician formerly of the University of Washington. Rush was apparently infuriated by El Naschie s unsubstantiated claims to have held senior positions at a variety of universities including Cambridge, Frankfurt, Surrey and Cornell. Since 2009 he has closely, perhaps a little obsessively, published a blog that chronicles the activities of El Naschie and those associated with him. Most of what is known about El Naschie and He was unearthed by his blog, El Naschie Watch.

Meanwhile, Thomson Reuters were preparing their analysis of citations for the THE rankings. They used the Incites system and compared the number of citations with benchmark scores representing the average for year and field.

This meant that for this criterion a high score did not necessarily represent a large number of citations. It could simply represent more citations than normal in a short period of time in fields where citation was infrequent and, perhaps more significantly since we are talking about averages here, a small total number of publications. Thus, Alexandria, with only a few publications but listed as the affiliation of an author who was cited much more frequently than usual in theoretical physics or applied mathematics, did spectacularly well.


This is rather like declaring Norfolk (very flat according to Oscar Wilde) the most mountainous county in England because of a few hillocks that were nonetheless relatively much higher than the surrounding plains.

Thomson Reuters would have done themselves a lot of good if they had taken the sensible course of using several indicators of research impact, such as total citations, citations per faculty, the h-index or references in social media or if they had allocated a smaller weighting to the indicator or if they had imposed a reasonable  threshold number of publications instead of just 50 or if they had not counted self-citations, or citations within journals or if they had figured out a formula to detect mutual citations..

So, in September  THE published its rankings with University of Alexandria in the top 200 overall and in fourth place for research impact, ahead of Oxford, Cambridge and most of the Ivy league. Not bad for a university that had not even been counted by HEEACT, QS or the Shanghai rankings and that in 2010 had lagged behind two other institutions in Alexandria itself in Webometrics.

When the rankings were published THE pointed out that Alexandria had once had a famous library and that a former student had gone on to the USA to eventually win a Nobel prize decades later. Still, they did concede that the success of Alexandria was mainly due  to one "controversial" author.

Anyone with access to the Web of Science could determine in a minute precisely who the controversial author was. For a while it was unclear exactly how a few dozen papers and a few hundred citations could put Alexandria among the world’s elite. Some observers wasted time wondering if  Thomson Reuters had been counting papers from a community college in Virginia or Minnesota, a branch of the Louisiana State University or federal government offices in the Greater Washington area. Eventually, it was clear that El Naschie could not, as he himself asserted, have done it by himself: he needed the help of the very distinctive features of Thomson Reuters’ methodology.

There were  other oddities in the 2010 rankings. Some might have accepted a high placing for Bilkent University in Turkey. It was well known for its Academic English programs. It also had one much cited article whose apparent impact was increased because it was classified as multidisciplinary, usually a low cited category, thereby scoring well above the world benchmark. However, when regional patterns were analyzed, the rankings began to look rather strange, especially the research impact indicator. In Australia, the Middle East, Hong Kong and Taiwan the order of universities, looked rather different from what local experts expected. Hong Kong Baptist University the third best in the SAR? Pohang University of Science and Technology so much better than Yonsei or KAIST? Adelaide the fourth best Australian university?

In the UK or the US these placings might seem plausible or at least not worth bothering about. But in the Middle East the idea of Alexandria as top university even in Egypt is a joke and the places awarded to the others look very dubious.

THE and Thomson Reuters tried to shrug off the complaints by saying that there were just a few outliers which they were prepared to debate and that anyone who criticized them had a vested interest in the old THE-QS rankings which had been discredited. They  dropped hints that the citations indicator would be reviewed but so far nothing specific has emerged.

A few days ago, however,  Phil Baty of THE seemed to imply that there was nothing wrong with the citations indicator.

Normalised data allow fairer comparisons, and that is why Times Higher Education will employ it for more indicators in its 2011-12 rankings, says Phil Baty.

One of the most important features of the Times Higher Education World University Rankings is that all our research citations data are normalised to take account of the dramatic variations in citation habits between different academic fields.

Treating citations data in an “absolute manner”, as some university rankings do, was condemned earlier this year as a “mortal sin” by one of the world’s leading experts in bibliometrics, Anthony van Raan of the Centre for Science and Technology Studies at Leiden University. In its rankings, Times Higher Education gives most weight to the “research influence” indicator – for our 2010-11 exercise, this drew on 25 million citations from 5 million articles published over five years. The importance of normalising these data has been highlighted by our rankings data supplier, Thomson Reuters: in the field of molecular biology and genetics, there were more than 1.6 million citations for the 145,939 papers published between 2005 and 2009; in mathematics, however, there were just 211,268 citations for a similar number of papers (140,219) published in the same period.

To ignore this would be to give a large and unfair advantage to institutions that happen to have more provision in molecular biology, say, than in maths. It is for this crucial reason that Times Higher Education’s World University Rankings examine a university’s citations in each field against the global average for that subject.

Yes, but when we are assessing hundreds of universities in very narrowly defined fields we start running into quite small samples that can be affected by deliberate manipulation or by random fluctuations.

Another point is that if there are many more journals, papers, citations and grants in oncology or genetic engineering than in the spatialization of gender performativity or the influence of Semitic syntax on Old Irish then perhaps society is telling us something about what it values and that is something that should not be dismissed so easily.

So, it could be  we are going to get the University of Alexandria in the top 200 again, perhaps joined by Donghua university.

At the risk of being repetitive, there are a few simple  things that Times Higher  and TR could do to make the citations indicator more credible. There are also  more ways of measuring research excellence.Possibly they are thinking about them but so far there is no sign  of this.

The credibility of last year's rankings has  declined further with the decisions of the judge presiding over the libel case brought by El Naschie against Nature (see here for commentary). Until now it could be claimed that El Naschie was a wll known scientist by virtue of the large numbers of citations that he had received or at least an interesting and controversial maverick.

El  Naschie is pursuing a case against  Nature for publishing an article that suggested his writings were not of a high quality and that those published in his journal did not appear to be properly peer reviewed

The judge has recently ruled  ruled that  El Naachie cannot proceed with a claim for specific damages since he has not brought any evidence for this. He can only go ahead with a claim for general damages for loss of reputation and hurt feelings. Even here, it looks like it will be tough going. El Naschie seems to be unwilling or unable to find expert witnesses to testify to the scientific merits of his papers.

"The Claimant is somewhat dismissive of the relevance of expert evidence in this case, largely on the basis that his field of special scientific knowledge is so narrow and fluid that it is difficult for him to conceive of anyone qualifying as having sufficient "expert" knowledge of the field. Nevertheless, permission has been obtained to introduce such evidence and it is not right that the Defendants should be hindered in their preparations."

He also seems to have problems with locating records that would demonstrate that his many articles published in Chaos, Solitons and Fractals were adequately reviewed.

46. The first subject concerns the issue of peer-review of those papers authored by the Claimant and published in CSF. It appears that there were 58 articles published in 2008. The Claimant should identify the referees for each article because their qualifications, and the regularity with which they reviewed such articles, are issues upon which the Defendants' experts will need to comment. Furthermore, it will be necessary for the Defendants' counsel to cross-examine such reviewers as are being called by the Claimant as to why alleged faults or defects in those articles survived the relevant reviews.
    
    
47. Secondly, further information is sought as to the place or places where CSF was administered between 2006 and 2008. This is relevant, first, to the issue of whether the Claimant has complied with his disclosure obligations. The Defendants' advisers are not in a position to judge whether a proportionate search has been carried out unless they are properly informed as to how many addresses and/or locations were involved. Secondly, the Defendants' proposed expert witnesses will need to know exactly how the CSF journal was run. This information should be provided.

It would therefore  seem to be getting more and more difficult for anyone to argue that TR's methodology has uncovered a pocket of excellence in Alexandria.

Unfortunately, it is beginning to look as though THE will not only use much the same method as last time but will apply normalisation to other indicators as well.

But what about the other performance indicators used to compare institutions? Our rankings examine the amount of research income a university attracts and the number of PhDs it awards. For 2011-12, they will also look at the number of papers a university has published that are co-authored by an international colleague.

Don’t subject factors come into play here, too? Shouldn’t these also be normalised? We think so. So I am pleased to confirm that for the 2011-12 World University Rankings, Times Higher Education will introduce subject normalisation to a range of other ranking indicators.
This is proving very challenging. It makes huge additional demands on the data analysts at Thomson Reuters and, of course, on the institutions themselves, which have had to provide more and richer data for the rankings project. But we are committed to constantly improving and refining our methodology, and these latest steps to normalise more indicators evidence our desire to provide the most comprehensive and rigorous tables we can.

What this might mean is that universities that spend modest amounts of money in fields where little money is usually spent would get a huge score. So what would happen if an eccentric millionaire left millions to establish a lavishly funded research chair in continental philosophy at Middlesex University?  There are no doubt precautions that Thomson Reuters could take but will they? The El Naschie business does not inspire very much confidence that they will.

The reception of the 2010 THE WUR rankings suggests that the many in the academic world have doubts about the wisdom of using normalised citation data without considering the potential for gaming or statistical anomalies. But the problem may run deeper and involve citations as such. QS, THE 's rival and former partner, have produced a series of subject rankings based on data from 2010. The overall results for each subject are based on varying combinations of the scores for academic opinion, employer opinion and citations per paper (not per faculty as in the general rankings).

The results are interesting. Looking at citations per paper alone we see that Boston College and Munich are jointly first in Sociology. Rutgers is third for politics and international studies. MIT is third for philosophy (presumably Chomsky and co). Stellenbosch is first for Geography and Area studies. Padua is first for linguistics. Tokyo Metropolitan University is second for biological sciences and Arizona State University first.

Pockets of excellence or statistical anomalies? These results may not be quite as incredible as Alexandria in the THE rankings but they are not a very good advertisement for the validity of citations as a measure of research excellence.

It appears that THE have not made their minds up yet. There is still time to produce a believable and rigorous ranking system. But whatever happens, it is unlikely that citations,  normalized or unnormalized, will continue to be the unquestionable gold standard of academic and scientific research.

What should India do about the rankings?

India seems to be suffering from ranking fever. This is a serious problem that periodically sweeps across countries with the national media echoing statements from university heads and bureaucrats about being in the top one hundred or two hundred of something in the next few years or passionate claims that rankings do not reflect the needs of local society or that the uniquely transformative features of this or that institution -- how dare they ignore our sensitivity training or sustainability programs! --  are not recognised by the rankers.

There is  now a lot of debate about which is the best university in India and also about why Indian institutions, especially the sometimes lauded Indian Institutes of Technology (IITs), have a modest impact on the international rankings.

So what do the various rankings say about the quality of Indian universities (counting the IITs and other Institutes)? Starting with Webometrics, which measures the Internet presence of universities,  first place in India goes to IIT Bombay, 517th in the world, followed by IIT Madras, The Indian Institute of Science (IISc) in Bangalore, IIT Kanpur and the University of Delhi.

Moving on to research based rankings, only one Indian university is ranked in  Shanghai Jiao Tong University's Academic Ranking of World Universities (ARWU) top 500, and that is the IISc in the 301-400 band.

Scimago Institutional Rankings in their 2013 default list of number of publications, also puts IISc in first place in India, followed by IIT Kharagur, IIT Delhi, the University of Delhi, and IIT Madras.

The Leiden Ranking has the IISc in first place for number of publications although IIT Roorkee is first for publications in high quality journals.

Looking at the research-only rankings then, the best bet for top place would be the IISc which is ranked first in India by ARWU, Scimago, and, for number of publications, by Leiden Ranking, although for quality of research the IITs at Roorkee, Delhi and Guwahati perform comparatively well.

Moving on to rankings that attempt to assess factors other than research, we find that in the most recent QS World and Asian University Rankings first place in India goes to IIT Delhi with IIT Bombay second and IIT Kanpur third.

Last year's Times Higher Education world rankings produced an unusual result. Panjab University (PU) was ranked in the 226-250 band well ahead of the IITs Delhi, Kanpur, Kharagpur and Roorkee in the 350 - 400. In this case, Panjab university's feat was entirely due to its massive score for citations, 84.7 compared to IIT Delhi's 38.5, a score that was in stark contrast to a very poor 14 for research.

The main reason for PU's whopping score for citations appears to be that a few of its physicists are involved in the Large Hadron Collider project, which involves more than 2000 physicists in more than 150 research centers and 37 countries and consequently produces a huge number of citations. PU gets the credit for all of those citations even though its contribution to the cited papers is extremely small.

This only works because the overall number of papers produced is low. Hundreds or even thousands of citations are of little incremental value if they are spread out over thousands or tens of thousands of papers.

It would be unwise for other Indian universities to emulate PU's approach to get into the THE rankings. For one thing they would have to keep total publications low. For another, they may find that highly cited researchers might be a tempting target for universities in the US or Australia. And it does not work for any other ranking.

It is noticeable that The IISc is not included in the QS or THE rankings, presumably as a result of its own choice.

Should India's universities try to improve their ranking performance? Perhaps, but it would be better if they focused on improving their research performance, admissions policies, administration and selection processes. And here there is a much bigger problem for India, the utterly dismal performance of the country's school system.

In 2009, students from Tamil Nadu and Himachel Pradesh, which do better than the Indian average on social and economic development measures, took the PISA test. They were just ahead of the lowest ranked Kirgystan.

Riaz Haq writes:

"In Tamil Nadu, only 17% of students were estimated to possess proficiency in reading that is at or above the baseline needed to be effective and productive in life. In Himachal Pradesh, this level is 11%. “This compares to 81% of students performing at or above the baseline level in reading in the OECD countries, on an average,” said the study. 
The average Indian child taking part in PISA2009+ is 40 to 50 points behind the worst students in the economic superstars. Even the best performers in Tamil Nadu and Himachal Pradesh - the top 5 percent who India will need in science and technology to complete globally - were almost 100 points behind the average child in Singapore and 83 points behind the average Korean - and a staggering 250 points behind the best in the best.
The average child in HP & TN is right at the level of the worst OECD or American students (only 1.5 or 7.5 points ahead). Contrary to President Obama's oft-expressed concerns about American students ability to compete with their Indian counterparts, the average 15-year-old Indian placed in an American school would be among the weakest students in the classroom, says Lant Pritchett of Harvard University. Even the best TN/HP students are 24 points behind the average American 15 year old."

If this does not change, there is very little that anyone can do to improve the status of India's universities, apart from importing large numbers of Finnish, Chinese or Korean students, teachers and researchers.

Unsolicited Advice

There has  been a lot of debate recently about the reputation survey component in the QS World University Rankings.

The president of University College Cork asked faculty to find friends at other universities who "understand the importance of UCC improving its university world ranking". The reason for the reference to other universities is that the QS survey very sensibly does not permit respondents to vote for their own universities, those that they list as their affiliation.  

This request appears to violate QS's guidelines which permit universities to inform staff about the survey but not to encourage them to nominate or refrain from nominating any particular university. According to an article in Inside Higher Ed QS are considering whether it is necessary to take any action.

This report has given Ben Sowter of QS sufficient concern to argue that it is not possible to effectively manipulate the survey.  He has set out a reasonable case why it is unlikely that any institution could succeed in marching graduate students up to their desktops to vote for favoured institutions to avoid being sent to a reeducation camp or to teach at a community college.

However, some of his reasons sound a little unconvincing: signing up, screening, an advisory board with years of experience. It would help if he were a little more specific, especially about the sophisticated anomaly detection algorithm, which sounds rather intimidating.

The problem with the academic survey is not that an institution like University College Cork is going to push its way into the global  top twenty or top one hundred  but that there could be a systematic bias towards those who are ambitious or from certain regions. It is noticeable that some universities in East and Southeast Asia do very much better on the academic survey than on other indicators. 

The QS academic survey is getting overly complicated and incoherent. It began as a fairly simple exercise. Its respondents were at first drawn form the subscription lists of World Scientific, an academic publishing company based in Singapore. Not surprisingly, the first academic survey produced a strong, perhaps too strong, showing for Southeast and East Asia and Berkeley. 

The survey turned out to be unsatisfactory, not least because of an extremely small response rate. In succeeding years QS has added respondents drawn from the subscription lists of Mardev, an academic database, largely replacing those from World Scientific, lists supplied by universities, academics nominated by respondents to the survey and those joining the online sign up facility. It is not clear how many academics are included in these groups or what the various response rates are. In addition, counting responses for three years unless overwritten by the respondent might enhance the stability of the indicator but it also means that some of the responses might be from people who have died or retired.

The reputation survey does not have a good reputation and it is time for QS to think about revamping the methodology. But changing the methodology means that rankings cannot be used to chart the progress or decline of universities over time. The solution to this dilemma might be to launch a new ranking and keep the old one, perhaps issuing it later in the year or giving it less prominence.

My suggestion to QS is that they keep the current methodology but call it the Original QS Rankings or the QS Classic Rankings. Then they could introduce the  QS Plus or New QS rankings or something similar which would address the issues about the academic survey and introduce some other changes. Since QS are now offering a wide range of products, Latin American Rankings, Asian Rankings, subject rankings, best student cities and probably more to come, this should  not impose an undue burden.

First, starting with the academic survey, 40 percent is too much for any indicator. It should be reduced to 20 per cent.

Next, the respondents should be divided into clearly defined categories, presented with appropriate questions and appropriately verified.

It should be recognised that subscribing to an online database or being recommended by another faculty member is not really a qualification for judging international research excellence. Neither is getting one’s name listed as corresponding author. These days that  can have as much to do with faculty politics as with ability.  I suggest that the academic survey should be sent to:

(a) highly cited researchers  or those with a high h-index who should be asked about international research excellence;

(b) researchers drawn from the Scopus database who should be asked to rate the regional or national research standing of universities.

Responses should be weighted according to the number of researchers per country.

This could be supplemented with a survey of student satisfaction with teaching based on a student version of the sign up facility and requiring a valid academic address with verification.

Also, a sign up facility could be established for anyone interested and asking a question about general perceived quality.

If QS ever do change the academic survey they might as well review the other indicators. Starting with the employer review, this should be kept since, whatever its flaws, it is an external check on universities. But it might be easier to manipulate than the academic survey. Something was clearly going on in the 2011 ranking when there appeared to be a disproportionate number of respondents from some Latin American countries, leading QS to impose caps on universities exceeding the national average by a significant amount. 

"QS received a dramatic level of response from Latin America in 2011, these counts and all subsequent analysis have been adjusted by applying a weighting to responses from countries with a distinctly disproportionate level of response."

It seems that this problem was sorted out in 2012. Even so, QS might consider giving   half the weighting for this survey to an invited panel of employers. Perhaps  they could also broaden their database by asking NGOS and non-profit groups about their preferences.

There is little evidence that overall the number of international students has anything to do with any measure of quality and it also may have undesirable backwash effects as universities import large numbers of less able students. The problem is that QS are doing a good business moving graduate students across international borders so it is unlikely that they will ever consider doing away this indicator.

Staff student ratio is by all accounts a very crude indicator of quality of teaching. Unfortunately, at the moment there does appear to be any practical alternative. 

One thing that QS could do is to remove research staff from the faculty side of the equation. At the moment a university that hires an army of underpaid research assistants  and sacks a few teaching staff, or packs them off to a branch campus, would be recorded as having brought about a great improvement in teaching quality.

Citations are a notoriously problematical way of measuring research influence or quality. The Leiden Ranking shows that there are many ways of measuring research output and influence. It would be a good  idea to combine several different ways of counting citations. QS have already started to use the h- index in their subject rankings starting this year and have used citations per paper in the Asian University Rankings.

With the 20 per cent left over from reducing the weighting for the academic survey QS might consider introducing a measure of research output rather than quality since this would help distinguish among universities outside the elite and perhaps use internet data from Webometrics as in the Latin American rankings.

Article in University World News

GLOBAL

Ranking’s research impact indicator is skewed

Richard Holmes15 November 2012 Issue No:247

The 2012 Times Higher Education World University Rankings consisted of 13 indicators grouped into five clusters. One of these clusters consisted of precisely one indicator, research impact, which is measured by normalised citations and which THE considers to be the flagship of the rankings.

It is noticeable that this indicator, prepared for THE by Thomson Reuters, gives some universities implausibly high scores. I have calculated the world's top universities for research impact according to this indicator.

Since it accounts for 30% of the total ranking it clearly can have a significant effect on overall scores. The data are from the profiles, which can be accessed by clicking on the top 400 universities. Here are the top 20:

1. Rice University
1. Moscow (State) Engineering Physics Institute (MEPhI)
3. University of California Santa Cruz
3. MIT
5. Princeton
6. Caltech
7. University of California Santa Barbara
7. Stanford
9. University of California Berkeley
10. Harvard
11. Royal Holloway London
12. Chicago
13. Northwestern
14. Tokyo Metropolitan University
14. University of Colorado Boulder
16. University of Washington Seattle
16. Duke
18. University of California San Diego
18. University of Pennsylvania
18. Cambridge

There are some surprises here, such as Rice University in joint top place, second-tier University of California campuses at Santa Cruz (equal third with MIT) and Santa Barbara (equal seventh with Stanford) placed ahead of Berkeley and Los Angeles, Northwestern almost surpassing Chicago, and Tokyo Metropolitan University ahead of Tokyo and Kyoto universities and everywhere else in Asia.

It is not totally implausible that Duke and the University of Pennsylvania might be overtaking Cambridge and Oxford for research impact, but Royal Holloway and Tokyo Metropolitan University?

These are surprising, but Moscow State Engineering Physics Institute (MEPhI) as joint best in the world is a definite head-scratcher.

Other oddities

Going down a bit in this indicator we find more oddities.

According to Thomson Reuters, the top 200 universities in the world for research impact include Notre Dame, Carleton, William and Mary College, Gottingen, Boston College, University of East Anglia, Iceland, Crete, Koc University, Portsmouth, Florida Institute of Technology and the University of the Andes.

On the other hand, when we get down to the 300s we find that Tel Aviv, National Central University Taiwan, São Paulo, Texas A&M and Lomonosov Moscow State University are assigned surprisingly low places. The latter is actually in 400th place for research impact among the top 400.

It would be interesting to hear what academics in Russia think about an indicator that puts MEPhI in first place in the world for research impact and Lomonosov Moscow State University in 400th place.

I wonder too about the Russian reaction to MEPhI as overall second among Russian and Eastern European universities. See here, here and here for national university rankings and here and here for web-based rankings.

Déjà vu

We have been here or somewhere near here before.

In 2010 the first edition of the THE rankings placed Alexandria University in the world's top 200 and fourth for research impact. This was the result of a flawed methodology combined with diligent self-citation and cross-citation by a writer whose lack of scientific credibility has been confirmed by a British court.

Supposedly the methodology was fixed last year. But now we have an indicator as strange as in 2010, perhaps even more so.

So how did MEPhI end up as world's joint number one for research impact? It should be emphasised that this is something different from the case of Alexandria. MEPhI is, by all accounts, a leading institution in Russian science. It is, however, very specialised and fairly small and its scientific output is relatively modest.

First, let us take a look at another source, the Scimago World Report, which gives MEPhI a rank of 1,722 for total publications between 2006 and 2010, the same period that Thomson Reuters counts.

Admittedly, that includes a few non-university institutions. It has 25.9 % of its publication in the top quartile of journals. It has a score of 8.8 % for excellence – that is, the proportion of publications in the most highly cited 10% of publications. It gets a score of 1.0 for ‘normalised impact’, which means that it gets exactly the world average for citations adjusted by field, publication type and period of publication.

Moving on to Thomson Reuters’ database at the Web of Science, MEPhI has had only 930 publications listed under that name between 2006 and 2010, although there were some more under other name variants that pushed it over the 200 papers per year threshold to be included in the rankings.

It is true that MEPhI can claim three Nobel prize winners, but they received awards in 1958 and 1964 and one of them was for work done in the 1930s.

So how could anyone think that an institution that now has a modest and specialised output of publications and a citation record that, according to Scimago, does not seem significantly different from the international average using the somewhat larger Scopus database – Thomson Reuters uses the more selective ISI Web of Science – could emerge at the top of Thomson Reuters' research impact indicator?

Furthermore, MEPhI has no publications listed in the ISI Social Science Citation Index and exactly one (uncited) paper in the Arts and Humanities Index on oppositional politics in Central Russia in the 1920s.

There are, however, four publications assigned to MEPhI authors in the Web of Science that are listed as being on the literature of the British Isles, none of which seem to have anything to do with literature or the British Isles or any other isles, but which have a healthy handful of citations that would yield much higher values if they were classified under literature rather than physics.

Citations

Briefly, the essence of Thomson Reuters' counting of citations is that a university's citations are compared to the average for a field in a particular period after publication.

So if the average for a field is 10 citations per paper one year after publication, then 300 citations of a single paper one year after publication would count as 30 citations. If the average for the field was one citation it would count as 300 citations.

To get a high score in the Thomson Reuters research impact indicator, it helps to get citations soon after publication, preferably in a field where citations are low or middling, rather than simply getting many citations.

The main cause of MEPhI's research impact supremacy would appear to be a biennial review that summarises research over two years in particle physics and is routinely referred to in the literature review of research papers in the field.

The life span of each review is short since it is superseded after two years by the next review so that the many citations are jammed into a two-year period, which could produce a massive score for ‘journal impact factor’.

It could also produce a massive score on the citations indicator in the THE rankings. In addition, Thomson Reuters then gives a weighting to countries according to the number of citations. If citations are generally low in their countries, then institutions get some more value added.

The 2006 “Review of Particle Physics” published in Journal of Physics G, received a total of 3,662 citations, mostly in 2007 and 2008. The 2008 review published in Physics Letters B had 3,841 citations, mostly in 2009 and 2010, and the 2010 review, also published in Journal of Physics G, had 2,592 citations, nearly all in 2011. Someone from MEPhI was listed as co-author of the 2008 and 2010 reviews.

It is not the total number of citations that matters, but the number of citations that occur soon after publication. So the 2008 review received 1,278 citations in 2009, but the average number of citations in 2009 to other papers published in Physics Letters B for 2008 was 4.4.

So the 2008 review received nearly 300 times as many citations in the year after publication as the mean for that journal. Add the extra weighting for Russia and there is a very large boost to MEPhI's score from just a single publication. Note that these are reviews of research so it is likely that there had already been citations to the research publications that are reviewed. Among the highlights of the 2010 review are 551 new papers and 108 mostly revised or new reviews.

If the publications had a single author or just a few authors from MEPhI then this would perhaps suggest that the institute had produced or made a major contribution to two publications of exceptional merit. The 2008 review in fact had 173 co-authors. The 2010 review listed 176 members of the Particle Data Group who are referred to as contributors.

It seems then that MEPhI was declared the joint best university for research impact largely because of two multi-author (or contributor) publications to which it had made a fractional contribution. Those four papers assigned to literature may also have helped.

As we go through the other anomalies in the indicator, we find that the reviews of particle physics contributed to other high research impact scores. Tokyo Metropolitan University, Royal Holloway London, the University of California at Santa Cruz, Santa Barbara and San Diego, Notre Dame, William and Mary, Carleton and Pisa also made contributions to the reviews.

This was not the whole story. Tokyo Metropolitan University benefited from many citations to a paper about new genetic analysis software and Santa Cruz had contributed to a massively cited multi-author human haplotype map.

Number of authors

This brings us to the total number of publications. There were more than 100 authors of or contributors to the reviews but for some institutions the number of citations had no discernible effect and for others not very much.

Why the difference? Here size really does matter and small really is beautiful.

MEPhI has relatively few publications overall. It only just managed to cross the 200 publications per year threshold to get into the rankings. This means that the massive and early citation of the reviews was averaged out over a small number of publications. For others the citations were absorbed by many thousands of publications.

These anomalies and others could have been avoided by a few simple and obvious measures. After the business with Alexandria in 2010 Thomson Reuters did tweak its system, but evidently this was not enough.

First, it would help if Thomson Reuters scrutinised the criteria by which specialised institutions are included in the rankings. If we are talking about how well universities spread knowledge and ideas, it is questionable whether we should count institutions that do research in one or only a few fields.

There are many methods by which research impact can be evaluated. The full menu can be found on the Leiden Ranking site. Use a variety of methods to calculate research impact, especially those like the h-index that are specifically designed to work around outliers and extreme cases.

It would be sensible to increase the threshold of publications for inclusion in the rankings. The Leiden Ranking excludes universities with fewer than 500 publications a year. If a publication has multiple authors, divide the number of citations by the number of authors. If that is too complex then start dividing citations when the number reaches 10 or a dozen.

Do not count reviews, summaries, compilations or other publications that refer to papers that may have already been cited, or at least put them in a separate publication type. Do not count self-citations. Even better, do not count citations within the same institution or the same journal.

Most importantly, calculate the indicator for the six subject groups and then aggregate them. If you think that a fractional contribution to two publications justifies putting MEPhI at the top for research impact in physics, go ahead and give them 100 for natural sciences or physical sciences. But is it reasonable to give the institution any more than zero for arts and humanities, the social sciences and so on?

So we have a ranking indicator that has again yielded some very odd results.

In 2010 Thomson Reuters asserted that it had a method that was basically robust, transparent and sophisticated but which had a few outliers and statistical anomalies about which they would be happy to debate.

It is beginning to look as though outliers and anomalies are here to stay and there could well be more on the way.

It will be interesting to see if Thomson Reuters will try to defend this indicator again.

* Richard Holmes is a lecturer at Universiti Teknologi MARA in Malaysia and author of the University Ranking Watch blog.

Apology for factual error

In an earlier version of this article in University World News I made a claim that Times Higher Education, in their 2012 World University Rankings, had introduced a methodological change that substantially affected the overall ranking scores. I acknowledge that this claim was without factual foundation. I withdraw the claim and apologise without reservation to Phil Baty and Times Higher Education.

Richard Holmes