I appreciate that measuring the research impact of an institution is difficult. Unfortunately, the THE seems to have got it quite badly wrong this time. The best evidence for this is not that Warwick doesn't make it into the top 200 (even though this hurts; yes I am faculty at Warwick), but the fact that the University of Alexandria makes it number 4, based on the output of one single person.
1. Do not count self-citations. Even better, it is common sense that a citation by someone further from your field should count more than a citation by your former PhD student. Of course, these things are difficult to figure out in an automated way. But one could for example use the collaboration distance (coauthors have distance one, coauthors of coauthors have distance two, etc) to weight citations (with a cap at 4, say).
2. Quality of research is paramount. As already pointed out, it is easy to get many citations for mediocre research if there are sufficiently many other mediocre researchers working in the same area. Again, this is vastly more common than you may think, for the simple reason that it is easier to perform mediocre research than world-class research. Furthermore, you get more recognition as counted by basic citation count, so why not doing it?
One way of taking this into account is to give higher weight to citations coming from articles published in highly respected journals. (Similarly, when measuring "research output", higher weight should be given to articles published in journals with high reputation.)
However, please *DO NOT* use the impact factor as a measure of the quality of a journal, as it can be (and is!) very easily manipulated, as the story of the journal "Chaos, Solitons and Fractals" shows. Instead, the only reliable way of assessing the quality of a journal within a given field is to ask researchers in that field to provide their own rankings. Yes, this seems subjective, but unfortunately that's all you are ever going to get, and I can assure you that you will get a very consistent picture within each area. The fact that the "Annals of Mathematics" is the most respected journal in mathematics simply cannot be measured in terms of impact factor.
3. Count current citations to older work. If someone's article turns out to spawn an entire new field of research five years later, it will not show up at all in the current metric. This makes simply no sense. Of course, this doesn't happen all that often, but the reason why top institutions have a reputation is precisely because of those instances in which it happens. Furthermore, there are areas of research (like mathematics) in which the "lifespan" of a good article is measured in decades, which goes way beyond the two to five years that you use as a rule. Counting current citations to older articles would be one small but absolutely essential step to correct this.
4. Measure the total impact of the institution in a field, and not its "average" impact. The only way I could see that the output of one single person can count so much is that this person somehow has an abnormally high weight, probably due to the fact that there is very little research output from the U. of Alexandria. If this suspicion is indeed correct (I hope that I am wrong on this one), then this would effectively mean that universities are penalised by having large (and influential!) departments and should rather strive to have only very few but quite prolific researchers on their payroll.
There is probably more, but I am getting hungry now ;-) I very much hope that you will take these comments to heart. Best wishes,
Martin
"“I believe we are well deserving of being on the list,” professor of economics
Amr Hussein told Bikya Masr. “We have worked hard to improve our system of
education and it is showing that we are succeeding in doing so.”...
Hend Hanafi, President of Alexandria University, told local media that she is proud
of the ranking and hopes the university will continue to make efforts to
consistently improve the quality of education at the university."
"Haaretz has learned that most Israeli universities were not on the list because they failed to respond to repeated requests for information, including on faculty and students, which is necessary for the listing.There are other surprising omissions such as all the Indian Institutes of Technology, the University at Buffalo: SUNY and the Catholic University of Louvain (the French one -- the Dutch one is there at 120)
TAU and the Hebrew University say that they never received such a request from THE. According to THE, only the Technion and Bar-Ilan University responded with information, but they were ranked 221 and 354 respectively.
As for the other universities, the editor of the ranking, Phil Baty, told Haaretz that although it is upsetting for Israel, he hoped that the Israeli universities would recognize the amount of serious work invested in creating the ranking and the degree to which the methodology was transparent, and would participate in the initiative, like other universities have done. He also expressed certainty that next year "they will be included."
Didn't do their homework
THE says that more than 6,000 universities participated in the ranking and most provided the necessary information.
A spokesperson for the Hebrew University responded that contrary to the claim of
the survey's editors, "following an examination we did not find any such request
[for information]. When we asked for the correspondence to the university on the
subject, they could not provide it. The university is saddened by the fact that
the editors of the ranking did not carry out their work responsibly, and thus
harmed the university." "
My bet is that TR uses the Leiden "Crown indicator" since this is what is embodied in their product InCites.
To cut it short, each paper is linked to a subdiscipline, a type of publication (letter, review, ...) and a year of publication. With this data for the whole world, it is easy to calculate the expected number of citations for a paper of a given type, in a given discipline, in a given year.
For a set of papers (e.g. all the papers of Alexandria university), the indicator is calculated as Sum(received citations)/Sum(expected citations).
This number can become very high if you have a small number of paper or if you look only at recent papers (if, on average you expect 0.1 citations for a recent paper in math, a single citation will give you a score of 10 for this paper!)
Note that Leiden as recently decided to change its favorite indicator for a mean(citations received/citations expected) which gives less weight for a few highly cited papers in a set. But it seems that TR has not implemented yet this new indicator.
Note also that, in order to avoid the overweight given to few papers in a small set, Leiden publish its own ranking of universities with thresholds on the total number of papers published.
This is not very convincing. Does Alexandria produce strong research? Overall, No. It is ranked 1014 in the world for total papers over a ten year period by SCImago.“Alexandria University is Egypt's only representative in the global top 200, in joint 147th place. Its position, rubbing shoulders with the world's elite, is down to an exceptional score of 99.8 in the "research-influence" category, which is virtually on a par with Harvard University.
Alexandria, which counts Ahmed H. Zewail, winner of the 1999 Nobel Prize for
Chemistry, among its alumni, clearly produces some strong research. But it is a
cluster of highly cited papers in theoretical physics and mathematics - and more
controversially, the high output from one scholar in one journal - that gives it
such a high score.Mr Pratt said: "The citation rates for papers in these fields may not appear exceptional when looking at unmodified citation counts; however, they are as high as 40 times the benchmark for similar papers. "The effect of this is particularly strong given the relatively low number of papers the university publishes overall."
Quacquarelli Symonds has continued to produce those rankings, now called the QS World University Rankings, and is partnering with U.S. News and World Report for their publication in the United States.
The relationship between the former collaborators has deteriorated into
barely veiled animosity. QS has accused Times Higher Education of unfairly
disparaging the tables they once published together. This week the company
threatened legal action against the magazine over what Simona Bizzozero, a QS
spokeswoman, described as "factually inaccurate" and misleading statements by
representatives of Times Higher Education. She said THE's role in the
collaboration was limited to publishing the rankings based on a methodology that
QS had developed. "What they're producing now is a brand-new exercise. A totally
brand-new exercise, with absolutely no links whatsoever to what QS produced and
is producing," she said. "So when they refer to their old methodology, that is
not correct."
Phil Baty, editor of the rankings for Times Higher Education, declined to respond to QS's complaints: "We are now looking forward, not looking backward."
"Really, nothing has changed," said Ellen Hazelkorn, executive director of the Higher Education Policy Research Unit at the Dublin Institute of Technology, whose book "Rankings and the Battle for Worldclass Excellence: The Reshaping of Higher Education" is due to be published in March.
Despite Times Higher Education's assurances that the new tables represent a much more rigorous and reliable guide than the previous rankings, the indicators on which the new rankings are based are as problematic in their own way, she believes. The heavily weighted measure of teaching, which she described as subjective and based on reputation, introduces a new element of
unreliability.
Gauging research impact through a subjective, reputation-based measure is troublesome enough, and "the reputational aspect is even more problematic once you extend it to teaching," she said.
Ms. Hazelkorn is also troubled by the role Thomson Reuters is playing through its
Global Institutional Profiles Project, to which institutions provide the data
used in the tables. She dislikes the fact that institutions are going to great
effort and expense to compile data that the company could then sell in various
ways.
"This is the monetarization of university data, like Bloomberg
made money out of financial data," she said.
Thomson Reuters is a private global information services
firm, and a highly respected one at that. Apart from ‘deep pockets’, they have
knowledgeable staff, and a not insignificant number of them. For example, on 14
September Phil Baty, of Times Higher Education sent out this fact via their
Twitter feed:
2 days to #THEWUR. Fact: Thomson Reuters involved more
than 100 staff members in its global profiles project, which fuels the rankings
The incorporation of Thomson Reuters into the rankings games by Times
Higher Education was a strategically smart move for this media company for it
arguably (a) enhances their capacity (in principle) to improve ranking
methodology and implementation, and (b) improves the respect the ranking
exercise is likely to get in many quarters. Thomson Reuters is, thus, an
analytical-cum-legitimacy vehicle of sorts.
Malaysia Star Online
