Penn State Law School

Malcolm Gladwell has an article in the current New Yorker about the US News and World Report college rankings. There is quite a lot there that I would like to discuss in another post. For the moment, I will just comment on an anecdote about the appearance of a non-existent law school in a ranking.

Gladwell descibes how Thomas Brennan, who edits a well known ranking of law schools, once sent out a questionnaire to other lawyers asking them to rank law schools and found that Penn State was, as Brennan is quoted as recalling, ranked around fifth. This was strange since there was no law school at Penn State until quite recently (1997 or 2000 in different sources).

This immediately struck me as odd since I remember a similar story about the Princeton Law School, which does not exist and which was also supposed to have made its appearance in a ranking.
The Princeton story is very probably apocryphal and might have  begun with a comment by the dean of New York University Law School in the Dartmouth Law Journal that Princeton would appear in the top twenty law schools if a questionnaire was asked about it.

This story was plausible since it was an apparent example of the halo effect with Princeton's general excellence being reflected in the perception of a school that did not exist.

The problem with Brennan's account retold by Gladwell, which does not appear to be supported by documentary evidence, is that it requires that many lawyers should not only have mistakenly thought that Penn State had  a law school (getting mixed up with the University of Pennsylvania?) but should have been in error about the general quality of the university. Penn State is nowhere near being a top ten or even a top fifty school.

Could this be another academic legend?

 Impact Assessment

The use of citations as a measure of research quality was highlighted by the remarkable performance of Alexandria University, Bilkent University, Hong Kong Baptist University and others in the 2010 Times Higher Education World University Rankings. As THE and Thomson Reuters review their methodology, perhaps they could take note of this post in Francis' World Inside Out, that refers to a paper by Arnold and Fowler.

'“Goodhart’s law warns us that “when a measure becomes a target, it ceases to be a good measure.” The impact factor has moved in recent years from an obscure bibliometric indicator to become the chief quantitative measure of the quality of a journal, its research papers, the researchers who wrote those papers and even the institution they work in. The impact factor for a journal in a given year is calculated by ISI (Thomson Reuters) as the average number of citations in that year to the articles the journal published in the preceding two years. It is widely used by researchers deciding where to publish and what to read, by tenure and promotion committees laboring under the assumption that publication in a higher impact-factor journal represents better work. However, it has been widely criticized on a variety of grounds (it does not determine paper’s quality, it is a crude and flawed statistic, etc.). Impact factor manipulation can take numerous forms. Let us follow Douglas N. Arnold and Kristine K. Fowler, “Nefarious Numbers,” Notices of the AMS 58: 434-437, Mach 2011 [ArXiv, 1 Oct 2010].



Editors can manipulate the impact factor by means of the following practices: (1) “canny editors cultivate a cadre of regulars who can be relied upon to boost the measured quality of the journal by citing themselves and each other shamelessly;” (2) “authors of manuscripts under review often were asked or required by editors to cite other papers from the journal; this practice borders on extortion, even when posed as a suggestion;” and (3) “editors raise their journals’ impact factors is by publishing review items with large numbers of citations to the journal.” “These unscientific practices wreak upon the scientific literature have raised occasional alarms. A counterexample should confirm the need for alarm.” '

Looking East

Shanghai is planning to persuade two Ivy League schools, Cornell and Columbia, to set up branch campuses there. They already a branch of New York University.

Would anybody like to make a prediction when a new Oxford or Cambridge college will be established in Shanghai (or Singapore or Hong Kong)?

Or when an entire American university will move to China?

More dumbing Down

De Paul University will make it optional for applicants to submit SAT or ACT scores. Instead they can write short essays that demonstrate non-cognitive traits such as "commitment to service", "leadership" and "ability to meet long term goals".

The university says:

'"Admissions officers have often said that you can't measure heart," said Jon Boeckenstedt, associate vice president for enrollment management. "This, in some sense, is an attempt to measure that heart."



Mr. Boeckenstedt expects the change to encourage applicants with high grade-point averages but relatively low ACT and SAT scores to apply—be they low-income students, underrepresented minorities, or otherwise. Moreover, he and his colleagues believe the new admissions option will allow them to better select applicants who are most likely to succeed—and graduate.'

De Paul's administrators are being extremely naive if they think that these attributes cannot be easily coached or faked. Bluntly, how much effort does it take to teach a student what to say on one of these essays compared to squeezing out a few more points on the SAT?

Another US News Ranking

This one is about the schools where congressmen received their bachelor degrees.

Here are the top 10. What might be more interesting is the party affiliation of the congressmen. D = Democrat, R = Republican, I = Independent.

1.    Harvard                                      D 13, R 2
2.    Stanford                                      D 9, R 2
3.    Yale                                            D 8, R 1, I 1
4.    UCLA                                        D 6, R 3
5=   Georgetown                                D 5, R 2
5=   Florida                                        D2, R5
5=   Georgia                                       D1, R6
5=   Wisconsin - Madison                   D6, R1
9.     North carolina -- Chapel Hill       D 5, R 1 
10=  Brigham Young                           R5
10=  George Washington                    D2, R5
10=  Louisiana State                           D1, R4
10=  Berkeley                                    D4, R1
10=  Missouri                                     D4, R1
10= Tennessee                                   D2, r3

The Fortune 500

The US News has produced a ranking of US universities according to the number of degrees awarded to the CEO s of the Fortune 500, the largest American corporations according to gross revenue.

Here are the top five.

1.  Harvard
2.  Columbia
3.  University of pennsylvania
4.  Unuiversity of Wisconsin -- Madison
5.  Dartmouth College

Ranking Education Schools

The US News and World Report, publishers of America's Best Colleges, are teaming up with the National Council on Teachers Quality to produce a rating of teacher preparation programs.

Many Education deans are strongly  opposed. See here.

We are all equal

I have come across an interesting article, "The equality of intelligence", in the philosopher's magazine by Nina Power. It is one of a series, "Ideas of the century" (I am not sure which one).

Power, whose dissertation is entitled From Theoretical Antihumanism to Practical Humanism: The Political Subject in Sartre, Althusser and Badiou and who is a senior lecturer at Roehampton University, refers to the work of Jacques Rancière,

"who never tires of repeating his assertion that equality is not just something to be fought for, but something to be presupposed, is, for me, one of the most important ideas of the past decade. Although Rancière begins the discussion of this idea in his 1987 text The Ignorant Schoolmaster, it is really only in the last ten years that others have taken up the idea and attempted to work out what it might mean for politics, art and philosophy. Equality may also be something one wishes for in a future to come, after fundamental shifts in the arrangement and order of society. But this is not Rancière’s point at all. Equality is not something to be achieved, but something to be presupposed, universally. Everyone is equally intelligent."

Just in case you thought she was kidding:

"In principle then, there is no reason why a teacher is smarter than his or her student, or why educators shouldn’t be able to learn alongside pupils in a shared ignorance (coupled with the will to learn). The reason why we can relatively quickly understand complex arguments and formulae that have taken very clever people a long time to work out lends credence to Rancière’s insight that, at base, nothing is in principle impossible to understand and that everyone has the potential to understand anything."

Power seems to be living in a different universe from those of us in the academic periphery. Perhaps she is actually pulling a Sokalian stunt but I suspect not. This sort of thing might be funny to many of us but it seems to be taken seriously in departments of education around the world. Just take a look at the model teaching philosophy statements found on the Internet.

Another example of her writing is Sarah Palin: Castration as Plenitude. Presumably that is  potentially understandable by everybody.

More on Citations

A column in the THE by Phil Baty indicates that there might be some change in the research impact indicator in the forthcoming THE World University Rankings. It is good that THE is considering changes but I have a depressing feeling that  Thomson Reuters, who collect the citations data, are going to have more weight in this matter than anyone or anything else.

Baty refers to a paper by Simon Pratt who manages the data for TR and THE.

The issue was brought up again this month in a paper to the RU11 group of 11 leading research universities in Japan. It was written by Simon Pratt, project manager for institutional research at Thomson Reuters, which supplies the data for THE’s World University Rankings.


Explaining why THE’s rankings normalise for citations data by discipline, Pratt highlights the extent of the differences. In molecular biology and genetics, there were more than 1.6 million citations for the 145,939 papers published between 2005 and 2009, he writes; in mathematics, there were just 211,268 citations for a similar number of papers (140,219) published in the same period.


Obviously, an institution with world-class work in mathematics would be severely penalised by any system that did not reflect such differences in citations volume.

This is correct but perhaps we should also consider whether the number of citations to papers in genetics is telling us something about the value that societies place on genetics rather than on mathematics and perhaps that is something that should not be ignored.


Also, in the real world are there many universities that are excellent in a single field, defined as narrowly as theoretical physics or applied mathematics, while being mediocre or worse in everything else? Anyone who thinks that Alexandria is the fourth best university in the world for research impact because of its uncontested excellence in mathematics should take a look here.

There are also problems with normalising by region. Precisely what the regions are for the purposes of this indicator is not stated. If Africa is a region, does this mean that Alexandria got another boost, one denied to other Middle Eastern universities? Is Istanbul in Europe and Bilkent in Asia? Does Singapore get an extra weighting because of the poor performance of its Southeastern neighbours?

There are two other aspects of the normalisation that are not foregrounded in the article. First, TR apparently use normalisation by year. In some disciplines it is rare for a paper to be cited within a year of publication..In others it is commonplace. An article that is classified  as being in a low citation field would get a massive boost if in addition it had a few citations within months of publication.

Remember also that the scores represent averages. A small number of total publications means an immense advantage for a university that has a few highly cited article in low cited fields and is located in a normally unproductive region. Alexandria's remarkable success was due to the convergence of four favourable factors: credit for publishing in a low citation sub-discipline, the frequent citation of recently published papers, being located in a continent whose scholars are not generally noticed and finally the selfless cooperation of hundreds of faculty who graciously refrained from sending papers to ISI indexed journals.

Alexandria University may not be open for the rest of this year and may not take part in the second THE WUR exercise. One wonders though how many universities around the world could benefit from these four factors and how many are getting ready to submit data to Thomson Reuters.

Training for Academics

The bureaucratisation of higher education continues relentlessly. Times Higher Education reports on moves to make all UK academics undergo compulsory training. This is not a totally useless idea: a bit of training in teaching methodology would do no harm at all for all those unprepared graduate assistants, part-timers and new Ph Ds that make up an increasing proportion of the work force in European and American universities.

But the higher education establishment has more than this in mind.

Plans to revise the UK Professional Standards Framework were published by the HEA in November after the Browne Review called for teaching qualifications to be made compulsory for new academics.

The framework, which was first published in 2006, is used to accredit universities' teaching-development activities, but the HEA has admitted that many staff do not see it as "relevant" to their career progression.

Under the HEA's proposals, the updated framework says that in future, all staff on academic probation will have to complete an HEA-accredited teaching programme, such as a postgraduate certificate in higher education. Postgraduates who teach would also have to take an HEA-accredited course.

A "sector-wide profile" on the number of staff who have reached each level of the framework would be published by the HEA annually.


Meanwhile, training courses would have to meet more detailed requirements.

A comment by "agreed" indicates just what is likely to happen.
 

I did one of these course a couple of years ago. I learnt nothing from the "content" that I couldn't have learnt in a fraction of the time by reading a book. The bulk of the course was an attempt to compel all lecturers to adopt fashionable models of teaching with no regard to the need for students to learn content. The example set by the lecturers on the course was apalling: ill prepared, dogmatic, and lacking in substance. A failure to connect with the "students" and a generally patronising tone was just one of the weaknesses. Weeks of potentially productive time were taken up by jumping through hoops and preparing assignments. This is not an isolated case, I know of several other such courses in other institutions that were equally shambolic. I'm all for improving the qulaity of teaching, but this is nonsensical. The only real benefit was the collegial relations with academics from other deaprtments forged through common bonds of disgust and mockery aimed at this ridiculous enterprise (presumably designed to justify the continued employment of failed academics from other disciplines given the role of teaching the reast of us how to teach).

Comparing Rankings 2

Number of Indicators

A ranking that contained only a single indicator would not be very interesting. Providing that the indicators are actually measuring different things, rankings with many indicators would contain more information. On the other hand, the more indicators there are the more likely it is that some will be redundant.

At the moment, the THE World University Rankings are in first place with 13 indicators and Paris Mines is last with only one. We should note, however, that the THE indicators are combined into 5 super-indicators and scores are given only for the latter.

So we have the following order.

1. THE World University Rankings: 13 indicators (scores are given for only 5 indicator groups)

2.  HEEACT: 8 indicators

3= Academic Ranking of World Universities (Shanghai):   6 indicators

3= QS World University Rankings: 6 indicators

5.  Leiden: 5 (strictly speaking, 5 separate rankings)

6=  Webometrics: 4

6=  Scimago Institutions Ranking: 4 (1 used for ranking)

8.  Paris Mines Tech: 1

All Ears

Times Higher Education and Thomson Reuters are considering changes to their ranking methodology. It seems that the research impact indicator (citations) will figure prominently in their considerations.  Phil Baty writes:

In a consultation document circulated to the platform group, Thomson Reuters suggests a range of changes for 2011-12.


A key element of the 2010-11 rankings was a "research influence" indicator, which looked at the number of citations for each paper published by an institution. It drew on some 25 million citations from 5 million articles published over five years, and the data were normalised to reflect variations in citation volume between disciplines.


Thomson Reuters and THE are now consulting on ways to moderate the effect of rare, exceptionally highly cited papers, which could boost the performance of a university with a low publication volume.


One option would be to increase the minimum publication threshold for inclusion in the rankings, which in 2010 was 50 papers a year.


Feedback is also sought on modifications to citation data reflecting different regions' citation behaviour.


Thomson Reuters said that the modifications had allowed "smaller institutions with good but not outstanding impact in low-cited countries" to benefit.

It would be very wise to do something drastic about the citations indicator. According to last year's rankings, Alexandria university is the fourth best university in the world for research impact, Hong Kong Baptist University is second in Asia, Ecole Normale Superieure Paris best in Europe with Royal Holloway University of London fourth, University of California Santa Cruz fourth in the USA and the University of Adelaide best in Australia.

If anyone would like to justify these results they are welcome to post a comment.

I would like to make these suggestions for modifying the citations indicator.

Do not count self-citations, citations to the same journal in which a paper is published or citations to the same university. This would reduce, although not completely eliminate, manipulation of the citation system. If this is not done there will be massive self citation and citation of friends and colleagues. It might even be possible to implement a measure of net citation by deducting citations from an institution from the citations to it, thus reduce the effect of tacit citation agreements.

Normalisation by subject field is probably going to stay. It is reasonable that some consideration should be given to scholars who work in fields where citations are delayed and infrequent. However, it should be recognised that the purpose of this procedure is to identify pockets of excellent and research institutions are not built around a few pockets or even a single one. There are many ways of measuring research impact and this is just one of them. Others that might be used include total citations, citations per faculty, citations per research income and h-index.

Normalisation by year is especially problematical and should be dropped. It means that a handful of citations  to an article classified as being in a low-citation discipline in the same year could dramatically multiply the score for this indicator. It also introduces an element of potential instability. Even if the methodology remains completely unchanged this year, Alexandria and Bilkent and others are going to drop scores of places as papers go on receiving citations but get less value from them as the benchmark number rises.

Raising the threshold of number of publications might not be a good idea. It is certainly true that Leiden University have a threshold of 400 publications a year but Leiden is measuring only research impact while THE and TR are measuring a variety of indicators. There are already too many blank spaces in these rankings and their credibility will be further undermined if universities are not assessed on an indicator with such a large weighting.

More Rankings

With acknowledgements to Registrarism, here is news about more new rankings.

Universities are being ranked according to the popularity of their Twitter accounts. According  to an article in the Chronicle of Higher Education:

Stanford earned a Klout score of 70, with Syracuse University, Harvard University, and the University of Wisconsin at Madison all following with a score of 64.


The top 10 is rounded out by University of California at Berkeley, Butler University, Temple University, Tufts University, the University of Minnesota, the University of Texas at Austin, and Marquette University.

Another is the Green Metric Ranking of World Universities, compiled by Universitas Indonesia. The criteria are  green statistics, energy and climate change, waste, water and transportation.

The top five are:

1.  Berkeley (Surprise!)
2.  Nottingham
3.  York (Canada)
4.  Northeastern
5.  Cornell

Universiti Putra Malaysia is 6th and Universitas Indonesia 15th.

Cambridge and Harvard

After the break with THE, QS decided to continue the old methodology of the 2004-2009 rankings. At least, that is what they said. It was therefore surprising to see that, according to data provided by QS,  there were in fact a number of noticeable rises and falls between 2009 and 2010 although nothing like as much as in previous years.


For example the University of Munich fell from 66th place to 98th place, the Free University of Berlin from 70th to 94th and Stockholm University from 168th to 215th while University College Dublin rose from 114th to 89th and Wurzburg from 309th to 215th.

But perhaps the most remarkable news was that Cambridge replaced Harvard as the world's best university. In every other ranking Harvard is well ahead.

So how did it happen? According to Martin Ince, “Harvard has taken more students since the last rankings were compiled without an equivalent increase in the number of academics.”

In other words there should have been a lower faculty student ratio and therefore a lower score for this indicator. This in fact happened. Harvard’s score went from 98 to 97.

Ince also says that there was an “improvement in staffing levels”, at Cambridge, presumably meaning that there was an increase in the number of faculty relative to the number of students. Between 2009 and 2010 Cambridge’s score for the student faculty remained the same at 100 which is consistent with Ince’s claim.

In addition to this, there was a "significant growth in the number of citations per faculty member" for Cambridge. It is not impossible that the number of citations racked up by Cambridge had  risen relative to Harvard but the QS indicator counts citations over a five year so even a substantial increase in publications or citations would take a few years to have an equivalent effect on this indicator. Also note that this indicator is citations per faculty and it appears that the number of faculty at Cambridge has gone up relative to Harvard. So we would expect any increase in citations to be cancelled out by a similar increase in faculty.

It looks a little odd then that for this indicator the Cambridge score rose from 89 to 93, four points, which is worth 0.8 in the weighted total score. That, by the way, was the difference between Harvard and Cambridge in 2009.

The oddity is compounded when we look at other high ranking universities. between 2009 and 2010 Leiden's score for citations per faculty rose from 97 to 99, Emory from 90 to 95, Oxford from 80 to 84, Florida from 70 to 75.

It would at first sight appear plausible that if Harvard, the top scorer in both years, did worse on this indicator then everybody or nearly everybody else would do better. But if we look at universities further down the table, we found the opposite. Between 2009 and 2010 for this indicator  Bochum fell from 43 to 34, Ghent from 43 to 37, Belfast from 44 to 35 and so on.

Could it be that there was some subtle and unannounced change in the method by which the raw scores were transformed into  indicator scores. Is it just a coincidence that the change was sufficient to erase the difference between Harvard and Cambridge?








http://www.wiziq.com/tutorial/90743-QS-World-University_Rankings-top-500

Comparing Rankings

International university rankings are now proliferating in much the same way that American rankings have multiplied over the last few decades although so far there is no global equivalent to top party schools or best colleges for squirrels.

It is now time for a cursory review and comparison of the major international rankings. I will omit recent rankings, those that look as though they may not be repeated or those that provide insufficient information about methodology.

The list is as follows:

Academic Ranking of World Universities

Higher Education Evaluation and Accreditation Council of Taiwan

International Professional Ranking of Higher Education Institutions (Paris Mines Tech)

Leiden Ranking

QS World University rankings

Scimago Institutions Ranking

THE World University Rankings

Webometrics Ranking of World Universities

The first attribute to be considered is simply the number of universities ranked. A ranking might have an impeccable methodology and analyse a score  of indicators with the strictest attention to current bibliometric theory and statistical technique. If, however, it only ranks a few hundred universities it is of no use to those interested in the thousands left outside the elite of the ranked.

I am counting the number of universities in published rankings. Here the winner is clearly Webometrics, followed by Scimago.

Webometrics 12,300

Scimago 1,955

QS WUR 616

ARWU 500

HEEACT 500
 
Leiden 500

THE WUR 400

Paris Mines 376

Shanghai Ranks Macedonia

I am not sure how accurate the folowing report is. The whole of Macedonia has never had a Nobel or Fields award winner or an ISI highly cited researcher, has published fewer articles than Loughborugh University and has no articles in Nature or Science. It is difficult to see just what a team of seven from Shanghai would evaluate, especially since ARWU is reluctant to get involved with teaching quality or publications in the arts and humanities. Still, it is perhaps indicative that a European country has turned to China to evaluate its universities.

"Shanghai Jiao Tong University, which analyzes the top universities in the world on quality of faculty, research output quality of education and performance, has been selected to evaluate the public and private institutions for higher education in Macedonia, Minister of Education and Science Nikola Todorov told reporters on Sunday.

The ranking team included the Shanghai University Director, Executive and six members of the University's Center, Todorov said, pointing out that Macedonia is to be the first country from the region to be part of the the Academic Ranking of World Universities (ARWU), commonly known as the Shanghai ranking.


"The Shanghai ranking list is the most relevant in the world, and being part of it is a matter of prestige. We shall be honored our institutions for higher education to be evaluated by this university. This is going to be a revolution in the education sector, as for the first time we are offered an opportunity to see where we stand in regard to the quality," Todorov said"

Full QS Rankings 2010

QS have published  full details including indicator scores of the top 400 universities in their 2010 rankings. In the transparency stakes this brings them level with THE who have an iphone/ipad app that provides these details for the main indicators but not the sub-indicators.

Microsoft Academic Search

Microsoft has developed a computer science research ranking. Organisations, mainly but not entirely universities, are ranked according to number of citations and there is also data on publications and the h-index.

The top five in the world are Stanford, MIT, Berkeley, Carnegie-Mellon and Microsoft. Harvard is seventh and Cambridge 18th.

Top regional universities are:
Africa -- Cape Town
Asia and Oceania -- Tel Aviv
Europe -- Cambridge
North America --  Stanford
South America --  Universidade de Sao Paulo

The Disposable Academic

An article in the Economist  (print edition, 18-31/12/2010, 146-8) analyses the plight of many of the world's Ph Ds. Many can expect nothing more a succession of miserable post-doc fellowships, short-term contracts or part-time jobs teaching remedial or matriculation classes. And those are the lucky ones who actually get their diploma.

It seems that the financial return for a Ph D is only marginally higher than that for a master's. Since there are undoubtedly variations by institution and discipline, it follows that for many joining a doctoral program is a losing proposition in every way.

One wonders whether countries like South Africa and some in Southeast Asia are creating future problems in the drive to boost the production of Ph Ds.

Value for Money

An article by Richard Vedder describes how the publication of data by Texas A and M University shows enormous variation in the cost of faculty members per student taught.

I recently asked my student research assistant to explore this data by choosing, more or less at random, 40 professors of the university's main campus at College Station — including one highly paid professor with a very modest teaching load in each department and one instructor who is modestly paid but teaches many students. The findings were startling, even for a veteran professor like myself.


The 20 high-paid professors made, on average, over $200,000 each, totaling a little over $5 million annually to the university. These professors collectively taught 125 students last year, or roughly $40,000 per student. Since a typical student takes about 10 courses a year, the average cost of educating a student exclusively with this group of professors would be about $400,000, excluding other costs beyond faculty salaries.

There are of course questions to be asked about whether the data included the supervision of dissertations and the difficulty of the courses taught. Even so, the results deserve close scrutiny and might even be a model for some sort of international comparison.

Dumbing Down of University Grades

An article in the London Daily Telegraph shows that the number of first and upper second class degrees awarded by British universities has risen steadily over the last few decades. Their value to employers as an indicator of student quality has accordingly diminished.

David Barrett reports that:

The latest data shows that the criteria for awarding degrees has changed dramatically - despite complaints from many universities that grade inflation at A-level has made it hard for them to select candidates.

Traditionally, first class honours have been awarded sparingly to students who show exceptional depth of knowledge and originality.


But the new figures add further weight to a report by MPs last year which found that "inconsistency in standards is rife" and accused vice-chancellors of "defensive complacency".

We might note that the THE-QS rankings until 2009 and the QS rankings of last year  have probably done quite a lot to encourage complacency by consistently overrating British universities especially Oxbridge and the London colleges.

Top US Colleges by Salary

PayScale has published its ranking of US colleges by mid-career median salary. The top five are:

1. Harvey Mudd College
2.  Princeton
3.  Dartmouth
4.  Harvard
5.  Caltech

The top schools in various categories are:

Engineering: Harvey Mudd
Ivy League: Princeton
Liberal Arts: Harvey Mudd
Party Colleges: Union College, NY
Private Research Universities: Princeton
State Universities: Colorado School of Mines

Can 25 Million Citations be Wrong?

Perhaps not but a few hundred might.

University World News has an article by Phil Baty, deputy editor of Times Higher Education, that discusses the recent THE World University Rankings. He is mainly concerned with the teaching component of the rankings and I hope to discuss this in a little while. However, there are some remarks about the citation component that are worth commenting on. He says:

"We look at research in a number of different ways, examining research reputation, income and research volume (through publication in leading academic journals indexed by Thomson Reuters). But we give the highest weighting to an indicator of 'research influence', measured by the number of times a university's published research is cited by academics around the globe.


We looked at more than 25 million citations over a five-year period from more than five million articles. All the data were normalised to reflect variations in citation volume between different subject areas.


This indicator has proved controversial, as it has shaken up the established order, giving high scores to some smaller institutions with clear pockets of research excellence, often at the expense of the larger research-intensive universities.


We make no apology for recognising quality over quantity, but we concede that our decision to openly include in the tables the two or three extreme statistical outliers, in the interests of transparency, has given some fuel for criticism, and has given us some food for thought for next year's table."

First, normalisation of data means that the number of citations is compared to a benchmark derived from the world average number of citations for a subject area. A large number of citations might mean that an article has been warmly received. It might equally well mean that the article was in a field where articles are typically cited a lot. Comparing simple numbers of citations in  a field like literary studies to those in medical research would be  unfair to the former since citations there are relatively scarce. So part of the reason for Alexandria  University's remarkable success in the recent THE WUR was not just the number of citations of the papers of Mohamed El Naschie but also that he was  publishing in a field with a low frequency of citations. Had he published papers on medicine nobody would have noticed.

It is also very likely -- although I cannot recall seeing direct confirmation -- that Thomson Reuters were benchmarking by year so that a university would score more for a citation to a recently cited article than one to an article published four years ago.

In the case of Alexandria and other universities that scored unexpectedly well we are not talking about millions of citations. We are talking about dozens of papers and hundreds of citations the effect of which has been enormously magnified because the papers were in a low-citation field  and were cited within months of publication.

Remember also that we are talking about averages. This means that a university will get a higher score the smaller the number of papers that are published in ISI- indexed journals. Alexandria did not do so well just because El Naschie published a lot and was cited a lot. It also did well because overall it published few articles. Had Alexandria researchers published more then its score would have been correspondingly lower..

Perhaps El Naschie constitutes a clear pocket of excellence, although that is not entirely uncontroversial. But he is a clear pocket of excellence who only became visible because of the modest achievement of the rest of the university. Conversely, there are probably many modestly cited researchers in Europe and the USA who might welcome a move to a  university in Asia or Latin America where a few papers and citations in a low cited discipline would blossom into such a pocket.

Is Aleaxandria one of two or three anomalies? There are in fact many more more anomalies perhaps not quite so obvious and this can be seen by comparing scores for research impact with other rankings of research output and impact such as HEEACT and Scimago or with the scores for research in the THE rankings themselves. It would also be interesting if THE released the released of the academic reputational survey.

Consider what would happen if we had  a couple of universities that were generally similar, with the same income, staff-student ratio and so on. One however had published two or three times the number of ISI indexed articles as the other. Both had a few researchers who had been cited more frequently than is usual for their discipline. Under the current system, the first university would get a much lower score than the second. Can this really be consider a preference for quality over quantity? Only if we think that publishing in ISI journals adds to quantity but does not indicate quality.

 I hope that food for thought means radical revision of the citations indicator.

A minimal list of changes would include adding more markers of research impact, removing self citations and citations to the same university and the same journal and combining the score for the various disciplinary fields.

 If this can be done then the THE rankings may become what was promised.

Is there a Future for Citations?

simplification administrative has some caustic comments on the role of self-citation and reciprocal citation  in the remarkable performance of Alexandria University in the 2010 THE rankings.

The title, 'Bibliometry -- already broken', is perhaps unduly pessimistic but THE and Thomson Reuters are going to have to move quickly if they are to rescue their rankings. An obvious remedy would include removing self-citations and intra-university and intra-journal citations  from the count.

Priorities

An article by William Patrick Leonard in the Korea Herald discusses how this year's rankings by Shanghai Jiao Tong University, THE and QS, whatever their faults, indicate a long term shift in academic excellence from the USA and Europe to China and other parts of Asia.

"All three release their findings as the school year begins. Each employs a similar blend of indicators, which purportedly measure the relative quality of the institutions surveyed. All emphasize institutional reputation expressed in the quality and quantity of faculty publications, peer assessments, faculty/student ratios, budgets and other input quality measures.



These rankings are clearly flawed; for example, it is not evident that the volume of scholarly publications or peer assessments reflects quality in the classroom. Nevertheless, the rankings show that other fast-growing countries are willing to apply their resources to higher education, just as the United States has been doing for years"

.
He then observes how in the US sport seems to take precedence over education.

"Yet, instead of strengthening our academic programming, some are planning costly recreational diversions. In May, the National Football Foundation & College Hall of Fame announced that “six new college football teams are set to take the field for the first time this season with 11 more programs set to launch between 2011 and 2013.” Such an announcement is simultaneously sad and humorous. The resources spent to implement and subsequently prop up these programs could be used to improve technology, science, and engineering programs. Sadly, some institutions have opted for stadiums over instructional infrastructure."