Would anyone notice if a small, old but awfully clever dog filled in a university ranking survey, and would it make a difference?

The Australian newspaper The Age has a piece by Erica Cervini on how she allowed her dog to complete the QS academic reputation survey on the quality of veterinary schools.

She doesn't elaborate on how the dog chose the schools. Was it by barking or tail wagging when shown pictures of the buildings?

Seriously though, she does have a point. Can QS stop people signing up just to support their employer or outvote their rivals?

To be fair, QS are aware that their surveys might be manipulated and have taken steps over the years to prevent this by such means as forbidding respondents from voting for their declared employer or repeat voting from the same computer. Even so, it seems that some universities, especially in Latin America, are getting scores in the reputation surveys that appear too high,especially when compared with their overall scores. In the employer survey the Pontifical Catholic University of Chile is 56th (overall 167) and the University of Buenos Aires 49th (overall 198). In the academic survey the University of Sao Paulo is 54th (overall 132 and the National Autonomous University of Mexico 55th (overall 175).

QS are apparently considering reforming their citations per faculty indicator and allowing unchanged responses to the surveys to be recycled for five instead of three years. This is welcome but a more rigorous overhaul of the reputation indicators is sorely needed.

Perfect Storm Heading for Tokyo Metropolitan University

Seen on the Times Higher Education website today:

Tokyo Metropolitan University

World's Best University

Scored a Perfect 100.00 for Two Years in Citations Sector

From TMU to the World

Tokyo Metropolitan University got its perfect score largely because it was one of hundreds of institutions contributing to a few publications from the Large Hadron Collider project. In their recent experimental African rankings THE started using fractionalized counting of citations. If THE use this method in the coming world rankings then TMU will surely suffer a dramatic fall in the citations indicator.

I would not like to be the president of TMU on September 30th.

Recommended Reading

Anybody interested in educational policy, especially the never ending campaign to close gaps of one sort or another or the oddities of university rankings should take a look at chapter four of Jordan Ellenberg's How not to be wrong: The  power of mathematical thinking which is about the obvious -- or ought to be obvious observation -- that smaller populations are more variable.

He notes that South Dakota is top of the league for brain cancer while North Dakota is near the bottom. What makes the difference? It is just that the bigger the population the more likely it is that outliers will be diluted by a great mass of mediocrity. So, extreme scores tend to crop up in small places or small samples.

Similarly when he tossed coins ten at a time he came up head counts ranging from 3 to 9 out of ten.

When he tossed them 100 at a time he got counts ranging from 45 to 60.

When he (actually his computer program) tossed them 1,000 times, the counts ranged from 462 to 537.

It is worth remembering this when a study with a double digit sample is published showing the latest way to close one of achievement gaps or a very small school in a rural state somewhere starts boosting the test scores of underperforming students or a few test takers reveal that the national IQ is imploding. Or the studies fail to be replicated, if indeed anyone tries.

Or university rankings that show very small or very unproductive institutions having an enormous research impact measured by citations.

In the QS BRICS rankings nearly everybody gets a prize

There is a growing trend towards specialised and regional university rankings. The magic of this is that they can provide something for almost everybody. QS recently published its latest BRICS rankings which combined data from five very different university systems. The result was a triumphant (almost) success for everybody (almost).

Here are some highlights that QS could use in selling the BRICS rankings or an expanded version.

Russian universities are ahead of everybody else for teaching quality.

The top 21 universities in the BRICS for Faculty Student Ratio (perhaps not a perfect proxy for teaching excellence) are Russian, headed by Bauman Moscow State Technical University. Imagine what Russian universities could do if QS recognised the importance of teaching and increased the weighting for this indicator.

India performs excellently for Faculty with a Ph D.

Out of the top 15 for this category, ten are Indian and all of these get the maximum score of 100. Of the other five, four are Brazilian and one Chinese. If only QS realised the importance of a highly qualified faculty, India would do much better in the overall rankings.

South Africa takes five out of the first six places for international faculty.

China has four out of five top places for academic reputation and employer reputation.

Meanwhile a Brazilian university is first for international faculty and another is third for academic reputation.

It seems that with rankings like these a lot depends on the weighting assigned to the various indicators.