Opening the Black Box

I have just seen an article by Mu-Hsuang Huan of National Taiwan University published by Research Evaluation, an Oxford University Press journal. Here is the abstract:

"In the era of globalization, the trend of university rankings gradually shifts from country-wide analyses to world-wide analyses. Relatively high analytical weightings on reputational surveys have led Quacquarelli Symonds (QS) World University Rankings to criticisms over the years. This study provides a comprehensive discussion of the indicators and weightings adopted in the QS survey. The article discusses several debates stirred in the academia on QS. Debates on this ranking system are presented in the study. Firstly, problems of return rate, as well as unequal distribution of returned questionnaires, have incurred regional bias. Secondly, some universities are listed in both domestic and international reputation questionnaires, but some others are listed only in the domestic part. Some universities were evaluated only by domestic respondents, limiting their performance of the ranking results. Thirdly, quite a few universities exhibit the same indicator scores or even full scores, rendering the assessment questionable. Lastly, enormous changes of single indicator scores suggest that the statistic data adopted by QS Rankings should be further questioned."

The article is useful and interesting, especially the table of rates of return for different countries. But it does not seem to go beyond what this blog and others have been saying for a long time.

This seems to be another case of mainstream academia lagging behind the mass media which in turn  is way behind the blogosphere.