Mysteries of the Rankings

One of the problems with interpreting the Times Higher Education (THE) World University Rankings is that some of the indicators are bundled together with others so that is very difficulty to work out exactly what happens from one year to the next.

One group of indicators, International Outlook: People, Research, includes the ratio of international to domestic staff, the ratio of international to domestic students and the proportion of papers with an international co-author.

Research: Volume, Income, Reputation includes the results of a reputation survey, research income, which is scaled against staff numbers and normalised for purchasing power parity and the number of papers per staff.

Teaching: the Learning Environment is based on five indicators: the academic reputation survey,  staff student ratio, the ratio of doctoral to bachelor's degrees, the number of doctorates divided by the number of academic staff and institutional income scaled against academic staff numbers.

This means that if a university enjoys a rise or suffers a fall for the international, teaching or research indicators it is impossible to see exactly what was the cause.

If, for example, the score for teaching for a given university increased it could result from one or more of the following:

• a rise in the number of votes received in the academic survey
• an increase in the number of teaching staff
• a fall in the number of students
• an increase in the number of doctoral degrees awarded
• a fall in the number of bachelor degrees awarded
• an increase in institutional income.

The published scores for the indicator groups do not represent raw data. They are z scores and therefore can change if the mean scores for all ranked universities change. So, a rise in the teaching indicators score might also result from one or more of the following, which could contribute to lowering the mean score from which the university z scores are calculated:

• a fall in the mean number of votes for ranked universities in the academic survey
• a reduction in the mean number of teaching staff at ranked universities
• an increase in the mean number of students at ranked universities
• a fall in the mean number of doctoral degrees awarded ranked universities
• an increase in the mean number of bachelor degrees awarded at ranked universities
• a fall in the mean institutional income of ranked universities.

Unless THE and TR disaggregate their data interpreting their results is going to be very problematical. It would also be unwise to judge the impact of changes in public or university policies by looking at changes in the rankings.

An example is the University of Hong Kong (UHK). Between 2012 and 2013 it fell from 35th to 43rd place in the THE world rankings, its overall score falling from 75 .6 to 65.5. There were slight falls for citations from 62.1 to 61.5 and international outlook from 81.7 to 80.3 and a somewhat larger fall for income from industry from 62.5 to 56.9. There were very large falls for the combined teaching indicators, from 78.4 to 61.6, and the research  indicators, from 85.9 to 69.9.

THE may be on to something. UHK has slipped in the QS and Shanghai rankings as well.  But exactly what they are on to is not clear.

At the recent World Class Universities conference in Shanghai, Gerard Postiglione referred to suggestions that the fall in the position of UHK and other Hong Kong research universities in the THE  rankings may have resulted from a conflict between indigenous and international leadership, difficulties in attracting funding from industry and the transition from a British-style 3 year degree programme to a 4 year American style programme.

It is difficult to see how the change from a three year  to a four year system would have a direct effect. Such a change could impact the THE rankings by increasing the number of undergraduate students compared to doctoral candidates or compared to faculty but this change began to come into effect in 2012 whereas the date in the 2013 rankings refers to 2011. It is possible though that arguments about the impending switch might have had an effect  on the reputation survey but unless THE and TR publish separate data for all their indicators it is not possible to be certain about the causes for the decline of Hong Kong universities. .