Friday, December 29, 2017

Getting ready for next year's university rankings

Getting ready for next year's university rankings.

More on Japan and the Rankings

The Japan Times recently published an article by Takamitsu Sawa, President and Distinguished Professor at Shiga University, discussing the apparent decline of Japan's universities in the global rankings.

He notes that in 2014 there were five Japanese universities in the top 200 of the Times Higher Education (THE) world rankings but only two in 2016. He attributes Japan's poor performance to the bias of the citations indicator towards English language publications and the inability or reluctance of Japanese academics to write in English. Professor Sawa seems to be under the impression that THE does not count research papers not written in English, which is incorrect. It is, however, true that the failure of Japanese scholars to write in English prevents their universities doing better in the rankings. He also blames lack of funding from the government and the Euro-American bias of the THE reputation survey.

The most noticeable thing about this article is that the author talks about exactly one table, the THE World University Rankings. This is unfortunately very common especially among Asian academics, There are now over a dozen global rankings of varying quality and some of them tell a different, and perhaps more accurate, story than THE's. For example, there are several well known international rankings in which there are more Japanese universities in the world top 200 than there are in THE's.

There are currently two in the THE top 200 but seven in the Shanghai Academic Ranking of World Universities (ARWU), ten in the QS World University Rankings, ten in the Russian Round University Rankings, seven in the CWTS Leiden Ranking total publications indicator and ten in the Nature Index.

Let's now take a look at the University of Tokyo (Todai), the country's best known university, and it's position in these rankings. Currently it is 46th in the world in THE but in ARWU it is 23rd, in QS 28th, in Leiden Ranking tenth for publications and tenth in the Nature Index. RUR put the university in 43rd place, still a little better than THE. It is very odd that Professor Sawa should focus on the rankings that puts Japanese universities in the worst possible light and ignore the others.

As noted in an earlier post, Tokyo's tumble in the THE rankings came suddenly in 2015 when THE made some drastic changes in its methodology, including switching to Scopus as data supplier, excluding papers with large numbers of authors such as those derived from the CERN projects, and applying a country adjustment to half instead of all the citations indicator. Then in 2016 THE made further changes for its Asian rankings that further lowered the scores of Japanese universities.

It is true that scores of leading Japanese universities in most rankings have drifted downwards over the last few years but this is a relative trend caused mainly by the rise of a few Chinese and Korean universities. Japan's weakest point, as indicated by the RUR and THE rankings, is internationalisation. These rankings show that the major Japanese universities still have strong reputations for postgraduate teaching and research while the Nature Index and the Leiden Ranking point to an excellent performance in research in the natural science at the highest levels.

Nobody should rely on a single ranking and changes caused mainly by methodological tweaking should be taken with a large bucket of salt.

Tuesday, December 19, 2017

Rankings Calendar

The US News Online Program Rankings will be published on January 9th, 2018

Sunday, December 17, 2017

Rankings in Hong Kong

My previous post on the City University of Hong Kong has been republished in the Hong Kong Standard.

So far I can find no reference to anyone asking about the City University of Hong Kong's submission of student data to THE or data about faculty numbers for any Hong Kong university.

I also noticed that the Hong Kong University of Science and Technology is not on the list of 500 universities in the QS Employability Rankings although it is 12th in the one published in THE. Is there a dot here? 

Saturday, December 16, 2017

Measuring graduate employability; two rankings

Global university rankings are now well into their second decade. Since 2003, when the first Shanghai rankings appeared, there has been a steady growth of global and regional rankings. At the moment most global rankings are of two kinds, those that focus entirely or almost entirely on research and those such as the Russian Round Rankings, Times Higher Education (THE) and Quacquarelli Symonds (QS) that claim to also measure teaching, learning or graduate quality in some way, although even those are biased towards research when you scratch the surface a little.

The ranking industry has become adept at measuring research productivity and quality in various ways. But the assessment of undergraduate teaching and learning is another matter.

Several ranking organisations use faculty student ratio as a proxy for quality of teaching which in turn is assumed to have some connection with something that happens to students during their programmes. THE also count institutional income, research income and income from industry, again assuming that there is a significant association with academic excellence. Indicators like this are usually based on those supplied by institutions. For examples of problems here see an article by Alex Usher and a reply by Phil Baty.

An attempt to get at student quality is provided by the CWUR rankings now based in UAE, counting alumni who win international awards or who are CEOs of major companies. But obviously this is relevant only for a very small number of universities. A new pilot ranking from Moscow also counts international awards.

The only attempt to measure student quality  by the well known rankers that is relevant to most institutions is the survey of employers in the QS world and regional rankings. There are some obvious difficulties here. QS gets respondents from a variety of channels and this may allow some universities to influence the survey. In recent years some Latin American universities have done much better on this indicator than on any other.

THE now publish a global employability ranking which is conducted by two European firms, Trendence and Emerging. This is based on two surveys of recruiters in Argentina, Australia, Austria, Brazil, Canada, China, Germany, France, India, Israel, Italy, Japan, Mexico, Netherlands, Singapore, Spain, South Africa, South Korea, Turkey, UAE, UK, and USA. There were two panels with a total of over 6,000 respondents.

A global survey that does not include Chile, Sweden, Egypt, Nigeria, Saudi Arabia, Russia, Pakistan, Indonesia, Bangladesh, Poland, Malaysia or Taiwan can hardly claim to be representative of international employers. This limited representation may explain some oddities of the rankings such as the high places of the American University of Dubai and and the National Autonomous University of Mexico.

The first five places in these rankings are quite similar to the THE world rankings: Caltech, Harvard, Columbia, MIT, Cambridge.  But there some significant differences after that and some substantial changes since last year. Here Columbia, 14th in the world rankings, is in third place, up from 12th last year. Boston University is 6th here but 70th in the world rankings. Tokyo Institute of Technology in 19th place is in the 251-300 band in the world rankings. CentraleSupelec, is 41st,  but in the world  401-500 group.

These rankings are useful only for a small minority of universities, stakeholders and students. Only 150 schools are ranked and only a small proportion of the world's employers consulted.

QS have also released their global employability rankings with 500 universities. These combine the employer reputation survey, used in their world rankings with  other indicators: alumni outcomes, based on lists of high achievers, partnership with employers, that is research collaboration noted in the Scopus database, employer-student connections, that is employers actively present on campus, and graduate employment rate. There seems to be a close association, at least at the top, between overall scores, employer reputation and alumni outcomes. Overall the top three are Stanford, UCLA, Harvard. For employer reputation they are Cambridge, Oxford, Harvard and for alumni outcomes Harvard, Stanford Oxford.

The other  indicators are a different matter. For employer-student connections the top three are Huazhong University of Science and Technology, Arizona State University, and New York University. In fact seven out of the top ten on this measure  are Chinese. For graduate employment rate they are Politecnico di Torino, Moscow State Institute of International Relations, and Sungkyunkwan University and for partnership with employers Stanford, Surrey and Politecnico Milano. When the front runners in indicators are so different one has to wonder about their validity.

There are some very substantial differences in the ranks given to various universities in these rankings. Caltech is first in the Emerging-Trendence rankings and 73rd in QS. Hong Kong University of Science and Technology is 12th in Emerging-Trendence but not ranked at all by QS. The University of Sydney is 4th in QS and 48th in Emerging-Trendence. The American University of Dubai is in QS's 301-500 band but 138th for Emerging-Trendence

The  rankings published by THE could be some value to those students contemplating careers with the leading companies in the richest countries.

The QS rankings may be more helpful for those students or stakeholders looking at universities outside the very top of the global elite. Even so QS have ranked only a fraction of the world's universities.

It still seems that the way forward in the assessment of graduate outcomes and employability is through standardised testing along the lines of AHELO or the Collegiate Learning Assessment.

Monday, December 11, 2017

Rankings Calendar

The Times Higher Education (THE) Asian Universities Summit will be held at the Southern University of Science and Technology, Shenzen, China, 5th-7th February, 2018. The 2018 THE Asian universities rankings will be announced.

Tuesday, November 28, 2017

Rankings Uproar in Hong Kong

There is a controversy brewing in Hong Kong about the submission of data to the QS World University Rankings. It seems that the City University of Hong Kong (CityU) has submitted a smaller figure for the total number of its students than that presented by the SAR's University Grants Committee (UGC). The objective of this was presumably to boost the score for faculty student ratio, which accounts for 20% of the total score in the QS rankings. The complaints apparently began with two other local universities and were reported in the Chinese language Apple Daily.

There is nothing new about this sort of thing. Back in 2006 I commented on the difference between the number of students at "Beijing University" on the university web site and that declared by QS. Ong Kian Ming has noted discrepancies between the number of students at Malaysian universities reported on web sites and that published by QS and there have been questions about the number of international students at Singapore universities

The first thing that strikes an outside observer about the affair is that the complaint seems to be just about QS and does not mention the THE rankings although exactly the same number of students, 9,240, appears on both the QS and THE pages. The original article in Chinese apparently makes no mention of THE.

This suggests that there might be a bit of politics going on here. THE seems to have a good relationship with some of the leading universities in Hong Kong such as the University of Hong Kong (UHK) and the Hong Kong University of Science and Technology (HKUST). In 2015 THE held a prestigious summit at HKUST where it announced after "feedback from the region" that it was introducing methodological changes that would dethrone the University of Tokyo from the number one spot in the Asian rankings and send it down to seventh place behind HKUST and UHK. It looks as though whoever is complaining about CityU is diverting their eyes from THE.

There is certainly a noticeable difference between the number of students submitted to QS and THE by CityU and that published by the UGC. This is not, however, necessarily nefarious. There are many ways in which a university could massage or trim data in ways compliant with the rankers' guidelines: using a specific definition of Full Time Equivalent, omitting or including branch campuses, research centres, affiliated institutions, counting students at the beginning or the end of the semester, counting or not counting exchange students or those in certificate, diploma, transitional or preparatory programmes. It is also not totally impossible that the government data may not be 100% accurate.

Other Hong Kong universities have also submitted student data that differs from that available at the UGC site but to a lesser extent. 

The UGC's data refers to 13,725 full time equivalent students in 2014-15. It is possible that City University has found legitimate ways of whittling down this number. If nothing else, they could claim that they had to use data from earlier years because of uncertainty about the validity of current data.

The real problem here is that it is possible that some universities have learned that success in the rankings is sometimes as much a matter of careful reading of statistics and guidelines as it is of improved teaching or research.

Another thing that has so far gone unnoticed is that CityU has also been reducing the number of faculty. The UGC reports a total of 2,380 full time equivalent faculty while QS reports 1,349. If the university had just used the raw UGC figures it would have a faculty student ratio of 5.77. The QS figure is 6.85. So by modifying the UGC data, if that is where the university started, CityU actually got a worse result on this indicator. They would, however, have done a bit better on the citations per faculty indicator.

This leads on to what the  Hong Kong  universities did with their faculty numbers.

For the University of Hong Kong the UGC reports a total of 5,093 FTE  staff but the QS site has 3,012. THE does not give a figure for the number of faculty but it is possible to calculate this from the number of students and the faculty student ratio, which are provided.  The current THE profile of UHK has 18,364 students and 18 students per staff, which gives us 1,020 staff.

For HKUST the UGC number of staff is 2,398. The number calculated from THE data is 442. QS has a total of 1,150. 

For the Chinese University of Hong Kong (CUHK) we have these numbers: UGC 5,070, QS 2,208, THE 1,044.

For the Polytechnic University of Hong Kong (PUHK): UGC, 3,356, QS 2,447, THE 809.

The UGC gives 2,380 FTE staff for CityU, QS 1,349, and THE 825.

The UGC also provides the number of  faculty wholly funded by the UGC and this number is  always much lower than the total faculty. The QS faculty numbers are generally quite similar to these although I do not know if there was a decision to exclude non-funded faculty. The calculated THE faculty numbers are much lower than those provided by the UGC and lower than the QS numbers.

I suspect that what is going on is that the leading Hong Kong universities have adopted the strategy of aiming for the THE rankings where their income, resources and international connections can yield maximum advantage. They presumably know that the weighting of the staff/student indicator, where it is better to have more faculty, is only 4.5% but the indicators where fewer total staff are better (international faculty, research income, research productivity, industry income, doctorates awarded, institutional income) have a combined weighting of 25.25%.

CityU in contrast has focussed on the QS rankings and looked for ways of reducing the number of students submitted.

It is possible that HKUST and UHK could justify the data the submitted to the rankers while CityU might not, It does, however, seem rather strange and unfair that City University's student data has come under such intense scrutiny while the faculty data of the other universities is so far unquestioned.

Ranking organisations should heed the suggestion by the International Rankings Experts Group (IREG) that indicators measure outcomes rather than inputs such as staff, facilities or income. They also should think about how much they should use data submitted by institutions. This may have been a good idea when they were ranking 200 or 300 places mainly in North America and Western Europe but now they are approaching 1,000 universities, sometimes very decentralised, and data collection is becoming more complicated and difficult.

QS used to talk about its "validation hierarchy" with central agencies such as HESA and NCES at the top, followed by direct contact with institutions, websites, and ending with "smart" averages. Perhaps this could be revived but with institutional data further down the hierarchy. The lesson of the latest arguments in Hong Kong and elsewhere is that data submitted by universities can often be problematical and unreliable.

Friday, November 17, 2017

Another global ranking?

In response to  suggestion by Hee Kim Poh of Nanyang Technological University, I have had a look at the Worldwide Professional University Rankings which appear to be linked to "Global World Communicator" and the "International Council of Scientists" and may be based in Latvia.

There is a methodology page but it does not include essential information. One indicator is "number of publications to number of academic staff" but there is nothing about how either of these are calculated or where the data comes from. There is a reference to a survey of members of the International Council of Scientists but nothing about the wording of the survey, date of survey, distribution of respondents or the response rate.

Anyway, here is the introduction to the methodology:

"The methodology of the professional ranking of universities is based on comparing universities and professional evaluation by level of proposed training programs (degrees), availability and completeness of information on activities of a university, its capacity and reputation on a national and international levels. Main task is to determine parameters and ratios needed to assess quality of the learning process and obtained specialist knowledge. Professional formalized ranking system based on a mathematical calculation of the relation of parameters of the learning process characterizing quality of education and learning environment. Professional evaluation criteria are developed and ranking is carried out by experts of the highest professional qualification in relevant fields - professors of universities, specialists of the highest level of education, who have enough experience in teaching and scientific activities. Professional rating of universities consists of three components.. "

The top five universities are 1. Caltech,  2. Harvard,  3. MIT,  4. Stanford,  5. ETH Zurich.

Without further information, I do not think that this ranking is worth further attention.