Bad Mood Rising

In 2006 I tried to get an article published in the Education section of the Guardian, that fearless advocate of radical causes and scourge of the establishment, outlining the many flaws and errors in the Times Higher Education Supplement -- Quacquarelli Symonds (as they were then) World University Rankings, especially its "peer review". Unfortunately, I was told that they would be wary of publishing an attack on a direct rival. That was how University Ranking Watch got started.

Since then QS and Times Higher Education have had an unpleasant divorce, with the latter now teaming up with Thomson Reuters. New rankings have appeared, some of them to rapidly disappear -- there was one from Wuhan and another from Australia but they seem to have vanished. The established rankings are spinning off subsidiary rankings at a bewildering rate.

As the higher education bubble collapses in the West everything is getting more competitive including rankings and everybody -- except ARWU -- seems to be getting rather bad-tempered.

Rankers and academic writers are no longer wary about "taking a pop" at each other. Recently, there has been an acrimonious exchange between Ben Sowter of QS and Simon Marginson of Melbourne University. This has gone so far as to include the claim that QS has used the threat of legal action to try to silence critics.

"[Ben] Sowter [of QS] does not mention that his company has twice threatened publications with legal action when publishing my bona fide criticisms of QS. One was The Australian: in that case QS prevented my criticisms from being aired. The other case was University World News, which refused to pull my remarks from its website when threatened by QS with legal action.

If Sowter and QS would address the points of criticism of their ranking and their infamous star system (best described as 'rent a reputation'), rather than attacking their critics, we might all be able to progress towards better rankings. That is my sole goal in this matter. As long as the QS ranking remains deficient in terms of social science, I will continue to criticise it, and I expect others will also continue to do so."

Meanwhile the Leiter Reports has a letter from "a reader in the UK".

THES DID drop QS for methodological reasons. The best explanation is here: http://www.insidehighered.com/views/2010/03/15/baty
But there may have been more to it? Clearly QS's business practices leave an awful lot to be desired. See: http://www.computerweekly.com/news/1280094547/Quacquarelli-Symonds-pays-80000-for-using-unlicensed-software
Also I understand that the "S" from QS -- Matt Symonds -- walked out on the company due to exasperation with the business practices.  He has been airbrushed from QS history, but can be foud at:  https://twitter.com/SymondsGSB
And as for the reputation survey, there was also this case of blantant manipulation: http://www.insidehighered.com/news/2013/04/08/irish-university-tries-recruit-voters-improve-its-international-ranking
And of course there's the high-pressure sales: http://www.theinternationalstudentrecruiter.com/how-to-become-a-top-500-university/
And the highly lucrative "consultancy" to help universities rise up the rankings: http://www.iu.qs.com/projects-and-services/consulting/
There are "opportunities" for branding -- a snip at just $80,000 -- with QS Showcase: http://qsshowcase.com/main/branding-opportunities/
Or what about some relaxing massage, or a tenis tournament and networking with the staff who compile the rankings: http://www.qsworldclass.com/6thqsworldclass/
Perhaps most distribing of all is the selling of dubious Star ratings: http://www.nytimes.com/2012/12/31/world/europe/31iht-educlede31.html?pagewanted=all&_r=0
Keep up the good work. Its an excellent blog.

All of this is true although I cannot get very excited about using pirated software and the bit about relaxing massage is rather petty -- I assume it is something to do with having a conference in Thailand. Incidentally, I don't think anyone from THE sent this since the reader refers to THES (The S for Supplement was removed in 2008).

This is all a long way from the days when journalists refused to take pops at their rivals, even when they knew the rankings were a bit rum.

Rankings Uproar in Hong Kong

There is a controversy brewing in Hong Kong about the submission of data to the QS World University Rankings. It seems that the City University of Hong Kong (CityU) has submitted a smaller figure for the total number of its students than that presented by the SAR's University Grants Committee (UGC). The objective of this was presumably to boost the score for faculty student ratio, which accounts for 20% of the total score in the QS rankings. The complaints apparently began with two other local universities and were reported in the Chinese language Apple Daily.

There is nothing new about this sort of thing. Back in 2006 I commented on the difference between the number of students at "Beijing University" on the university web site and that declared by QS. Ong Kian Ming has noted discrepancies between the number of students at Malaysian universities reported on web sites and that published by QS and there have been questions about the number of international students at Singapore universities

The first thing that strikes an outside observer about the affair is that the complaint seems to be just about QS and does not mention the THE rankings although exactly the same number of students, 9,240, appears on both the QS and THE pages. The original article in Chinese apparently makes no mention of THE.

This suggests that there might be a bit of politics going on here. THE seems to have a good relationship with some of the leading universities in Hong Kong such as the University of Hong Kong (UHK) and the Hong Kong University of Science and Technology (HKUST). In 2015 THE held a prestigious summit at HKUST where it announced after "feedback from the region" that it was introducing methodological changes that would dethrone the University of Tokyo from the number one spot in the Asian rankings and send it down to seventh place behind HKUST and UHK. It looks as though whoever is complaining about CityU is diverting their eyes from THE.

There is certainly a noticeable difference between the number of students submitted to QS and THE by CityU and that published by the UGC. This is not, however, necessarily nefarious. There are many ways in which a university could massage or trim data in ways compliant with the rankers' guidelines: using a specific definition of Full Time Equivalent, omitting or including branch campuses, research centres, affiliated institutions, counting students at the beginning or the end of the semester, counting or not counting exchange students or those in certificate, diploma, transitional or preparatory programmes. It is also not totally impossible that the government data may not be 100% accurate.

Other Hong Kong universities have also submitted student data that differs from that available at the UGC site but to a lesser extent. 

The UGC's data refers to 13,725 full time equivalent students in 2014-15. It is possible that City University has found legitimate ways of whittling down this number. If nothing else, they could claim that they had to use data from earlier years because of uncertainty about the validity of current data.

The real problem here is that it is possible that some universities have learned that success in the rankings is sometimes as much a matter of careful reading of statistics and guidelines as it is of improved teaching or research.

Another thing that has so far gone unnoticed is that CityU has also been reducing the number of faculty. The UGC reports a total of 2,380 full time equivalent faculty while QS reports 1,349. If the university had just used the raw UGC figures it would have a faculty student ratio of 5.77. The QS figure is 6.85. So by modifying the UGC data, if that is where the university started, CityU actually got a worse result on this indicator. They would, however, have done a bit better on the citations per faculty indicator.

This leads on to what the  Hong Kong  universities did with their faculty numbers.

For the University of Hong Kong the UGC reports a total of 5,093 FTE  staff but the QS site has 3,012. THE does not give a figure for the number of faculty but it is possible to calculate this from the number of students and the faculty student ratio, which are provided.  The current THE profile of UHK has 18,364 students and 18 students per staff, which gives us 1,020 staff.

For HKUST the UGC number of staff is 2,398. The number calculated from THE data is 442. QS has a total of 1,150. 

For the Chinese University of Hong Kong (CUHK) we have these numbers: UGC 5,070, QS 2,208, THE 1,044.

For the Polytechnic University of Hong Kong (PUHK): UGC, 3,356, QS 2,447, THE 809.

The UGC gives 2,380 FTE staff for CityU, QS 1,349, and THE 825.

The UGC also provides the number of  faculty wholly funded by the UGC and this number is  always much lower than the total faculty. The QS faculty numbers are generally quite similar to these although I do not know if there was a decision to exclude non-funded faculty. The calculated THE faculty numbers are much lower than those provided by the UGC and lower than the QS numbers.

I suspect that what is going on is that the leading Hong Kong universities have adopted the strategy of aiming for the THE rankings where their income, resources and international connections can yield maximum advantage. They presumably know that the weighting of the staff/student indicator, where it is better to have more faculty, is only 4.5% but the indicators where fewer total staff are better (international faculty, research income, research productivity, industry income, doctorates awarded, institutional income) have a combined weighting of 25.25%.

CityU in contrast has focussed on the QS rankings and looked for ways of reducing the number of students submitted.

It is possible that HKUST and UHK could justify the data the submitted to the rankers while CityU might not, It does, however, seem rather strange and unfair that City University's student data has come under such intense scrutiny while the faculty data of the other universities is so far unquestioned.

Ranking organisations should heed the suggestion by the International Rankings Experts Group (IREG) that indicators measure outcomes rather than inputs such as staff, facilities or income. They also should think about how much they should use data submitted by institutions. This may have been a good idea when they were ranking 200 or 300 places mainly in North America and Western Europe but now they are approaching 1,000 universities, sometimes very decentralised, and data collection is becoming more complicated and difficult.

QS used to talk about its "validation hierarchy" with central agencies such as HESA and NCES at the top, followed by direct contact with institutions, websites, and ending with "smart" averages. Perhaps this could be revived but with institutional data further down the hierarchy. The lesson of the latest arguments in Hong Kong and elsewhere is that data submitted by universities can often be problematical and unreliable.

What's Happened to Malaysia?


It's that time of year again. A few days ago I received an invitation to take part in the THES-QS "peer review" Last year it came via World Scientific, the Singapore-based publishing company whose subscription list has been used by QS to construct their list of "smart people " who pass judgement on the world's universities. This year it came directly from QS. I do not know whether this means that QS is now using the THES subscription list as its database. If so, we can expect to see some wild fluctuations in the "peer review" and hence the overall rankings in October.



Anyway, here is message from QS.





It's that time of year again. Each year the response to the academic peer review for the Times Higher - QS World University Rankings goes from strength to strength.
QS and Times Higher Education are committed to making these rankings as strong and robust as they can be. Many enhancements have been made to the system in the last 12 months (you can read about them on http://www.topuniversities.com/) but amongst the most important has been your help in increasing the response to our academic peer review questionnaire.
Put simply... your opinion counts. Please share it with us.
You'll notice some slight differences to the survey this year - mainly in that we have included a lot more universities - as a result there are two questions about universities - one asking about those around the world, and a second asking about your own country specifically.
Please be as accurate and honest as possible. Help us make sure that your university contributes a representative response to the survey this year.
http://research.qsnetwork.com/academic
The deadline for response is July 15th.
At the end of the survey, from a selection of offers, you will have the chance to either...
Opt for a $100 discount on delegate's fee for QS APPLE (Asia Pacific Professional Leaders in Education Conference & Exhibition) 2009 in Kuala Lumpur
Enter a draw to win your university a free exhibition table at the QS World Grad School Tour (the world's leading circuit of Masters & PhD recruitement fairs) in a city of your choice
Receive a month month trial subscription to Times Higher Education
Thank you for taking the time to share your opinions with us, and please look out for the results of the Times Higher - QS World University Rankings 2008 - due to be published on October 9th.
Many thanks,
Ben SowterHead of ResearchQS



I was disappointed that I would not have a chance of getting a BlackBerry this time around. I started to fill out the form but stopped when I noticed something very odd. There are no Malaysian universities listed this year. Possible explanations are:



A. For some reason, QS have decided that no Malaysian university is of sufficient quality to even be included in the survey. This is unlikely considering some of the others that are included.



B. QS state at the start of the survey that the respondent's own university will be excluded from consideration. Since I work at a Malaysian university it is to be expected that that particular university would not show up. Perhaps some sort of error has meant that all Malaysian universities have been excluded from the list presented to me.



C. QS have made a principled decision that respondents are not allowed to choose any university in the country in which they work. This would be a good idea and therefore can probably be ruled out straightaway.



D. QS just forgot about Malaysia.



E. A computer error that affected me and nobody else.



Based on past experience, D seems the most likely, followed by B. If D, then in October Malaysian universities are going to get zero on this year's "peer review" and therefore will fall even further in the rankings. There will no doubt be an mass outbreak of soul searching in Malaysian universities and jeering by the opposition. If B, The fall will not be so great but Malaysian universities would still suffer if they are the only ones who cannot receive votes from within the country.



I appeal to any reader of this blog who has completed or is about to complete the THES-QS survey to let me know whether they have also noticed the omission of Malaysian universities or of any other country.

Comparing Two Surveys

Over the last few years university rankings have acquired a large audience. Each year since 2003 , when the first Shanghai index came out, the ups and downs of universities, especially in East and Southeast Asia, have commanded almost as much attention as the fortunes of national football teams.

This year it seems that competition between the rankers, Times Higher Education and their former partners, QS, will be get as much attention as that between universities and a lot of that attention will go to the merits or flaws of the surveys that are now under way.

Times Higher have just announced the launching of the new reputational survey while QS have started a sign -up facility. If THE are going to start the survey now then they could create a problem for QS since after one e-mail message plus a few follow-ups (I expect Ipsos MORI will tell them about this) and, for some people, a form from the EU rankings, severe ranking fatigue will set in and the later survey forms will go unanswered.

Here are some points of comparison of the two main surveys that will be filling academic e-mail boxes in the next few weeks or months.


Indicator Weighting

QS have stated that their survey will continue to have a weighting of 40 percent. Times Higher say that theirs will have a smaller weighting but have not said exactly how small. Probably the reduction will not be too great if the expense and effort of conducting a survey is to be justified.

Participants

The bulk of QS's survey respondents have come from the mailing lists of World Scientific, a Singapore based publishing company that is linked with Imperial College London and has had a close relationship with Peking University. Others, mainly in the humanities and social sciences, have come from Mardev, a company that collects academic addresses. Some no doubt have been identified during QS's various seminars and tours. This year QS have added a sign up facility that will screen those who wish to take part.

THE will get most of their respondents from the Thomson Reuters internal database by which they presumably mean authors of papers in ISI-indexed journals and conference proceedings, supplemented by so far unidentified third party sources.

The basic qualification then for participating in the QS survey is therefore to subscribe to a newsletter from World Scientific. For the THE survey it will be to to have published a paper in a reputable academic journal or conference proceedings. The THE respondents should then be better qualified to comment on research quality, although one might note that the assigning of the role of first or corresponding author is sometimes a political decision rather than a recognition of actual contributions to a research project.

Numbers

THE have said that they are aiming at a target of 25,000 participants. QS appear to be aiming at close to 17,000 this year.

Regional and Disciplinary Balance

QS have stated that they weight by discipline and subject when selecting potential respondents from the World Scientific and Mardev databases. After data collection they balance responses between three super- egions, the Americas, Asia and the Pacific, and Africa, Europe and the Middle east, but not apparently within those regions. THE have stated they will distribute the survey forms to reflect the world distribution of academic researchers geographically and in terms of discipline.

Questions

THE have stated that they will be asking questions about teaching and research and that the questions about research will be more focused than in the past. QS will continue to ask only about research, which is a little odd since their respondents probably include many who teach but do not do research.

Languages

Last year the THE- QS forms could be answered in English or Spanish. QS may be including other language options this year. So far, it looks as though the THE forms will be entirely in English.

General

It appears that THE may produce a valid survey of the opinion of recently published researchers that reflects the current global distribution of academic research activity. The main problem may well be that there will be a serious conflict between quantity and quality. Academic e-mail addresses are highly degradable and THE may find that many of their published researchers have retired, been downsized, moved, died, forgotten their password or just got fed up with filling out online survey forms. If, in pursuit of the targeted 25,000, they are forced to start trying to contact scientists who published an article (or just put their names on the work of graduate students) several years ago the validity of the survey may become questionable.


On the other hand, it would seem an error for QS to insist on continuing to ask only about research. The THE-QS survey was a dubious measure of research performance but it might have more credibility if it also measured teaching quality or social and economic contributions.

On balance, it would seem that THE, if it can get the the number of respondents it needs, will produce a more accurate and credible survey of opinion about research, although QS might claim that by reaching out to university teachers and non-English speakers they are providing a platform for those whose views ought to be considered in any opinion survey.

More About Duke

On January 23rd I wrote to John O’Leary, Editor of the Times Higher Education Supplement concerning the data for Duke University in the 2006 world university rankings. I had already pointed out that the 2006 data appeared to be inaccurate and that, since Duke had the best score in the faculty–student section against which the others were benchmarked, all the scores in this section and therefore all the overall scores were inaccurate. There has to date been no reply. I am therefore publishing this account of how the data for Duke may have been constructed.

It has been clear for some time that the score given to Duke for this section of the rankings and the underlying data reported on the web sites of THES’s consultants, QS Quacquarelli Symonds, were incorrect and that Duke should not have been the highest scorer in 2006 on this section. Even the Duke administration has expressed its surprise at the published data. What has not been clear is how QS could have come up with data so implausible and so different from those provided by Duke itself. I believe I have worked out how QS probably constructed these data, which have placed Duke at the top of this part of the rankings so that it has become the benchmark for every other score.

In 2005 Duke University made an impressive ascent up the rankings from 52nd to 11th. This rise was due in large part to a remarkable score for faculty-student ratio. In that year Duke was reported by QS on their topgraduates site to have a total of 12,223 students, comprising 6,248 undergraduates and 5, 975 postgraduates, and 6,244 faculty, producing a ratio of 1.96 students per faculty. The figure for faculty was clearly an error since Duke itself claimed to have only
1,595 tenure and tenure-track faculty and was almost certainly caused by someone entering the number of undergraduate students at Duke, 6,244 in the fall of 2005, into the space for faculty on the QS database. In any case someone should have pointed out that large non-specialist institutions, no matter how lavishly they are funded, simply do not have fewer than two students per faculty

In 2006 the number of faculty and students listed on QS’s topuniversities site was not so obviously incredible and erroneous but was still quite implausible.

According to QS, there were in 2006 11,106 students at Duke, of whom 6,301 were undergraduates and 4,805 postgraduates. It is unbelievable that a university could reduce the number of its postgraduate students by over a thousand, based on QS’s figures, or about two thousand, based on data on the Duke web site, in the course of a single year.

There were in 2006, according to QS, 3,192 faculty at Duke. This is not quite as incredible as the number claimed in 2005 but is still well in excess of the number reported on the Duke site.

So where did the figures, which have placed Duke at the top of the faculty student ratio component in 2006, come from? The problem evidently faced by whoever compiled the data is that the Duke site has not updated its totals of students and faculty since the fall of 2005 but has provided partial information about admissions and graduations which were used in an attempt to estimate current enrollment for the fall of 2006.

If you look at the Duke site you will notice that there is some information about admissions and graduations. At the start of the academic year of 2005 – 2006 (the “class of 2009”) 1,728 undergraduates were admitted and between July 1st, 2005 and June 30th, 2006 1,670 undergraduate degrees were conferred.

So, working from the information provided by Duke about undergraduate students we have;

6,244-
1,670+
1,728
=
6,302

The QS site indicates 6,301 undergraduate students in 2006.

It seems likely that the number of undergraduates in the fall of 2006 was calculated by adding the number of admissions in the fall of 2005 (it should actually have been the fall of 2006) to the number enrolled in the fall of 2005 and deducting the number of degrees conferred between July 2005 and June 2006. The total thus obtained differs by one digit from that listed by the QS site. This is most probably a simple data entry error. The total obtained by this method would not of course be completely valid since it did not take account of students leaving for reasons other than receiving a degree. It would, however, probably be not too far off the correct number.

The number of postgraduate students is another matter. It appears that there was a botched attempt to use the same procedure to calculate the number of graduate students in 2006. The problem, though, was that the Duke site does not indicate enrollment of postgraduate students in that year. In the fall of 2005 there were 6,844 postgraduate students. Between July 2005 and June 2006 2,348 postgraduate and professional degrees were awarded, according to the Duke site. This leaves 4,496 postgraduate students. The QS topuniversities site reports that there were 4,805 postgraduates in 2006. This is a difference of 309.

So where did the extra 309 postgraduates come from? Almost certainly the answer is provided by the online Duke news of September 6, 2006 which refers to a total of 1,687 first year undergraduate students composed of 1,378 entering the Trinity College of Arts and Science (Trinity College is the old name of Duke retained for the undergraduate school) and 309 undergraduates entering the Pratt School of Engineering. The total number of admissions is slightly different from the number given on the Duke main page but this may be explained by last minute withdrawals or a data entry error.

So it looks like someone at QS took the number of postgraduate students in 2005 , deducted the number of degrees awarded and added students admitted to the Pratt School of Engineering in the fall of 2006 and came up with the total of 4,805 in 2006. This is way off the mark because the 309 students admitted to the School of Engineering are not postgraduates, as is evident from their inclusion in the class of 2010, and no postgraduate admissions of any kind were counted. The result is that Duke appears erroneously to have lost about 2,000 postgraduate students between 2005 and 2006.

The undergraduate and postgraduate students were then apparently combined on the QS site to produce a total of 11,106 students, or about 1,000 less than QS reported in 2005 and about 2,000 less than indicated by Duke for that year.

What about the number of faculty? Here, QS’s procedure appears to get even dodgier. The Duke site refers to 1,595 tenure and tenure track faculty. The QS site refers to 3,192 faculty. Where does the difference come from? The answer ought to be obvious and I am embarrassed to admit that it took me a couple of hours to work it out. 1,595 multiplied by 2 is 3190 or exactly 2 less than QS’s figure. The slight difference is probably another data entry error or perhaps an earlier error of addition.

The Duke site contains a table of faculty classified according to school – Arts and Sciences, Engineering, Divinity and so on adding up to 1,595 and then classified according to status – full, associate and assistant professors, again adding up to 1,595. It would seem likely that someone assumed that the two tables referred to separate groups of faculty and then added them together.

So, having reduced the number of students by not including postgraduate admissions and doubling the number of faculty by counting them twice, QS seem to have come up with a a ratio of 3.48 students per faculty This gave Duke the best score for this part of the ranking against which all other scores were calibrated. The standardized score of 100 should in fact have been given to Yale, assuming, perhaps optimistically, that this ratio has been calculated correctly.

It follows that every score for the faculty student ratio is incorrect and therefore that every overall score is incorrect.

If there is another explain for the unbelievable Duke statistics I would be glad to hear it. But I think that if there is going to be a claim that an official at Duke provided information that is so obviously incorrect then the details of the communication should be provided. If information was obtained from another source, although I do not see any way that it could be, it should be indicated. Whatever the source of the error, someone at QS ought to have checked the score of the top university in each component and should have realized immediately that major universities do not reduce the number of their students so dramatically in a single year and keep it secret. Nor is it plausible that a large general university could have a ratio of 3.48 students per faculty.

To show that this reconstruction of QS’s methods is mistaken would require nothing more than indicating the source of the data and an e-mail address or citation by which it could be verified.

Student Faculty Ratios

Something especially striking about the THES~QS rankings this year is that British universities have done spectacularly well overall while getting miserable scores, comparatively speaking, on the citations section. We have to remember that this component does not measure the absolute numbers of citations but the number per faculty. It is then worth investigating whether the high score for student faculty ratios are the result of inflated faculty numbers which have also led to a reduced score for citations per faculty. First, I want to look at the faculty data for the top British and American universities.

Cambridge

Looking at the QS website we find that they claim that Cambridge has a total of 3,765 Full Time Equivalent (FTE) faculty. The data was entered on 23/8/07 by Saad Shabbir, presumably an employee of QS.

Going to the Cambridge site we find that as of July, 2005, Cambridge had 1,558 academic staff, 1,167 academic-related staff (presumably in computers, administration, libraries and so on and probably also research) and 2,497 contract research staff. Adding the first and third categories and leaving out the second, gives us 4,055, close to QS’s figure for total faculty.

It seems reasonable then to conclude that QS added academic staff to research contract staff and made an adjustment to arrive at a Full Time Equivalent (FTE) number to come up with the total faculty. No doubt they got more up to date information than is available on the university website.

With 18,309 FTE students this gives us a student faculty ratio of 4.9. This is much better than the data from third party sources. The Higher Education Statistics Agency (HESA) provides a figure of 11.9.

It looks like QS have counted both teaching staff and contract research staff who do little or no teaching as faculty members.

Oxford

According to QS Oxford has 3,942 FTE faculty (data entered by Saad Shabbir 21/08/07) and 18,667 FTE students, a ratio of 4.7 students per faculty.

According to Oxford there were (July 2006) 1,407 academic staff, 612 in administration and computing, 169 library and museum staff, 753 in university funded research, 2,138 in externally funded research and 15 in self-funded research (all FTE). All this adds up to 4,094, very close to QS’s figure. It seems that for Oxford, QS has included research and other staff in the total faculty.

According to HESA Oxford has 13 students per faculty.

Imperial College London

The QS site indicates that Imperial has 2,963 FTE faculty and 12,025 FTE students (data entered by Saad Shabbir 21/08/07), a ratio of 3.03.

The Imperial site indicates 1,114 academic staff and 1,856 research staff (FTE 2006-7), a total of 2,970 academic and research staff combined. It would seem that QS have again counted research staff as faculty. This site refers to a 12,509 student load and a student staff ratio of 11.2. The HESA ratio is 9.4.

Harvard

According to QS, the Harvard faculty headcount is 3,369 (data entered by Baerbel Eckelmann 8/07/07). There were 29,000 students by headcount (FTE 16,520).The headcount student faculty ratio is 8.6.

According to the United States News and World Report (USNWR), 8% of Harvard’s faculty are part-time. If part time means doing a quarter of a full time teaching load this means that Harvard’s FTE faculty would be 3,406.The FTE student faculty would then be 4.8.

The Harvard site, however, refers to a much smaller number of faculty, 2,497 non-medical faculty and to 20,042 students, making a ratio of 8.0.The USNWR indicates a ratio of 7 for Harvard (2005).

Something strange about QS’s data is that it refers to a headcount of 13,078 and 3,593 FTE undergraduates. This is something that definitely needs explaining.

Yale

According to QS, the number of faculty by headcount is 3,248. The number of students is 11, 851 by headcount and 10,845 FTE. The headcount student faculty ratio is then 3.6.

According to the Yale site, there are 3,384 faculty and 11,358 students, a ratio of 3.4. (All figures from the 2006-7 academic year.)

For the fall of 2006 the faculty headcount included:

Tenured faculty 906

Term 966

Nonladder 903

Research 609

The USNWR ratio for Yale is 6.

Princeton

According to QS, the faculty headcount was 1,263 (entered by Baerbel Eckelmann 09/07/07). The number of students was 6,708 by headcount and 6,795 FTE. The headcount ratio is 5.3

According to the Princeton site, there are more than 850 FTE faculty and 7,055 students, a ratio of 8.3. USNWR has a ratio of 5.

Conclusion

It seems that QS’s policy is to include any sort of research staff, whether or not they do any teaching, in the category of faculty. In some cases, other professional non-teaching staff are also included. This produces student faculty ratios that are noticeably better than those that can be calculated from, and sometimes specifically stated in, the universities’ web sites or that are provided by other sources. It looks as though British universities have benefited from this more than their American counterparts.

This means, very ironically, that this measure, which is supposed to be a proxy for teaching quality, is to a large extent a reflection of a university’s commitment to research since the employment of large numbers of researchers, or even librarians and computer programmers, would lead to an improvement in this ratio.

It also looks as though leading British universities are favoured disproportionately by this procedure although a definite conclusion would have to wait more extensive analysis.

I think that we can put forward a working hypothesis that British universities have been ascribed inflated faculty numbers and that this contributes to high scores for teaching quality as measured by student faculty radio and to low scores for research as measured by citations per faculty.

QS Stars

University World News (UWN) has published an article by David Jobbins about QS Stars, which are awarded to universities that pay (most of them anyway) for an audit and a three year licence to use the stars and which are shown alongside the listings in the  QS World University Rankings. Participation is not spread evenly around the world and it is mainly medioce universities or worse that have signed up according to a QS brochure. Nearly half of the universities that have opted for the stars are from Indonesia.

Jobbins refers to a report in Private Eye which in turn refers to the Irish Examiner. He writes:

The stars appear seamlessly alongside the listing for each university on the World University Rankings, despite protestations from QS that the two are totally separate operations.

The UK magazine Private Eye reported in its current issue that two Irish universities – the University of Limerick and University College Cork, UCC – had paid “tens of thousands” of euro for their stars.

The magazine recorded that UCC had told the Irish Examiner that the €22,000 (US$26,600) cost of obtaining the stars was worthwhile, as it could be recouped through additional international student recruitment.

The total cost for the audit and a three-year licence is US$30,400, according to the scheme prospectus.

 The Irish Examiner article by Neil Murray is quite revealing about the motivation for signing up for an audit:

UCC paid almost €22,000 for its evaluation, which includes a €7,035 audit fee and three annual licence fees of €4,893. It was awarded five-star status, which it can use for marketing purposes for the next three years.

The audit involved a visit to the college by QS researchers but is mostly based on analysis of data provided by UCC on eight criteria. The university’s five-star rating is largely down to top marks for research, infrastructure, internationalisation, innovation, and life science, but it got just three stars for teaching and engagement.
About 3,000 international students from more than 100 countries earn UCC approximately €19 million a year.

UCC vice-president for external affairs Trevor Holmes said there are plans to raise the proportion of international students from 13% — one of the highest of any Irish college — to 20%.

"Should UCC’s participation in QS Stars result in attracting a single additional, full-time international student to study at UCC then the costs of participation are covered," he said.

"In recent times, unlike many other Irish universities, UCC has not been in a position to spend significant sums on marketing and advertising domestically or internationally. QS Stars represents a very cost-effective approach of increasing our profile in international media and online."

So now we know how much a single international student adds to the revenue of an Irish university.

So far, there is nothing really new here. The QS Stars system has been well publicised and it probably was a factor in Times Higher Education dropping QS as its data collecting partner and replacing them with Thomson Reuters.

What is interesting about the UWN article is that a number of British and American universities have been given the stars without paying anything. These include Oxford and Cambridge and 12 leading American institutions that are described by QS as "independently audited based on publicly available information". It would be interesting to know whether the universities gave permission to QS to award them stars in the rankings. Also, why are there differences between the latest rankings and the QS brochure? Oxford does not have any stars in last year's rankings but is on the list in the brochure. Boston University has stars but is not on the list. It may be just a matter of updating.

It would probably be a good idea for QS to remove the stars from the rankings and keep them in the university profiles.

Universiti Malaya Again

In many countries performance in international university rankings has become as much a symbol of national accomplishment as winning Olympic medals or qualifying for the World Cup. When a local university rises in the rankings it is cause for congratulations for everyone, especially for administrators. When they fall it is an occasion for soul-searching and a little bit of schadenfreude for opposition groups.

Malaysia has been particularly prone to this syndrome. There was a magical moment in 2004 when the first THES-QS ranking put Universiti Malaya (UM), the country's first university, in the world's top 100. Then it went crashing down . Since then it has moved erratically up and down around the 200th position.

Lim Kit Siang, leader emeritus of the Malaysian opposition has this to say in his blog:

At the University of Malaya’s centennial celebrations in June 2005, the then Deputy Prime Minister Datuk Seri Najib Razak threw the challenge to University of Malaya to raise its 89th position among the world’s top 100 universities in THES-QS (Times Higher Education Supplement-Quacquarelli Symonds) ranking in 2004 to 50 by the year 2020.

Instead of accepting Najib’s challenge with incremental improvement of its THES ranking, the premier university went into a free fall when in 2005 and 2006 it fell to 169th and 192nd ranking respectively, and in the following two years in 2007 and 2008, fell out of the 200 Top Universities ranking altogether.

In 2009, University of Malaya made a comeback to the 200 Top Universities Ranking when it was placed No. 180, but in 2010 it again fell out of the 200 Top Universities list when it dropped to 207th placing.

For the 2011 QS Top 200 Universities Ranking, University of Malaya returned to the Top 200 Universities Ranking, being placed at No. 167.

In the THES-QS World University Rankings 2009, University of Malaya leapfrogged 50 places from No. 230 placing in 2008 to No. 180 in 2009; while in the 2011 QS World University Ranking, University of Malaya leapt 40 places from No. 207 in 2010 to No. 167 in 2011.

The QS World University Rankings 2012 will be released in 20 days’ time. Can University of Malaya make another leapfrog as in 2009 and 2011 to seriously restore her place as one of the world’s top 100 universities by before 2015?

The government has announced that in addition to Najib’s challenge to University of Malaya in 2005 to be among the world’s Top 50 universities by 2020, the National Higher Education Strategic Plan called for at least three Malaysian universities to be ranked among the world’s top 100 universities.

Recently, the U.S. News World’s Best Universities Rankings included five local universities in its Top 100 Asian Universities, but this is not really something to celebrate about.

The U.S. News World’s Best Universities Ranking is actually based on the QS 2012 Top 300 Asian University Rankings released on May 30 this year, which commented that overall, although University of Malaya improved its ranking as compared to 2011 ranking, the majority of Malaysian universities dropped in their rankings this year as compared to 2011.

There is a lot of detail missing here. UM"s fluctuating scores had nothing to do with failed or successful policies but resulted from errors, corrections of errors, or "clarification of data", changes in methodology and variations in the collecting and reporting of data .

UM was only in the top 100 of the THES-QS rankings because of a mistake by QS, the data collectors, who thought that ethnic minority students and faculty were actually foreigners and therefore handed out a massive and undeserved boost for the international faulty and international student indicators.

Its fall in 2005 was the result of QS's belated realisation of its mistake.

The continued decline in 2007 may have been because QS changed its procedures to prevent respondents to the academic survey voting for their own institutions or because of the introduction of Z scores which had the effect of substantially boosting the scores in citation per faculty for mediocre universities like Peking but only slightly for laggards like UM.

The rise in 2009 from 230th to 180th position was largely the result of a big improvement in the score for faculty student ratio comprising both a reported fall in the number of students and a reported rise in the number of faculty. It is unlikely that the university administration had thrown 6000 students into the Klang River:more probably somebody told somebody that diploma and certificate students need not be included in the data reported to QS.

Whether UM rises again in the QS rankings is less interesting than its performance in the Shanghai Academic Ranking of World Universities. In 2011 it moved into the top 500 with.scores of 3.4 for highly cited researchers and 34.6 for ISI-indexed publications (compared to 100 for the front-runner Harvard) and 16 for per capita productivity (in this case the top scorer was Caltech).

In 2012 UM had the same score for highly cited researchers and registered a score of 38.6 for publications and a slight improvement to 16.7 for productivity. This meant that UM was now ranked in 439th place and that reaching the 300-400 band in  a few years time would not be impossible.

UM has managed to make it into the Shanghai rankings by actively encouraging research among its faculty and by recruiting international researchers, policies that are unpopular and in marked contrast to those of other Malaysian universities.

What will happen in the QS rankings when they come out next month? Something to watch out for is the employer survey, which has a weighting of ten per cent. In 2011 something odd was going on . Apparently there had been an enthusiastic response to the rankings in Latin America especially the employer survey so that QS resorted to capping the scores for many universities. They reported that:

"QS received a dramatic level of response from Latin America in 2011, these counts and all subsequent analysis have been adjusted by applying a weighting to responses from countries with a distinctly disproportionate level of response."

It seems that one effect of the inflated number of responses was to raise the mean score so that universities with below average scores saw a dramatic fall in their adjusted scores. If there is a further increase in responses this year universities like UM may see a further reduction for this indicator.

Why Beijing University is not the Best in Asia

According to the Times Higher Education Supplement's (THES) recent ranking of world universities, Beijing University ( the correct name is actually Peking University, but never mind) is the best university in Asia and 14th in the world.

Unfortunately, it is not. It is just another mistake by QS Quacquarelli Symonds, THES's consultants. Unless, of course, they have information that has been kept secret from everybody else.

In 2005, Beijing University was, according to THES, ranked 15th in the world. This was partly due to remarkably high scores for the peer review and the recruiter ratings. It also did quite well on the faculty/student section with a score of 26. In that year the top score on that part of the ranking was Ecole Polytechnique in Paris whose 100 score appears to represent a ratio of 1.3 students per faculty. It seems that QS derived this ratio from their datafile for the Ecole, although they also give other figures in another part of their page for this institution. Comparing the Ecole's score to others confirms that this was the data used by QS. It is also clear that for this measure QS was counting all students, not just undergraduates, although there is perhaps some inconsistency about the inclusion of non-teaching faculty. It seems then that, according to QS, Beijing University had a ratio of five students per faculty.

Here is the page from QS's web site with the 2005 data for Beijing University.

Datafile
Demographic
No. of faculty:
15,558
No. of international
faculty:
617
No. of students:
76,572
No. of international
students:
2,015
No. of undergraduates:
15,182
No. of
international
undergraduates:
1,025
No. of postgraduates:
13,763
No. of
international postgrads:
308
Financial
Average
undergrad course
fees:
USD$ 3,700
Average postgrad course fees:
USD$
4,700


Annual library spend:
USD$ 72,000
Source:
World University Research (QS & Times Higher Education Supplement)

Postgraduate Course List
For information on undergraduate
courses, please look out for thelaunch of TopUniversities.com in March
2006

Notice that it indicates that there are 76,572 students and 15, 558 faculty, which would give a ratio of 4.92, very close to 5. We can therefore safely assume that this is where QS got the faculty/student ratio.

But there is something wrong with the data. QS gives a total of 76,572 students but there are only 15,182 undergraduates and 13,763 postgraduates, a total of 28, 945. So where did the 46,000 plus students come from? When there is such a glaring discrepancy in a text it usually means that two different sources were used and were imperfectly synthesised. If we look at Beijing University's web site (it calls itself Peking University), we find this data.

Faculty
At present, Peking University has over 4,574 teachers, 2,691 of whom
are full or associate professors. Among the teachers are not only a number
of senior professors of high academic standing and world fame, but also a host
of creative young and middleaged experts who have been working at the forefront
of teaching and research

And this.

At present, Peking University has 46,074 students.
15,001
undergraduates8,119 master candidates3,956 doctoral candidates18,998 candidates
for a correspondence courses or study at the night school1,776 international
students from 62 countries and regions

QS's data were used for the 2005 ranking exercise. The information on Peking University's web site has no doubt been updated since then. However, it looks like QS obtained the numbers of undergraduates and post graduates from Peking University's site although they left out the 18,998 correspondence and night school students that the university counted.

According to the university's definition of students and teachers, the faculty student ration would be 10.07. Excluding correspondence and night school students but counting international students gives us a ratio of 6.31. The former ratio would probably be the correct one to use. THES's definition of a student is someone "studying towards degrees or substantial qualifications" and there is no indication that these students are studying for anything less. Therefore, it seems that the correct ratio for ratio for Beijing University should be around 10 students per faculty.

Looking at the reference work The World of Learning 2003 (2002) we find that Beijing University had 55,000 students and 4,537 teachers. Probably the data reported to this reference included several thousand students from research institutes or branch campuses or was simply an overstatement. The number of teachers is however, almost identical. But whatever the exact numbers, it is clear that QS made a serious mistake and this meant that the score for faculty/student ratio in 2005 was incorrect. Since it appears that a similar or identical ratio was used for this year's ranking as well, the ratio for 2006 is also wrong.

We still have the problem of where QS came up with the figure of 76,572 students and 15, 558 faculty on its web site. It did not come from Peking University.

Or maybe it did. This is from a brief history of Peking University on its site.

After the readjustment, Peking University became a university comprising
departments of both liberal Arts and Sciences and emphasizing the teaching and
research of basic sciences. By 1962, the total enrollment grew to 10,671
undergraduate students and 280 graduate students. Since 1949, Peking University
has trained for the country 73,000 undergraduates and specialty students,
10,000 postgraduates and 20,000 adult-education students, and many of them have become the backbones on all fronts in China.

There has evidently been a massive expansion in the number of postgraduate students recently. The figure of 73,000 undergraduates who ever completed studies at Peking University is close enough to QS's total of students to arouse suspicion that somebody may have interpreted the data for degrees awarded as that for current enrollment.

There is another possible source. There are several specialist universities in the Beijing area, which is one reason why it is rather silly of THES and QS to refer to Peking University as Beijing University. These include the Beijing Foreign Studies University, the Beijing University of Aeronautics and Astronautics, the Beijing University of Business and Technology and so on.

The sum total of students at these institutions, according to the World of Learning is 75,746 students and 12, 826 teachers. The first is very close to QS's figure and the latter somewhat so. A bit of double counting somewhere might have brought the number of teachers closer to that given by QS. I am inclined to suspect that the figures resulted from an enquiry that was interpreted as a request for information about the specialist Beijing universities.

So what about 2006? Wherever the numbers came from this much is clear. Using Yale as a benchmark for 2006 ( there are problems discussed already with top scoring Duke) it would appear that the ratio of 5 students per faculty was used in the latter year as well as in 2005. But according to the data on the university web site, the ratio should be around 10.

What this means is that Beijing University should have got a score for faculty/student ratio of 31 and not 69. I calculate that Beijing University's overall score, applying THES's weighting, dividing by Harvard's total score and then multiplying by 100, should be 57.3. This would put Beijing University in 28th position and not 14th. It would also mean that Beijing University is not the best in the Asia-Pacific region. That honour belongs to the Australian National University. Nor is it the best in Asia. That would be the National University of Singapore. Also Tokyo and Melbourne are ahead of Beijing University.

If there is a mistake in these calculations please tell me and I will correct it.

This is of course assuming that the data for these universities is correct. We have already noted that the score for Duke is too high but if there are no further errors (a very big assumption I admit) then Beijing should have a much lower position than the one assigned by QS. If QS have information from Beijing University that has not been divulged to the public then they have a duty to let us know.


In a little while I shall write to THES and see what happens.

Conference in Shanghai

I hope to post something in a day or two on the recent World Class Universities conference in Shanghai. Meanwhile, there is an interesting comment by Alex Usher of Higher Education Strategy Associates, a Canadian consulting firm.

"In discussions like this the subject of rankings is never far away, all the more so at this meeting because its convenor, Professor Nian Cai Liu, is also the originator of the Academic Ranking of World Universities, also known as the Shanghai Rankings. This is one of three main competing world rankings in education, the others being the Times Higher Education Supplement (THES) and the QS World Rankings.

The THES and QS rankings are both commercially-driven exercises. QS actually used to do rankings for THES, but the two parted ways a couple of years ago when QS’s commercialism was seen to have gotten a little out of hand. After the split, THES got a little ostentatious about wanting to come up with a “new way” of doing rankings, but in reality, the two aren’t that different: they both rely to a considerable degree on institutions submitting unverified data and on surveys of “expert” opinion. Shanghai, on the other hand, eschews surveys and unverified data, and instead relies entirely on third-party data (mostly bibliometrics).

In terms of reliability, there’s really no comparison. If you look at the correlation between the indicators used in each of the rankings, THES and QS are very weak (meaning that the final results are highly sensitive to the weightings), while the Shanghai rankings are very strong (meaning their results are more robust). What that means is that, while the Shanghai rankings are an excellent rule-of-thumb indicator of concentrations of scientific talent around the world, the QS and THES rankings in many respects are simply measuring reputation.

(I could be a bit harsher here, but since QS are known to threaten academic commentators with lawsuits, I’ll be circumspect.)

Oddly, QS and THES get a lot more attention in the Canadian press than do the Shanghai rankings. I’m not sure whether this is because of a lingering anglophilia or because we do slightly better in those rankings (McGill, improbably, ranks in the THES’s top 20). Either way, it’s a shame, because the Shanghai rankings are a much better gauge of comparative research output, and with its more catholic inclusion policy (500 institutions ranked compared to the THES’s 200), it allows more institutions to compare themselves to the best in the world – at least as far as research is concerned. "

Some technical points. First, Times Higher Education Supplement changed its name to Times Higher Education when it converted to a magazine format in 2008.

Second, the Shanghai rankings are not entirely free from commercial pressures themselves although that has probably had the laudable effect of maintaining a stable methodology since 2003.

Third, both THE and QS accept data from institutions but both claim to have procedures to validate them. Also, the Shanghai rankings do include data from government agencies in their productivity per capita criterion and in some places that might not be any more valid than data from universities.

Fourth, until recently there has been a significant difference in the expert opinion used by THE and by QS. Most of QS's survey respondents were drawn from the mailing lists of the Singapore- and London- based academic publishers, World Scientific,  while THE's are drawn from those who have published papers in the ISI indexes. All other things being equal, we would expect THE's respondents to be more expert. This year the difference has been reduced somewhat as QS are getting most of their experts from the Mardev lists supplemented by a sign up facility.

Fifth, although THE publish a list of 200 universities in print and on their site, there is a fairly easily downloadable iphone app available that lists 400 universities.

The most important point though is the question of consistency. It is quite true that the various indicators in the Shanghai rankings correlate quite closely or very closely with one another (.46 to .90 in 2011 according to a conference paper by Ying Chen  and Yan Wu of the Shanghai Center for World- Class Universities) while some of those in the QS and THE rankings have little or no relation to one another. However, it could be argued that if two indicators show a high correlation with one another then they are to some extent measuring the same thing and one of them is redundant. Still, that is probably better than indicators which statistically have little to do with one another.

What is more important perhaps is the consistency from one year to another. The main virtue of the Shanghai rankings is that changes in position can be assumed to reflect actual real world changes whereas those in the THE and QS rankings could easily be the result of methodological changes or, in the case of THE, omissions or inclusions.

‘again!?’ Yep... Quacquarelli Symonds Ltd (QS) did it again.


Eric Beeekens at Bog,u +S has written some excellent posts on the internationalization of higher education.

A recent one concerns QS Quacquarelli Symonds Ltd (QS) who were responsible for collecting data for a ranking of business schools by Fortune magazine. It seems that QS committed a major blunder by leaving out the Kenan-Flagler School at the University of North Carolina at Chapel Hill, one of the top American business schools and one that regularly appears among the high fliers in other business school rankings. Apparently QS got mixed up with North Carolina State University’s College of Management. They also left out the Boston University School of Business. Beerkens refers to an article in the Economist (subscription required) and remarks:

“After reading the first line, I thought: 'again!?' Yep... Quacquarelli Symonds Ltd (QS) did it again.”

Beerkens then points out that this is not the first time that QS has produced flawed research, referring – for which many thanks – to this blog and others. He concludes:

“It's rather disappointing that reputable publications like THES and Forbes use the services of companies like QS. QS clearly doesn't have any clue about the global academic market and has no understanding of the impact that their rankings are having throughout the world. There has been a lot of critique about the indicators that they use, but at least we can see these indicators. It are the mistakes and the biases that are behind the indicators that make it unacceptable!”


There was a vigorous response from the University of North Carolina. They pointed out that QS had admitted to not contacting the university about the rankings, using outdated information and getting the University of North Carolina mixed up with North Carolina State University. QS did not employ any proper procedures for verification and validation, apparently failed to check with other rankings, gave wrong or outdated information about salaries and provided data from 2004 0r 2005 although claiming that these referred to 2006.

Fortune has done the appropriate and honest, although probably expensive, thing and removed the rankings from its website.

What is remarkable about this is the contrast between Fortune and the THES All of the errors committed by QS with regard to the Fortune rankings are parallelled in the World University Rankings. They have, for example grossly inflated the scores of Ecole Normale Superieure in Paris in 2004 and Ecole Polytechnique in 2005 by counting part-time faculty as full time, and done the same for Duke University – QS does seem to have bad luck in North Carolina, doesn’t it? -- in 2005 by counting undergraduate students as faculty and in 2006 by counting faculty twice, used a database from a Singapore based academic publishing company that specializes in Asia-Pacific publications to produce a survey to represent world academic opinion, conducted a survey with an apparent response rate of less than one per cent and got the names of universities wrong – Beijing University and the Official University of California among others.

It is probably unrealistic for THES to remove the rankings from its website. Still, they could at the very least start looking around for another consultant.

More on the Duke and Beijing Scandals

Sorry, there's nothing here about lacrosse players or exotic dancers. This is about how Duke supposedly has the best faculty-student ratio of any university in the world and how Beijing (Peking) university is supposedly the top university in Asia.

In previous posts I reported how Duke, Beijing and Ecole Polytechnique in Paris (see srchives) had apparently been overrated in the Times Higher Educational Supplement (THES) world university rankings because of errors in counting the number of faculty and students.

QS Quacquarelli Symonds, the consultants who conducted the collection of data for THES, have now provided links to data for each of the universities in the top 200 in the latest THES ranking.
Although some errors have been corrected, it seems that new ones have been committed.

First of all, this year Duke was supposed to be top for faculty-student ratio. The QS site gives a figure of 3,192 faculty and 11,106 students, that is a ratio of 3.48, which is roughly what I suspected it might be for this year. Second placed Yale, with 3,063 faculty and 11,441 students according to QS, had a ratio of 3.74 and Beijing (Peking University -- congratulations to QS for getting the name right this year even if THES did not), with 5,381 faculty and 26,912 students, a ratio of 5.01.

But are QS's figures accurate? First of all, looking at the Duke site, there are 13,088 students. So how did QS manage to reduce the number by nearly 2,000? No doubt, the site needs updating but universities do not lose nearly a sixth of their students in a year.

Next, the Duke site lists 1,595 tenure and tenure track faculty and 925 non-teaching faculty. Even counting the latter we are still far short of QS's 3,192.

If we count only teaching faculty the Duke faculty-student ration would be 8.21 students per faculty. Counting non-teaching faculty would produce a ratio of 5.20, still a long way behind Yale.
It is clear then from data provided by QS themselves that Duke should not be in first place in this part of the rankings. This means that all the data for this component are wrong since all universities are benchmarked against the top scorer in each category and, therefore, that all the overall scores are wrong. Probably not by very much, but QS does claim to be the best.

Where did the incorrect figures come from? Perhaps Duke gave QS a different set of figures from those on its web site. If so, this surely is deliberate deception. But I doubt if that is what happened for the Duke administration seems to have been as surprised as anyone by the THES rankings.

I am wondering if this has something to do with Duke in 2005 being just below Ecole Polytechnique Paris in the overall ranking, The Ecole was top scorer for the faculty-student component in 2005. Is it possible that the data for 2006 was entered into a form that also included the 2005 data and that the Ecole's 100 for 2005 was typed in for Duke for 2006? Is it possible then that the data for numbers of students and faculty were constructed to fit the score of 100 for Duke?

As for Beijing (Peking), QS this year provides a drastically reduced number of faculty and students, 5,381 and 26,912 respectively. But even these figures seem to be wrong. The Peking University site indicates 4,574 faculty. So where did the other 800 plus come from?

The number of students provided by QS is roughly equally to the number of undergraduates, master's and doctoral students listed on Peking University's site. It presumably excludes night school and correspondence students and international students. It could perhaps be argued that the first two groups should not be counted but this would be a valid argument only if the the university itself did not count them in the total number of students and if their teachers were not counted in the number of faculty. It still seems that the most accurate ratio would be about 10 students per faculty and that Beijing's overall position is much too high.

Finally, QS has now produced much more realistic data for the number of faculty at the Ecole Polytechnique Paris, Ecole Normale Superieure Paris and Ecole Polytechnique Federale Lausanne. Presumably, this year part-time staff were not counted.

More about Student-Faculty Ratios

I have just discovered a very good site by Ben Wilbrink, Prestatie-indicatoren (indicator systems). He starts off with "Een fantastisch document voor de kick-off", referring to a monograph by Sharon L. Nichols and David C. Berliner (2005), The Inevitable Corruption of Indicators and Educators Through High-Stakes Testing. Education Policy Studies Laboratory, Arizona State University pdf (180 pp.).

The summary of this study reports that:

"This research provides lengthy proof of a principle of social science known as Campbell's law: "The more any quantitative social indicator is used for social decisionmaking, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." "

This insight might well be applied to current university ranking systems. We have seen, for example, some US universities making it optional for applicants to submit their SAT results. It is predictable that good scores will be submitted to admissions officers, but not bad ones. Universities will then find that the average scores of their applicants will rise and therefore so will their scores on rankings that include SAT data.

I would like to propose a new law, an inversion of Gresham's. Good scores drive out bad.

Wilbrink has some good comments on the THES-QS rankings but I would like to focus on what he says about the student-faculty ratio.

"The faculty/student score (20%)The scores in this rubric are remarkable, to say the least. I do not think the student/staff ratio is less reliable than the other indicators, yet the relation to the world rank score seems to be nil. The first place is for (13) Duke, the second for (4=) Yale, the third for (67) Eindhoven University of Technology. Watch who have not made it here in the top twenty: Cambridge is 27th, Oxford 31st, Harvard 37th, Stanford 119, Berkeley 158. This is one more illustration that universities fiercely competing for prestige (see Brewer et al.) tend to let their students pay at least part of the bill.
"We measure teaching by the classic criterion of staff-to-student ratio." Now this is asking for trouble, as Ince is well aware of. Who is a student, who is a teacher? In the medieval universities these were activities, not persons. Is it much different nowadays? How much? ...

Every administration will creatively fill out the THES/QS forms asking them for the figures on students and teachers, this much is absolutely certain. If only because they will be convinced other administrations will do so. Ince does not mention any counter-measure, hopefully the THES/QS people have a secret plan to detect fraudulent data."

It is possible to test whether Wilbrink's remarks are applicable to the student-faculty scores for the 2006 THES-QS rankings. THES have published a table of student-faculty ratios at British universities from the University and College Union that is derived from data from the Higher Education Statistics Agency (HESA). These include further education students and exclude research-only staff. These results can be compared to the data in the THES-QS rankings


In 2006 QS reported that the top scorer for student-faculty ratio was Duke. Looking at QS's website we find that this represents a ratio of 3.48 students per faculty. Cross-checking shows that QS used the data on their site to construct the scores on the 2006 rankings. Thus, the site reports that Harvard had 3,997 faculty and 24,648 students , a ratio of 6.17 students per faculty, ICL 3,090 faculty and 12,185 students, a ratio 0f 3.94, Peking 5,381 faculty and 26,972 students, a ratio of 5.01, Cambridge 3,886 faculty and 21,290 students, a of ratio of 5 .48. These ratios yielded scores of 56, 88, 69 and 64 on the student-faculty component of the 2006 rankings.


Now we can compare the QS data with those from HESA for the period 1005-06. Presumably ,this represents the period covered in the rankings. If Wilbrink is correct, then we would expect the ratios of the rankings to be much lower and more favourable than those provided by HESA.

That in fact is the case. Seven British universities have lower ratios in the HESA statistics. The se are Cranfield, Lancaster, Warwick, Belfast, Swansea, Strathclyde and Goldsmith's College. In 35 cases the THES-QS score was much better. The most noticeable differences were ICL, 3.95 and 9.9, Cambridge , 5,48 and 12,.30, Oxford 5.70 and 11.9, LSE 6.57 and 13, Swansea, 8.49 and 15.1 and Edinburgh, 8.29 and 14.

It is possible that the differences are the result of different consistent and principled conventions. Thus one set of data might specifically include people excluded by the other. The HESA data, for example, includes further education students, presumably meaning non-degree students, but the THES-QS data apparently does not. This would not, however, seem to make much of a difference between the two sets of data for places like Oxford and LSE.

Both HESA`and QS claim not to count staff engaged only in research.

It is possible then that the data provided by universities to QS has been massaged a bit to give favourable scores. I suspect that this does not amount deliberate lying. It is probably more a case of choosing the most beneficial option whenever there is any ambiguity.

Overall, the ratios provided by QS`are much lower, 11.37 compared to 14.63.

Malaysia and the Rankings: The Saga Continues

Malaysia has had a long and complicated relationship with global rankings ever since that wonderful moment in 2004 when the Times Higher Education Supplement (THES) -- Quacquarelli Symonds (QS) World University Rankings, as they were then, put Universiti Malaya (UM), the country's oldest institution, in the top 100 universities of the world.

It turned out that UM was only in the top 100 because of a ridiculous error by data gatherers QS who counted ethnic Indians and Chinese as international and so boosted the score for the international faculty and international student indicators. This was followed in 2005 by the correction of the error, or "clarification of data" as THES put it, and UM's extraordinary fall out of the top 100, often explained by higher education experts as a change in methodology.

There was another fall in 2007 when QS introduced several methodological changes including the use of z scores, that is calibrating scores against the indicator means, and prohibiting survey respondents from voting for their own universities.

In 2009 UM made something of a recovery rising from 230th in the Times Higher Education (the Supplement bit was dropped in 2008) charts to 180th, largely because of an increase in the number of faculty and a reduction in the number of students reported to QS.

In 2010 THE and QS went their separate ways, publishing their own rankings with different methodologies. UM dropped out of the top 200 of the QS rankings but was back again in 2011 and  has now reached 151st place. It has stayed clear of the THE World University Rankings, which require the annual resubmission of data.

Every time UM or any of the other Malaysian universities rises or falls, it becomes a political issue. Ascent is seen as proof of the strength of Malaysian higher education, decline is the result of policy failures.

Recently, the Malaysian second Minister for Education argued that Malaysian higher education was now world class and on a par with countries such as the United Kingdom, Germany and Australia because of its improved performance in the QS rankings and because it had attracted 135,000 foreign students.

Not everyone was impressed by this. Opposition MP Tony Pua criticised the reliance on the QS rankings, saying that they had been condemned by prominent academics such as Simon Marginson and that UM was not ranked by THE and performed much less well in the other rankings such as the Academic Ranking of World Universities.

The minister has riposted by noting that four Malaysian researchers were included in Thomson Reuters' list of influential scientific minds and that UM had been given five stars by QS.

So, who's right about  Malaysian higher education?

First, Tony Pua is quite right about the inadequacies of the QS world rankings. It can be unstable since the number of universities included in the rankings changes from year to year and this can affect the scores for each indicator. Many of the scores for the objective indicators such as faculty student ratio and international students seem exaggerated and appear to have had a bit of massaging somewhere along the line.

The biggest problem with the QS rankings is the academic and employer  reputation surveys. These collect data from a variety of sources, have low response rates and are very volatile. They include respondents whose names are submitted by universities and those who nominate themselves. There were some suspiciously high scores for the  academic reputation indicator in 2014: Peking University in 19th place, National Taiwan University in 37th, University of Buenos Aires in 52nd and the Pontifical Catholic University of Chile in 78th.

The employer survey also produces some counter-intuitive results: Universitia Commerciale Luigi Bocconi in 33rd place, Indian Institute of Technology Bombay in 60th, the American University of Beirut in 85th and Universidiad de los Andes in 98th.

The QS world rankings can therefore be considered a poor reflection of overall quality.

Some critics have asserted that the THE rankings are superior and that Malaysian universities are being evasive by staying away from them. It is true that  THE  have won the approval of the British political and educational establishment. David Willetts, former universities and science minister, has joined the advisory board of  THE's parent company. THE has highlighted comments on its recent reputation rankings by Greg Clark, universities, science and cities minister, Vince Cable, the Business Secretary and Wendy Piatt, director of the Russell Group of research intensive universities.

However, more informed and observers such as Alex Usher of Higher Education Strategy Associates and Isidro Aguillo of the Cybermetrics Lab have little regard for these rankings.

Even Simon Marginson, who has moved to the London Institute of Education, now accepts that they are "fatally flawed once outside the top 50 universities".

The THE rankings have some serious methodological flaws. They assign a 33% weighting to a reputation survey. After the top six universities the number of responses drops off rapidly. After we leave the top 100 the number of votes on the survey is small and so it is quite normal for a few additional responses to have a disproportionate effect on the indicator scores and consequently on overall scores. QS does give an even greater weighting for reputation -- 50% -- but reduces annual fluctuations by carrying over responses for a further two years if they are not updated. The new Best Global Universities produced by US News takes a five year average of their reputation scores.

In addition, the THE rankings assign a 30 % weighting to their Citations: Research Impact indicator which  is constructed so that it allows contributions to publications, usually in physics , astronomy or medicine, with hundreds of contributing institutions to give a university an undeserved score for citations. Since its beginning, the THE rankings have  shown bizarre results for research impact by putting places like Alexandria University, Moscow State Engineering Physics Institute, Federico Santa Maria Technical University in Valparaiso and the University of Marrakech Cadi Ayyad in the top ranks of the world for research impact.

Yes, QS putting Tsinghua  University in overall 47th place is questionable (the Shanghai Academic Ranking of World Universities has it in 101-150 band)  but on balance this is more plausible than putting, as THE does, Scuola Normale Superiore di Pisa in 63rd place (Shanghai has it in the 301-400 band)

The only situation in which it would make sense for UM to take part in the THE rankings would be if it was going to start a first rate particle physics programme including participation in the Large Hadron Collider project with multi-author publications that would bring in thousands of citations.

Rather than relying on the questionable QS and THE rankings, it would be a good idea to look at the progress of UM according to the less well known but technically competent research-based rankings. The Scimago Institution Rankings show that in a ranking of higher education institutions by research output  UM was in 718th place in 2009.  Since then it has risen to 266th place, behind 16 British (out of 189), 15 German and seven Australian universities and institutes.

This is similar to the CWTS Leiden Ranking which has UM in 270th place for number of publications (calculated with default settings) or the Shanghai rankings, where Nobel prizes are indicators, which place it in the 301-400 band.

This does not necessarily mean that there has been similar progress in graduate employability or in the quality of research. It does, however, mean that for research output Universiti Malaya, and maybe two or three other Malaysian universities, are now competitive with second tier universities in Britain and Germany.

This is probably not quite what most people mean by world-class but it is not impossible that in a decade UM could, if it sticks to current policies, be the rival of universities like Sheffield, Cardiff or Leeds.

But such progress depends on Malaysian universities focussing on their core missions and not falling into the quagmire of mission creep.

It also depends on something being done to remedy the very poor performance of Malaysian secondary schools. If that does not happen then the future of Malaysian higher education and the Malaysian economy could be very bleak.