Comparing Two Surveys

Over the last few years university rankings have acquired a large audience. Each year since 2003 , when the first Shanghai index came out, the ups and downs of universities, especially in East and Southeast Asia, have commanded almost as much attention as the fortunes of national football teams.

This year it seems that competition between the rankers, Times Higher Education and their former partners, QS, will be get as much attention as that between universities and a lot of that attention will go to the merits or flaws of the surveys that are now under way.

Times Higher have just announced the launching of the new reputational survey while QS have started a sign -up facility. If THE are going to start the survey now then they could create a problem for QS since after one e-mail message plus a few follow-ups (I expect Ipsos MORI will tell them about this) and, for some people, a form from the EU rankings, severe ranking fatigue will set in and the later survey forms will go unanswered.

Here are some points of comparison of the two main surveys that will be filling academic e-mail boxes in the next few weeks or months.


Indicator Weighting

QS have stated that their survey will continue to have a weighting of 40 percent. Times Higher say that theirs will have a smaller weighting but have not said exactly how small. Probably the reduction will not be too great if the expense and effort of conducting a survey is to be justified.

Participants

The bulk of QS's survey respondents have come from the mailing lists of World Scientific, a Singapore based publishing company that is linked with Imperial College London and has had a close relationship with Peking University. Others, mainly in the humanities and social sciences, have come from Mardev, a company that collects academic addresses. Some no doubt have been identified during QS's various seminars and tours. This year QS have added a sign up facility that will screen those who wish to take part.

THE will get most of their respondents from the Thomson Reuters internal database by which they presumably mean authors of papers in ISI-indexed journals and conference proceedings, supplemented by so far unidentified third party sources.

The basic qualification then for participating in the QS survey is therefore to subscribe to a newsletter from World Scientific. For the THE survey it will be to to have published a paper in a reputable academic journal or conference proceedings. The THE respondents should then be better qualified to comment on research quality, although one might note that the assigning of the role of first or corresponding author is sometimes a political decision rather than a recognition of actual contributions to a research project.

Numbers

THE have said that they are aiming at a target of 25,000 participants. QS appear to be aiming at close to 17,000 this year.

Regional and Disciplinary Balance

QS have stated that they weight by discipline and subject when selecting potential respondents from the World Scientific and Mardev databases. After data collection they balance responses between three super- egions, the Americas, Asia and the Pacific, and Africa, Europe and the Middle east, but not apparently within those regions. THE have stated they will distribute the survey forms to reflect the world distribution of academic researchers geographically and in terms of discipline.

Questions

THE have stated that they will be asking questions about teaching and research and that the questions about research will be more focused than in the past. QS will continue to ask only about research, which is a little odd since their respondents probably include many who teach but do not do research.

Languages

Last year the THE- QS forms could be answered in English or Spanish. QS may be including other language options this year. So far, it looks as though the THE forms will be entirely in English.

General

It appears that THE may produce a valid survey of the opinion of recently published researchers that reflects the current global distribution of academic research activity. The main problem may well be that there will be a serious conflict between quantity and quality. Academic e-mail addresses are highly degradable and THE may find that many of their published researchers have retired, been downsized, moved, died, forgotten their password or just got fed up with filling out online survey forms. If, in pursuit of the targeted 25,000, they are forced to start trying to contact scientists who published an article (or just put their names on the work of graduate students) several years ago the validity of the survey may become questionable.


On the other hand, it would seem an error for QS to insist on continuing to ask only about research. The THE-QS survey was a dubious measure of research performance but it might have more credibility if it also measured teaching quality or social and economic contributions.

On balance, it would seem that THE, if it can get the the number of respondents it needs, will produce a more accurate and credible survey of opinion about research, although QS might claim that by reaching out to university teachers and non-English speakers they are providing a platform for those whose views ought to be considered in any opinion survey.