Does Size Really Matter?
Times Higher Education (THE) are keeping the "peer review" but possibly with new questions. According to a recent article they will be using the British pollsters Ipsos MORI to collect data.
"So we are delighted to confirm that for the 2010 Times Higher Education World University Rankings, our new rankings partner Thomson Reuters has commissioned one of the world's leading polling companies, Ipsos Mori, to carry out research to support the peer-review element of the tables. Using a professional polling company means that we can inject proper targeting and transparency into the process while ensuring that we get a much larger response rate than in the past - the aim is for at least 25,000 responses in 2010. It also means that the questions in the opinion survey can be carefully crafted to elicit meaningful and consistent responses while ensuring that every respondent knows what is being asked of them. "
THE seems to be overly concerned with the number of respondents, claiming that the 9,000 plus of the 2009 THE-QS rankings was an inadequate number to represent the millions of academics of one sort or another around the world. They are right to be concerned but the number of respondents is not the main determinant of the validity of any survey. What matters more is the extent to which the sample is representative of the population about which data is sought. If THE and if Ipsos MORI are going to do no more than get a lot of people to fill out online forms then their new survey will be little better than the old one.
If the rankings industry is going to descend into a squabble about who's got the biggest survey then QS might be able to trump THE. They could revive their retired respondents from 2004-06, purchase a large stash of email addresses from Mardev, make the survey more user-friendly (tick boxes instead of typing names) and they might well be able to get above the 25,000 mark.
The choice of Ipsos MORI, whose offices are in London, Harrow, Manchester, Edinburgh, Belfast and Dublin might be an indicator of a narrowing of vision. THE's editorial board, which seems to have become more active of late, is predominantly British with a heavy bias towards officialdom. Discussion about rankings in THES seems rather anglocentric. A subtle slip was Phil Baty's recent reference to "overseas" universities. They may be overseas to you but you are overseas to them and everybody else.
3 comments:
Bravo, your idea is useful
Richard,
It is great that blogs like yours exist to hold us "rankers" to account and to subject all rankings to scrutiny.
But I'm concerned that you seem to jump to speculative conclusions witout engaging directly with us. The whole point of using Ipsos Mori is that they are a professional research company who know what they are doing and who can inject proper social science into the peer review exercise for our 2010 world university rankings.
We are clear that the number of respondents to the old QS rankings were far too low, and we will improve on that. But we are also clear that getting this element right is much more than a numbers game. Ipsos Mori will properly target and properly sample the responses, so that they correctly reflect world higher education demogrphics. This will be a huge improvement on the old QS method.
Also, Thomson Reuters, who will collect the data for us, are a US-based company with offices around the world, and the staff they will have working on this project will be drawn from around the world.
Best wishes, and thanks for your interest. Feel free to speak to me directy at phil.baty@tsleducation.com
Richard,
It is great that blogs like this exist to hold us "rankers" to account and to subject all rankings to scrutiny. I think transparency is essential.
But I am concerned that you appear to jump to speculative conclusions without engaging directly with us.
The point of using Ipsos MORI is that they are a professional polling company who know what they are doing and who can inject some proper social science into the peer review element of the rankings. We are clear that the response rate to the peer review element of the old THE-QS rankings was far too low, and we intend to improve on that with the new 2010 Times Higher Education world university rankings. But we are also clear that surveying is not simply a numbers game. Ipsos will ensure we have a properly targetted and correctly sampled response, to truly reflect the demographics of world higher education.
It is correct that THE is a UK-based publication. But Thomson Reuters, as you will know, are based in the US and have offices around the world. They will be collecting all the data for our rankings, and will be drawing on staff from their world-wide network.
If you would like to speak to me directly, I'm at phil.baty@tsleducation.com
Thanks.
Post a Comment