Is There a q Factor in University Ranking?
Although the Shanghai rankings show a high correlation with other rankings (based on a tiny sample of US universities) the HEEACT rankings from Taiwan (Performance Ranking for Scientific Papers for World Universities) do somewhat better. The correlation with THE-QS is .740, the Shanghai ARWU .984, the USNWR America's best Colleges .711, Professional Ranking of World Universities .920 and the Center for College Affordability and Productivity .700.
All these rankings measure diffent things. The USNWR measures a variety of indicators related directly or indirectly to the quality of instruction, the CCAP is quite definitely a consumer-orintated ranking, the THES-QS World University Rankings are largely a measure of research performance (reputational survey, citations per faculty and student faculty ratio where researchers are counted in the faculty), the Professional Renking of World Universities counts CEOs of top companies while the Shanghai and Taiwan rankings focus entirely on research, mainly in the natural sciences.
The ability of the Taiwan rankings to predict scores on the other rankings suggests that underlying various measures of university quality is a single q factor, the average intelligence of its faculty. If there is one single number that would tell you about the general quality of a school than it would probably be the average IQ of the faculty, although performance on standardised tests, publications and citations (especially in the hard sciences) and postgraduate degrees might be goood proxies. The strength of the Taiwan rankings would be their focus on research productivity alone.
Incidently, if anyone from HEEACT reads this, please think of a new name for your rankings. PROSPWU is not exactly a memorable acronym.
Discussion and analysis of international university rankings and topics related to the quality of higher education. Anyone wishing to contact Richard Holmes without worrying about ending up in comments can go to rjholmes2000@yahoo.com
Tuesday, June 30, 2009
Sunday, June 28, 2009
Ranking the Rankings
University rankings are popping up everywhere. So how do they compare with one another? One way is to check the correlation between the total scores of the rankings. Here, correlations have been calculated for the scores of ten US universities (every tenth university in the Shanghai rankings excluding those not in the THE-QS top 400).
It seems that the Shanghai ARWU is the most valid of five rankings Correlations for total scores are .796 with the THE-QS, .712 with the USNWR America's Best Colleges, .896 for the Professional Ranking of World Universities (Paris) and .628 for the Center for College Affordability and Productivity.
It looks like on the basis of this extremely small and unrepresentative sample that if you had to pick just one ranking to rely on then it would have to be the Shanghai ARWU.
University rankings are popping up everywhere. So how do they compare with one another? One way is to check the correlation between the total scores of the rankings. Here, correlations have been calculated for the scores of ten US universities (every tenth university in the Shanghai rankings excluding those not in the THE-QS top 400).
It seems that the Shanghai ARWU is the most valid of five rankings Correlations for total scores are .796 with the THE-QS, .712 with the USNWR America's Best Colleges, .896 for the Professional Ranking of World Universities (Paris) and .628 for the Center for College Affordability and Productivity.
It looks like on the basis of this extremely small and unrepresentative sample that if you had to pick just one ranking to rely on then it would have to be the Shanghai ARWU.
Thursday, June 25, 2009
International Rankings Expert Group Conference in Kazakhstan
I have just returned from the International Rankings Expert Group’s fourth conference in Astana, Kazakstan. There were some positive developments at the conference but also a few disappointments.
Starting with the negative aspects, there seems to be a global trend to the proliferation of national rankings which are increasingly and unnecessarily detailed and which impose a serious burden on teachers and researchers.. A case in point is the new ranking produced for Kazakhstan which includes just about every variable imaginable from "the number of Dissertation Councils" to "the availabilty of medical centers, sport halls, preventoriums, recreation zones". Very few at the conference seemed aware of the backwash effect of the rankings boom as universities outside the top 500 create their own rankings or compete for irrelevant awards, medals or certificates. Drudges in the periphery of the world university system now face an endless round of form filling, office tidying, meetings, committees and professional development activities which make teaching difficult and genuine research, as opposed to research-like behaviour, close to impossible.
The European Union ranking project was presented in some detail but I suspect is going to make little impact since it appears largely concerned with making fine distinctions between the research capabilities of faculties and departments.
There was a presentation about the Lisbon project which proposed to ignore research altogether and measure teaching excellence. This is an interesting idea but it seems to miss two important points. One reason for emphasizing the measurement of research is that the qualities required for research, general cognitive ability, reading and writing skills, conscientiousness and interest in a subject also correlate to some extent with teaching ability, however that is measured. Also, the assumption that learning is dependent on teaching which in turn must be regulated by a centralized bureaucracy is surely false, at least for the more able students
Positive developments include a trend towards personalized rankings where consumers assign their own weighting to indicators. There is an interesting project under way in Taiwan.
Richard Vedder introduced a ranking that has the merit of being based largely on publicly accessible data. The basic idea is excellent but there are some issues to be dealt with. Using RateMyProfessors is not a bad way to assess the quality of teaching but to be really valid there needs to be some adjustment for the grades awarded by the instructor. Using the American Who’s Who is also potentially interesting – and could well be applied internationally -- but there are of course obvious issues of bias.
He also gave a presentation without using PowerPoint. I must remember that next time I fill in a form about my innovative teaching methodology.
One measure presented was to create an IREG seal of approval. The logo is ready. I am not sure though whether this is going to be effective.
Overall, the conference has strengthened my conviction that if ranking is to be done it should not be by administrators or businesses but by universities themselves.
I have just returned from the International Rankings Expert Group’s fourth conference in Astana, Kazakstan. There were some positive developments at the conference but also a few disappointments.
Starting with the negative aspects, there seems to be a global trend to the proliferation of national rankings which are increasingly and unnecessarily detailed and which impose a serious burden on teachers and researchers.. A case in point is the new ranking produced for Kazakhstan which includes just about every variable imaginable from "the number of Dissertation Councils" to "the availabilty of medical centers, sport halls, preventoriums, recreation zones". Very few at the conference seemed aware of the backwash effect of the rankings boom as universities outside the top 500 create their own rankings or compete for irrelevant awards, medals or certificates. Drudges in the periphery of the world university system now face an endless round of form filling, office tidying, meetings, committees and professional development activities which make teaching difficult and genuine research, as opposed to research-like behaviour, close to impossible.
The European Union ranking project was presented in some detail but I suspect is going to make little impact since it appears largely concerned with making fine distinctions between the research capabilities of faculties and departments.
There was a presentation about the Lisbon project which proposed to ignore research altogether and measure teaching excellence. This is an interesting idea but it seems to miss two important points. One reason for emphasizing the measurement of research is that the qualities required for research, general cognitive ability, reading and writing skills, conscientiousness and interest in a subject also correlate to some extent with teaching ability, however that is measured. Also, the assumption that learning is dependent on teaching which in turn must be regulated by a centralized bureaucracy is surely false, at least for the more able students
Positive developments include a trend towards personalized rankings where consumers assign their own weighting to indicators. There is an interesting project under way in Taiwan.
Richard Vedder introduced a ranking that has the merit of being based largely on publicly accessible data. The basic idea is excellent but there are some issues to be dealt with. Using RateMyProfessors is not a bad way to assess the quality of teaching but to be really valid there needs to be some adjustment for the grades awarded by the instructor. Using the American Who’s Who is also potentially interesting – and could well be applied internationally -- but there are of course obvious issues of bias.
He also gave a presentation without using PowerPoint. I must remember that next time I fill in a form about my innovative teaching methodology.
One measure presented was to create an IREG seal of approval. The logo is ready. I am not sure though whether this is going to be effective.
Overall, the conference has strengthened my conviction that if ranking is to be done it should not be by administrators or businesses but by universities themselves.
Monday, June 22, 2009
A New Atlantic Alliance?
Recently, the US News and World Report expanded its rankings portfolio to include the World’s 100 Best Universities. This turned out to be nothing more than the THE-QS World University Rankings with a rebranding for the US market. Now the USNWR has gone a step further and produced a list of the world’s top 400 universities along with sundry regional and subject rankings. Once again, this is the QS rankings with a new name.
This is no doubt a shrewd move for QS who are now marketing their rankings on both sides of the Atlantic and appear to be on the way to establishing a near monopoly over the international ranking business. It could, however, be risky for USNWR. People are bound to wonder why it should link up with a company that has a history of errors where American universities are concerned. In 2007 QS got their North Carolina business schools mixed up and as a result caused Fortune magazine to withdraw its business school rankings based on QS data. Will US students and stakeholders forgive the USNWR if its data includes things like a near zero for research for Washington University in St Louis or an unbelievably good score for Duke for student faculty ratio?
Recently, the US News and World Report expanded its rankings portfolio to include the World’s 100 Best Universities. This turned out to be nothing more than the THE-QS World University Rankings with a rebranding for the US market. Now the USNWR has gone a step further and produced a list of the world’s top 400 universities along with sundry regional and subject rankings. Once again, this is the QS rankings with a new name.
This is no doubt a shrewd move for QS who are now marketing their rankings on both sides of the Atlantic and appear to be on the way to establishing a near monopoly over the international ranking business. It could, however, be risky for USNWR. People are bound to wonder why it should link up with a company that has a history of errors where American universities are concerned. In 2007 QS got their North Carolina business schools mixed up and as a result caused Fortune magazine to withdraw its business school rankings based on QS data. Will US students and stakeholders forgive the USNWR if its data includes things like a near zero for research for Washington University in St Louis or an unbelievably good score for Duke for student faculty ratio?
Tuesday, June 16, 2009
An Alternative Global Ranking
This is from GLOBALHIGHERED.
This is from GLOBALHIGHERED.
Finally the decision on who has won the European Commission’s million euro tender – to develop and test a global ranking of universities – has been announced.
The successful bid – the CHERPA network (or the Consortium for Higher Education and Research Performance Assessment), is charged with developing a ranking system to overcome what is regarded by the European Commission as the limitations of the Shanghai Jiao Tong and the QS-Times Higher Education schemes. The final product is to be launched in 2011.
CHERPA is comprised of a consortium of leading institutions in the field within Europe; all have been developing and offering rather different approaches to ranking over the past few years (see our earlier stories here, here and here for some of the potential contenders):CHE – Centre for Higher Education Development (Gütersloh, Germany)Center for Higher Education Policy Studies (CHEPS) at the University of Twente (Netherlands)Centre for Science and Technology Studies (CWTS) at Leiden University (Netherlands)Research division INCENTIM at the Catholic University of Leuven (Belgium)Observatoire des Sciences et des Techniques (OST) in ParisEuropean Federation of National Engineering Associations (FEANI)European Foundation for Management Development (EFMD)
Monday, June 15, 2009
Rankings do matter
One of the most dangersous things about university rankings is that they are becoming -- in parts of Asia at any rate -- symbols of national grandeur or decline, attracting almost as much public concern and interest as the World Cup.
Dr Hsu has an interesting post on the divergent histories of Singapore and Malaysia that contains this comment:
One of the most dangersous things about university rankings is that they are becoming -- in parts of Asia at any rate -- symbols of national grandeur or decline, attracting almost as much public concern and interest as the World Cup.
Dr Hsu has an interesting post on the divergent histories of Singapore and Malaysia that contains this comment:
Incidentally, I think this university ranking [almost certainly he means THE-QS] can be taken as representative of everything comparative among the 2 countries.
Sunday, June 14, 2009
Publish and Pay
There is a growing trend towards open access academic publishing where researchers have to pay for publication. Open access is in principle a good idea but the idea of authors rather than subscribers footing the bill has its dangers.
Firstly, it poses a threat to new academic journals in emerging countries. There are, I suspect quite a few researchers who would find it more convenient to spend a few hundred dollars, especially if comes out of grant money, for speedy and "prestigious" international publication rather than writing for a local journal with limited impact.
Secondly, there is a definite threat to standards if criteria for publication are to relaxed or perhaps even abandoned altogether.
Recently, Philip Davis and Kent Anderson sent a totally nonsensical computer generated paper to the Open Information Science Journal. It was accepted, supposedly after peer review, with a request for the payment of $800 in author's fees. In this case, at least, the peer review process had apparently been dropped altogther.
For more information see The Scientist and the "authors'" blog, The Scholarly Kitchen.
In all fairness, it must be pointed out that another computer generated paper submitted to another journal run by the same company journal was rejected and that one reviewer at least figured out what was going on.
Still, this does have disturbing implications. If publication becomes influenced or even determined by ability to pay then we are heading for the complete corruption of the peer review system.
It would be a good idea if universities refused to consider articles in pay for publication journals as evidence for selection or promotion. Perhaps also, Scopus and other databases could list such journals in a separate category.
Anyway, here is an extract from the first paper:
"In this section, we discuss existing research into red-black trees, vacuum tubes, and courseware [10]. On a similar note, recent work by Takahashi suggests a methodology for providing robust modalities, but does not offer an implementation [9]."
Thursday, June 11, 2009
Ranking the Rankings
University rankings are popping everywhere now. It is time to start comparing them with each other. First, here are the number of results from a Yahoo! search using the official names of the rankings. In the lead is the THE-QS World University Rankings, followed by the USNWR America's Best Colleges. The Shanghai rankings have made a much smaller impact and the Webometrics rankings even less. No doubt a search in languages other than English would lead to different results as would a search using different names.
Still, it seems that in the webosphere THE-QS have a strong lead among the international rankings.
"World University Rankings" (THES-QS) 942,000
"America's Best Colleges" (US News and World Report) 892,000
"Times Good University Guide" (UK) 252,000
"Academic Ranking of World Universities" (Shanghai) 123,000
"Guardian Good University Guide" (UK) 87,300
"Maclean's University Rankings" (Canada) 14,200
"World University Ranking on the Web" (Webometrics) 9,890
"CHE/Die Zeit University Ranking" (Germany) 4,570
"Ranking of Scientific Papers for World Universities" (Taiwan) 2,120
University rankings are popping everywhere now. It is time to start comparing them with each other. First, here are the number of results from a Yahoo! search using the official names of the rankings. In the lead is the THE-QS World University Rankings, followed by the USNWR America's Best Colleges. The Shanghai rankings have made a much smaller impact and the Webometrics rankings even less. No doubt a search in languages other than English would lead to different results as would a search using different names.
Still, it seems that in the webosphere THE-QS have a strong lead among the international rankings.
"World University Rankings" (THES-QS) 942,000
"America's Best Colleges" (US News and World Report) 892,000
"Times Good University Guide" (UK) 252,000
"Academic Ranking of World Universities" (Shanghai) 123,000
"Guardian Good University Guide" (UK) 87,300
"Maclean's University Rankings" (Canada) 14,200
"World University Ranking on the Web" (Webometrics) 9,890
"CHE/Die Zeit University Ranking" (Germany) 4,570
"Ranking of Scientific Papers for World Universities" (Taiwan) 2,120
Sunday, June 07, 2009
Is this an error?
According to the QS.com ranking of Asian universities, the best university in Asia for Student/Faculty Ratio is "College of Medicine, Pochon Cha University". (it seems that it is actually Pochon CHA, with CHA being the name of a private medical conglomerate).
This is a little odd since the institution is clearly a single subject one and therefore presumably should not have been included in the rankings at all. This was the rationale for the University of California at San Francisco being removed after a brief appearance in the world rankings.
It is possible though that QS has different requirements for being included in the world and the regional rankings. If this is the case then countries can now use a new strategy for getting excellent scores in the rankings. Just designate medical schools or faculties as independent universities. They will get good scores for publications and citations since medical researchers tend to publish short articles that are cited more frequently and more quickly than in other disciplines and for student/faculty ratio since they have a lot of clinical faculty who can be added to the faculty totals.
It will be interesting to see how long Pochon CHA University remains in the Asian rankings or whether it will appear in the forthcoming world rankings.
According to the QS.com ranking of Asian universities, the best university in Asia for Student/Faculty Ratio is "College of Medicine, Pochon Cha University". (it seems that it is actually Pochon CHA, with CHA being the name of a private medical conglomerate).
This is a little odd since the institution is clearly a single subject one and therefore presumably should not have been included in the rankings at all. This was the rationale for the University of California at San Francisco being removed after a brief appearance in the world rankings.
It is possible though that QS has different requirements for being included in the world and the regional rankings. If this is the case then countries can now use a new strategy for getting excellent scores in the rankings. Just designate medical schools or faculties as independent universities. They will get good scores for publications and citations since medical researchers tend to publish short articles that are cited more frequently and more quickly than in other disciplines and for student/faculty ratio since they have a lot of clinical faculty who can be added to the faculty totals.
It will be interesting to see how long Pochon CHA University remains in the Asian rankings or whether it will appear in the forthcoming world rankings.
Wednesday, June 03, 2009
Not a Good Start
The Malaysian government has awarded Universiti Sains Malaysia the coverted APEX University status, meaning that it gets a lot of money to try and get in the top 100 world. universities.
Unfortunately, things went wrong last Friday when the university's website informed 8,000 plus students that they had been accepted. In fact, only 3,599 had been and it took 24 hours for the university to correct the error. Not a good start but it will probably boost USM's scores in the Webometrics rankings.
See Education in Malaysia for more coverage
The Malaysian government has awarded Universiti Sains Malaysia the coverted APEX University status, meaning that it gets a lot of money to try and get in the top 100 world. universities.
Unfortunately, things went wrong last Friday when the university's website informed 8,000 plus students that they had been accepted. In fact, only 3,599 had been and it took 24 hours for the university to correct the error. Not a good start but it will probably boost USM's scores in the Webometrics rankings.
See Education in Malaysia for more coverage
Ranking News
The European Union is planning on introducing a rival to the Shanghai and THES-QS rankings. This is a good idea in principle but who is going to get the contract? It is a pity that "internationalisation" is going to be an indicator and what exactly does "community outreach" mean?
The European Union is planning on introducing a rival to the Shanghai and THES-QS rankings. This is a good idea in principle but who is going to get the contract? It is a pity that "internationalisation" is going to be an indicator and what exactly does "community outreach" mean?
Odile Quintin, the European Commission's director-general for education, told the HES that the Shanghai Jiao Tong was "firmly concentrated on research", anchored to the production of Nobel laureates, and narrow in scope.
"We think that universities have a strong role in research but also in teaching and employability so we are promoting an alternative ranking to measure all these dimensions," she said.
The ranking would be handled by a consortium working independently of the EC, and work would begin after the results of a tendering process were revealed next week.
The plan is to develop the ranking throughout 2009 and 2010, for implementation a
year later. The project will have a budget of E1.1 million ($1.9m).
Ms Quintin said the new ranking, while based in Europe, would have a global reach.
She added that the new European survey would be focused much more on
disciplinary strength, "because you can be the best university in nanotechnology
but not in psychology".She said the alternative world ranking system would be independent, run neither by governments nor universities and provide a multidimensional measure of education, research, innovation, internationalisation and community outreach.