Worth Reading
Retraction Watch is a blog that deals with the retraction of scientific papers because of plagiarism, duplication, fabrication of data and so on.
Friday, April 22, 2011
Second Edition
The National Research Council in the US has released a new version of its 2010 doctoral program rankings. It seems that there were a large number of errors first time around According to an article by David Glenn in the Chronicle of Higher Education:
The National Research Council in the US has released a new version of its 2010 doctoral program rankings. It seems that there were a large number of errors first time around According to an article by David Glenn in the Chronicle of Higher Education:
The National Research Council released on Thursday a revised edition of its 2010 rankings of American doctoral programs that corrects four types of errors discovered in the original report, which was issued last September. But the new rankings do not deal with certain other concerns that scholars have raised about the project.
In the revised edition, almost all programs' positions on the council's "ranges of rankings" have changed at least slightly, but in most cases the changes are not substantial. In a few academic fields, however, the numbers have changed significantly for at least 20 percent of the programs. Those include geography, linguistics, and operations research.
A spreadsheet of the new rankings is available for download at the council's Web site. The council has also released a separate, much smaller spreadsheet that summarizes the changes in programs' "R" and "S" rankings. (R rankings reflect how similar a program is to the programs in its field with the strongest reputations. S rankings more directly reflect a program's performance on variables that scholars in the field say are most important, such as faculty research productivity or student diversity.)
The new edition makes four kinds of corrections. The original report in many cases undercounted faculty members' honors and awards, the proportion of new graduates who find academic jobs, and the proportion of first-year students who are given full financial support. In nonhumanities fields, the report also used faulty data for faculty members' 2002 publications, which in turn caused errors in calculations of citation counts.
Tuesday, April 19, 2011
Reviewing the THE Rankings
An article by Phil Baty in Times Higher Education looks at the various components of last year's THE World University Rankings and gives some hints about changes to come this year. Some good points but also some problems. My comments are in red.
We look at research in a number of different ways, examining reputation, income and volume (through publication in leading academic journals indexed by Thomson Reuters). But we give the highest weighting to an indicator of “research influence”, measured by the number of times published research is cited by academics across the globe.
We looked at more than 25 million citations over a five-year period from more than five million articles.
Yes, but when you normalise by field and by year, then you get very low benchmark figures and a few hundred citations to a few dozen articles can acquire disproportionate influence.
All the data were normalised to reflect variations in citation volume between different subject areas, so universities with strong research in fields with lower global citation rates were not penalised.
The lower the global citation rates the more effect a strategically timed and placed citation can have and the greater the possibility of gaming the system.
We also sought to acknowledge excellence in research from institutions in developing nations, where there are less-established research networks and lower innate citation rates, by normalising the data to reflect variations in citation volume between regions. We are proud to have done this, but accept that more discussion is needed to refine this modification.
In principle this sounds like a good idea but it could just mean that Singapore, Israel, South Africa and the south of Brazil might be rewarded for being located in under-achieving regions of which they are not really a part.
The “research influence” indicator has proved controversial, as it has shaken up the established order, giving high scores to smaller institutions with clear pockets of research excellence and boosting those in the developing world, often at the expense of larger, more established research-intensive universities.
Here is a list of universities that benefited disproportionately from high scores for the "research influence" indicator. Are they really smaller, are they really in the developing world? And as for those clear pockets of excellence, that would certainly be the case for Bilkent (you can find out who he is in five minutes) but for Alexandria...?
Boston College
University of California Santa Cruz
Royal Holloway, University of London
Pompeu Fabra
Bilkent
Kent State University
Hong Kong Baptist University
Alexandria
Barcelona
Victoria University WellingtonTokyo Metropolitan University
University of Warsaw
Something else about this indicator that nobody seems to have noticed is that even if the methodology remains completely unchanged, it is capable of producing dramatic changes from year to year. Suppose that an article in a little cited field like applied math was cited ten times in its first year of publication. That could easily be 100 times the benchmark figure. But in the second year that might be only ten times the benchmark. So if the clear pocket of research excellence stops doing research and becomes a newspaper columnist or something like that, the research influence score will go tumbling down.
We judge knowledge transfer with just one indicator – research income earned from industry – but plan to enhance this category with other indicators.
This is a good idea since it represents, however indirectly, an external assessment of universities.
Internationalisation is recognised through data on the proportion of international staff and students attracted to each institution.
Enough has been said about the abuses involved in recruiting international students. Elsewhere THE have said that they are adding more measures of internationalisation.
The flagship – and most dramatic – innovation is the set of five indicators used to give proper credit to the role of teaching in universities, with a collective weighting of 30 per cent.
But I should make one thing very clear: the indicators do not measure teaching “quality”. There is no recognised, globally comparative data on teaching outputs at present. What the THE rankings do is look at the teaching “environment” to give a sense of the kind of learning milieu in which students are likely to find themselves.
The key indicator for this category draws on the results of a reputational survey on teaching. Thomson Reuters carried out its Academic Reputation Survey – a worldwide, invitation-only poll of 13,388 experienced scholars, statistically representative of global subject mix and geography – in early 2010.
It examined the perceived prestige of institutions in both research and teaching. Respondents were asked only to pass judgement within their narrow area of expertise, and we asked them “action-based” questions (such as: “Where would you send your best graduates for the most stimulating postgraduate learning environment?”) to elicit more meaningful responses.
In some ways, the survey is an improvement on the THE-QS "peer review" but the number of responses was lower than the target and we still do not know how many survey forms were sent out. Without knowing the response rate we cannot determine the validity of the survey.
The rankings also measure staff-to-student ratios. This is admittedly a relatively crude proxy for teaching quality, hinting at the level of personal attention students may receive from faculty, so it receives a relatively low weighting of just 4.5 per cent.
Wait a minute. This means the measure is the number of faculty or staff per student. But the THE web site says "undergraduates admitted per academic", which is the complete opposite. An explanation is needed
We also look at the ratio of PhD to bachelor’s degrees awarded, to give a sense of how knowledge-intensive the environment is, as well as the number of doctorates awarded, scaled for size, to indicate how committed institutions are to nurturing the next generation of academics and providing strong supervision.
Counting the proportion of postgraduate students is not a bad idea. If nothing else, it is a crude measure of the maturity of the students. However, counting doctoral students may well have serious backwash effects as students who would be quite happy in professional or masters programs are coerced or cajoled into Ph D courses that they may never finish and which will lead to a life of ill-paid drudgery if they do.
The last of our teaching indicators is a simple measure of institutional income scaled against academic staff numbers. This figure, adjusted for purchasing-price parity so that all nations compete on a level playing field, gives a broad sense of the general infrastructure and facilities available.
Yes, this is important and it's time someone started counting it.
An article by Phil Baty in Times Higher Education looks at the various components of last year's THE World University Rankings and gives some hints about changes to come this year. Some good points but also some problems. My comments are in red.
We look at research in a number of different ways, examining reputation, income and volume (through publication in leading academic journals indexed by Thomson Reuters). But we give the highest weighting to an indicator of “research influence”, measured by the number of times published research is cited by academics across the globe.
We looked at more than 25 million citations over a five-year period from more than five million articles.
Yes, but when you normalise by field and by year, then you get very low benchmark figures and a few hundred citations to a few dozen articles can acquire disproportionate influence.
All the data were normalised to reflect variations in citation volume between different subject areas, so universities with strong research in fields with lower global citation rates were not penalised.
The lower the global citation rates the more effect a strategically timed and placed citation can have and the greater the possibility of gaming the system.
We also sought to acknowledge excellence in research from institutions in developing nations, where there are less-established research networks and lower innate citation rates, by normalising the data to reflect variations in citation volume between regions. We are proud to have done this, but accept that more discussion is needed to refine this modification.
In principle this sounds like a good idea but it could just mean that Singapore, Israel, South Africa and the south of Brazil might be rewarded for being located in under-achieving regions of which they are not really a part.
The “research influence” indicator has proved controversial, as it has shaken up the established order, giving high scores to smaller institutions with clear pockets of research excellence and boosting those in the developing world, often at the expense of larger, more established research-intensive universities.
Here is a list of universities that benefited disproportionately from high scores for the "research influence" indicator. Are they really smaller, are they really in the developing world? And as for those clear pockets of excellence, that would certainly be the case for Bilkent (you can find out who he is in five minutes) but for Alexandria...?
Boston College
University of California Santa Cruz
Royal Holloway, University of London
Pompeu Fabra
Bilkent
Kent State University
Hong Kong Baptist University
Alexandria
Barcelona
Victoria University WellingtonTokyo Metropolitan University
University of Warsaw
Something else about this indicator that nobody seems to have noticed is that even if the methodology remains completely unchanged, it is capable of producing dramatic changes from year to year. Suppose that an article in a little cited field like applied math was cited ten times in its first year of publication. That could easily be 100 times the benchmark figure. But in the second year that might be only ten times the benchmark. So if the clear pocket of research excellence stops doing research and becomes a newspaper columnist or something like that, the research influence score will go tumbling down.
We judge knowledge transfer with just one indicator – research income earned from industry – but plan to enhance this category with other indicators.
This is a good idea since it represents, however indirectly, an external assessment of universities.
Internationalisation is recognised through data on the proportion of international staff and students attracted to each institution.
Enough has been said about the abuses involved in recruiting international students. Elsewhere THE have said that they are adding more measures of internationalisation.
The flagship – and most dramatic – innovation is the set of five indicators used to give proper credit to the role of teaching in universities, with a collective weighting of 30 per cent.
But I should make one thing very clear: the indicators do not measure teaching “quality”. There is no recognised, globally comparative data on teaching outputs at present. What the THE rankings do is look at the teaching “environment” to give a sense of the kind of learning milieu in which students are likely to find themselves.
The key indicator for this category draws on the results of a reputational survey on teaching. Thomson Reuters carried out its Academic Reputation Survey – a worldwide, invitation-only poll of 13,388 experienced scholars, statistically representative of global subject mix and geography – in early 2010.
It examined the perceived prestige of institutions in both research and teaching. Respondents were asked only to pass judgement within their narrow area of expertise, and we asked them “action-based” questions (such as: “Where would you send your best graduates for the most stimulating postgraduate learning environment?”) to elicit more meaningful responses.
In some ways, the survey is an improvement on the THE-QS "peer review" but the number of responses was lower than the target and we still do not know how many survey forms were sent out. Without knowing the response rate we cannot determine the validity of the survey.
The rankings also measure staff-to-student ratios. This is admittedly a relatively crude proxy for teaching quality, hinting at the level of personal attention students may receive from faculty, so it receives a relatively low weighting of just 4.5 per cent.
Wait a minute. This means the measure is the number of faculty or staff per student. But the THE web site says "undergraduates admitted per academic", which is the complete opposite. An explanation is needed
We also look at the ratio of PhD to bachelor’s degrees awarded, to give a sense of how knowledge-intensive the environment is, as well as the number of doctorates awarded, scaled for size, to indicate how committed institutions are to nurturing the next generation of academics and providing strong supervision.
Counting the proportion of postgraduate students is not a bad idea. If nothing else, it is a crude measure of the maturity of the students. However, counting doctoral students may well have serious backwash effects as students who would be quite happy in professional or masters programs are coerced or cajoled into Ph D courses that they may never finish and which will lead to a life of ill-paid drudgery if they do.
The last of our teaching indicators is a simple measure of institutional income scaled against academic staff numbers. This figure, adjusted for purchasing-price parity so that all nations compete on a level playing field, gives a broad sense of the general infrastructure and facilities available.
Yes, this is important and it's time someone started counting it.
Monday, April 18, 2011
Art Imitates Life: Update
It is time to bring the unbearable suspense to an end. The first text is the April Fool's Joke. Go here. The second is an article in Notices of the AMS by Douglas Arnold and Kristine Fowler.
It is time to bring the unbearable suspense to an end. The first text is the April Fool's Joke. Go here. The second is an article in Notices of the AMS by Douglas Arnold and Kristine Fowler.
Thursday, April 14, 2011
Art Imitates Life
Which of these texts is an April Fool joke? How can you tell? Links will be posted in a few days.
TEXT 1
The Federal Intelligence Service discovered a Ponzi scheme of academic citations lead by an unemployed particle physicist. A house search conducted in Berlin last week revealed material documenting the planning and administration of a profitable business of trading citations for travel reimbursement.
According to the Federal Intelligence Service, the hint came from researchers at Michigan University, Ann Arbor, who were analyzing the structure of citation networks in the academic community. In late 2010, their analysis pointed towards an exponentially growing cluster originating from a previously unconnected researcher based in Germany's capital. A member of the Ann Arbor group, who wants to remain unnamed, inquired about the biography of the young genius, named Al Bert, sparking such amount of activity. The researcher was easily able to find Dr. Bert scheduled for an unusual amount of seminars in locations all over the world, sometimes more than 4 per week. However, upon contacting the respective institutions, nobody could remember the seminars, which according to Prof. Dr. Dr. Hubert at The Advanced Institute is "Not at all unusual." The network researcher from Ann Arbor suspected Dr. Bert to be a fictitious person and notified the university whose email address Dr. Bert was still using.
It turned out Dr. Bert is not a fictitious person. Dr. Bert's graduated in 2006, but his contract at the university run out in 2008. After this, colleagues lost sight of Dr. Bert. He applied for unemployment benefits in October 2008. As the Federal Intelligence Service reported this Wednesday, he later founded an agency called 'High Impact' (the website has since been taken down) that offered to boost a paper's citation count. A user registered with an almost finished, but not yet published, paper and agreed to pay EUR 10 to Dr. Bert's agency for each citation his paper received above the author's average citation count at the time of registration. The user also agreed to cite 5 papers the agency would name. A registered user would earn EUR 10 for each recruitment of a new paper, possibly their own.
This rapidly created a growing network of researchers citing each others papers, and encouraged the authors to produce new papers, certain they would become well cited. Within only a few months, the network had spread from physics to other research fields. With each citation, Dr. Bert made an income. The algorithm he used to assign citations also ensured his own works became top cites. Yet, with many researchers suddenly having papers with several hundred citations above their previously average citation count, their fee went into some thousand dollars. On several instances Dr. Bert would suggest they invite him for a seminar at their institution and locate it in a non-existent room. He would then receive reimbursement for a fraudulent self-printed boarding pass, illegible due to an alleged malfunctioning printer.
Names of researchers subscribed to Dr. Bert's agency were not accessible at the time of writing.
TEXT 2
The field of applied mathematics provides an illuminating case in which we can study such impact-factor distortion. For the last several years, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS) has dominated the impact-factor charts in the “Mathematics, Applied” category. It took first place in each year 2006, 2007, 2008, and 2009, generally by a wide margin, and came in second in 2005. However, as we shall see, a more careful look indicates that IJNSNS is nowhere near the top of its field. Thus we set out to understand the origin of its large impact factor.
Which of these texts is an April Fool joke? How can you tell? Links will be posted in a few days.
TEXT 1
The Federal Intelligence Service discovered a Ponzi scheme of academic citations lead by an unemployed particle physicist. A house search conducted in Berlin last week revealed material documenting the planning and administration of a profitable business of trading citations for travel reimbursement.
According to the Federal Intelligence Service, the hint came from researchers at Michigan University, Ann Arbor, who were analyzing the structure of citation networks in the academic community. In late 2010, their analysis pointed towards an exponentially growing cluster originating from a previously unconnected researcher based in Germany's capital. A member of the Ann Arbor group, who wants to remain unnamed, inquired about the biography of the young genius, named Al Bert, sparking such amount of activity. The researcher was easily able to find Dr. Bert scheduled for an unusual amount of seminars in locations all over the world, sometimes more than 4 per week. However, upon contacting the respective institutions, nobody could remember the seminars, which according to Prof. Dr. Dr. Hubert at The Advanced Institute is "Not at all unusual." The network researcher from Ann Arbor suspected Dr. Bert to be a fictitious person and notified the university whose email address Dr. Bert was still using.
It turned out Dr. Bert is not a fictitious person. Dr. Bert's graduated in 2006, but his contract at the university run out in 2008. After this, colleagues lost sight of Dr. Bert. He applied for unemployment benefits in October 2008. As the Federal Intelligence Service reported this Wednesday, he later founded an agency called 'High Impact' (the website has since been taken down) that offered to boost a paper's citation count. A user registered with an almost finished, but not yet published, paper and agreed to pay EUR 10 to Dr. Bert's agency for each citation his paper received above the author's average citation count at the time of registration. The user also agreed to cite 5 papers the agency would name. A registered user would earn EUR 10 for each recruitment of a new paper, possibly their own.
This rapidly created a growing network of researchers citing each others papers, and encouraged the authors to produce new papers, certain they would become well cited. Within only a few months, the network had spread from physics to other research fields. With each citation, Dr. Bert made an income. The algorithm he used to assign citations also ensured his own works became top cites. Yet, with many researchers suddenly having papers with several hundred citations above their previously average citation count, their fee went into some thousand dollars. On several instances Dr. Bert would suggest they invite him for a seminar at their institution and locate it in a non-existent room. He would then receive reimbursement for a fraudulent self-printed boarding pass, illegible due to an alleged malfunctioning printer.
Names of researchers subscribed to Dr. Bert's agency were not accessible at the time of writing.
TEXT 2
The Case of IJNSNS
The field of applied mathematics provides an illuminating case in which we can study such impact-factor distortion. For the last several years, the International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS) has dominated the impact-factor charts in the “Mathematics, Applied” category. It took first place in each year 2006, 2007, 2008, and 2009, generally by a wide margin, and came in second in 2005. However, as we shall see, a more careful look indicates that IJNSNS is nowhere near the top of its field. Thus we set out to understand the origin of its large impact factor.
In 2008, the year we shall consider in most detail, IJNSNS had an impact factor of 8.91, easily the highest among the 175 journals in the applied math category in ISI’s Journal Citation Reports (JCR). As controls, we will also look at the two journals in the category with the second and third highest impact factors, Communications on Pure and Applied Mathematics (CPAM) and SIAM SIAM
Evaluation based on expert judgment is the best alternative to citation-based measures for journals. Though not without potential problems of its own, a careful rating by experts is likely to provide a much more accurate and holistic guide journal quality than impact factor or similar metrics. In mathematics, as in many fields, researchers are widely in agreement about which are the best journals in their specialties. The Australian Research Council recently released such an evaluation, listing quality ratings for over 20,000 peer reviewed journals across disciplines. The list was developed through an extensive review process involving learned academies (such as the Australian Academy of Science Australia
• A*: one of the best in its field or subfield
• A*: one of the best in its field or subfield
• A: very high quality
• B: solid, though not outstanding, reputation
• C: does not meet the criteria of the higher tiers.
The ERA list included all but five of the 175 journals assigned a 2008 impact factor by JCR in the category “Mathematics, Applied”. Figure 1 shows the impact factors for journals in each of the four rating tiers. We see that, as a proxy for expert opinion, the impact factor does rather poorly. There are many examples of journals with a higher impact factor than other journals that are one, two, and even three rating tiers higher. The red line is drawn so that 20% of the A* journals are below it; it is notable that 51% of the A journals have an impact factor above that level, as do 23% of the B journals and even 17% of those in the C category. The most extreme outlier is IJNSNS, which, despite its relatively astronomical impact factor, is not in the first or second but, rather, third tier.
The ERA rating assigned its highest score, A*, to 25 journals. Most of the journals with the highest impact factors are here, including CPAM and SIREV, but, of the top 10 journals by impact factor, two were assigned an A, and only IJNSNS was assigned a B. There were 53 A-rated journals and 69 B-rated journals altogether. If IJNSNS were assumed to be the best of the B journals, there would be 78 journals with higher ERA ratings, whereas if it were the worst, its ranking would fall to 147. In short, the ERA ratings suggest that IJNSNS is not only not the top applied math journal but also that its rank should be somewhere in the range 75–150. This remarkable mismatch between reputation and impact factor needs an explanation.
Makings of a High Impact Factor
A first step to understanding IJNSNS’s high impact factor is to look at how many authors contributed substantially to the counted citations and who they were. The top-citing author to IJNSNS in 2008 was the journal’s editor-in-chief, Ji-Huan He, who cited the journal (within the two-year window) 243 times. The second top citer, D. D. Ganji, with 114 cites, is also a member of the editorial board, as is the third, regional editor Mohamed El Naschie, with 58 cites. Together these three account for 29% of the citations counted toward the impact factor.
For comparison, the top three citers to SIREV contributed only 7, 4, and 4 citations, respectively, accounting for less than 12% of the counted citations, nd none of these authors is involved in editing the journal. For CPAM the top three citers (9, 8, and 8) contributed about 7% of the citations and, again, were not on the editorial board.
Another significant phenomenon is the extent to which citations to IJNSNS are concentrated within the two-year window used in the impact factor calculation. Our analysis of 2008 citations to articles published since 2000 shows that 16% of the citations to CPAM fell within that two-year window and only 8% of those to SIREV did; in contrast, 71.5% of the 2008 citations to IJNSNS fell within the two-year window. In Table 1, we show the 2008 impact factors for the three journals, as well as a modified impact factor, which gives the average number of citations in 2008 to articles the journals published not in 2006 and 2007 but in the preceding six years. Since the cited half-life (thetime it takes to generate half of all the eventual citations to an article) for applied mathematics is nearly 10 years,12 this measure is at least as reasonable as the impact factor. It is also independent, unlike JCR’s 5-Year Impact Factor, as its time period does not overlap with that targeted by the Journal.
Table 1. 2008 impact factor with normal 2006–7 window Modified 2008 “impact factor” with 2000–5 window
IJNSNS 8.91 1.27
CPAM 3.69 3.46 S
IREV 2.8 10.4
2008 impact factors computed with the usual two-preceding-years window, and with a window going back eight years but neglecting the two immediately preceding.
Note that the impact factor of JNSNS drops precipitously, by a factor of seven, when we consider a different citation window. By contrast, the impact factor of CPAM stays about the same, and that of SIREV increases markedly. One may simply note that, in distinction to the controls, the citations made to IJNSNS in 2008 greatly favor articles published in precisely the two years that are used to calculate the impact factor. Further striking insights arise when we examine the high-citing journals rather than high-citing authors. The counting of journal self-citations in the impact factor is frequently criticized, and indeed it does come into play in this case. In 2008 IJNSNS supplied 102, or 7%, of its own impact factor citations.
The corresponding numbers are 1 citation (0.8%) for SIREV and 8 citations (2.4%) for CPAM. The disparity in other recent years is similarly large or larger. However, it was Journal of Physics: Conference Series that provided the greatest number of IJNSNS citations. A single issue of that journal provided 294 citations to IJNSNS in the impact factor window, accounting for more than 20% of its impact factor. What was this issue? It was the proceedings of a conference organized by IJNSNS editor-in-chief He at his home university. He was responsible for the peer review of the issue. The second top-citing journal for IJNSNS was Topological Methods in Nonlinear Analysis, which contributed 206 citations (14%), again with all citations coming from a single issue. This was a special issue with Ji-Huan He as the guest editor; his co-editor, Lan Xu, is also on the IJNSNS editorial board. J.-H. He himself contributed a brief article to the special issue, consisting of three pages of text and thirty references. Of these, twenty were citations to IJNSNS within the impact-factor window. The remaining ten consisted of eight citations to He and two to Xu.
Continuing down the list of IJNSNS high-citing journals, another similar circumstance comes to light: 50 citations from a single issue of the Journal of Polymer Engineering (which, like IJNSNS, is published by Freund), guest edited by the same pair, Ji-Huan He and Lan Xu. However, third place is held by the journal Chaos, Solitons and Fractals, with 154 citations spread over numerous issues. These are again citations that may be viewed as subject to editorial influence or control. In 2008 Ji-Huan He served on the editorial board of CS&F, and its editor-in-chief was Mohamed El Naschie, who was also a coeditor of IJNSNS. In a highly publicized case, the entire editorial board of CS&F was recently replaced, but El Naschie remained coeditor of IJNSNS.
Many other citations to IJNSNS came from papers published in journals for which He served as editor, such as Zeitschrift für Naturforschung A, which provided forty citations; there are too many others to list here, since He serves in an editorial capacity on more than twenty journals (and has just been named editor-in-chief of four more journals from the newly formed Asian Academic Publishers). Yet another source of citations came from papers authored by IJNSNS editors other than He, which accounted for many more. All told, the aggregation of such editor-connected citations, which are time-consuming to detect, account for more than 70% of all the citations contributing to the IJNSNS impact factor.
Tuesday, April 12, 2011
QS Engineering Rankings
QS have started to published detailed subject rankings based on citations per paper over five years and their surveys of academics and employers. The first of these is engineering. There are five subfields: Computer Science and Information Systems, Chemical Engineering, Civil and Structural Engineering, Electrical and Electronic Engineering and Mechanical, Aeronautical and Manufacturing.
For Civil and Structural Engineering the weighting is 50% for the academic survey, 30% for the employers' survey and 20 % for citations per paper. For the others it is 40%, 30% and 30%.
MIT, not surprisingly, is top in each of the five engineering fields that are ranked. In general, the upper levels of these rankings seem reasonable. However, a look at the details, especially in the bottom half, 100-200 places, raises some questions.
One basic problem is that as QS make finer distinctions, they have to rely on smaller sets of data. There were 285 respondents to the academic survey for chemical engineering and 394 for civil and structural engineering. For the employer survey there were 836 for computer science. Each respondent to the academic survey was allowed to nominate up to 40 universities but usually the number was much lower than this. Around the 151-200 level the number of responses would surely have been very low. Similarly, the number of papers counted in each field varied considerably from 43,222 in civil and structural engineering to 514,95 in electrical and electronic engineering. We should therefore be rather sceptical about these rankings.
Something that is noticeable is that there is a reasonably high correlation between the scores for the academic survey and the employer survey. For electrical engineering it is .682, chemical engineering .695, civil engineering .695, computer science .722.
But there is no correlation at all between the citations per paper indicator and the surveys. For electrical engineering it is .064 between citations and academic survey and -.004 between citations and the employer survey. It is the same for the other subfields. None of the correlations are statistically significant.
Looking at the top universities for the three indicators, we see the same familiar places in each of the subfields according to the surveys: MIT, Stanford, Cambridge, Berkeley, Oxford, Harvard, Imperial College London, Melbourne, Caltech.
But looking at the top scorers for citations per paper, we find a much more varied and unfamiliar array of institutions: New York University, Wageningen, Dartmouth College, Notre Dame, Aalborg, Athens, Lund, Uppsala, Drexel, Tufts, IIT Roorkee, University of Washington, Rice, University of Massachusetts.
The agreement of employers and academic about the quality of engineering programs, even though they refer to different aspects, research and graduate employability, suggests that the surveys are moderately accurate, at least for the top hundred or so.
However, the lack of any correlation at all between the citations indicator and the surveys needs to be raised. It could be that citations have identified up and coming superstars. Perhaps the number of papers is so low in the various subfields that the indicator does not mean very much. Perhaps citations have been so manipulated in recent years -- see the case of Alexandria University -- that they are no longer a robust indicator of quality.
QS have started to published detailed subject rankings based on citations per paper over five years and their surveys of academics and employers. The first of these is engineering. There are five subfields: Computer Science and Information Systems, Chemical Engineering, Civil and Structural Engineering, Electrical and Electronic Engineering and Mechanical, Aeronautical and Manufacturing.
For Civil and Structural Engineering the weighting is 50% for the academic survey, 30% for the employers' survey and 20 % for citations per paper. For the others it is 40%, 30% and 30%.
MIT, not surprisingly, is top in each of the five engineering fields that are ranked. In general, the upper levels of these rankings seem reasonable. However, a look at the details, especially in the bottom half, 100-200 places, raises some questions.
One basic problem is that as QS make finer distinctions, they have to rely on smaller sets of data. There were 285 respondents to the academic survey for chemical engineering and 394 for civil and structural engineering. For the employer survey there were 836 for computer science. Each respondent to the academic survey was allowed to nominate up to 40 universities but usually the number was much lower than this. Around the 151-200 level the number of responses would surely have been very low. Similarly, the number of papers counted in each field varied considerably from 43,222 in civil and structural engineering to 514,95 in electrical and electronic engineering. We should therefore be rather sceptical about these rankings.
Something that is noticeable is that there is a reasonably high correlation between the scores for the academic survey and the employer survey. For electrical engineering it is .682, chemical engineering .695, civil engineering .695, computer science .722.
But there is no correlation at all between the citations per paper indicator and the surveys. For electrical engineering it is .064 between citations and academic survey and -.004 between citations and the employer survey. It is the same for the other subfields. None of the correlations are statistically significant.
Looking at the top universities for the three indicators, we see the same familiar places in each of the subfields according to the surveys: MIT, Stanford, Cambridge, Berkeley, Oxford, Harvard, Imperial College London, Melbourne, Caltech.
But looking at the top scorers for citations per paper, we find a much more varied and unfamiliar array of institutions: New York University, Wageningen, Dartmouth College, Notre Dame, Aalborg, Athens, Lund, Uppsala, Drexel, Tufts, IIT Roorkee, University of Washington, Rice, University of Massachusetts.
The agreement of employers and academic about the quality of engineering programs, even though they refer to different aspects, research and graduate employability, suggests that the surveys are moderately accurate, at least for the top hundred or so.
However, the lack of any correlation at all between the citations indicator and the surveys needs to be raised. It could be that citations have identified up and coming superstars. Perhaps the number of papers is so low in the various subfields that the indicator does not mean very much. Perhaps citations have been so manipulated in recent years -- see the case of Alexandria University -- that they are no longer a robust indicator of quality.
Monday, April 04, 2011
First They Came For the Fire Buffs, Then They Came for the Science Nerds....
An interesting aspect of the 2009 court case brought by firemen in New Haven, Connecticut, was the not very subtle disdain shown by members of the American academic elite towards the pretensions of those who thought that fire fighting required a degree of knowledge and intelligence. The aggrieved firemen had been denied promotion because the test resulted in white firemen doing better than their African American and Hispanic colleagues. See here for an insightful account.
An article by Nicole Allan and Emily Bazelon, graduate of Yale Law School, granddaughter of a judge of the US Court of Appeals, former law clerk for a judge of the US Court of Appeals and Senior Research Scholar in Law and Truman Capote Fellow for Creative Writing and Law at Yale Law School, reported without noticeable irony complaints about the unfairness of the test that passed too many white firefighters: it favored "fire buffs" (enthusiasts according to the dictionary) who read firefighting manuals in their spare time or who came from families with lots of firefighters.
One wonders whether Bazelon ever wondered whether she had derived an unfair advantage from her family background or felt guilty because she had had read books about law when she did not have to.
The article concluded:
That has in fact been done in Chicago.
Meanwhile, admission into North American graduate and professional schools has followed the cities of the US by becoming increasingly less selective as intelligence and knowledge are downgraded and admission is increasingly dependent on vague and unverifiable personality traits.
Educational Testing Service, producers of the Graduate Record Exam, have now introduced a Personal Potential Index that will supplement (and perhaps eventually replace?) the use of the GRE and undergraduate grades for admissions to graduate school.
Basically, this new tool involves applicants nominating five evaluators who will provide assessments of "Knowledge and Creativity; Communication Skills; Teamwork; Resilience; Planning and Organization; and Ethics and Integrity". I can see Newton, Darwin, Einstein and James Watson all tripping up on at least one of these.
The latest development is that the Medical College Admissions Test will do away with its writing test because it does not add much information beyond undergraduate grades. It will be replaced with a section on "behavioural and social sciences principles."
It seems that the point of this is to increase the number of minorities in medical schools although it is not clear why it is assumed that they will do better in answering questions about the social sciences and critical thinking than in writing an essay and verbal analysis.
More changes may be coming soon. Already there are pilot projects in which schools are "doing brief interviews of applicants involving various ethical and social scenarios to learn more about would-be students".
It seems that these developments are a response to criticism of the MCAT from organisations like The National Center for Fair and Open Testing:
According to Wikipedia, "Nerd is a term that refers to a social perception of a person who avidly pursues intellectual activities, technical or scientific endeavors, esoteric knowledge, or other obscure interests, rather than engaging in more social or conventional activities."
Will the time come when the likes of Emily Bazelon are denied promotion or appointment because of their inappropriate buffiness or avid pursuit of intellectual activity? Not to worry. They could probably get jobs as firefighters somewhere.
Here is a prediction. As American universities increasingly select students and faculty because they are communicative, culturally sensitive, resilient and and so on while cleansing themselves of all those buffs and nerds, China, Korea and a few other countries will catch up and then overtake them first in scientific output and then in quality.
An interesting aspect of the 2009 court case brought by firemen in New Haven, Connecticut, was the not very subtle disdain shown by members of the American academic elite towards the pretensions of those who thought that fire fighting required a degree of knowledge and intelligence. The aggrieved firemen had been denied promotion because the test resulted in white firemen doing better than their African American and Hispanic colleagues. See here for an insightful account.
An article by Nicole Allan and Emily Bazelon, graduate of Yale Law School, granddaughter of a judge of the US Court of Appeals, former law clerk for a judge of the US Court of Appeals and Senior Research Scholar in Law and Truman Capote Fellow for Creative Writing and Law at Yale Law School, reported without noticeable irony complaints about the unfairness of the test that passed too many white firefighters: it favored "fire buffs" (enthusiasts according to the dictionary) who read firefighting manuals in their spare time or who came from families with lots of firefighters.
One wonders whether Bazelon ever wondered whether she had derived an unfair advantage from her family background or felt guilty because she had had read books about law when she did not have to.
The article concluded:
If New Haven could start over, maybe it could also admit outright that it has more deserving firefighters than it has rewards. The city could come up with a measure for who is qualified for the promotions, rather than who is somehow best. And then it could choose from that pool by lottery. That might not exactly be fair, either. But it would recognize that sometimes there may be no such thing.
That has in fact been done in Chicago.
Meanwhile, admission into North American graduate and professional schools has followed the cities of the US by becoming increasingly less selective as intelligence and knowledge are downgraded and admission is increasingly dependent on vague and unverifiable personality traits.
Educational Testing Service, producers of the Graduate Record Exam, have now introduced a Personal Potential Index that will supplement (and perhaps eventually replace?) the use of the GRE and undergraduate grades for admissions to graduate school.
Basically, this new tool involves applicants nominating five evaluators who will provide assessments of "Knowledge and Creativity; Communication Skills; Teamwork; Resilience; Planning and Organization; and Ethics and Integrity". I can see Newton, Darwin, Einstein and James Watson all tripping up on at least one of these.
The latest development is that the Medical College Admissions Test will do away with its writing test because it does not add much information beyond undergraduate grades. It will be replaced with a section on "behavioural and social sciences principles."
It seems that the point of this is to increase the number of minorities in medical schools although it is not clear why it is assumed that they will do better in answering questions about the social sciences and critical thinking than in writing an essay and verbal analysis.
More changes may be coming soon. Already there are pilot projects in which schools are "doing brief interviews of applicants involving various ethical and social scenarios to learn more about would-be students".
It seems that these developments are a response to criticism of the MCAT from organisations like The National Center for Fair and Open Testing:
Robert Schaeffer, public education director of the center, said that the MCAT has been viewed as encouraging "memorization and regurgitation" and is "better at identifying science nerds than candidates who would become capable physicians well-equipped to serve their patients." The changes being proposed appear to be "responding directly" to these critiques, he said.
According to Wikipedia, "Nerd is a term that refers to a social perception of a person who avidly pursues intellectual activities, technical or scientific endeavors, esoteric knowledge, or other obscure interests, rather than engaging in more social or conventional activities."
Will the time come when the likes of Emily Bazelon are denied promotion or appointment because of their inappropriate buffiness or avid pursuit of intellectual activity? Not to worry. They could probably get jobs as firefighters somewhere.
Here is a prediction. As American universities increasingly select students and faculty because they are communicative, culturally sensitive, resilient and and so on while cleansing themselves of all those buffs and nerds, China, Korea and a few other countries will catch up and then overtake them first in scientific output and then in quality.
Sunday, April 03, 2011
The View from Hong Kong
University World News has an article by Kevin Downing of the City University of Hong Kong. It begins:
University World News has an article by Kevin Downing of the City University of Hong Kong. It begins:
Are Asian institutions finally coming out of the shadow cast by their Western counterparts? At the 2010 World Universities Forum in Davos, a theme was China's increasing public investment in higher education at a time when reductions in public funding are being seen in Europe and North America. China is not alone in Asia in increasing public investment in higher education, with similar structured and significant investment evident in Singapore, South Korea and Taiwan.
While in many ways this investment is not at all surprising and merely reflects the continued rise of Asia as a centre of global economic power, it nonetheless raises some interesting questions in relation to the potential benefits of rankings for Asian institutions.
Interest in rankings in Asian higher education is undoubtedly high and the introduction of the QS Asian University Rankings in 2009 served to reinforce this. The publication of ranking lists is now greeted with a mixture of trepidation and relief by many university presidents and is often followed by intense questioning from media that are interested to know what lies behind a particular rise or fall on the global or regional stage.
Friday, April 01, 2011
Best Grad Schools
The US News Graduate School Rankings were published on March 15th. Here are the top universities in various subject areas.
Business: Stanford
Education: Vanderbilt
Engineering: MIT
Law: Yale
Medical: Harvard
Biology: Stanford
Chemistry: Caltech, MIT, UC Berkeley
Computer Science; Carnegie-Mellon, MIT, Stanford, UC Berkeley
Earth Sciences: Caltech, MIT
Mathematics: MIT
Physics: Caltech, Harvard, MIT, Stanford
Statistics: Stanford
Library and Information Studies: Illinois at Urbana-Champagne
Criminology: Maryland -- College Park
Economics: Harvard, MIT, Princeton, Chicago
English: UC Berkeley
History: Princeton
Political Science: Harvard, Princeton, Stanford
Psychology: Stanford, UC Berkeley
Sociology: UC Berkeley
Public Affairs: Syracuse
Fine Arts: Rhode Island School of Design
The US News Graduate School Rankings were published on March 15th. Here are the top universities in various subject areas.
Business: Stanford
Education: Vanderbilt
Engineering: MIT
Law: Yale
Medical: Harvard
Biology: Stanford
Chemistry: Caltech, MIT, UC Berkeley
Computer Science; Carnegie-Mellon, MIT, Stanford, UC Berkeley
Earth Sciences: Caltech, MIT
Mathematics: MIT
Physics: Caltech, Harvard, MIT, Stanford
Statistics: Stanford
Library and Information Studies: Illinois at Urbana-Champagne
Criminology: Maryland -- College Park
Economics: Harvard, MIT, Princeton, Chicago
English: UC Berkeley
History: Princeton
Political Science: Harvard, Princeton, Stanford
Psychology: Stanford, UC Berkeley
Sociology: UC Berkeley
Public Affairs: Syracuse
Fine Arts: Rhode Island School of Design
Sunday, March 27, 2011
Say It Loud
Phil Baty has an article in THE, based on a speech in Hong Kong entitled 'Say it loud: I'm a ranker and I'm proud'. Very interesting but personally I prefer the James Brown version.
Phil Baty has an article in THE, based on a speech in Hong Kong entitled 'Say it loud: I'm a ranker and I'm proud'. Very interesting but personally I prefer the James Brown version.
Saturday, March 26, 2011
Growth of Academic Publications: Southwest Asia, 2009-2010
One of several surprises in last year's THE rankings was the absence of any Israeli university from the Top 200. QS had three and, as noted earlier on this blog, over a fifth of Israeli universities were in the Shanghai 500, a higher proportion than any other country. It seems that in the case of at least two universities, Tel Aviv and the Hebrew University of Jerusalem, there was a failure of communication that meant that data was not submitted to Thomson Reuters, who collect and analyse data for THE.
The high quality of Israeli universities might seem rather surprising since Israeli secondary school students perform poorly on international tests of scholastic attainment and the national average IQ is mediocre. It could be that part of the reason for the strong Israeli academic performance is the Psychometric Entrance Test for university admission that measures quantitative and verbal reasoning and also includes an English test. The contrast with the trend in the US, Europe and elsewhere towards holistic assessment, credit for leadership, community involvement, overcoming adversity and being from the right post code area is striking.
Even so, Israeli scientific supremacy in the Middle East is looking precarious. Already the annual production of academic papers in Israel has been exceeded by Iran and Turkey.
Meanwhile, the total number of papers produced in Israel is shrinking while that of Iran and Turkey continues to grow at a respectable rate. The fastest growth in Southwest Asia comes from Saudi Arabia and the smaller Gulf states of Qatar and Bahrain.
Countries ranked by percentage increase in publications in the ISI Science, Social Science and Arts and Humanities indexes and Conference Proceedings between 2009 and 2010. (total 2010 publications in brackets)
1. Saudi Arabia 35% (3924)
2. Qatar 31% (453)
3. Syria 14% (333)
4. Bahrain 13% (184)
5. Palestine 9% (24)
6. UAE 6% (303)
7. Turkey 5% (26835)
8. Lebanon 4% (2058)
9. Iran 4% (21047)
10. Oman 4% (494)
11. Jordan 1% (1637)
12. Iraq -3% (333)
13. Israel -4% (17719)
14. Yemen -8% (125)
15. Kuwait -13% (759)
(data collected 23/3/11)
This of course may not say very much about the quality of research. A glance at the ISI list of highly cited researchers shows that Israel is ahead, for the moment at least, with 50 compared to 29 for Saudi Arabia and one each for Turkey and Iran.
One of several surprises in last year's THE rankings was the absence of any Israeli university from the Top 200. QS had three and, as noted earlier on this blog, over a fifth of Israeli universities were in the Shanghai 500, a higher proportion than any other country. It seems that in the case of at least two universities, Tel Aviv and the Hebrew University of Jerusalem, there was a failure of communication that meant that data was not submitted to Thomson Reuters, who collect and analyse data for THE.
The high quality of Israeli universities might seem rather surprising since Israeli secondary school students perform poorly on international tests of scholastic attainment and the national average IQ is mediocre. It could be that part of the reason for the strong Israeli academic performance is the Psychometric Entrance Test for university admission that measures quantitative and verbal reasoning and also includes an English test. The contrast with the trend in the US, Europe and elsewhere towards holistic assessment, credit for leadership, community involvement, overcoming adversity and being from the right post code area is striking.
Even so, Israeli scientific supremacy in the Middle East is looking precarious. Already the annual production of academic papers in Israel has been exceeded by Iran and Turkey.
Meanwhile, the total number of papers produced in Israel is shrinking while that of Iran and Turkey continues to grow at a respectable rate. The fastest growth in Southwest Asia comes from Saudi Arabia and the smaller Gulf states of Qatar and Bahrain.
Countries ranked by percentage increase in publications in the ISI Science, Social Science and Arts and Humanities indexes and Conference Proceedings between 2009 and 2010. (total 2010 publications in brackets)
1. Saudi Arabia 35% (3924)
2. Qatar 31% (453)
3. Syria 14% (333)
4. Bahrain 13% (184)
5. Palestine 9% (24)
6. UAE 6% (303)
7. Turkey 5% (26835)
8. Lebanon 4% (2058)
9. Iran 4% (21047)
10. Oman 4% (494)
11. Jordan 1% (1637)
12. Iraq -3% (333)
13. Israel -4% (17719)
14. Yemen -8% (125)
15. Kuwait -13% (759)
(data collected 23/3/11)
This of course may not say very much about the quality of research. A glance at the ISI list of highly cited researchers shows that Israel is ahead, for the moment at least, with 50 compared to 29 for Saudi Arabia and one each for Turkey and Iran.
Thursday, March 24, 2011
Growth in Academic Publications: Southeast Asia 2009-2010
Countries ranked by percentage increase in publications in the ISI Science, Social Science and Arts and Humanities indexes and Conference Proceedings between 2009 and 2010. (total 2010 publications in brackets)
1. Malaysia 31% (8603)
2. Laos 30% (96)
3. Indonesia 30% (1631)
4. Brunei 16% (88)
5. Papua New Guinea 5% (67)
6. Vietnam 5% (1247)
7. Singapore 4% (11900)
8. Thailand 2% (2248)
9. Timor Leste 0% (4)
10. Cambodia -5% (158)
11. Myanmar -12% (78)
Singapore is still the dominant research power in Southeast Asia but Malaysia and Indonesia, admittedly with much larger populations, are closing fast. Thailand is growing very slowly and Myanmar is shrinking.
(data collected 23/3/11)
Countries ranked by percentage increase in publications in the ISI Science, Social Science and Arts and Humanities indexes and Conference Proceedings between 2009 and 2010. (total 2010 publications in brackets)
1. Malaysia 31% (8603)
2. Laos 30% (96)
3. Indonesia 30% (1631)
4. Brunei 16% (88)
5. Papua New Guinea 5% (67)
6. Vietnam 5% (1247)
7. Singapore 4% (11900)
8. Thailand 2% (2248)
9. Timor Leste 0% (4)
10. Cambodia -5% (158)
11. Myanmar -12% (78)
Singapore is still the dominant research power in Southeast Asia but Malaysia and Indonesia, admittedly with much larger populations, are closing fast. Thailand is growing very slowly and Myanmar is shrinking.
(data collected 23/3/11)
Tuesday, March 22, 2011
Comparing Rankings 3: Omissions
The big problem with the Asiaweek rankings of 1999-2000 was that they relied on data submitted by universities. This meant that if enough were dissatisfied they could effectively sabotage the rankings by withholding information, which is in fact what happened.
The THES-QS rankings, and since 2010 the QS rankings, avoided this problem by ranking universities whether they liked or not. Nonetheless, there were a few omissions in the early years: Lancaster, Essex, Royal Holloway University of London and the SUNY campuses at Binghamton, Buffalo and Albany.
In 2010 THE decided that they would not rank universities that did not submit data, a principled decision but one that has its dangers. Too many conscientious objectors (or maybe poor losers) and the rankings would begin to lose face validity.
When the THE rankings came out last year, there were some noticeable absentees, among them the Chinese University of Hong Kong, the University of Queensland, Tel Aviv University, the Hebrew University of Jerusalem, the University of Texas at Austin, the Catholic University of Louvain, Fudan University, Rochester, Calgary, the Indian Institutes of Technology and Science and Sciences Po Paris.
As Danny Byrne pointed out in University World News, Texas at Austin and Moscow State University were in the top100 in the Reputation Rankings but not in the THE World University Rankings. Producing a reputation-only ranking without input from the institutions could be a smart move for THE.
Monday, March 21, 2011
QS comments on the THE Reputation Ranking
In University World News, Danny Byrne from QS comments on the new THE reputation ranking.
In University World News, Danny Byrne from QS comments on the new THE reputation ranking.
So why has THE decided to launch a world ranking based entirely on institutional reputation? Is it for the benefit of institutions like Moscow State University, which did not appear in THE's original top 200 but now appears 33rd in the world?
The data on which the new reputational ranking is based has been available for six months and comprised 34.5% of the world university rankings published by THE in September 2010.
But this is the first time the magazine has allowed anyone to view this data in isolation. Allowing users to access the data six months ago may have attracted less attention, but it would perhaps have been less confusing for prospective students.
The order of the universities in the reputational rankings differs from the THE's overall ranking. But no new insights have been offered and nothing has changed. This plays into the hands of those who are sceptical about university rankings.
Wednesday, March 16, 2011
Worth Reading
Ellen Hazelkorn, 'Questions Abound as the College-Rankings Race Goes Global' in Chronicle of Higher Education
"It is amazing that more than two decades after U.S. News & World Report first published its special issue on "America's Best Colleges," and almost a decade since Shanghai Jiao Tong University first published the Academic Ranking of World Universities, rankings continue to dominate the attention of university leaders. Indeed, the range of people watching them now includes politicians, students, parents, businesses, and donors. Simply put, rankings have caught the imagination of the public and have insinuated their way into public discourse and almost every level of government. There are even iPhone applications to help individuals and colleges calculate their ranks.
More than 50 country-specific rankings and 10 global rankings are available today, including the European Union's new U-Multirank, due this year. What started as small-scale, nationally focused guides for students and parents has become a global business that heavily influences higher education and has repercussions well beyond academe."
Tuesday, March 15, 2011
Bright Ideas Department
This is from today's Guardian:
This is from today's Guardian:
The coalition is considering a Soviet-style central intervention policy to effectively fine individual universities if they impose unreasonable tuition fees next year.Next bright idea? A Gulag for recalcitrant vice-chancellors? Re-education camps for those who don't take their teaching philosophy statements seriously enough?
Vince Cable, the business secretary whose department is responsible for universities, and David Willetts, the universities minister, are looking at allowing colleges that charge a modest fee to expand and constraining those that are charging too much.
The government, through the Higher Education Funding Council, sets the grant and numbers for each university and has the power to fine a university as much as £3,000 per student if it over-recruits in a single year.
Ministers are looking at cutting funding from universities that unreasonably charge the maximum £9,000 fee from 2012-13. They admit it is likely most universities will charge well over £8,000 a year.
One minister said: "A form of dramatic centralisation is under active consideration - a form of Gosplan if you like," a reference to the Russian state planning committee set up in the 1920s.
Saturday, March 12, 2011
Going Global Hong Kong 2011
Speeches about rankings by Martin Davidson, British Council, Phil Baty, THE, John Molony, QS, and others can be seen here.
Speeches about rankings by Martin Davidson, British Council, Phil Baty, THE, John Molony, QS, and others can be seen here.
Thursday, March 10, 2011
A Bit More on the THE Reputation Rankings
There is a brief article in the Guardian with a lot of comments.
Incidentally, I don't see Alexandria, Hong Kong Baptist and Bilkent Universities in the top 100 for reputation despite the outstanding work that gave them high scores for research impact in the 2010 THE WUR. Perhaps I'm not looking hard enough.
There is a brief article in the Guardian with a lot of comments.
Incidentally, I don't see Alexandria, Hong Kong Baptist and Bilkent Universities in the top 100 for reputation despite the outstanding work that gave them high scores for research impact in the 2010 THE WUR. Perhaps I'm not looking hard enough.
The THE Reputation Rankings
Times Higher Education have constructed a reputation ranking from the data collected for last year's World University Rankings. There is a weighting of two thirds for research and one third for postgraduate teaching. The top five are:
1. Harvard
2. MIT
3. Cambridge
4. UC Berkeley
5. Stanford
Scores are given only for the top fifty universities. Then another fifty are sorted in bands of ten without scores. Evidently, the number of responses favouring universities outside the top 100 was so small that it was not worth listing.
This means that the THE reputational survey reveals significant differences between Harvard and MIT or between Cambridge and Oxford but it would be of no help to those wondering whether to study or work at the University of Cape Town or the University of Kwazulu-Natal or Trinity College Dublin or University College Dublin.
The scores for research reputation (top fifty for total reputation scores only) show a moderate correlation with the THE citations indicator (.422) and, perhaps surprisingly, a higher correlation with the citations per faculty score on the QS World University Rankings of 2010 (.538).
looking at the QS academic survey, which asked only about research, we can see that there was an insignificant correlation of .213 between the QS scores and the score for citations per faculty in the QS rankings (THE reputation ranking top 50 only). However, there was a higher correlation between the QS survey and the THE citations indicator of .422, the same as that between the THE research reputation scores and the the THE citations indicator.
Comparing the two research surveys with a third party, the citations indicator in the Scimago 2010 rankings, the THE research reputation survey did better with a correlation of .438 compared to an insignificant .188 for the QS academic survey.
This seems to suggest that the THE reputational survey does a better job at differentiating between the world's elite universities. But once we leave the top 100 it is perhaps less helpful and there may still be a role for the QS rankings
Times Higher Education have constructed a reputation ranking from the data collected for last year's World University Rankings. There is a weighting of two thirds for research and one third for postgraduate teaching. The top five are:
1. Harvard
2. MIT
3. Cambridge
4. UC Berkeley
5. Stanford
Scores are given only for the top fifty universities. Then another fifty are sorted in bands of ten without scores. Evidently, the number of responses favouring universities outside the top 100 was so small that it was not worth listing.
This means that the THE reputational survey reveals significant differences between Harvard and MIT or between Cambridge and Oxford but it would be of no help to those wondering whether to study or work at the University of Cape Town or the University of Kwazulu-Natal or Trinity College Dublin or University College Dublin.
The scores for research reputation (top fifty for total reputation scores only) show a moderate correlation with the THE citations indicator (.422) and, perhaps surprisingly, a higher correlation with the citations per faculty score on the QS World University Rankings of 2010 (.538).
looking at the QS academic survey, which asked only about research, we can see that there was an insignificant correlation of .213 between the QS scores and the score for citations per faculty in the QS rankings (THE reputation ranking top 50 only). However, there was a higher correlation between the QS survey and the THE citations indicator of .422, the same as that between the THE research reputation scores and the the THE citations indicator.
Comparing the two research surveys with a third party, the citations indicator in the Scimago 2010 rankings, the THE research reputation survey did better with a correlation of .438 compared to an insignificant .188 for the QS academic survey.
This seems to suggest that the THE reputational survey does a better job at differentiating between the world's elite universities. But once we leave the top 100 it is perhaps less helpful and there may still be a role for the QS rankings
Wednesday, March 09, 2011
The Second Wave
It seems that another wave of rankings is coming. The new edition of America's best graduate schools will be out soon, QS will be releasing detailed subject rankings and, according to Bloomberg Businessweek, THE's ranking by reputation is imminent. It seems that the academic anglosphere dominates when reputation alone is considered.
It seems that another wave of rankings is coming. The new edition of America's best graduate schools will be out soon, QS will be releasing detailed subject rankings and, according to Bloomberg Businessweek, THE's ranking by reputation is imminent. It seems that the academic anglosphere dominates when reputation alone is considered.
Tuesday, March 08, 2011
Comment on the Paris Ranking
Ben Wildavsky in the Chronicle of Higher Education says:
The Mines ParisTech ranking is an explicitly chauvinistic exercise, born of French unhappiness with the dismal showing of its universities in influential surveys such as the Academic Ranking of World Universities created at Shanghai Jiao Tong University in 2003. When designing the Mines ParisTech ranking, with a view to influencing the architects of the Shanghai methodology, the college says in the FAQ section of its survey results, “we believed it was useful to highlight the good results of French institutions at a time when the Shanghai ranking was widely and is still widely discussed, and not always to the advantage of our own schools and universities.” What’s more, it goes on, “these results constitute a genuine communication tool at an international level, both for the recruitment of foreign students as well as among foreign companies which are not always very familiar with our education system.” Given the genesis of the ranking, it doesn’t seem too surprising that three French institutions made it into this year’s top 10 — École Polytechnique and École Nationale d’Administration joined HEC Paris — while Mines ParisTech itself placed 21st in the world.
Sunday, March 06, 2011
The Paris rankings
The fifth edition of the Professional Ranking of World Universities from Paris Mines Tech has just been published. This is based on one indicator, the number of top corporate CEOs. Here are the top ten:
1. Harvard
2. Tokyo
3. Keio
4. HEC, France
5. Kyoto
5. Oxford
7. Ecole Polytechnique
8. Waseda
9. ENA
10. Seoul National University
The fifth edition of the Professional Ranking of World Universities from Paris Mines Tech has just been published. This is based on one indicator, the number of top corporate CEOs. Here are the top ten:
1. Harvard
2. Tokyo
3. Keio
4. HEC, France
5. Kyoto
5. Oxford
7. Ecole Polytechnique
8. Waseda
9. ENA
10. Seoul National University
Saturday, February 26, 2011
Surveys and Citations
I have just finished calculating the correlation between the scores for the academic survey and citations per faculty on the 2010 QS world University rankings.
Since the survey asked about research and since citations are supposed to be a robust indicator of research excellence we would expect a high correlation between the two.
It is in fact .391, which is on the low side. There could be valid reasons why it is so low. Citations, by definition, must follow publication which follows research which in turn is preceded by proposals and a variety of bureaucratic procedures. A flurry of citations might be indicative of the quality of research begun a decade ago. The responses to the survey might, on the other hand, be based on the first signs of research excellence long before the citations start rolling in.
Still, the correlation does not seem high enough. At first glance one would suspect that the survey is faulty but it could be that citations do not mean very much any more as a measure of excellence.
It would be very interesting to calculate the correlation between the score for research reputation on the Times Higher Education WUR and its citation indicator.
We would expect the THE survey to be more valid since the basic qualification for being included in the survey is being the corresponding author of an article included in the ISI indexes whereas for QS it is signing up for a journal published by World Scientific. But it can no longer be assumed that authorship of any article means very much . Does it always require more initiative and interest to get on the list of co-authors than to sign up for an online subscription?
It should also be noted that there is an overlap between the two surveys as both are supplemented with arts and humanities respondents from the Mardev mailing lists.
I have calculated the correlation between the citations indicator (normalised average citations per paper) in the THE 2010 rankings and the research indicator -- volume ( 4.5 % of the total score) income (6%) and reputation (19.5%)
This is .562, quite a bit better than the QS correlation . However, the research indicator combines a survey with other data.
It would be very interesting if THE and/or Thomson Reuters released the scores of the individual components of the research indicator.
I have just finished calculating the correlation between the scores for the academic survey and citations per faculty on the 2010 QS world University rankings.
Since the survey asked about research and since citations are supposed to be a robust indicator of research excellence we would expect a high correlation between the two.
It is in fact .391, which is on the low side. There could be valid reasons why it is so low. Citations, by definition, must follow publication which follows research which in turn is preceded by proposals and a variety of bureaucratic procedures. A flurry of citations might be indicative of the quality of research begun a decade ago. The responses to the survey might, on the other hand, be based on the first signs of research excellence long before the citations start rolling in.
Still, the correlation does not seem high enough. At first glance one would suspect that the survey is faulty but it could be that citations do not mean very much any more as a measure of excellence.
It would be very interesting to calculate the correlation between the score for research reputation on the Times Higher Education WUR and its citation indicator.
We would expect the THE survey to be more valid since the basic qualification for being included in the survey is being the corresponding author of an article included in the ISI indexes whereas for QS it is signing up for a journal published by World Scientific. But it can no longer be assumed that authorship of any article means very much . Does it always require more initiative and interest to get on the list of co-authors than to sign up for an online subscription?
It should also be noted that there is an overlap between the two surveys as both are supplemented with arts and humanities respondents from the Mardev mailing lists.
I have calculated the correlation between the citations indicator (normalised average citations per paper) in the THE 2010 rankings and the research indicator -- volume ( 4.5 % of the total score) income (6%) and reputation (19.5%)
This is .562, quite a bit better than the QS correlation . However, the research indicator combines a survey with other data.
It would be very interesting if THE and/or Thomson Reuters released the scores of the individual components of the research indicator.
Wednesday, February 23, 2011
Reputation, reputation, reputation!
As the world (or some of it) waits for the ranking survey forms to appear in its mail boxes, both THE and QS are promoting their surveys.
According to Phil Baty of THE:
"But in our consultation with the sector, there was strong support for the continued use of reputation information in the world rankings. Some 79 per cent of respondents to a survey by our rankings data provider Thomson Reuters rated reputation as a “must have” or “nice to have” measure. We operate in a global market where reputation clearly matters."
He then indicates several ways in which the THE survey is an improvement over the THE-QS, now QS, survey.
"We received a record 13,388 usable responses in just three months, making the survey the biggest of its kind in the world.
We promised a transparent approach. The methodology and survey instrument were published in full and this week, the thousands of academics who took part in the survey were sent a detailed report on the respondent profile. It makes reassuring reading:
• Responses were received from 131 countries"
It would, however, be interesting if the number of respondents from all countries were indicated. There are some people who wonder whether THE's sampling technique means that Singapore got the lion's share of responses in Southeast Asia.
Also, will THE publish the scores for the reputation surveys? At the moment they are bundled in with the other teaching and research indicators. What is the correlation between the score for research reputation and the citations indicator? Is there any sign that Alexandria, Bilkant or Hong Kong Baptist University have reputations that match their scores for research impact?
Meanwhile QS also has an item on its survey. They find that there is a similar demand for data on reputations.
"An impressive 79% of respondents, voted reputation for research as one of their top three criteria, with 60% choosing international profile of faculty, essentially another indicator of international reputation for research. This is in stark contrast to the 26% and 30% that prioiritised citations as a key measure.
Furthermore, when breaking these results out by broad faculty area, we can see consistent support across disciplines for the reputation measure but a marked dip in support for citations as a measure amongst respondents in the Arts & Humanities area – which tends to be the area least recognized by traditional measures of research output."
As the world (or some of it) waits for the ranking survey forms to appear in its mail boxes, both THE and QS are promoting their surveys.
According to Phil Baty of THE:
"But in our consultation with the sector, there was strong support for the continued use of reputation information in the world rankings. Some 79 per cent of respondents to a survey by our rankings data provider Thomson Reuters rated reputation as a “must have” or “nice to have” measure. We operate in a global market where reputation clearly matters."
He then indicates several ways in which the THE survey is an improvement over the THE-QS, now QS, survey.
"We received a record 13,388 usable responses in just three months, making the survey the biggest of its kind in the world.
We promised a transparent approach. The methodology and survey instrument were published in full and this week, the thousands of academics who took part in the survey were sent a detailed report on the respondent profile. It makes reassuring reading:
• Responses were received from 131 countries"
It would, however, be interesting if the number of respondents from all countries were indicated. There are some people who wonder whether THE's sampling technique means that Singapore got the lion's share of responses in Southeast Asia.
Also, will THE publish the scores for the reputation surveys? At the moment they are bundled in with the other teaching and research indicators. What is the correlation between the score for research reputation and the citations indicator? Is there any sign that Alexandria, Bilkant or Hong Kong Baptist University have reputations that match their scores for research impact?
Meanwhile QS also has an item on its survey. They find that there is a similar demand for data on reputations.
"An impressive 79% of respondents, voted reputation for research as one of their top three criteria, with 60% choosing international profile of faculty, essentially another indicator of international reputation for research. This is in stark contrast to the 26% and 30% that prioiritised citations as a key measure.
Furthermore, when breaking these results out by broad faculty area, we can see consistent support across disciplines for the reputation measure but a marked dip in support for citations as a measure amongst respondents in the Arts & Humanities area – which tends to be the area least recognized by traditional measures of research output."
Comment on Internationalisation
International Focus, the newsletter of UK HE international Unit, has an article by Jane Knight on myths of internationalisation. The second myth is:
"Myth two rests on a belief that the more international a university is – in terms of students, faculty, curriculum, research, agreements, network memberships – the better its reputation is.
This is tied to the false notion that a strong international reputation is a proxy for quality. Cases of questionable admission and exit standards for universities highly dependent on the revenue and ‘brand equity’ of international students are concrete evidence that internationalisation does not always translate into improved quality or high standards.
This myth is further complicated by the quest for higher rankings on a global or regional league table such as the Times Higher Education or Academic World Ranking of Universities (AWRU). It is highly questionable whether the league tables accurately measure the internationality of a university and more importantly whether the international dimension is always a robust indicator of quality."
Also, it is much easier to be international in Switzerland or Singapore than in Central China or the Midwest of the US.
International Focus, the newsletter of UK HE international Unit, has an article by Jane Knight on myths of internationalisation. The second myth is:
"Myth two rests on a belief that the more international a university is – in terms of students, faculty, curriculum, research, agreements, network memberships – the better its reputation is.
This is tied to the false notion that a strong international reputation is a proxy for quality. Cases of questionable admission and exit standards for universities highly dependent on the revenue and ‘brand equity’ of international students are concrete evidence that internationalisation does not always translate into improved quality or high standards.
This myth is further complicated by the quest for higher rankings on a global or regional league table such as the Times Higher Education or Academic World Ranking of Universities (AWRU). It is highly questionable whether the league tables accurately measure the internationality of a university and more importantly whether the international dimension is always a robust indicator of quality."
Also, it is much easier to be international in Switzerland or Singapore than in Central China or the Midwest of the US.
Subscribe to:
Posts (Atom)