Recently, there has been a lot of finger wagging about King Abdulaziz University (KAU), Jeddah, signing up highly cited researchers as secondary affiliations. The idea behind this was to climb up the ladder of the Shanghai rankings, the Academic Ranking of World Universities. These rankings include an indicator, based on Thomson Reuters' (TR) lists of highly cited researchers, which until now gave universities credit for those researchers who list them as secondary affiliation.
The Shanghai Ranking Consultancy decided that this year they would count secondary affiliations in the old but not the new list "at the suggestion of many institutions and researchers including some Highly Cited Researchers".
It is possible that the highly cited researchers mentioned may have upset their primary affiliations who might have noticed that the indicator points accruing to KAU would come out of their own scores. Just counting the primary affiliations in the new list meant that institutions such as Stanford, the Tokyo Institute of Technology, Indiana University Bloomington, the University of Sydney and the Indian Institute of Science have lost several points for this indicator.
The highly cited indicator is unique among the well known international rankings because when a researcher changes his or her affiliation all of his or her papers go with him or her. It does not matter whether a university has employed a researcher for a day or a decade it will still get the same credit in this indicator. Everything depends on what the researcher puts down as his or her affiliation or affiliations.
All of this is just one manifestation of a problem that has been latent in academic publishing for some years, namely the issue of the affiliation that researchers use when submitting papers or articles. There has probably been quite a bit of small scale fiddling going on for years, with researchers with doctorates from selective universities giving those places as affiliations rather than the technical or education colleges where they are teaching or adjuncts picking the most prestigious of the several institutions where they work.
The best known case of creative affiliation was that of Mohammed El Naschie whose publication career included questionable claims to affiliation with Cambridge, Frankfurt, Surrey and Alexandria Universities (see High Court of Justice Queen's Bench Division: Neutral Citation
Number: [2012] EWHC 1809 (QB)).
Most of these claims did no one any good or any harm, apart from a little embarrassment. However, the Alexandria affiliation, combined with Thomson Reuters' distinctive method of counting citations and the university's relatively few publications, propelled Alexandria into the worlds top 5 for research impact and top 200 overall in the the 2010 Times Higher Education (THE) World University Rankings.
It is possible that a few of the researchers who have signed up for KAU will start showing up in massively cited multi-contributor publications, many of them in physics, that can boost otherwise obscure places into the upper sections of the research impact indicator of the THE rankings.
TR have said that they did not count physics articles with more than 30 authors when they prepared their recent list of highly cited researchers. This could reduce the scores obtained by KAU, Panjab University and some other institutions if TR follow the same procedure in the coming THE world rankings. The issue, however, is not confined to physics.
It is time that journals, databases and ranking organisations began to look carefully at affiliations. At the least, journals should start checking claims and rankers might consider counting only one affiliation per author.
Monday, September 01, 2014
Saturday, August 23, 2014
The Shanghai Rankings Part 2
The Shanghai Rankings have had a reputation for reliability and consistency. The latest rankings have, however, undermined that reputation a little. There have been two methodological changes of which one, not counting Proceedings Papers in the Nature and Science and Publications indicators, may not be of any significance. The other is the use of a new list of Highly Cited Researchers prepared by Thomson Reuters covering citations between 2002 and 2012. In this year's rankings this was combined with the old list which had not been updated since 2004.
One result of this is that there have been some very dramatic changes in the scores for Highly Cited Researchers this year. University of California Santa Cruz's score has risen from 28.9 to 37.9, Melbourne's from 24 to 29.3 and China University of Science and Technology's from 7.2 to 24.5 while that of the Australian National University has fallen from 32.3 to 24.8 and Virginia Polytechnic Institute's from 22.9 to 11.4.
This has had a noticeable impact on total scores. Santa Cruz has risen from the 101-150 band to 93rd place, Melbourne from 54th to 44th and China University of Science and Technology from the 201 - 300 band to the 150-200 band. The Australian National University has fallen from 66th place to 74th and Indiana University at Bloomington has dropped from 85th place to the 101-150 band.
In the top 20, this year's ARWU is more volatile than the two previous editions but still not as much as any other international ranking. The top 20 universities in 2013 rose or fell an average of 0.65 places.
Looking at the top 100 universities, the ARWU is more volatile than last year's QS rankings with the average institution moving up or down 4.92 places.
One result of this is that there have been some very dramatic changes in the scores for Highly Cited Researchers this year. University of California Santa Cruz's score has risen from 28.9 to 37.9, Melbourne's from 24 to 29.3 and China University of Science and Technology's from 7.2 to 24.5 while that of the Australian National University has fallen from 32.3 to 24.8 and Virginia Polytechnic Institute's from 22.9 to 11.4.
This has had a noticeable impact on total scores. Santa Cruz has risen from the 101-150 band to 93rd place, Melbourne from 54th to 44th and China University of Science and Technology from the 201 - 300 band to the 150-200 band. The Australian National University has fallen from 66th place to 74th and Indiana University at Bloomington has dropped from 85th place to the 101-150 band.
In the top 20, this year's ARWU is more volatile than the two previous editions but still not as much as any other international ranking. The top 20 universities in 2013 rose or fell an average of 0.65 places.
| Ranking | Average Place Change of Universities in the top 20 |
|---|---|
| ARWU 2013 -2014 | 0.65 |
| ARWU 2012-2013 | 0.25 |
| ARWU 2011 - 2012 | 0.15 |
| Webometrics 2013-2014 | 4.25 |
| Center for World University Ranking (Jeddah) 2013-2014 |
0.90 |
| THE World Rankings 2012-2013 | 1.20 |
| QS World Rankings 2012-2013 | 1.70 |
Looking at the top 100 universities, the ARWU is more volatile than last year's QS rankings with the average institution moving up or down 4.92 places.
| Ranking | Average Place Change of Universities in the top 100 |
|---|---|
| ARWU 2013 -2014 | 4.92 |
| ARWU 2012-2013 | 1.66 |
| ARWU 2011 - 2012 | 2.01 |
| Webometrics 2013-2014 | 12.08 |
| Center for World University Ranking (Jeddah) 2013-2014 | 10.59 |
| THE World Rankings 2012-2013 | 5.36 |
| QS World Rankings 2012-2013 | 3.97 |
Sunday, August 17, 2014
The Shanghai Rankings (Academic Ranking of World Universities) 2014 Part 1
Publisher
Center for World-Class Universities, Shanghai Jiao Tong University
Scope
Global. 500 institutions.
Methodology
See ARWU site.
In contrast to the other indicators, the Highly Cited Researchers indicator has undergone substantial changes in recent years, partly as a result of changes by data provider Thomson Reuters. Originally, ARWU used the old list of highly cited researchers prepared by Thomson Reuters (TR), which was first published in 2001 and updated in 2004. Since them no names have been added although changes of affiliation submitted by researchers were recorded.
Until 2011 when a researcher listed more than one institution as his or her affiliation then credit for the highly cited indicator would be equally divided. Following the recruitment of a large number of part time researchers by King Abdulaziz University, ARWU introduced a new policy of asking researchers how their time was divided. When there was no response, secondary affiliations were counted as 16%, which was the average time given by those who responded to the survey.
In 2013 TR announced that they were introducing a new list based on field-normalised citations over the period 2002-2012. However, problems with the preparation of the new list meant that it could not be used in the 2013 rankings. Instead, the Shanghai rankings repeated the 2012 scores.
During 2013, KAU recruited over 100 highly cited researchers who nominated the university as a secondary affiliation. That caused some comment by researchers and analysts. A paper by Lutz Bornmann and Johann Bauer concluded that to " counteract attempts at manipulation, ARWU should only consider primary institutions of highly cited researchers."
It seems that Shanghai has acted on this advice: "It is worth noting that, upon the suggestion of many institutions and researchers including some Highly Cited Researchers, only the primary affiliations of new Highly Cited Researchers are considered in the calculation of an institution’s HiCi score for the new list."
As a result, KAU has risen into the lower reaches of the 150-200 band on the basis of publications, some papers in Nature and Science and a modest number of primary affiliations among highly cited researchers. That is a respectable achievement but one that would have been much greater if the secondary affiliations had been included.
Perhaps Shanghai should also take note of the suggestion in a paper by Lawrence Cram and Domingo Docampo that " [s]ignificant acrimony accompanies some published comparisons between ARWU and other rankings (Redden, 2013) driven in part by commercial positioning . Given its status as an academic ranking , it may be prudent for ARWU to consider replacing its HiCi indicator with a measure that is nit sourced from a commercial provider if such a product can be found that satisfies the criteria (objective, open, independent ) used by ARWU."
.
Top Ten
Countries With Universities in the Top 100
Center for World-Class Universities, Shanghai Jiao Tong University
Scope
Global. 500 institutions.
Methodology
See ARWU site.
In contrast to the other indicators, the Highly Cited Researchers indicator has undergone substantial changes in recent years, partly as a result of changes by data provider Thomson Reuters. Originally, ARWU used the old list of highly cited researchers prepared by Thomson Reuters (TR), which was first published in 2001 and updated in 2004. Since them no names have been added although changes of affiliation submitted by researchers were recorded.
Until 2011 when a researcher listed more than one institution as his or her affiliation then credit for the highly cited indicator would be equally divided. Following the recruitment of a large number of part time researchers by King Abdulaziz University, ARWU introduced a new policy of asking researchers how their time was divided. When there was no response, secondary affiliations were counted as 16%, which was the average time given by those who responded to the survey.
In 2013 TR announced that they were introducing a new list based on field-normalised citations over the period 2002-2012. However, problems with the preparation of the new list meant that it could not be used in the 2013 rankings. Instead, the Shanghai rankings repeated the 2012 scores.
During 2013, KAU recruited over 100 highly cited researchers who nominated the university as a secondary affiliation. That caused some comment by researchers and analysts. A paper by Lutz Bornmann and Johann Bauer concluded that to " counteract attempts at manipulation, ARWU should only consider primary institutions of highly cited researchers."
It seems that Shanghai has acted on this advice: "It is worth noting that, upon the suggestion of many institutions and researchers including some Highly Cited Researchers, only the primary affiliations of new Highly Cited Researchers are considered in the calculation of an institution’s HiCi score for the new list."
As a result, KAU has risen into the lower reaches of the 150-200 band on the basis of publications, some papers in Nature and Science and a modest number of primary affiliations among highly cited researchers. That is a respectable achievement but one that would have been much greater if the secondary affiliations had been included.
Perhaps Shanghai should also take note of the suggestion in a paper by Lawrence Cram and Domingo Docampo that " [s]ignificant acrimony accompanies some published comparisons between ARWU and other rankings (Redden, 2013) driven in part by commercial positioning . Given its status as an academic ranking , it may be prudent for ARWU to consider replacing its HiCi indicator with a measure that is nit sourced from a commercial provider if such a product can be found that satisfies the criteria (objective, open, independent ) used by ARWU."
.
Top Ten
| Place | University |
|---|---|
| 1 | Harvard |
| 2 | Stanford |
| 3 | MIT |
| 4 | University of California Berkeley |
| 5 | Cambridge |
| 6 | Princeton |
| 7 | California Institute of Technology (Caltech) |
| 8 | Columbia |
| 9= | Chicago |
| 9= | Oxford |
Countries With Universities in the Top 100
| Country | Number of Universities |
|---|---|
| United States | 52 |
| United Kingdom | 8 |
| Switzerland | 5 |
| Germany | 4 |
| France | 4 |
| Netherlands | 4 |
| Australia | 4 |
| Canada | 4 |
| Japan | 3 |
| Sweden | 3 |
| Belgium | 2 |
| Israel | 2 |
| Denmark | 2 |
| Norway | 1 |
| Finland | 1 |
| Russia | 1 |
Tuesday, August 05, 2014
Webometrics: Ranking Web of Universities 2nd 2014 Edition
The Webometrics rankings are based on web-derived data. They cover more than 22,000 institutions, far more than conventional rankings, and should always be consulted as a check on the plausibility of the others. They are, however, extremely volatile and that reduces their reliability considerably.
Publisher
Cybermetrics Lab, CSIC, Madrid
Scope
Global. 22,000+ institutions.
Methodology
From the Webometrics site.
The current composite indicator is now built as follows:
Visibility (50%)
IMPACT. The quality of the contents is evaluated through a "virtual referendum", counting all the external inlinks that the University webdomain receives from third parties. Those links are recognizing the institutioof conventinal prestige, the academic performance, the value of the information, and the usefulness of the services as introduced in the webpages according to the criteria of millions of web editors from all over the world. The link visibility data is collected from the two most important providers of this information:Majestic SEO and ahrefs. Both use their own crawlers, generating different databases that should be used jointly for filling gaps or correcting mistakes. The indicator is the product of square root of the number of backlinks and the number of domainsoriginating those backlinks, so it is not only important the link popularity but even more the link diversity. The maximum of the normalized results is the impact indicator.
Activity (50%)
PRESENCE (1/3). The total number of webpages hosted in the main webdomain (including all the subdomains and directories) of the university as indexed by the largest commercial search engine (Google). It counts every webpage, including all the formats recognized individually by Google, both static and dynamic pages and other rich files. It is not possible to have a strong presence without the contribution of everybody in the organization as the top contenders are already able to publish millions of webpages. Having additional domains or alternative central ones for foreign languages or marketing purposes penalizes in this indicator and it is also very confusing for external users.
OPENNESS (1/3). The global effort to set up institutional research repositories is explicitly recognized in this indicator that takes into account the number of rich files (pdf, doc, docx, ppt) published in dedicated websites according to the academic search engine Google Scholar. Both the total files Both the total records and those with correctly formed file names are considered (for example, the Adobe Acrobat files should end with the suffix .pdf). The objective is to consider recent publications that now are those published between 2008 and 2012 (new period).
EXCELLENCE (1/3). The academic papers published in high impact international journals are playing a very important role in the ranking of Universities. Using simply the total number of papers can be misleading, so we are restricting the indicator to only those excellent publications, i.e. the university scientific output being part of the 10% most cited papers in their respective scientific fields. Although this is a measure of high quality output of research institutions, the data provider Scimago groupsupplied non-zero values for more than 5200 universities (period 2003-2010). In future editions it is intended to match the counting periods between Scholar and Scimago sources.
Top Ten
1. Harvard University
2. MIT
3. Stanford University
4. Cornell University
5. University of Michigan
6. University of California Berkeley
7= Columbia University
8= University of Washington
9. University of Minnesota
10. University of Pennsylvania
Countries with Universities in the Top Hundred
USA 66
Canada 7
UK 4
Germany 3
China 3
Japan 2
Switzerland 2
Netherlands 1
Australia 1
Italy 1
South Korea 1
Taiwan 1
Belgium 1
Hong Kong 1
Brazil 1
Austria 1
Czech Republic 1
Singapore 1
Mexico 1
Top Ranked in Region
USA: Harvard
Canada: Toronto
Latin America: Sao Paulo
Caribbean University of the West Indes
Europe: Oxford
Africa: University of Cape Town
Asia: Seoul National University
South Asia IIT Bombay
Southeast Asia National University of Singapore
Middle East: Hebrew University of Jerusalem
Arab World: King Saud University
Oceania Melbourne
Noise Index
Average position change of universities in the top 20 in 2013:
4.25
Comparison
Center for World University Rankings -- 0.90
Shanghai Rankings (ARWU): 2011-12 -- 0.15
Shanghai Rankings (ARWU) 2012-13 -- 0.25
THE WUR: 2012-13 -- 1.20
QS WUR 2012-13 -- 1.70
Average position change of universities in the top 100 in 2013
12.08
Comparison
Center for World University Rankings -- 10.59
Shanghai Rankings (ARWU): 2011-12 -- 2.01
Shanghai Rankings 2012-13 -- 1.66
THE WUR: 2012-13 -- 5.36
QS WUR 2012-13 -- 3.97
Subscribe to:
Comments (Atom)