Shanghai Rankings: Correlations Between Indicators

This is, I hope, the first of  a series. Maybe THE and QS next week.

If we want to compare  the utility of university rankings one attribute to consider is internal consistency. Here, the correlation between the various indicators can tell us a lot. If the correlation between a pair of indicators is 0.90 or above we can assume that these indicators are essentially measuring the same thing.

On the other hand, if  there is no correlation or one that is low, insignificant or even negative we might have doubts about the validity of one or both of the indicators. It is reasonable that if a university scores well for one metric it will do well for others providing they both represent highly valued attributes. A university producing high quality research or collecting large numbers of citations should also score well for reputation. If it does not there might be a methodological problem somewhere.

So, we can assume that if the indicators are valid and are not measuring the same thing the correlation between indicators will probably be somewhere between 0.5 and 0.9.

Let's have a look at the Shanghai ARWU for 2019. The indicator scores were extracted and analysed using PSPP. (It is very difficult to analyse the 2020 edition because of a recent change in presentation.) These rankings have six indicators: alumni and faculty receiving Nobel and Fields awards, papers in Nature and Science, highly cited researchers, publications in the Web of Science, and productivity per capita.

Looking at all 1000 institutions in the Shanghai Rankings, Alumni, Awards, and Nature and Science all correlate well with each other Highly Cited Researchers correlates well with Nature and Science and Publications but less so with Alumni and Awards. Nature and Science correlates well with all the other indicators.

The Publications indicator does not correlate well with Alumni and Awards. This is to be expected since Publications refers to 2018 while the Alumni and Awards indicators go back several decades.

Overall, the correlations are quite good although there is a noticeable divergence between Publications and Alumni and Awards, which cover very different time periods. 

CORRELATIONS

CORRELATION
	/VARIABLES =  alumni awards highlycited naturescience publications pcp finaltotal
	/PRINT = TWOTAIL NOSIG.

Correlations

		alumni	awards	highlycited	naturescience	publications	pcp	finaltotal
alumni	Pearson Correlation	1.00	.78	.51	.72	.45	.63	.76
	Sig. (2-tailed)		.000	.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
awards	Pearson Correlation	.78	1.00	.57	.75	.44	.67	.82
	Sig. (2-tailed)	.000		.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
highlycited	Pearson Correlation	.51	.57	1.00	.79	.72	.64	.87
	Sig. (2-tailed)	.000	.000		.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000
naturescience	Pearson Correlation	.72	.75	.79	1.00	.69	.73	.93
	Sig. (2-tailed)	.000	.000	.000		.000	.000	.000
	N	992	992	992	992	992	992	992
publications	Pearson Correlation	.45	.44	.72	.69	1.00	.50	.81
	Sig. (2-tailed)	.000	.000	.000	.000		.000	.000
	N	1000	1000	1000	992	1000	1000	1000
pcp	Pearson Correlation	.63	.67	.64	.73	.50	1.00	.78
	Sig. (2-tailed)	.000	.000	.000	.000	.000		.000
	N	1000	1000	1000	992	1000	1000	1000
finaltotal	Pearson Correlation	.76	.82	.87	.93	.81	.78	1.00
	Sig. (2-tailed)	.000	.000	.000	.000	.000	.000
	N	1000	1000	1000	992	1000	1000	1000

Most observers of ARWU and other global rankings are interested in the top levels where elite schools and national flagships jostle for dominance. Analysing correlations among indicators for the top 200 in ARWU, there are high correlations between Alumni, Awards, Nature and Science, and Productivity per Capita, ranging from .69 to .79.

There is also a high correlation of .72 between Nature and Science and Highly Cited Researchers. It is, however, noticeable that the correlation between Publications and other indicators is low for Highly Cited Researchers and very low for Productivity per Capita, Alumni and Awards.

It seems that, especially among the top 200 places, there is a big gap opening between the old traditional elite of Oxbridge, the Ivy League and the like who continue to get credit for long dead Nobel laureates and the new rising stars of Asia and Europe who are surging ahead for WOS papers and beginning to produce or recruit superstar researchers.

Correlations

		alumni	awards	highlycited	naturescience	publications	pcp	finaltotal
alumni	Pearson Correlation	1.00	.79	.36	.69	.21	.62	.78
	Sig. (2-tailed)		.000	.000	.000	.003	.000	.000
	N	200	200	200	199	200	200	200
awards	Pearson Correlation	.79	1.00	.44	.74	.14	.67	.84
	Sig. (2-tailed)	.000		.000	.000	.044	.000	.000
	N	200	200	200	199	200	200	200
highlycited	Pearson Correlation	.36	.44	1.00	.72	.57	.49	.78
	Sig. (2-tailed)	.000	.000		.000	.000	.000	.000
	N	200	200	200	199	200	200	200
naturescience	Pearson Correlation	.69	.74	.72	1.00	.44	.65	.92
	Sig. (2-tailed)	.000	.000	.000		.000	.000	.000
	N	199	199	199	199	199	199	199
publications	Pearson Correlation	.21	.14	.57	.44	1.00	.12	.55
	Sig. (2-tailed)	.003	.044	.000	.000		.083	.000
	N	200	200	200	199	200	200	200
pcp	Pearson Correlation	.62	.67	.49	.65	.12	1.00	.72
	Sig. (2-tailed)	.000	.000	.000	.000	.083		.000
	N	200	200	200	199	200	200	200
finaltotal	Pearson Correlation	.78	.84	.78	.92	.55	.72	1.00
	Sig. (2-tailed)	.000	.000	.000	.000	.000	.000
	N	200	200	200	199	200	200	200