Sunday, March 27, 2016

The Goldman Sachs Rankings

From efinancialcareers

"Goldman Sachs recruits primarily from US Ivy League universities. This might sound entirely predictable, but Goldman does have some surprises in the places it recruits from. In Asia, for example, it recruits mostly from local universities. In the UK, it seems to draw most of its operations hires from Warwick.
Based on the 22,000 Goldman Sachs CVs in the eFinancialCareers CV database, we’ve created rankings of the top universities the bank hires from by sector and region.
Globally, the best university for getting into Goldman Sachs is the London School of Economics (LSE), our data suggests, followed by Columbia University and the University of Pennsylvania – both of which have a strong emphasis on financial services careers."
Globally the best places for a career with Goldman Sachs are:

1.  London School of Economics
2.  Columbia University
3.  University of Pennsylvania
4.  Princeton University
5.  Harvard University.

In the US they are:

1.  Columbia University
2.  University of Pennsylvania
3.  Princeton University
4.  Cornell University
5.  Harvard University.

In Asia they are:

1.  University of Hong Kong
2.  National University of Singapore
3.  Nanjing University
4.  London School of Economics
5.  Indian Institute of Technology (which one is not stated).

In Europe they are:
1.  London School of Economics
2.  Cambridge University
3.  Imperial College London
4.  Oxford University
5.  Warwick University.

Saturday, March 26, 2016

Ranking Rankings Again

Magnus Gunnarsson of the University of Gothenburg has reminded me of a 2010 report which included an assessment of methodology based on IREG's Berlin principles.

There were 16 Berlin principles, 14 of which were given weighted subscores in the report. The values and weighting were determined  subjectively although the rankers were evidently well informed.

The Berlin principles were grouped in four categories, Purpose and Goals of Rankings, Design and Weighting of Indicators, Collection and Processing of Data and Presentation of Ranking Results. For more information see here.

The ranking of rankings  by methodology is as follows. It is obviously out of date.

Overall method score
1= CHE (Germany) 3.10
1=  Webometrics 3.10
HEEACT (now Taiwan National
4 ARWU (Shanghai) 2.80
5= High Impact Universities (Australia) 2.60
5= Observatory (sustainable development)2.60
7= Scimago 2.40
7= CWTS Leiden ranking 2.40
9 THE 2.30
10 Mines Paris Tech 2.20
11 QS 2.10
12 Rater (Russia) 1.70

Thursday, March 24, 2016

Update on Trinity College Dublin

Some things have been clarified about the now famous message sent by John Boland, vice-president and dean of research at Trinity College Dublin to various stakeholders.

It seems that the message was about both the QS and the THE academic surveys, referring to the sign up facility for the former and the one dollar donation to charity for the latter.

Apparently THE has no problems with the message.

On the other hand, QS thinks that its guidelines have been breached.

"QS has been made aware of recent communications from Trinity College Dublin (TCD) that appear to contravene the guidelines used by QS to ensure the accuracy of its university rankings. In accordance with our standard operating procedure, we have notified TCD that its “awareness” campaign is in breach of these guidelines."

The issue is not suggesting that anyone should go the sign up facility. That is permitted in clearly stated guidelines. What is not permitted is suggesting what they do after they have signed up.

"It is acceptable and even encouraged for institutions to communicate with employers and academics worldwide to showcase their achievements. Institutions are welcome to invite contacts to sign up for possible selection for our survey using our Academic or Employer Sign Up Facilities, but any message soliciting a specific response in our surveys, represents unfair manipulation of the results and will not be tolerated. "

It is fairly clear what the vice-dean was suggesting people should do although I can see how he might have thought he was being subtle enough to get around the QS guidelines.

One interesting aspect of this affair is that QS is much stricter about influencing the surveys than THE.

I think the main lesson to be learnt from this affair is that it is unwise to allow senior administrators to get involved with any sort of rankings initiative or strategy.

Monday, March 21, 2016

Turning the Tide: Contributing to the Decline of American Universities

Harvard Graduate School of Education has come out with a plan to radically overhaul the admissions process in American universities. The document has been endorsed by a huge, if that word won't trigger a trauma for someone, number of certified academic gatekeepers.

The report argues that the current university admissions process emphasises personal success at the expense of community engagement, puts too much stress on applicants and discriminates against students from disadvantaged communities.

Proposals include "promoting more meaningful contributions to others", "assessing students  ethical engagement and contributions to others in ways that reflect varying types of family and community contributions across race, culture and class" and  "redefining achievement in ways that both level the playing field for economically diverse students and reduce excessive achievement pressure."

Detailed recommendations include students doing at least a year of "sustained service or community engagement", which could include working to contribute family finances. The report recommends that what should count for university admissions is whether " whether students immersed themselves in an experience and the emotional and ethical awareness and skills generated by that experience.'

It is predictable that this will greatly increase the stress experienced by applicants who would have to find some sort of service for a year, immerse themselves in it, generate emotional and ethical skills and then be able to convince admissions officers that they have done so. Or, let us be realistic, find teachers or advisors who will show them how to do this. It will also prompt a massive invasion of privacy if colleges are really going to demand evidence that community service is authentic and meaningful.

Students will also be encouraged to undertake activities that  "deepens their appreciation of diversity." Would it be too cynical to suspect that this is code for political indoctrination?

The report also urges that students take fewer AP and IB courses and that colleges should consider making the SAT and ACT optional.

None of these are particularly novel but put together they are likely to cause a shift in the qualities required for admission to America's elite schools. Students will care for others, be passionate about community engagement, appreciate diversity and have authentic extra-curricular activities or at least they will be highly skilled at pretending that they are. They will also be less academically able and prepared and universities will inevitably have to adjust to this.

Also, admissions offices will require more money and responses to make sure that those diversity experiences are meaningful and that community engagement is authentic. But that shouldn't be a problem. Somehow money is always available when it is really needed.

Over the next few years look for a migration of talented students and researchers from the US.

Sunday, March 20, 2016

Trinity College Dublin: Waiting for Clarification

Yesterday I published a post about Trinity College Dublin that referred to a report in the London Sunday Times about TCD encouraging its graduates to respond to questionnaires  from THE.

Now I have seen a report in Inside Higher Education that says that TCD has been encouraging people to sign up for the QS academic survey sign up facility and to vote for the college.

I have therefore deleted the post until it is clear whether TCD has been sending messages about about the QS or the THE surveys or both and whether it has violated the guidelines of either or both.

Friday, March 18, 2016

"What we are seeing worldwide, from India to the UK to the US, is the rebellion against the inner circle of no-skin-in-the-game policymaking "clerks" and journalists-insiders, that class of paternalistic semi-intellectual experts with some Ivy league, Oxford-Cambridge, or similar label-driven education who are telling the rest of us 1) what to do, 2) what to eat, 3) how to speak, 4) how to think... and 5) who to vote for.
With psychology papers replicating less than 40%, dietary advice reversing after 30y of fatphobia, macroeconomic analysis working worse than astrology, microeconomic papers wrong 40% of the time, the appointment of Bernanke who was less than clueless of the risks, and pharmaceutical trials replicating only 1/5th of the time, people are perfectly entitled to rely on their own ancestral instinct and listen to their grandmothers with a better track record than these policymaking goons."

And, one might add, the publishers of sophisticated and prestigious rankings who would have us believe that a university can be world-class one year and at the bottom of the table the next.

Thursday, March 17, 2016

Trinity College Dublin Gets Upset About Rankings

Trinity College Dublin has done remarkably well in global university rankings over the last decade. Since 2004 it has steadily risen from the 201-300 band to the 151-200 band in the Shanghai Academic Ranking of World Universities (ARWU). Its score for publications went from 27.1 in 2004 to 31 in 2015 (Harvard is 100) and for productivity per capita (scores for Nobel and Fields awards, papers in Nature and Science, highly cited researchers and publications divided by number of faculty) it rose from 13.9 to 19 (Caltech is 100).

The Shanghai rankings measure only research. Nonetheless, this is genuine progress even if slow and boring: at this rate Trinity will catch up with Harvard for publications in another 170 years or so.

So why is Trinity not celebrating this excellent achievement? Instead it is getting very excited about its poor and declining performance in the QS and THE world rankings.

It is a serious mistake to be concerned about falling in these rankings. Last year QS and THE made significant methodological changes so it is meaningless to make year on year comparisons.

Even if QS and THE make no further changes, these rankings are likely to be unacceptably volatile. Both rely heavily on reputation surveys for which scores tend to be very low once you get outside the top fifty or so and consequently are susceptible to short term fluctuations, although QS does damp down short term changes by recycling unchanged survey responses. THE has three income based indicators, institutional income, research income and income from industry and commerce so it is exposed to fluctuations resulting from exchange rate changes. If THE were serious about producing valid and reliable rankings they would use three or five year averages for the income indicators.

And so, as you might have guessed, Trinity is developing a rankings strategy

"The Rankings Steering Group, set up as part of the strategy, is chaired by the Provost, Patrick Prendergast, and has identified the QS World University Rankings and the Times Higher Education rankings as a priority. The strategy will focus on areas such as outputs, citations, funding levels, staff composition and reputation."

Plagiarism Data from Russia

The Russian website Dissernet has published information about about plagiarism in Russian universities. The worst offender, according to the Moscow Times, is the Financial University followed by the St Petersburg State University of Economics and the Plekhanov Economic University.

It seems that plagiarism is a big problem in Russian universities although it could be that some Russians are more serious about academic fraud than other countries.

Saturday, March 12, 2016

I'm glad somebody's noticed

Uwe Brandenburg, Managing Director of CHE Consult in Times Higher Education:

"Frankly, I think the success of certain institutions in rankings is more to do with the rankings’ methodology than anything else. They inevitably favour factors that are statistically more likely to be found among certain universities than others."

Ranking Rankings 1: Stability

Updated 13/03/16, 15/03/16

Making a start on ranking global university rankings, here is the average change in position of the top 20 twenty universities in 2014 in seven global university rankings between 2014 and 2015.

Note: both QS and THE introduced major methodological changes in 2015.

The table refers only to the top 20. Things might (or might not) be different if the top 100 or 500 were considered.

The Shanghai ARWU and URAP  are so far well ahead  of the others for stability.

Rankings Mean position change of
 top 20 universities
Shanghai Academic Ranking of World Universities (ARWU)  0.30
Center for World University Rankings (CWUR) (Jeddah)   0.40
University Ranking By Academic Performance (URAP)
(Middle East Technical University)  
THE World University Rankings 2.10
Round University Rankings (Russia) 2.35
QS World University Rankings 3.50
Webometrics 8.85

News Lite from Times Higher

The mainstream media in the UK and parts of Europe is getting very excited about the latest ranking from Times Higher Education, the top  200 European universities.

There is nothing new here. THE have just pulled the European universities out of last year's world rankings.

Here are some of the headlines that have greeted the remarkable revelations.

Leicester Mercury
Leicester Mercury
Google News
Quarter of top 200 European universities are in UK - league table

Wessex Scene

World News Online

Dundee University 'on a global scale' after European rankings accolade

Bailiwick Express
A quarter of Europe's top 200 universities are in the UK

Flanders Today

The Tab
UCL ranked 5th best university in Europe

Plymouth Herald

Shanghai Daily

Monday, March 07, 2016

Is it possible to rank rankings?

At a seminar recently at  Ural Federal University in Ekaterinburg the question was raised whether we could evaluate and rank rankings.

That's a tough one. Different rankings have different approaches, audiences and methodologies. The Shanghai rankings embody the concerns of the Chinese ruling class, convinced that salvation lies in science and technology and disdainful  -- not entirely without reason --  of the social sciences and humanities. The Times Higher Education world rankings have claimed to be looking for quality, recently tempered by a somewhat unconvincing concern for inclusiveness.

But it is possible that there are some simple metrics that could be used to compare global rankings. here are some suggestions.

Universities are big places typically with thousands of students and hundreds of staff. In the absence of administrative reorganisation or methodological changes we should not expect dramatic change from one year to another. Apparently a change of four places over a year is normal for the US News America's Best Colleges so nobody should get excited about going up or down a couple of places.

It would seem reasonable then that rankings could be ordered according to the average change in position over a year. I have already done some calculations with previous years' rankings (see posts 09/07/14, 17/07/14, 04/11/14).

So we could rank these international rankings according to the mean number of position changes in the top 100 between 2013 and 2014. The fewer the more stable the rankings.

1.   Quacquarelli Symonds World University Rankings        3.94
2.   Times Higher Education World University Rankings     4.34
3.   Shanghai ARWU 4.92
4.   National Taiwan University Rankings   7.30
5.   CWUR (Jeddah)   10.59
6.   Webometrics   12.08

Consistency and Redundancy
It is reasonable that if the various ranking indicators are measuring quality or highly valued attributes there should be at least a modest correlation  between them. Good students will attract good teachers who might also have the attributes, such as an interest in their field or reading comprehension skills, required to do research. Talented researchers will be drawn to places that are generously funded or highly reputed.

On the other hand, if  there is a very high correlation between two indicators, perhaps above .850, then this probably means that they are measuring the same thing. One of them could be discarded.

Some rankings have adopted the practice of putting universities into bands rather than giving them individual scores. This is, I suppose, a sensible way of discouraging people from getting excited about insignificant fluctuation but it might also suggest a lack of confidence in the rankers' data or the intention of selling the data in some way, perhaps in the guise of benchmarking. Since 2010 THE have bundled indicator scores into clusters making it very difficult to figure out exactly what is causing universities to rise or fall. Rankings could be ranked according to the number of universities for which overall scores and indicator scores are provided.

It would be very easy to rank rankings according to the number of universities that they include. This is something where they vary considerably. The Shanghai ARWU ranks 500 universities while Webometrics ranks close to 24,000.

Some rankings such as ARWU, the NTU rankings (Taiwan), URAP (Middle East Technical University) measure only research output and impact. The THE and QS rankings attempt to include metrics related, perhaps distantly, to teaching quality and innovation. QS has an indicator that purports

Some rankings award a disproportionate weighting to a single indicator, QS's  academic survey (40%), THE's citation indicator (30%).  Also, if a university or universities are getting disproportionately high scores for a specific indicator this might mean that the rankings are being manipulated or are seriously flawed in some way.

External Validation
How do we know that rankings measure what they are supposed to measure? It might be possible to measure the correlation between international rankings and national rankings which often include more data and embody local knowledge about the merits of universities.

How long would it take to check whether the rankings have given your university the correct indicator score? Try it for yourself with the Shanghai highly cited researchers indicator. Go here and find the number of highly cited researchers with Harvard as their primary affiliation and the number with your university. Find the square root of both numbers. Then give Harvard a score of 100 and adjust your university's score accordingly.

Now for the THE citations impact indicator. This is normalised by field and by year of citation so that the what matters is not the number of citations that a publication gets but the number of publications compared to the world average in 334 fields and in the first, second, third, fourth, fifth or sixth year of publication.

Dead simple isn't it?

And don't forget the regional modification.

I hope the next post will be a ranking of rankings according to stability.

Wednesday, March 02, 2016

The Decline of Free Speech in American Universities

The Foundation for Individual Rights in Education has just released its list of the ten worse colleges for free speech in the US. Here they are along with the incidents that put them on the list.

Mount St Mary's University, Maryland

Two faculty were sacked for criticising the President's plan to get rid of low performing students, "drowning the bunnies" as he so charmingly put it. They were later reinstated.

Northwestern University

Laura Kipnis was investigated for sexual harassment for writing an essay criticising the sexual harassment mania sweeping US colleges. She was cleared only after writing an account of her persecution in the Chronicle of Higher Education.

Louisiana State University

Theresa Buchanan was fired for using profanity in the classroom for a pedagogical reason.

University of  California San Diego

The administration attempted to defund a  student newspaper for making fun of "safe spaces".

St Mary's University of Minnesota

An adjunct classics professor was fired for sexual harassment which have have something to do with an authentic production of Seneca's Medea. He was also fired from his other job as a janitor (!).

University of Oklahoma

Two fraternity members for leading a racist chant. The supreme court has ruled that offensive speech is protected by the first amendment

Marquette University
John Mcadams was for criticising an instructor for suppressing a student's negative comments about same sex marriage.

Colorado College 

A student was suspended for unchivalrous remarks about African American women on Yik Yak.

University of Tulsa 

A student was removed from class because of Facebook posts written by his fiance criticising a professor.

Wesleyan University

The student government voted to remove funding from a student newspaper that was mildly critical of Black Lives Matter.