Ranking MENA Universities

Interest in the ranking of Arab and/or Middle Eastern and North African universities seems to be growing. There is definitely  a feeling  in many countries that higher education needs drastic reform and should open up to objective evaluation. Of course, there is also a realisation that rankings can be helpful for marketing and career advancement.

But there are problems. Most Arab universities do not have sufficient staff and organisation  to provide adequate data about  faculty and student numbers, let alone about things like employment of graduates or sources of income. Surveys are a possible source of information. for teaching quality but it is unlikely that at present they could produce reliable and accurate information.

The collection of data about research has probably reached the point where reasonable research rankings can be created for the Arab world and also for regions like South Asia, Southeast Asia and Africa. Unfortunately, the majority of Arab or Middle Eastern universities  produce very little significant research so such research based rankings would probably be of relevance to about fifty institutions.

See Higher Education Strategy Associates for more discussion.

The US News has produced a variation on its new Best Global Universities rankings. This is based on research and citations data from the Scopus database. As a ranking of universities according to research output and impact, it looks quite plausible.

The top five universities are:

1. King Saud University, Saudi Arabia
2. King Abdulaziz University,Saudi Arabia
3. King Abdullah University of Science and Technology, Saudi Arabia
4. Cairo University, Egypt
5. American University of Beirut, Lebanon.

QS has produced rankings that are "a pilot version of the ranking that has been developed to reflect priorities and challenges for universities in the region."

The top five are:

1. King Fahd University of Petroleum and Minerals, Saudi Arabia
2. American University of Beirut, Lebanon
3. King Saud University, Saudi Arabia
4. American University of Cairo, Egypt
5. King Abdulaziz University, Saudi Arabia.

This is a bit different from the US News rankings but the QS table does include indicators like faculty student ration, web impact and international faculty and students.

Times Higher Education has produced what it calls a "snapshot", which is in fact only one indicator, citations per paper normalised by field and year. The top five are:

1.  Texas A&M University Qatar
2.  Lebanese American University
3.  King Abdulaziz University,Saudi Arabia
4.  Qatar University
5.  American University in Beirut.

Texas A&M Qatar is a single subject branch campus which does not have a doctoral programme. It is difficult to believe that it will actually be in the full rankings that are scheduled for next year. THE's top thirty does not include Cairo University or King Abdullah University of Science and Technology.

As a simple check on the validity of these three rankings I have calculated their correlation with the results from a search of Google Scholar on the 4th of April. I have also included the scores for "publications" provided by THE next to their "snapshot" indicator.

There is a very strong correlation between the Google Scholar and the THE publication scores, a somewhat smaller correlation with the US News scores, which combine publication and citation data and a moderate correlation with the QS scores, which include web impact, internationalisation and institutional data.

There is no significant correlation with the THE citation indicator. In fact, this "snapshot" correlates with nothing, not even THE's own publication data.


Correlations

	Google Scholar	USN	QS	THE "snapshot"	THE publications
Google Scholar		.757**       .000  87	.570**            .000  49	.075  .692  30	.852** .000 30
USN	.757** .000  87		.419* .017  32	.266 .170  28	.869** .000 28
QS	570** .000   49	.419* .017  32		.267 .337  15	.412 .128  15
THE snapshot	.075 .692   30	.266 .170  28	.267 .337  15		.109 .567 30
THE publications	.852** .000 30	.869** .000  28	.412 .128  15	.109 .567 30

A Fictional, but Honest, College Rejection Letter

By Mimi Evans

Posted at Timothy MacSweeney's Blog

"Dear Applicant,

The Admissions Committee has carefully considered your application and we regret to inform you that we will not be able to offer you admission in the entering class of 2015, or a position on one of our alternate lists. The applicant pool this year was particularly strong, and by that I mean the Admissions Committee once again sent candidates like you multiple enticing pamphlets encouraging you to apply, knowing full well we had no intention of accepting you.

However, you will be pleased to know that you have contributed to our declining admissions rate, which has helped our university appear exclusive. This allows us to attract our real candidates: upper-class kids and certified geniuses who will glean no new information from our courses or faculty, whose parents can incentivize us with a new swimming pool or lacrosse stadium."

Moscow Ranking seminar

A seminar, "Road to Academic Excellence: Russian Universities in Rankings by Subject" was held in Moscow on April 9th and 10th.

QS Subject Rankings Postponed

From the QS topuniversities site

The planned publication of the QS World University Rankings by Subject 2015 has been postponed for the next few weeks.

In 2015, we have introduced minor methodological refinements which have allowed for improved discrimination, particularly among specialist institutions that are now featuring more materially in our work.  Our new subjects for this year include several - such as Veterinary Sciences, Art & Design, Architecture, Dentistry, Business & Management Studies – that are delivered by single-faculty institutions as well as large, comprehensive universities.

We approach our work with passion, dedication, integrity and a strong sense of responsibility. In response to feedback, we have decided to extend the consultation process, to fully articulate the methodological refinements of the QS World University Rankings by Subject.

Please check back on TopUniversities.com for details on the revised release date of the QS World University Rankings by Subject 2015.

Malaysia and the Rankings: The Saga Continues

Malaysia has had a long and complicated relationship with global rankings ever since that wonderful moment in 2004 when the Times Higher Education Supplement (THES) -- Quacquarelli Symonds (QS) World University Rankings, as they were then, put Universiti Malaya (UM), the country's oldest institution, in the top 100 universities of the world.

It turned out that UM was only in the top 100 because of a ridiculous error by data gatherers QS who counted ethnic Indians and Chinese as international and so boosted the score for the international faculty and international student indicators. This was followed in 2005 by the correction of the error, or "clarification of data" as THES put it, and UM's extraordinary fall out of the top 100, often explained by higher education experts as a change in methodology.

There was another fall in 2007 when QS introduced several methodological changes including the use of z scores, that is calibrating scores against the indicator means, and prohibiting survey respondents from voting for their own universities.

In 2009 UM made something of a recovery rising from 230th in the Times Higher Education (the Supplement bit was dropped in 2008) charts to 180th, largely because of an increase in the number of faculty and a reduction in the number of students reported to QS.

In 2010 THE and QS went their separate ways, publishing their own rankings with different methodologies. UM dropped out of the top 200 of the QS rankings but was back again in 2011 and  has now reached 151st place. It has stayed clear of the THE World University Rankings, which require the annual resubmission of data.

Every time UM or any of the other Malaysian universities rises or falls, it becomes a political issue. Ascent is seen as proof of the strength of Malaysian higher education, decline is the result of policy failures.

Recently, the Malaysian second Minister for Education argued that Malaysian higher education was now world class and on a par with countries such as the United Kingdom, Germany and Australia because of its improved performance in the QS rankings and because it had attracted 135,000 foreign students.

Not everyone was impressed by this. Opposition MP Tony Pua criticised the reliance on the QS rankings, saying that they had been condemned by prominent academics such as Simon Marginson and that UM was not ranked by THE and performed much less well in the other rankings such as the Academic Ranking of World Universities.

The minister has riposted by noting that four Malaysian researchers were included in Thomson Reuters' list of influential scientific minds and that UM had been given five stars by QS.

So, who's right about  Malaysian higher education?

First, Tony Pua is quite right about the inadequacies of the QS world rankings. It can be unstable since the number of universities included in the rankings changes from year to year and this can affect the scores for each indicator. Many of the scores for the objective indicators such as faculty student ratio and international students seem exaggerated and appear to have had a bit of massaging somewhere along the line.

The biggest problem with the QS rankings is the academic and employer  reputation surveys. These collect data from a variety of sources, have low response rates and are very volatile. They include respondents whose names are submitted by universities and those who nominate themselves. There were some suspiciously high scores for the  academic reputation indicator in 2014: Peking University in 19th place, National Taiwan University in 37th, University of Buenos Aires in 52nd and the Pontifical Catholic University of Chile in 78th.

The employer survey also produces some counter-intuitive results: Universitia Commerciale Luigi Bocconi in 33rd place, Indian Institute of Technology Bombay in 60th, the American University of Beirut in 85th and Universidiad de los Andes in 98th.

The QS world rankings can therefore be considered a poor reflection of overall quality.

Some critics have asserted that the THE rankings are superior and that Malaysian universities are being evasive by staying away from them. It is true that  THE  have won the approval of the British political and educational establishment. David Willetts, former universities and science minister, has joined the advisory board of  THE's parent company. THE has highlighted comments on its recent reputation rankings by Greg Clark, universities, science and cities minister, Vince Cable, the Business Secretary and Wendy Piatt, director of the Russell Group of research intensive universities.

However, more informed and observers such as Alex Usher of Higher Education Strategy Associates and Isidro Aguillo of the Cybermetrics Lab have little regard for these rankings.

Even Simon Marginson, who has moved to the London Institute of Education, now accepts that they are "fatally flawed once outside the top 50 universities".

The THE rankings have some serious methodological flaws. They assign a 33% weighting to a reputation survey. After the top six universities the number of responses drops off rapidly. After we leave the top 100 the number of votes on the survey is small and so it is quite normal for a few additional responses to have a disproportionate effect on the indicator scores and consequently on overall scores. QS does give an even greater weighting for reputation -- 50% -- but reduces annual fluctuations by carrying over responses for a further two years if they are not updated. The new Best Global Universities produced by US News takes a five year average of their reputation scores.

In addition, the THE rankings assign a 30 % weighting to their Citations: Research Impact indicator which  is constructed so that it allows contributions to publications, usually in physics , astronomy or medicine, with hundreds of contributing institutions to give a university an undeserved score for citations. Since its beginning, the THE rankings have  shown bizarre results for research impact by putting places like Alexandria University, Moscow State Engineering Physics Institute, Federico Santa Maria Technical University in Valparaiso and the University of Marrakech Cadi Ayyad in the top ranks of the world for research impact.

Yes, QS putting Tsinghua  University in overall 47th place is questionable (the Shanghai Academic Ranking of World Universities has it in 101-150 band)  but on balance this is more plausible than putting, as THE does, Scuola Normale Superiore di Pisa in 63rd place (Shanghai has it in the 301-400 band)

The only situation in which it would make sense for UM to take part in the THE rankings would be if it was going to start a first rate particle physics programme including participation in the Large Hadron Collider project with multi-author publications that would bring in thousands of citations.

Rather than relying on the questionable QS and THE rankings, it would be a good idea to look at the progress of UM according to the less well known but technically competent research-based rankings. The Scimago Institution Rankings show that in a ranking of higher education institutions by research output  UM was in 718th place in 2009.  Since then it has risen to 266th place, behind 16 British (out of 189), 15 German and seven Australian universities and institutes.

This is similar to the CWTS Leiden Ranking which has UM in 270th place for number of publications (calculated with default settings) or the Shanghai rankings, where Nobel prizes are indicators, which place it in the 301-400 band.

This does not necessarily mean that there has been similar progress in graduate employability or in the quality of research. It does, however, mean that for research output Universiti Malaya, and maybe two or three other Malaysian universities, are now competitive with second tier universities in Britain and Germany.

This is probably not quite what most people mean by world-class but it is not impossible that in a decade UM could, if it sticks to current policies, be the rival of universities like Sheffield, Cardiff or Leeds.

But such progress depends on Malaysian universities focussing on their core missions and not falling into the quagmire of mission creep.

It also depends on something being done to remedy the very poor performance of Malaysian secondary schools. If that does not happen then the future of Malaysian higher education and the Malaysian economy could be very bleak.

Reactions to the Times Higher Education Reputation Rankings

Tsinghua, Peking climb up

Durham University rated among the world's top 100 by scholars

Australia's most respected university named

Delft drops down Times best reputation university rankings

Moscow State University and St. Petersburg State University rank among thetop 100 universities

HK universities suffer drop in world rankings

India fails to make it to world's top 100 reputed universities list

Oxbridge up in university rankings

National Taiwan University falls in world reputation rankings

Lomonosov University Brings Prestige to Russia in World Ranking

Johns Hopkins Was Ranked One of the Best Universities in the World. Again.

Thai universities not recognised in worldwide ranking of institutes

Now an authority on cranberries

I just noticed that among the keywords that lead people to this blog are "asian sex diary" and "cranberry sciense paper".

The former probably has something to do with a post somewhere that mentioned gender bias in Asian universities, but cranberries?

The THE MENA Ranking: Interesting Results.

Times Higher Education (THE) has just published its ranking of universities in the MENA (Middle East and North Africa) region. It is, according to THE's rankings editor Phil Baty, "just a snapshot" of what MENA rankings might look like after consultation.with interested parties.

The ranking contains precisely one indicator, field-normalised citations, meaning that it is not the number of citations that matter but the number compared to the world average in specific fields. This was the flagship indicator in the THE world rankings and it is surprising that THE should continue using it in a region where it is inappropriate and produces extremely implausible results.

Number one in MENA is Texas A & M University at Qatar. This is basically an engineering school, evidently of a very high quality, and it is not clear whether is a genuinely independent institution. It offers undergraduate courses in engineering and has master's programmes in chemical engineering. Its output of research is meagre, as THE obligingly indicates in its press release.

How then did it get to the top of a research impact ranking? Easily. One of its faculty, with a joint appointment with the mother campus in Texas, is one of the collaborators on a multi-contributor paper emanating from CERN. I will leave it somebody else to count the number of contributors.

Another CERN collaborator, Cadi Ayyad University in Morocco is in sixth place. King Abdulaziz University is third.

There are ten Egyptian universities in the top thirty, including Alexandria but not Cairo.

Slipping down the curve

The ETS Center for Research on Human Capital and Education has produced an analysis of the performance of American millenials (young adults born after 1980 and aged 16-34 at the time of assessment) on the  Programme for the International Assessment of Adult Competencies (PIAAC) conducted by the OECD. The analysis  may be over-optimistic in places but in general it is a devastating forecast of a coming crisis for American higher education and very probably for American society.

American millenials are by historical and international standards well educated, at least in terms of the number of years of schooling but also, on average  less literate, less numerate and less able to solve problems than their international counterparts. To be blunt, they appear to be relatively less intelligent.

Let's start with the literacy scores for adults aged 16 to 65 tested by PIAAC, that is basically the adults making up the current work force.

The  average score for OECD countries is 273. There is nothing unusual at the top -- Japan 296, Finland 288, the Netherlands 284.

The score for the  USA is 270,  just below average.and  better than six OECD countries. Overall the USA is at the moment  mediocre compared with other developed nations.

Turning to numeracy, the OECD average is 269.  Once again the top is dominated by East Asia and the shores of the Baltic and North Seas: Japan (288), Finland (282), Flanders (280), the Netherlands (280). The USA at 253 is well below average.  Only Italy and Spain have lower scores.

For problem solving in technology-rich environments, the USA, with a score of 277 is again below the OECD average of 283.

This is the current work force, below average for literacy and problem solving, well below average for numeracy. It includes many who will soon die or retire and will be replaced by the millenial and post-millennial generations.

Take a look at the millenials. The gap is widening. For literacy the 6 point gap between the OECD average and the USA for 16-65 year olds becomes 8 points for the millennials.

For numeracy, the 13 point gap for 16 -65 year olds has become 21 points for the millenials and for problem solving 6 points becomes 9.

The situation becomes bleaker when we look at those who fail to meet minimum proficiency standards. Fifty percent of US milllenials score below literacy level 3,  64% below numeracy level 3, figures exceeded only by Spain, and 56% below level 2 proficiency in problem solving, the worst among developed countries reporting data.

Nor is there any hope that there may be a recovery from the younger section of the cohort, those aged between 16 and 24. The literacy gap remains the same at eight points but the numeracy and problem solving gaps each increase by an additional point.

The report also emphasises the large and increasing gap between the high and low skilled. Here there is a big danger. A gap can be closed from two ends and in the US it is easy to drag down high achievers by curtailing Advanced Placement programs, grade inflation, removal of cognitive content from college courses, group projects, holistic admissions and assessment and so on. The problem is that the closing of domestic gaps in this way just widens the international gap.

Free Speech Ranking

As the First Armoured Division made its way across Libya towards Tunisia at the end of 1942 and in early 1943, the troops were kept busy with early morning PT, lectures on "my county" and "the Everest expedition" and debates on things like "the Channel Tunnel would be a benefit". In his diary, my father, then a humble signalman, recounted another debate on whether "permanent conscription is a national asset: "Horace as usual made a vociferous speech and he said, ' There, bang goes my two stripes'"    

The spectacle of soldiers in the middle of a war  arguing against government policy with no more penalty than forfeiting two stripes -- if, in fact, Horace ever did lose them -- sounds slightly surreal today. Especially so, now that western schools, universities and other organisations appear to be becoming more and more hostile to "dangerous" ideas, a category that seems to be expanding relentlessly.

The British online magazine Spiked has just published its first Free Speech University Rankings, which are worth reading in detail.

These are actually ratings, not rankings, and divide universities into three categories:

• Red: has actively banned and censored ideas on campus
• Amber: has chilled free speech through intervention
• Green: has a hand-off approach to free speech.

Only 23 universities fall into the Green category and of those only three -- Exeter, Southampton and York are in the Russell Group and only three -- South Bank, Metropolitan and City Universities  -- in London.

Just a few examples:

Birkbeck University Students Union has apparently banned UKIP, because "homophobia, Islamophobia, disablism, xenophobia, misogyny, racism, fascism, and general discrimination [sic!] is rife amongst its members, supporters, officials, and prospective candidates".  If that wasn't bad enough, "John Sullivan, UKIP candidate for Forest of Dean and West Gloucestershire, said that regular physical exercise for boys released tension and thus avoided homosexuality."

The University of East London Students Union has banned materials opposing unrestricted abortion because "any material displayed in the Union building should adhere to the principle of ‘safe space’ and which resolves to ‘ensure an accessible environment in which every student feels comfortable, safe and able to get involved in all aspects of the organisation free from intimidation or judgement".

The University of Warwick noting the protected characteristics of "age, disability, gender reassignment, race, religion or belief, sex, sexual orientation, pregnancy and maternity and marriage or civil partnership" prohibits "displaying material that is likely to cause offence to others" or "spreading malicious rumours or insulting someone."

Affiliation in the News Again

Haaretz has published a story about Ariel University, in the occupied West Bank, suggesting that it is offering to pay researchers for adding its name to papers and grant proposals.

The report may be biased and the offer, which seems to apply to only one field, is probably an attempt to get round local and international ostracism. It is a much less blatant attempt to buy affiliations and therefore citations than the wholesale distribution of part time contracts by King Abdulaziz University in Jeddah to researchers on the Thomson Reuters Highly Cited lists.

Another case of affiliation abuse was that of Mohamed El Naschie, formerly editor of the journal Chaos, Solitons & Fractals, and writer of many articles that were cited frequently by himself and a few friends. El Naschie was also fond of giving himself affiliations that had little or no substance: Cambridge where he was a Visiting Scholar, allowed to use the library and other facilities, the University of Surrey for no discernible reason, and Alexandria University with which he had a tenuous connection.

Most of El Naschie's affiliation did not mean very much.  Cambridge was getting lots of citations anyway and did not need him. But Alexandria University produced a modest amount of research and El Naschie's self-citations went a long way and took Alexandria into the top 200 of the THE 2010 world university rankings.

This sort of thing is likely to continue, especially since there is now a stream of papers and reviews in physics and sometimes in medicine and genetics that have hundreds  of contributors and scores of contributing institutions. A part-time contract with a contributor to the Review of Particle Physics that includes adding the institution as a secondary affiliation could give an enormous boost to citation counts, especially if they are field and year- normalised..

It would be a good idea for academic editors and publishers to review their policies about the listing of affiliations. Perhaps second (or more) affiliations should only be allowed if documentary evidence of a significant connection is provided.

Likewise rankers ought to think about not counting secondary affiliations, as Shanghai Center for World Class Universities did last year, or giving them a reduced weighting.

Ranking Universities is a Really Serious Business

It seems that Webometrics has been hacked. Let's hope the problem is sorted out soon.


Ranking Web of Universities was attacked by external hackers. They did published hate messages and they had access to the Ranking, changing significantly the rank of at least one university and altering the structure and arrangement of the system. We are trying to fix the problems and sincerely apologize for any inconvenience. We hope to be able to be back in a few days with very exciting news and updated information. Thanks for your patience.

New York Times said it was a really big gap closing. It didn't seem so big then.

[Apologies to Bob Dylan, 'Talkin' New York']

The world of education is obsessed with gaps. Every time the PISA results come out there is renewed concern about the stubborn and growing gap between the United States and some Asian and Eastern European countries although the US should perhaps be congratulated for every large ethnic group doing as well or better than its international counterparts.

At the same time, there is recurrent anguish over the failure of African Americans and Hispanics to match the academic achievements of Whites and (East?) Asians.

According to the New York Times (NYT), the huge achievement gap between wealthy and poor American children is a major cause of the mediocre performance of the American economy. Just closing the American gap and going up a few points in the PISA rankings would apparently boost the economy significantly and create billions of tax revenues.

So how to do this? The NYT reports that a recent study by the Washington Center for Equitable Growth claims that things like more early childhood education, reducing lead paint exposure and letting students sleep a bit more will do the trick.

And has anyone managed to close the gap? Yes, according to the study, Montgomery County in Maryland, an affluent, racially mixed county near Washington DC, 

 "was able to reduce the gap and increase scores after instituting all-day kindergarten programs, reducing class size, investing in teacher development and reducing housing -based segregation in its schools and a host of other reforms, Montgomery County, Maryland was successful in both improving average achievement test scores and reducing achievement gaps. The percentage of 5th graders reading at or above the proficient level on the Maryland State Assessment rose for all racial and ethnic groups between 2003 and 2009. In addition, gaps between the disproportionately lower-income black and Hispanic students and the disproportionately higher-income white and Asian students narrowed.”

So we should all go to Montgomery County to find out how to close the gap, bring America up to OECD or even Finnish or Korean standards and achieve Chinese rates of economic growth?

Perhaps not.

The good news from Montgomery was a little surprising because I was sure that I had read a story that painted a rather less cheerful picture of the school system there.

Here it is. From the Washington Post of March 12, 2013. 'In Montgomery schools, the achievement gap widens in some areas', by Donna St George

"The achievement gap that separates white and Asian students from black and Latino students has grown wider in Montgomery County in several measures of academic success, according to a report released Tuesday."

“The 130-page report points to progress in five of 11 performance indicators in recent years. The school system improved on gaps in school readiness and high school graduation, for example. But disparities widened in advanced-level scores for state math exams in third, fifth and eighth grades. There were mixed results in two categories.”

Locals were baffled how the school system could spend so much money and still do so badly.

' “We still rank as one of the top spenders nationally in education, and then to lose ground is extremely concerning,” said Council Vice President Craig Rice (D-Upcounty), who called for more urgency. “It just boggles my mind that this can be so far below the radar.” '

But evidently the preferred solution is more money.

Montgomery Superintendent Joshua P. Starr said he agrees with most of the analysis. He wrote,

  “much of the $10 million the school system is seeking above mandatory funding levels in its budget proposal would help address achievement disparities, including 30 “focus” teachers to reduce class sizes in English and math at middle and high schools where students are struggling."

Anyway, here are some extracts from the report from Montgomery County itself.

“This report finds that since 2008 MCPS has made progress, but significant achievement gaps remain, particularly among measures of at-risk academic performance. Over the same period, MCPS also lost ground in narrowing the achievement gap among several measures of above grade level performance that align with MCPS’ Seven Keys initiative and the Common Core State Standards.”

and

“MCPS narrowed the achievement gap across five measures: school readiness, MSA proficiency, suspensions, academic ineligibility, and graduation rates. These gaps narrowed by increasing the performance of most subgroups while accelerating the performance of the lowest performing subgroups."

and

“MCPS achieved mixed or no progress in narrowing the gap on two measures: dropout rates and completion of USM or CTE program requirements among graduates. For these two measures, MCPS tended to narrow the gap by race and ethnicity, but did not achieve the same progress among service groups. “

and

“MCPS’ achievement gap widened across four measures: MSA advanced scores, Algebra 1 completion by Grade 8 with C or higher, AP/IB performance, and SAT/ACT performance. Among these four measures of above grade level performance that align with MCPS’ Seven Keys, high performing subgroups made greater gains on these benchmarks than low performing subgroups, thus widening the gap. More specifically: • The MSA Advanced Gaps in Grade 3 narrowed across most subgroups for reading by 2-7% but widened for math by 5-33% from 2007 to 2012; the Grade 5 gaps narrowed across most subgroups for reading by 2-16% but widened for math by 3-37%, and the Grade 8 gaps widened for both reading and math by 9-56%. • The Algebra 1 by Grade 8 with C or Higher Gap widened by 7-19% by race, ethnicity, special education, and FARMS status from 2010 to 2012, but narrowed by 7% by ESOL status. • The AP/IB Performance Gap among graduates widened by 6-37% by race, ethnicity, and service group status from 2007 to 2012. • The SAT/ACT Performance Gap among graduates held constant by special education and ESOL status from 2010 to 2012, but increased by race, ethnicity, and income by 3-6%."

So. Montgomery County reduced the gap for grade 5 reading for most subgroups but it widened for math. By Grade 8 it widened for reading and math, as did the AP/IB performance gap and the SAT/ACT gap for most groups.

When we get to the Center for Equitable growth and the New York Times, only the grade 5 reading improvement remains and the  failures in other areas have disappeared. How very careless of them.

Quality and Bias in University Rankings

I have just finished reading a very interesting unpublished paper, 'Measuring University Quality' by Christopher Claassen of the University of Essex.

He finds that all the major international rankings tap to some extent an underlying unidimensional trait of university quality and that this is measured more accurately by the US News Best Global Universities, the Center for World University Rankings (Jeddah) and the Academic Ranking of World Universities (Shanghai).

He also finds that these rankings are not biased towards their home countries, in contrast to the Times Higher Education, QS and Webometrics rankings.

Who says university isn't worth it?

In Malaysia it might be.

According to a local blog the customary dowry ("hantaran," distinct from the religiously sanctioned "mas kahwin" which is very modest) paid to the family of the bride varies significantly according to the bride's level of education.

For a woman with UPSR ( primary school certificate) it is 2-4,000 Ringgit.
For SPM (secondary school certificate) holders it is 4-8,000 Ringgit.
For STPM holders (equivalent to 'A' levels) it is 8-12,000 Ringgit.
For degree holders it is  12-15,000 Ringgit.
For master's holders it is  15-20,000 Ringgit.
For Ph Ds it is 20-30,000 Ringgit.

As far as I know, there is no premium for international universities or for those with a high place in the global rankings. Yet.

THE Under Fire in the Far North and Down Under

Times Higher Education  (THE) has been attacked for the latest spin-off from its world rankings. Alex Usher of Higher Education Strategy Associates has tweeted "your "most international unis" rankings lack even the barest face validity. Galway more intl than Harvard? C'mon"

The higher education editor of the Australian, Julie Hare, reports that Australian observers are surprised that Monash university, reputed to be the most international Australian university, has been ranked so low and quotes a comment at THE: "What cretin can assert that LSE and Cambridge are less "international" than Brunel and Canterbury?"

I wonder if there will be similar comments on THE's preview, in advance of a summit in Qatar, of its forthcoming MENA rankings. This is the top five of a research impact indicator based on field and year normalised citations. The first place goes to Texas A and M University Qatar. The other four are the Lebanese American University, King Abdulaziz University, Jeddah, Qatar University and the American University of Beirut.

In case you are wondering, Texas A and M Qatar does have someone on  the Large Hadron Collider project. 

The Most International Universities

Times Higher Education has released its list of the top 100 most international universities. This is simply the International Outlook indicator extracted from last year's world rankings. It counts the proportion of international staff and faculty and the percentage of papers with  international collaborators.

The top three are in Switzerland. The National University of Singapore is fourth and Ecole Polytechnique in Paris fifth. The Ecole is up from 28th place last year which probably means a "clarification of data" of some sort.

So what makes a university international?

It helps a lot to be located in an English speaking country. The UK, and Australia get high scores.

What is more noticeable is that many  small countries to do well especially if they are located next to a big country with the the same or similar language and culture -- Switzerland, Singapore, Macau, Hong Kong, Austria, Denmark,  Big countries like Mainland China, the USA and India do not.

Perhaps THE (and QS) should think about the implications of its methods. Does it make sense to count as international a student who moves a few miles from Fermanagh to Galway, Bavaria to Austria or Johor to Singapore?

Perhaps the THE should count the whole of the EU as a single country or give extra points for students and faculty who cross an ocean rather than an increasingly meaningless line on a map. Perhaps also, Macau and Hong Kong should be reunited methodologically with the Mainland. What about counting out of state students at US universities?

More University Mission Creep

Demands on western universities continue to increase and so do calls for more and more indicators in national and global rankings. Universities, it seems, are underemployed if they just provide instruction in academic, professional and technical subjects and promote research and scholarship.

Now they are supposed to support diversity and inclusiveness, build character, grit and resilience, promote anti-racism, combat sexism, homophobia, cisgender normativity and weightism, boycott Israel, reward students for overcoming adversity, engage with communities, combat terrorism, transform lives, provide gender free bathrooms, sponsor near-professional level sports teams, boycott fossil fuels, make everybody safe and comfortable except for those whose privilege needs continued confrontation.

All this is now spilling over into the rankings business. We have already seen Universitas 21, which ranks national university systems, give countries a score for the number of female students and faculty.and there have been repeated proposals  that the US News law school rankings should include faculty  and student diversity among their criteria.

US News has published a diversity index that consists simply of calculating the percentage of minority students. First place goes to Cornell, followed by the University of Hawaii - Manoa, Whitter College, California, the University of the District of Columbia and Nova Southeastern University in Florida. A quick calculation of the correlation between the diversity index and overall scores in the law school rankings shows no significant relationship between diversity so defined and overall quality as measured by the rankings.

To incorporate such an index into the law school rankings would be a pointless exercise.  If Nova Southeastern Unversity Law School, which accepts nearly half of those who apply and a third of whose graduates are not employed nine months after graduation, were to get a high ranking, this would be seriously misleading for everybody.

The US federal government is now proposing to rate colleges according to their admission of low income and first generation students, affordability, and outcomes such as graduation rates, graduate employment and entry into postgraduate programs. The problem here is that in the US and  almost everywhere else these objectives are mutually exclusive. Low income students are, on average, likely to be less academically capable, and that means, if academic standards remain unchanged, that fewer will graduate and fewer will go on to graduate study.

If the ratings plan ever happens the likeliest consequence of the colliding demands is that it will become much easier to get a degree or get into graduate school. There are dozens of ways in which academic standards can be eroded, most of which we have seen already somewhere.

Another kind of creep is the rising chorus that universities should encourage and promote civic engagement, a rather slippery concept that is difficult to describe but covers  a variety of worthy activities reaching beyond the campus such as promoting local economic development, employing women and minority groups, helping poor students succeed, buying local products and encouraging students to be volunteer teachers, . A recent conference in South Africa ended with a call for action that included a proposal that rankings should take account of such activities.

Adam Habib, Vice-Chancellor of the University of the Witwatersrand, even proposed to boycott rankings unless they included civic engagement as an indicator.

"Gather a group of universities and tell the rankings that you'll collectively withdraw if they don't take in civic engagement in the future. I guarantee that every one of them will listen".

But why should universities be required to do what other institutions have failed to do even though they are far better qualified. If entrepreneurs cannot promote economic growth, revolutionary parties cannot achieve social justice, trade unions cannot help the poor then just why should should universities be expected to do so?

Part of the drive for new indicators is probably rooted in the realisation that universities are losing much of their reason for existence. Research is increasingly done by specialised institutes, companies and hospitals, for- profit organisations offer no frills instruction at low prices, online learning is replacing traditional seminars and lectures. Civic engagement looks like the new quality audit, a way of keeping busy those who are reluctant to teach or research.

If all the demands for new indicators are met we will end up with hugely bloated rankings that fail to make any meaningful distinctions.

New Resource from IREG

A new resource for anyone interested in university rankings is available at the International Rankings Experts Group (IREG) site.

The Inventory of National Rankings has been prepared by the Perspektywy Education Foundation, Poland, and provides basic data about a variety of national ranking systems. Two of them have been approved by IREG.

US Federal Ratings Plan: A Few Answers, More Questions

The US Department of Education has just revealed the progress that it has made towards its planned ratings for colleges and universities. There has been over a year of public discussion since the Obama administration announced that it was planning on introducing a new system. Unfortunately, it seems that there is still a long way to go before a final product emerges and the administration’s forecast of a launch in August or September 2015 may be too optimistic.

Since the 1980s, the US News& World Report’s ‘America’s Best Colleges’ has been followed avidly by students and other stakeholders. These rankings have been criticised, sometimes with justification, but they do provide a reasonably accurate guide to some of the things that students, parents, employers and counsellors want to know: how likely a student is to graduate once admitted, the typical academic ability of fellow students, reputation among peers, resources available for teaching.

There is, of course, much that the US News rankings do not tell us. The international rankings produced by Shanghai Jiao Tong University’s Center for World-Class Universities, Quacquarelli Symonds (QS), Times Higher Education and now US News with its Best Global Universities are probably even more limited since they focus largely or entirely on research and postgraduate training. There is also a widespread feeling that existing rankings are unfair to schools that try to educate students from non-traditional backgrounds or underrepresented groups.

The demand for more information and for greater accountability comes when American universities are entering a time of increasing pressure and constraint. Costs are rising inexorably, even though many students are taught not by hugely expansive superstar professors but by poorly paid adjuncts and untrained graduate assistants. Many students graduate late or not at all and incur a large and growing debt burden from which bankruptcy rarely provides an escape. Meanwhile the more reliable global university rankings show American universities steadily losing ground to Asian institutions.

Many colleges and universities are facing a death spiral as stagnant or declining admissions lead to a fall in the number of graduates which in turn erodes reputation and undermines alumni contributions. Underlying everything is the grim reality that the overall quality of graduate of American high schools is apparently insufficient to supply colleges and universities with students capable of completing a degree within a reasonable length of time.

The federal government has become increasingly concerned over these trends and the failure of American higher education to provide a route to secure employment and middle class status. The new plan had its origins in a speech by President Obama at the University at Buffalo: SUNY in August 2013.

A succession of hearings and forums has been held and finally the Department of Education has come out with a draft framework. The department has indicated that it will publish ratings, not rankings, so that colleges and universities will be divided into three categories, high performers, low performers and those in between. Two year institutions such as community colleges and four year colleges and universities will be assessed separately and institutions that teach only postgraduates or do not grant degrees will not be included. The main source of data will be information collected by the federal government.

According to the document, ‘For Public feedback: A College Ratings Framework’, the objectives are to help colleges measure and make progress towards the objectives of access, affordability and outcomes, to provide information for students and families, and to help the government ensure that financial aid is well used.

The department has announced the indicators it is thinking about using. These include enrolment of low income and first generation students, family income levels and the average net price of an institution. Student outcomes could be measured by completion rates, transfer rates and the number of students going on to graduate school. More details can be found in ‘A New System of College Ratings– Invitation to Comment’.

Several questions remain unanswered. A rating consisting of three categories may be too crude. There will almost certainly be a large gap between those at the top of the high performing group and those on the edge of the intermediate category. Membership of the same large group will not help anyone trying to compare two universities. A small change in one or two indicators might push colleges out of the intermediate group into the underperformers where they could suffer financial sanctions.

If the department provides the scores or raw data for each indicator then it would be relatively simple for analysts or journalists to calculate numerical rankings.

The most disquieting thing about this document is that the department seems to have given little thought about how easy it would be to game much of the data. There are, for example, dozens of ways in which colleges and universities could increase the number of students who graduate on time, even if it means undermining the quality of their degrees and their value to potential employers. Rating institutions according to the repayment of student loans might encourage universities to close humanities and social science departments while coaxing students into programs for which they might not be suited.

It is likely that there will be more arguments and discussion before the ratings are launched and it remains to be seen how much credibility they will have when they do appear.

The Times Higher Education BRICS Plus Rankings 2014-15

The Times Higher Education BRICS Plus Rankings 2014-15 has been published in Study International.

Study International

Study International is a new web site published by Hybrid News that will provide news, analysis and advice about international higher education. Some posts from this blog have been republished there, See for example.

Is Asia really rising?

Over the last few years there has been a lot of talk about the continuing rise of Asian universities. In The Conversation, Gerard Postiglione of the University of Hong Kong has pointed out that Asian universities now take one in eight of the top 200 places in the Times Higher Education World University Rankings and he predicts that by 2040 a quarter of the top universities will be Asian.

It is interesting that he apparently regards the THE rankings as the arbiter of excellence despite an eccentric methodology that, among other oddities, claims that an excellent but small research institute is the best university in Italy and among the best in the world.

Exactly what progress in the THE rankings means is difficult to decide. A rise in the score for the research indicator, for example, could result from an increase in the number of publications, a fall in the number of academic and/or research staff, an increase in research income, an improvement in the research reputation survey or a combination of some or all of these.

Predicting what will happen in the THE rankings has become even more difficult since THE broke up with their data supplier, Thomson Reuters, raising the possibility that there will be another round of methodological changes.

There are some sceptics such as Alex Usher of Higher Education Strategy Associates but in general the rise of Asia and the decline of the US and UK seems to have become part of the accepted wisdom of Western pontificators.

So is Asia rising? And if it is, is it the whole of the continent or just parts of it?

The problem is that the rankings vary in their ability to identify medium term trends. QS and THE give a large weighting to reputation surveys that are inherently volatile,They also use an unstable number of institutions to generate means from which processed scores are calculated and this can led to fluctuations in the final overall scores

The Academic Ranking of World Universities (ARWU) produced by the Shanghai Center for World-Class Universities is probably the most useful for identifying changes over the last decade since there were no significant changes between 2004 (when schools with strengths in the social sciences  were helped by exemption from the Nature and Science indicator) and 2014 (when Thomson Reuters issued a new list of highly cited researchers).

The number of universities in the Shanghai top 500 provides strong evidence that some parts of Asia are making rapid progress. The number of mainland Chinese universities (excluding Hong Kong and Taiwan) has risen from 8 to 33, The number of Korean universities has gone from eight to ten, Taiwanese from three to six, Saudi from zero to four, and Malaysian from zero to two.

But some parts of Asia appear to be stagnant or in relative decline. The number of Japanese universities has fallen from 36 to 19 and Indian from three to one while the number of Hong Kong institutions has remained the same at five.

Looking at the performance of some national flagships in the ARWU publications indicator provides more evidence of an expansion of research in some Asian countries. Compared to Harvard's benchmark score of 100, Peking University has risen from 49.8 in 2004 to 63.6 in 2014. Other Asian universities have also had substantial  growth over the decade. Seoul National University went from 62.6 to 67.8, National Taiwan University from 52.6 to 57.9 and Istanbul University from 30.7 to 34.9.

Note that the raw numbers of publications have been modified by a logarithm so that in 2004 Peking was in fact publishing about a quarter of the number of papers produced by Harvard rather than a half.

On the other hand, Tokyo University, which in 2004 had the second highest publications score in the world, fell from 91.9 to 73 and the University of Hong Kong from 46.4 to 44.

If we look at Shanghai's Productivity per Capita indicator, which measures quality by dividing  five combined indicators by the number of faculty, we find some Asian universities doing well. Peking goes from 5.9 to 16.5, Seoul National University from 19 to 23.4 and National Taiwan University from 17.5 to 19.9. Tokyo, meanwhile, has fallen from 49.8 to 29.2. Hong Kong University, on the other hand,has risen from 13.1 to 22.4.

Confirmation of the trends for research output comes from the Output indicator in the Scimago rankings, which is based on data provided by Scopus. Peking, Seoul National University, National Taiwan University and Universiti Malaya rose between 2009 and 2014. However, the scores for Tokyo and Hong Kong both fell.

On the other hand, the evidence of Scimago' normalised impact indicator, which might measure research quality, shows Peking rising but Seoul National University and Hong Kong falling.

It would seem that China and the overseas Chinese communities and Korea are expanding the quantity of research but progress at higher levels is slower. There are also islands of research productivity in West and Southeast Asia.  In Central Asia, the Indian Subcontinent and Indonesia there is very little significant research activity while Japan is actually declining.

One Way of Rising in the Rankings

One way of rising in the rankings is to amalgamate. Watch out for Paris-Saclay in next year's Shanghai rankings.

From BBC Online News:

"Dominique Vernay, the president of this new university, says that within a decade he wants Paris-Saclay to be among the top ranking world universities.

"My goal is to be a top 10 institution," he says. In Europe, he wants Paris-Saclay to be in the "top two or three".

In university rankings, big is beautiful, and the Paris-Saclay will have 70,000 students and 10,000 researchers. There will be an emphasis on graduate courses and recruiting more international students and staff.

The idea of bringing together individual colleges into a "federal university" has been borrowed from the UK.

"Our model isn't that far from the Oxbridge model," he says.

To put it into scale, Mr Vernay says Paris-Saclay is going to be twice the size of the University of California, Berkeley, one of the flagships of the US university system."

Ranking Status Wars 4

Meanwhile in Saudi Arabia, scholarships granted under the Custodian of the Two Holy Mosques Programme will go to students at 200 scientific universities chosen from the Big Four rankings, US News, Times Higher Education, QS and Shanghai.