Outcomes-based Education Watch

Outcomes-based Education (OBE) is sweeping across Australia, the UK, Malaysia and other countries. I am sure that OBE is not the whole story, but if this news from South Africa is any guide the results are likely to be very negative.

"South African vice-chancellors warned the government last week to expect more students to drop out, as the shocking results of pilot national benchmark tests revealed that only 7% of first-year students are proficient in mathematics, only a quarter are fully quantitatively literate and fewer than half have the academic literacy skills needed to succeed without support.

.............


SOUTH AFRICA: Shocking results from university tests
Karen MacGregor16 August 2009 Issue: 0035
South African vice-chancellors warned the government last week to expect more students to drop out, as the shocking results of pilot national benchmark tests revealed that only 7% of first-year students are proficient in mathematics, only a quarter are fully quantitatively literate and fewer than half have the academic literacy skills needed to succeed without support."The challenge faced by higher education institutions in relation to mathematics is clearly enormous," according to a draft report produced for the vice-chancellors' association Higher Education South Africa (HESA) by the National Benchmark Tests Project."With the current emphasis on the production of graduates in scarce skills areas such as engineering and science, the need for curriculum responsiveness and remediation in this area is urgent," said the report, obtained by University World News, which is still to be considered by HESA.Last week HESA chairman, Professor Theuns Eloff, told parliament's higher education committee that most first-year students could not adequately read, write or comprehend - and universities that conduct regular competency tests have reported a decline in standards.While undergraduate enrolments had been growing by about 5% a year, and black students now comprised 63% of enrolment, there was concern about high drop out (around 50%) and low graduation rates, especially among black students. Only a third of students obtain their degrees within five years.HESA's findings from the benchmark project make it clear that South Africa's school system is continuing to fail its pupils and the country, and that universities will need to do a lot more to tackle what appear to be growing proficiency gaps.One reason for declining educational performance, Eloff argued, was flaws in the country's outcomes based education system. "You don't learn to spell and comprehend, and that's nonsense," he said. The Times newspaper commented: "So far, the only outcome from the outcomes-based education system is university students who can't read and write." "

The Discipline of Rankings

This is the title of an article by Michael Sauder and Wendy Nelson Espeland in the American Sociological Review. The Foucaultian jargon is not to my taste but underneath it there is a sensible and data-backed article on the prevasive and negative effects of rankings on US law schools.

Here is the abstract:


"The Discipline of Rankings: Tight
Coupling and Organizational Change

Michael Sauder Wendy Nelson Espeland
University of Iowa Northwestern University

This article demonstrates the value of Foucault’s conception of discipline for
understanding organizational responses to rankings. Using a case study of law schools,
we explain why rankings have permeated law schools so extensively and why these
organizations have been unable to buffer these institutional pressures. Foucault’s
depiction of two important processes, surveillance and normalization, show how
rankings change perceptions of legal education through both coercive and seductive
means. This approach advances organizational theory by highlighting conditions that
affect the prevalence and effectiveness of buffering. Decoupling is not determined solely
by the external enforcement of institutional pressures or the capacity of organizational
actors to buffer or hide some activities. Members’ tendency to internalize these
pressures, to become self-disciplining, is also salient. Internalization is fostered by the
anxiety that rankings produce, by their allure for the administrators who try to
manipulate them, and by the resistance they provoke. Rankings are just one example of
the public measures of performance that are becoming increasingly influential in many
institutional environments, and understanding how organizations respond to these
measures is a crucial task for scholars.

A Case of Dyslexia

The nomination of Sonia Sotomayor to the Supreme Court of the United States has focused attention on two cases that came before her, both involving people afflicted with dyslexia.

The better known of these is that of Frank Ricci, a New Haven, Connecticut fireman denied promotion because not enough members of racial minorities were able to pass the firefighters’ test along with him.

The other involved Marilyn Bartlett who wanted to be a lawyer. Before attempting to switch professions she had a notably successful academic career, earning her first degree in Education from the State College at Worcester, Massachusetts, her master’s in Special Education from Boston University (84th in the 2008 Shanghai rankings) and a Ph.D. in Educational Administration from New York University (32nd). She has taught English in Germany and has been an Associate Professor of Educational Leadership at the University of South Florida (201-302) and Director of Graduate Studies as well as Chair and Associate Professor in the Department of Educational Leadership and Technology at the New York Institute of Technology. She is now Dean of the School of Education at Texas A & M University - Kingsville.

She has also been a law clerk, an assistant superintendent of schools and a special education coordinator

We are further told that:

“She has seven articles in progress and has published numerous articles in encyclopedias, proceedings, periodicals, book chapters and reports. Her first book – an examination of education law in Florida – is due to be published this fall.
In 2006, she received the Teaching Excellence Award from the College of Education at the University of Florida St. Petersburg and in 1999, Bartlett received the Lifetime Achievement Award presented by LD Access, a foundation focused on needs of learning disabled adults.
Bartlett is a member of the American Education Research Association, the American Association for School Administrators, the Educational Law Association and the National Council of Professors of Educational Administrators.”

However, three students on ratemyprofessors speak of her in less than glowing terms: “not very student friendly”, “horrible and incompetent” and "incompetent and not up to date on the educational needs".

After taking a degree in law at Vermont Law School, Bartlett took the New York bar exam four times and failed on each occasion. She then requested special accommodation because of a claimed disability, dyslexia. She was allowed to use a computer, to have an assistant to read the answers and to have 50 per cent extra time. However, she still failed.

The case then came before Sotomayer and evidence was presented that

“The effect of plaintiff's reading impairment on her life, even with all of her self-accommodations, is profound. Cf. 29 C.F.R. Pt. 1630, App. A § 1630.2(j) ("The determination of whether an individual has a disability is . . . based on . . . the effect of that impairment on the life of the individual."). Plaintiff has difficulty with tasks that most people perform effortlessly, including reading short e-mails, using a telephone directory or electronic database, writing a shopping list, or following a recipe. (Bartlett Aff. PP 11, 12, 13, 22.) Plaintiff generally avoids reading any unnecessary material and does not read for pleasure. (Bartlett Aff. PP9, 10, 14.) As plaintiff and her experts stated, plaintiff consistently tries to find alternative routes around reading. Dr. Hagin [*119] testified that based on her experience, plaintiff's "reading was more limited than the average person I might see, even the average person with a learning disability." (Tr. at. 163.) “

However, even with the accommodations mandated by Judge Sotomayor, Bartlett could not pass the bar exam and apparently has now stopped trying


I am surely not alone in wondering about the common sense involved in requiring somebody to demonstrate serious incompetence in a key professional skill so that they may be assisted to gain entrance to a profession. Perhaps a lawyer can explain why people should not be allowed to show extreme cowardice or pyrophobia to be fast tracked into a fire department or serious myopia to become an airline pilot.

What I am concerned with here is what the case says about basic academic standards at American schools of education.

I assume that Bartlett really is dyslexic, although faking is apparently not impossible, indeed not uncommon, and that the accommodations granted during graduate school and after were not a substitute for intelligence but devices necessary to allow it to function. (G. H. Harrison, M.J. Edwards, K. C. H. Parker, Identifying students feigning dyslexia: Preliminary findings and strategies fordetection. Dyslexia 14/3, 228-246)

We still have to explain how it is possible for someone who cannot acquire the basic knowledge or skills to enter the legal profession can not only complete a doctoral degree and therefore be certified as an authority on education but go on to become a recognized academic leader.

The only answer I can think of is that the minimum intellectual ability required to start a legal career are very much higher than those needed to rise to the top in education.

Commentary on Best Colleges 2010

Rankings Frenzy 09 in Inside Higher Ed

Elyse Ashburn in the Chronicle of higher Education

Bob Morse on Makiing Sure the Coillege Data Are Correct in Morse Code

Best Colleges 2010

The US News and WorldReport's Best Colleges 2010 is out. The top ten national universities are :

1 = Harvard and Princeton
3. Yale
4 = Caltech, MIT, Stanford, University of Pennsylvania
8 = Columbia, Chicago
10. Duke

The CCAP/Forbes Rankings

The CCAP (Center for College Affordability and Productivity)/Forbes rankings are rather different from the rest, being emphatically based on outcomes rather than spending.

One quarter of the weighting of these rankings is for student satisfaction, based on scores from the ratemyprofessors site, another quarter on graduate success derived from Who's Who in America and payscale.com, a quarter from current students success -- graduation rates and winners of national student awards, a fifth for the debt incurred by students and five per cent for faculty quality.

Richard Vedder the director of CCAP claims that the rankings are relatively difficult to manipulate. Up to a point this is true. I cannot see much that anyone could do about Who's Who. But if these rankings ever overtook the USNWR rankings there could well be a lot of fiddling with graduation rates and innovative financial aid packages .

Anyway, the overall top five are .

1. US Military Aacadamy
2. Princeton
3. Caltech
4. Williams College
5. Harvard


The top five best value colleges are

1. Berea College, Kentucky
2. New College of Florida
3. US Miltary Academy
4. US Air Force Academy
5. University of Wyoming


The top five national research universities are:

1. Princeton
2. Caltech
3. Harvard
4. Yale
5. Stanford

Only Three Days Left

This is from The Morse Code


"It's getting very close to the launch of the new America's Best Colleges rankings. The 2010 edition will be published on Thursday, August 20, which is the day the new rankings go live on our website. The site will have the most complete version of the rankings, tables, and lists, plus extensive profiles on each school. The America's Best Colleges website also will have wide-ranging interactivity as well as a newly upgraded search feature to enable students and parents to find the school that best fits their needs.
These exclusive rankings will also be published in the magazine's September 2009 issue and in our newsstand guidebook, both of which will go on sale around August 20. The main rankings include the national universities, liberal arts colleges, master's universities, and baccalaureate colleges by region. In addition, there will be one new ranking to show which schools have the greatest "commitment to undergraduate teaching." For the second year in row, we will publish the very popular list of "Up-and-Coming Institutions"—the colleges making innovative improvements. In addition, we will have our third annual ranking of Historically Black Colleges and Universities. "

Life Imitates Art

In 2005 Duke University made an impressive showing in the THES-QS World University Rankings largely because someone at Quacquarelli Symonds counted undergraduate students as faculty. (see post January 29, 2007)

Perhaps it was not really an error. It looks like at least one Duke professor is intent on handing over over her teaching duties to her students

Cathy Davidson, a Duke professor, has told us about her "innovative' grading policies.

"I loved returning to teaching last year after several years in administration . . . except for the grading. I can't think of a more meaningless, superficial, cynical way to evaluate learning in a class on new modes of digital thinking (including rethining [sic or perhaps not -- maybe she means making even less substantial] evaluation) than by assigning a grade. It turns learning (which should be a deep pleasure, setting up for a lifetime of curiosity) into a crass competition: how do I snag the highest grade for the least amount of work? how do I give the prof what she wants so I can get the A that I need for med school? That's the opposite of learning and curiosity, the opposite of everything I believe as a teacher, and is, quite frankly, a waste of my time and the students' time. There has to be a better way . . .

So, this year, when I teach "This Is Your Brain on the Internet," I'm trying out a new point system. Do all the work, you get an A. Don't need an A? Don't have time to do all the work? No problem. You can aim for and earn a B. There will be a chart. You do the assignment satisfactorily, you get the points. Add up the points, there's your grade. Clearcut. No guesswork. No second-guessing 'what the prof wants.' No gaming the system. Clearcut. Student is responsible.

And how to judge quality, you ask? Crowdsourcing. Since I already have structured my seminar (it worked brilliantly last year) so that two students lead us in every class, they can now also read all the class blogs (as they used to) and pass judgment on whether they are satisfactory. Thumbs up, thumbs down. If not, any student who wishes can revise. If you revise, you get the credit. End of story. Or, if you are too busy and want to skip it, no problem. It just means you'll have fewer ticks on the chart and will probably get the lower grade. No whining. It's clearcut and everyone knows the system from day one. (btw, every study of peer review among students shows that students perform at a higher level, and with more care, when they know they are being evaluated by their peers than when they know only the teacher and the TA will be grading). "

So, every class is led by two students. An A is awarded for showing up for class, doing the work and having it judged as satisfactory by classmates or revising it after being judged unsatisfactory.

If classes are led by students, who also chose the reading and writing assignments and evaluate class contributions, and work is graded by students, then what is Professor Davidson being paid for?

Another point. Professor Davidson claims that all studies show that students perform at a higher level when they know they are being evaluated by peers rather than only by a teacher and a teaching assistant. We could of course argue about whether every study shows this and what a higher level means. But note that the studies are comparing students graded by peers and, presumably, instructors with those graded only by teacher and TA. From what Professor Davidson tells us grading in her class is done only by students and therefore the results of such studies cannot be used to support her claims.

note -- acknowledgement to Durham-in Wonderland

The New Taiwan Rankings

The new Performance Ranking of Scientific papers for World Universities is out.

This is based on a variety of measures derived from Essential Science Indicators database. It is therefore more orientated towards quality than the THE-QS rankings which use the more comprehensive but less selective Scopus database.

These rankings may become more influential in the future so it might be worthwhile making a few comments. First, like nearly all rankings there is a bias towards the citation-heavy natural sciences. Second, it may be that the number of indicators, eight, is too many since some at least may simply be counting the same thing. Third, there is no attempt to measure anything other than research.

Still, the current rankings are important. Looking at the overall index, we find that Oxbridge and some of the Ivy League schools are definitely slipping, deprived of the support from the THE-QS academic survey and the aging or dead laureates of the Shanghai index. Cambridge is in 15th place, Yale 16th, Oxford 17th and Princeton 38th.

Here are the top five.

1. Harvard
2. Johns Hopkins
3. Stanford
4. University of Washington at Seattle
5. UCLA

I am wondering about the University of Washington, which is 16th in the Shanghai rankings and 59th in the THE-QS.

Selecting for the Supreme Court ... and Perhaps Universities

It took a while for me to decide that this article by David G. Savage in the San Francisco Chronicle was not a parody. It is nonetheless worth reading carefully. Much of it will sound familiar to those who are aware of the ongoing debate about how university students and faculty should be selected.

The article begins:

"Justice Sonia Sotomayor will bring something new to the U.S. Supreme Court, far beyond her being its first Latina member."

And what will she bring? Savage approvingly lists the attributes that will justify her appointment to the Supreme Court.

• She will be the only judge whose first language is not English.
• She is diabetic.
• She grew up in a housing project where drugs and crime were more common than "Ivy League scholarly success".
• Her SAT scores were not very good but she managed to graduate first in her class at Princeton.
• "[She] is also a divorced woman with no children but a close relationship with an extended family.
  "She is a modern woman with a nontraditional family," said Sylvia Lazos, a law professor at the University of Nevada at Las Vegas. "She is much more reflective of contemporary American society than the other justices, like Alito and Roberts."
  She was referring to Chief Justice John Roberts and Justice Samuel Alito, both of whom are married and have two children. The court soon is expected to face a series of cases involving the legal rights of other nontraditional families with gay and lesbian couples. "
• She has had trouble paying her mortgage and credit cards.
• She has been a prosecutor and a trial judge.
• She will be one of two minorities on the court, the other being Clarence Thomas, and the only one who supports Affirmative Action. Apparently Jews, Italians and WASPs are not minorities.

So Sotomayor is qualified for the highest judicial office in the United States because she is a speaker of English as a second language, a diabetic, not a good test taker but hard working, divorced, childless, a member of a recognised minority, a supporter of Affirmative Action and a poor financial manager.

The time will come, I suspect, when these will be essential qualifications for faculty positions in the US and elsewhere.

And will someone please explain to me why Sotomayor's childlessness is more reflective of contemporary American society than Roberts's and Alito's two children apiece. Or is Professor Lazos living in a parallel universe where the American fertility rate is zero?

Sharing the Pain

From the Hawaii Star-Bulletin

David Ross, chairman of the University of Hawaii-Manoa Faculty Senate's Executive Committee, claims that the university's ranking performance means that they should not have to take a pay cut.


"Recently we heard the good news that the University of Hawaii Foundation had raised $330 million in charitable donations over a six-year period. What got less press attention was that the UH faculty had raised over $400 million in grant support, not over six years but in a single year. At the same time we learned that top UH executives, who earn mainly at or above the national average, were taking voluntary pay cuts by up to 10 percent, while lower-level executives would be cut 6 percent to 7 percent. Meanwhile, UH faculty, who (despite some recent raises) still earn well below our colleagues at peer institutions, are being asked to take a 15 percent cut. ...

By many independent measures, UH-Manoa remains one of the great universities in the world. We're one of only 63 public universities in the country with the highest Carnegie Foundation classification. The best-known international ranking of universities ranks us as tied for 59th in the Western Hemisphere.

These rankings are based on the quality of our faculty and programs, not our buildings or athletic records. At this level UHM is in intense competition for the best faculty, grants and students. It is not a coincidence that our successes in recent years, academic and financial, have followed the rebuilding of our faculty, both in size and in salary. We are worried that
decisions being made right now by the state and the system will not only undo the recent progress we have made, but cause irreversible harm to our competitive standing. We are already losing faculty, and the cuts will make it di cult to recruit outstanding new faculty members and the research programs that they can develop. Since university rankings are based primarily on a faculty's reputation and grants, our hard-earned status as a research-extensive university could fall into jeopardy. "

Sad, but there's an army of adjuncts and underemployed Ph Ds out there who would work at those reduced salaries or less and who are just as or better qualified.

Russia Loves Rankings

From Newsweek

" Paradoxical as it seems, one of the former Soviet states with a more pragmatic approach is Russia itself. To be sure, the Kremlin's fear of a color revolution means that touchy political subjects aren't taught. And there are no U.S.-accredited universities or American academic programs. But Russian authorities have recently begun allowing universities to open up—even if that means greater exposure to outside ideas. Many Russian schools, for example, have started reviving academic exchanges with Western universities. Their motivation is simple: desperation. Last year, not a single Russian university made it into the top 100 of a world ranking put out by Quacquarelli Symonds, a U.S.-based compiler of international university standards. Even Moscow State University, the pride of Russia's education system, slid from 97th place in 2007 to 180th in 2008.

To stop the rot, last year Prime Minister Vladimir Putin founded two new universities, bankrolling them to the tune of $300 million. More important, "education policymakers gave a signal to Russian universities to quickly embrace all the most innovative international programs, and now nothing is stopping them from inviting or hiring as many U.S. professors as they can," says Andrei Volkov, an adviser to the minister of education and rector of Moscow's Skolkovo School of Management. Accordingly, Moscow University recently signed a cooperation deal with the State University of New York to share students and award joint diplomas, and 65 U.S. visiting professors are working in Moscow this year. Another joint agreement with the University of Southern California is due to be inked this fall. "

I wonder if somone should tell Putin that until last year three out of four university centers in the SUNY system were not even listed in the THE-QS rankings.

More Dumbing Down

This is from TES Connect via butterfiesandswheels

"Exams for an Evangelical Christian curriculum in which pupils have been taught that the Loch Ness monster disproves evolution and racial segregation is beneficial have been ruled equivalent to international A- levels by a UK government agency.

The National Recognition Information Centre (Naric), which guides universities and employers on the validity of different qualifications, has judged the International Certificate of Christian Education (ICCE) officially comparable to qualifications offered by the Cambridge International exam board.

Hundreds of teenagers at around 50 private Christian schools in Britain study for the certificates, as well as several home-educated students."

This is the interesting bit:


"Mrs Lewis [spokesperson for the International Certificate of Christian Education]had not noticed the Loch Ness monster claims, which she suggested may have been a “slip at the typewriter”, adding that the science curriculum had helped a student to gain a place to study natural sciences at Oxford University[4th in the world according to THE-QS."

Dumbed Down Degrees

This is from today's Observer

"Universities were yesterday embroiled in a furious row over dumbing down after a parliamentary inquiry revealed the number of first-class degrees had almost doubled in a decade. Amid the war of words, senior Tories vowed to publish data that they claimed would reveal the true value of degrees.

Different universities demand "different levels of effort" from students to get similar degrees, according to the report from the commons select committee on innovation, universities and skills, suggesting that top grades from some colleges were not worth the same as others. "

And what is the cause of this grade inflation?

"Gillian Evans, a lecturer in mediaeval theology at Oxford University and an expert in university regulation, attributed the rise to universities' desire to move up published league tables.
"I am quite sure the reason proportions have gone up is exactly the same as the reasons A-levels have gone up: it's straightforward grade inflation, chasing a place in league tables," she said."

Highlights from the New Webometrics Rankings

Top USA and Canada

1. MIT
2. Harvard
3. Stanford
4. Berkeley
5. Cornell


Top Europe

1. Cambridge
2. Oxford
3. Federal Institute of Technology ETH Zurich
4. University College London
5. University of Helsinki

Top Oceana

1. Australian National University
2. University of Queensland
3. Monash University
4. University of Melbourne
5. University of Sydney

Top South East Asia

1. National University of Singapore
2. Prince of Songkhla University
3. Chulalongkorn University
4. Kasetsart Universiy
5. Mahidol University

Top Arab World

1. King Saud University
2. King Fahd University of Petroleum and Minerals
3. Imam Muhamed bin Saud University
4. King Faisal University
5. King Abdulaziz university

Webometrics Rankings

This is from Ranking Web of Universities

"The July edition of the Ranking Web of World Universities (http://www.webometrics.info) shows important news. Most of them are due to changes done to improve the academic impact of the open web contents and to reduce the geographical bias of search engines. As a result, the US universities still lead the Ranking (MIT with its huge Open Courseware is again the first, followed by Harvard, Stanford and Berkeley), but the digital gap with their European counterparts (Cambridge and Oxford are in the region’s top) has been reduced. Even more important, some of the developing countries institutions reach high ranks, especially in Latin America where the University of Sao Paulo (38th) and UNAM (44th) benefits from the increasingly interconnected Brazilian and Mexican academic webspaces.Several countries improves their performance including Taiwan and Saudi Arabia with strong web oriented strategies, Czech Republic (Charles), the leader for Eastern Europe, Spain (Complutense) and Portugal (Minho, Porto) with huge repositories and strong Open Access initiatives. Norway (NTNU, Oslo), Egypt could be also mentioned.On the other side, the underrated are headed by France, with a very fragmented system, Korea, whose student-oriented websites are frequently duplicated, New Zealand, India or Argentina.Africa is still monopolized by South African universities (Cape Town is the first, 405th), as well as Australian Universities are the best ranked for Oceania (Australian National University, 77th)Other well performing institutions include Cornell or Caltech in the USA, Tokyo (24th) Toronto (28th), Hong Kong (91st), or Peking (104th). On the contrary, in positions below expected we find Yale, Princeton, Saint Petersburg, Seoul and the Indian Institutes of Science or Technology."

Ranking Schools by Salary

The Payscale site has produced a ranking of American schools and colleges by the salaries that its graduates earn.

Here are the top five engineering colleges by median mid-career salary:

1. MIT
2. Harvey Mudd
3. Stanford
4. Bucknell University
5. Rensselaer Polytechnic Institute

the top five Ivy League schools;

1. Dartmouth College
2. Harvard
3. Princeton
4. Yale
5. University of Pennsylvania


the top five liberal arts colleges;

1. Colgate University
2. Bucknell University
3. Swarthmore University
4. Amherst College
5. Haverford College

and the top five state universities;

1. Berkeley
2. Colorado School of Mines
3. Georgia Institute of Technology
4. New Jersey Institute of technology
5. University of California at San Diego.

What does Internationalisation Mean?

In THE-QS rankings universities get 10 per cent for the proportion of international students and international faculty. Does this measure say anything about the quality of a university?

Maybe. But one thing it says something about is simply the size of the country in which the university is located. There is a moderate negative correlation between the score for international faculty of the top 400 universities in the 2008 rankings and population of .332 and of .326 between international students and population.

This may help to explain why Hong Kong universities have been doing so well lately compared with those in Mainland China.

The Paris Rankings are out

L'Ecole des Mines de Paris has produced its third Professional Ranking of World Universities. This is based solely on the number of CEOs of Fortune's top 500 companies. The top 5 in order are Tokyo, Harvard, Stanford, Waseda and Seoul national universities. Five French schools are in the top 20 and in general France performs much better on these rankings than any other, which, one suspects, might be the whole point of the enterprise.

According to University World News

Interviewed in the online higher education publication Educpros, Nicolas Cheimanoff, director of studies of Mines Paris
Tech, explained the aims of the rankings: "In France we were challenged into taking action, to say we could not base arguments exclusively on the Shanghai ranking and construct higher education policy solely on this ranking.

"We wanted to show at an international level that France is a country where you can study. Our ranking gives visibility to a school, but also to the system of French higher education as a whole."Cheimanoff said Mines Paris Tech had been in contact with Professor Liu, originator of the Shanghai rankings, to suggest Jiao Tong should incorporate the Mines crterion. "He was a priori in favour but only if we included the academic careers of company heads since 1920 as he did for
the Nobel prizewinners. But that's totally impossible."

The Paris rankings do correlate quite well with others indicating they are measuring some sort of quality. However, the performance of French, Japanese and Korean schools may say more about the recruitment and immigration policies of their countries than anything else.

Also, one wonders whether producing the CEO of General Motors is indicative of the real quality of Duke and Harvard.

The frightening thing is that it probably is.

A Ranking System for the Philippines

Another national ranking system is on the way.

The Commission on Higher Education (Ched) will come up with a ranking system of the best schools in specific fields of study or discipline, an official said today.

“We may come up (with the ranking system) within the
year,” said Ched executive director Julito Vitriolo said in a phone
interview. As of the moment, Vitriolo said Ched is now compiling the
licensure examination results on different fields of study in various colleges and universities nationwide.

See here for more.

IREG-4

Presentations from the fourth International Rankings Expert Group Conference in Astana, Kazakhstan, are available here.

Is There a q Factor in University Ranking?

Although the Shanghai rankings show a high correlation with other rankings (based on a tiny sample of US universities) the HEEACT rankings from Taiwan (Performance Ranking for Scientific Papers for World Universities) do somewhat better. The correlation with THE-QS is .740, the Shanghai ARWU .984, the USNWR America's best Colleges .711, Professional Ranking of World Universities .920 and the Center for College Affordability and Productivity .700.

All these rankings measure diffent things. The USNWR measures a variety of indicators related directly or indirectly to the quality of instruction, the CCAP is quite definitely a consumer-orintated ranking, the THES-QS World University Rankings are largely a measure of research performance (reputational survey, citations per faculty and student faculty ratio where researchers are counted in the faculty), the Professional Renking of World Universities counts CEOs of top companies while the Shanghai and Taiwan rankings focus entirely on research, mainly in the natural sciences.

The ability of the Taiwan rankings to predict scores on the other rankings suggests that underlying various measures of university quality is a single q factor, the average intelligence of its faculty. If there is one single number that would tell you about the general quality of a school than it would probably be the average IQ of the faculty, although performance on standardised tests, publications and citations (especially in the hard sciences) and postgraduate degrees might be goood proxies. The strength of the Taiwan rankings would be their focus on research productivity alone.

Incidently, if anyone from HEEACT reads this, please think of a new name for your rankings. PROSPWU is not exactly a memorable acronym.

Ranking the Rankings

University rankings are popping up everywhere. So how do they compare with one another? One way is to check the correlation between the total scores of the rankings. Here, correlations have been calculated for the scores of ten US universities (every tenth university in the Shanghai rankings excluding those not in the THE-QS top 400).

It seems that the Shanghai ARWU is the most valid of five rankings Correlations for total scores are .796 with the THE-QS, .712 with the USNWR America's Best Colleges, .896 for the Professional Ranking of World Universities (Paris) and .628 for the Center for College Affordability and Productivity.

It looks like on the basis of this extremely small and unrepresentative sample that if you had to pick just one ranking to rely on then it would have to be the Shanghai ARWU.

International Rankings Expert Group Conference in Kazakhstan

I have just returned from the International Rankings Expert Group’s fourth conference in Astana, Kazakstan. There were some positive developments at the conference but also a few disappointments.

Starting with the negative aspects, there seems to be a global trend to the proliferation of national rankings which are increasingly and unnecessarily detailed and which impose a serious burden on teachers and researchers.. A case in point is the new ranking produced for Kazakhstan which includes just about every variable imaginable from "the number of Dissertation Councils" to "the availabilty of medical centers, sport halls, preventoriums, recreation zones". Very few at the conference seemed aware of the backwash effect of the rankings boom as universities outside the top 500 create their own rankings or compete for irrelevant awards, medals or certificates. Drudges in the periphery of the world university system now face an endless round of form filling, office tidying, meetings, committees and professional development activities which make teaching difficult and genuine research, as opposed to research-like behaviour, close to impossible.

The European Union ranking project was presented in some detail but I suspect is going to make little impact since it appears largely concerned with making fine distinctions between the research capabilities of faculties and departments.

There was a presentation about the Lisbon project which proposed to ignore research altogether and measure teaching excellence. This is an interesting idea but it seems to miss two important points. One reason for emphasizing the measurement of research is that the qualities required for research, general cognitive ability, reading and writing skills, conscientiousness and interest in a subject also correlate to some extent with teaching ability, however that is measured. Also, the assumption that learning is dependent on teaching which in turn must be regulated by a centralized bureaucracy is surely false, at least for the more able students

Positive developments include a trend towards personalized rankings where consumers assign their own weighting to indicators. There is an interesting project under way in Taiwan.

Richard Vedder introduced a ranking that has the merit of being based largely on publicly accessible data. The basic idea is excellent but there are some issues to be dealt with. Using RateMyProfessors is not a bad way to assess the quality of teaching but to be really valid there needs to be some adjustment for the grades awarded by the instructor. Using the American Who’s Who is also potentially interesting – and could well be applied internationally -- but there are of course obvious issues of bias.

He also gave a presentation without using PowerPoint. I must remember that next time I fill in a form about my innovative teaching methodology.

One measure presented was to create an IREG seal of approval. The logo is ready. I am not sure though whether this is going to be effective.

Overall, the conference has strengthened my conviction that if ranking is to be done it should not be by administrators or businesses but by universities themselves.

A New Atlantic Alliance?

Recently, the US News and World Report expanded its rankings portfolio to include the World’s 100 Best Universities. This turned out to be nothing more than the THE-QS World University Rankings with a rebranding for the US market. Now the USNWR has gone a step further and produced a list of the world’s top 400 universities along with sundry regional and subject rankings. Once again, this is the QS rankings with a new name.

This is no doubt a shrewd move for QS who are now marketing their rankings on both sides of the Atlantic and appear to be on the way to establishing a near monopoly over the international ranking business. It could, however, be risky for USNWR. People are bound to wonder why it should link up with a company that has a history of errors where American universities are concerned. In 2007 QS got their North Carolina business schools mixed up and as a result caused Fortune magazine to withdraw its business school rankings based on QS data. Will US students and stakeholders forgive the USNWR if its data includes things like a near zero for research for Washington University in St Louis or an unbelievably good score for Duke for student faculty ratio?