Anybody interested in educational policy, especially the never ending campaign to close gaps of one sort or another or the oddities of university rankings should take a look at chapter four of Jordan Ellenberg's How not to be wrong: The power of mathematical thinking which is about the obvious -- or ought to be obvious observation -- that smaller populations are more variable.
He notes that South Dakota is top of the league for brain cancer while North Dakota is near the bottom. What makes the difference? It is just that the bigger the population the more likely it is that outliers will be diluted by a great mass of mediocrity. So, extreme scores tend to crop up in small places or small samples.
Similarly when he tossed coins ten at a time he came up head counts ranging from 3 to 9 out of ten.
When he tossed them 100 at a time he got counts ranging from 45 to 60.
When he (actually his computer program) tossed them 1,000 times, the counts ranged from 462 to 537.
It is worth remembering this when a study with a double digit sample is published showing the latest way to close one of achievement gaps or a very small school in a rural state somewhere starts boosting the test scores of underperforming students or a few test takers reveal that the national IQ is imploding. Or the studies fail to be replicated, if indeed anyone tries.
Or university rankings that show very small or very unproductive institutions having an enormous research impact measured by citations.
No comments:
Post a Comment