Current events have sharpened the difference of opinion of two thoughtful people on a controversial inference problem, specifically, whether to use base rate information on individuals. Richard Posner, economist and legal scholar, thinks that statistical information on groups is relevant for evaluating individual members of that group. Steve Pinker, a linguist, thinks using base rate information on individuals is at least immoral, and probably illogical.

In Posner’s recent review (registration required)of Malcom Gladwell’s Blink: The Power of Thinking without Thinking, he writes:

It would not occur to Gladwell, a good liberal, that an auto salesman's discriminating on the basis of race or sex might be a rational form of the "rapid cognition" that he admires. If two groups happen to differ on average, even though there is considerable overlap between the groups, it may be sensible to ascribe the group's average characteristics to each member of the group, even though one knows that many members deviate from the average. An individual's characteristics may be difficult to determine in a brief encounter, and a salesman cannot afford to waste his time in a protracted one, and so he may quote a high price to every black shopper even though he knows that some blacks are just as shrewd and experienced car shoppers as the average white, or more so. Economists use the term "statistical discrimination" to describe this behavior. It is a better label than stereotyping for what is going on in the auto-dealer case, because it is more precise and lacks the distracting negative connotation of stereotype, defined by Gladwell as "a rigid and unyielding system." But is it? Think of how stereotypes of professional women, Asians, and homosexuals have changed in recent years. Statistical discrimination erodes as the average characteristics of different groups converge.

In Steven Pinker’s interview with the Harvard Crimson over Larry Summer’s controversial remarks about women and science, Pinker says:

First, let’s be clear what the hypothesis is—every one of Summers’ critics has misunderstood it. The hypothesis is, first, that the statistical distributions of men’s and women’s quantitative and spatial abilities are not identical—that the average for men may be a bit higher than the average for women, and that the variance for men might be a bit higher than the variance for women (both implying that there would be a slightly higher proportion of men at the high end of the scale). It does not mean that all men are better at quantitative abilities than all women! That’s why it would be immoral and illogical to discriminate against individual women even if it were shown that some of the statistical differences were innate.

In The Blank Slate Pinker writes:

The point is not that group differences may never be used as a basis for discrimination. The point is that they do not have to be used that way, and sometimes we can decide on moral grounds that they must not be used that way.

Posner basically argues that base rate information, also known as “prior” distributional assumptions, are relevant and rational to use, as in Bayesian updating. It may not be fair, but it’s rational, it affects the efficiency or accuracy of the forecast. Pinker here says ‘discriminate’ in his quote from the Crimson, but given his other writings (see above) I think means treating people according to their individual merits irrespective their group’s statistical properties. Pinker seems to be saying that in some circumstances (eg, concerning race and sex) one should ignore base rates (not just immoral, but also illogical).

I tend to think we use base rate information all the time to prioritize the vast avalanche of information we see everyday. For example, if you have a job opening and get 50 resumes, you can’t interview them all, so you use a useful but incomplete metric to cull the list down to 5 people you can interview. That metric (advanced degrees, GPA, years experience) is incomplete but informative, just like a “statistical prior”, and is eventually superseded by the information gathered subsequently—but only for those who passed the initial filter. Indeed, not using base rate information is a major theme in Kahneman and Tversky’s behavioral economics literature where those who ignore or underweight base rate information are considered irrational or biased. I think Pinker is displaying wishful thinking that you can document base rate information on groups, but this should not affect how individuals in those groups are evaluated. I agree it isn’t fair, but it seems logical and rational, and therefore expected and probable.