Ever wonder why those talking heads on TV who seem to know all the answers, about sports or the markets, aren't getting fabulously wealthy investing their own money? Well, so does the CIA!

The CIA recently declassified Rob Johnston's Integrating Methodologists into Teams of Substantive Experts, and it's a fascinating paper about how to integrate information from experts. It's applied to the intelligence gathering, but his results seem pretty general.

Key part:

Intelligence analysis, like other complex tasks, demands considerable expertise. It requires individuals who can recognize patterns in large data sets, solve complex problems, and make predictions about future behavior or events. To perform these tasks successfully, analysts must dedicate a considerable number of years to researching specific topics, processes, and geographic regions. ...

The very method by which one becomes an expert explains why experts are much better at describing, explaining, performing tasks, and problem-solving within their domains than are novices, but, with a few exceptions, are worse at forecasting than actuarial tables based on historical, statistical models.