Here's a very interesting paper from a guy named Ioannadis, which was recently written up in the WSJ: http://medicine.plosjournals.org/perlserv?request=get-document&doi=10.1371/journal.pmed.0020124

The controversial title of his paper is "Why Most Published Research Findings are False." It's published in journal meant more for a medical audience, but I think statistically speaking, the findings apply to all sorts of research.

The argument sort of rests on the Bayesian interpretation of what probability is. I don't want to get into the gory mathematical details because that would be uninteresting but let the following stylized example illustrate:

Suppose you have a medical test that is 99% accurate. That is, if a person has a certain ailment (call it grooteditis), the test will come back positive 99% of the time. If the person on the other hand, does NOT have grooteditis, the test will show negative 99% of the time. So it is 99% correct.Now suppose you are a doctor and a person comes into your office and tests positive for grooteditis. Can you say with 99% certainty that this person has grooteditis?

Well, not really. The certainty of whether or not this person has grooteditis depends on what the prevalence of grooteditis is in the population. For example, what if there are SO few people with the disease in the population, that even when the test shows positive, among those times, it is often wrong? So for example, if only 1% of the population had grooteditis in reality, there's a good chance it's a false positive.

If you do the calculation it turns out that you can only say with 50% certainty that this person has the disease.

So what I think Ionnadis is mostly saying is an extended, detailed and sophisticated version of the above. Statistics obviously don't lie. But even if the data looks great, most theories are reductionist by necessity. In fact, one might say that the probability that a theory is true is probably very low. So the implication of this would be that we're probably, on average, too trusting of research findings.

I think this is good reason to pause before being sold on a "statistically significant" study done by this or that scholar or interest group. The probability that their research findings are true is probably even lower, as they generally have a more worldly vested interest in their subject matter.