Don’t make the logical error known as “selection on dependent variable”
Jay Green has a great blog today on the logical error known as "selection on dependent variable". He quotes Russ Whitehurst as follows.
"A while back, I read a journalistic account of a small island off Italy in which people live to be quite old. Could it be because they sleep late, or drink lots of wine, or live a communal existence, or don’t eat refined sugar? Unfortunately there is no way to know based on the information provided in the article if it is one or more of the lifestyle characteristics the author identified as distinctive, or whether something else is going on. Even the claim of unusual longevity is questionable since there is no birth registry. And taking everything at face value, maybe there are other places in Italy in which people live as long or longer with different lifestyles."
If you just substitute the word "Finland" for "Italy" and "where children do well on the PISA tests" for "in which people live to be quite old", you will get that those who "make policy recommendations based on the Finland du jour should be made to hold up a giant sign that says 'I do not understand the basic research methods of the field in which I claim to be an expert'."
"It is a matter of basic logic that one cannot know what causes success only by looking at a successful place (or set of successful places). You cannot know whether any factor contributes to success without also considering unsuccessful places and examining whether that same factor tends to be more present or absent in successful relative to unsuccessful places."




Share this article