Wednesday, July 1, 2009

Causation... seriously!

Over two months ago, I wrote an article entitled Why Take Causation Seriously? In that piece, I repeated the statistician's adage -- correlation does not imply causation -- and then I argued that when we use data, we actually want to know about causation. Because statistics won't tell us what causes what, we need to use good-old-fashioned logic to learn more about causation. Otherwise, we could make a big mistake with our data.

Here's an example to illustrate my point on causation:

McDonalds makes you fat. An example. Suppose you observe that overweight people eat at McDonalds. Does that fact alone allow you to conclude that eating at McDonalds makes you fat? No. After all, most people who sign up for Weight Watchers are overweight, and we wouldn't dare conclude that Weight Watchers makes people fat. If you conclude that McDonalds food makes you fat ("X causes Y") just on the basis of correlation, critics can claim one of two classic alternatives:

1. McDonalds meals attract naturally obese people ("Y causes X"). Maybe McDonalds isn't a problem at all. It's just that obese people like Big Macs.
2. Some third factor causes McDonalds patrons to be obese ("Z causes X and Y"). Maybe McDonalds restaurants are always next to Dairy Queen, which is the true source of obesity. But, people like to ruin a good healthy burger with an unheathly shake from DQ.

If you want to demonstrate that McDonalds makes people fat, you have to rule out these two options. This requires some mental gymnastics. What kind of mental gymnastics am I talking about?

On the McDonalds obesity problem, the movie Super Size Me is an excellent example. In that movie, Morgan Spurlock ate three meals per day at McDonalds for 30 days and chronicled the resulting adverse health effects. In starting the documentary with a clean bill of health and leaving with several serious medical maladies, Spurlock effectively rules out the other two options for what caused his weight gain. As a result, Spurlock made a point about causation: McDonalds does make people fat -- at least if you eat every meal there.

Don't people get this point? The McDonalds and obesity example is a familiar example to illustrate my point about causation, but it isn't the reason for this post. A recent Economist Do It with Models article references a blog post by Derek Sivers. The point is nicely summarized in its title, "Shut up! Announcing your plans makes you less motivated to accomplish them."

This title reads like causation gone wrong, and that bothers me.

Most of us have announced a plan that turned out to be unsuccessful. Such a plan is usually called a New Years Resolution. We hear about these failed plans all the time because they're announced. But, we don't see many of the plans that people keep private. Of the private plans we actually observe, those are almost always successful because their success is why we saw them.

To see this point more clearly, suppose that 20 of your friends announce New Years Resolutions to lose weight. By June, you check on the status of those resolutions, discovering that 18 of your friends failed. Suppose that you also notice that two of your other friends went on silently-willful diet plans that were successful. Of the 22 plans you notice, 100 percent of the privately-held plans were successful compared to 10 percent of the announced plans. Does this mean that announcing your resolution is what dooms it to failure?

Definitely not. The only reason you see your friends' unannounced plans is that they were successful. To make a valid statement about relative success rates, we would have to know how many of your friends made silent plans. But, in your everyday life, there's no way for you to know this number. Therefore, the data you can gather in your everyday experience does not tell you whether you should believe Sivers' statement.

Fortunately, the statement made by Sivers is not as sloppy as it looks. He cites academic work that is somewhat careful about causation (work found here). But, my biggest concern is that people will see this advice in the title, look around at their world, and conclude that it has to be right. Then again, maybe this is a case where the advice is just too strange to believe.

Given that the paper he cites is academic research, I don't want to take any chances. I read the report, but I am still worried about the underlying academic research. You should go read the article if you're technically inclined or if you can't wait to see what I object to. I still have two big concerns. Tomorrow, I address one of these concerns. Monday, I address the other.

No comments:

Post a Comment

Please feel free to share your ideas about this post in the open forum. Be mindful that comments in this blog are moderated. Please keep your comments respectful and on point.