Lies, damn lies, and statistics

One nice (scary) thing about the job I started last year is that I get (am forced) to teach statistics 1 and 2 now. As you can tell, I have mixed feelings about this. As an undergrad I was terrified by stats, but I turned out to have a natural faculty for it. Now I get to inflict that fear on others and hopefully help them discover that it's really not that bad. The one lesson that I try to convey to ALL (stats and econ) my classes is that correlation DOES NOT imply causation! Just because two things are related DOES NOT prove that one causes the other. Though they may ignore that lesson later, they seem to understand it in a classroom setting at least.

A tougher thing for them to recognize is the more nuanced problem that pops up quite often in the world of stats -- the failure to follow through to the end of the analysis.

Okay I admit that I just now made up the name for this particular problem, but it is real and it is dangerous. Well, not dangerous in the same way a rattlesnake or a hurricane is dangerous, but it can cause problems nonetheless. What exactly is this problem I'm so worried about? Let me explain. I'm talking about situations where the authorities identify a problem, usually through the correct usage of statistics, and then implement a solution to the problem without ever checking to see if the ultimate outcome has improved. Let me just try a couple of "for instances".

In recent years lots of folks have figured out that people talking on cell phones while driving are involved in more car accidents (a good use of statistics). Governments have now begun to implement laws to fix the problem. Laws that prohibit talking on a cell phone while driving? No, of course not. They're enacting laws that prevent talking on cell phones WITHOUT A HANDS FREE ADAPTER of some sort. See, they've identified the problem (lots of folks crashing cars while talking on cell phones) and they've implemented a solution (make them use hands free adapters), but not very many have followed through to see if the solution has improved the ultimate outcome.

Do folks with hands free adapters really have fewer car crashes? Isn't that something that should be asked? I think so, but it seems that cities and states are content with the knowledge that they've "done something" about the problem rather than asking whether their something really reduced traffic accidents. Many studies show that hands free devices do not, in fact, reduce traffic accidents. Instead, researchers suggest, "it's the cognitive overload that sometimes occurs when you're engaging in a conversation that is the source of the distraction more so than the manipulation of the device."

The impetus for this post was a similar example I saw in the local paper the other day. In case you've been high on drugs or in a VERY deep cave for 5 years or more, you've probly heard the news -- American kids are overweight (the problem). Some studies have shown this is more a function of lack of physical activity rather than a diet issue. Still, the popular approach is to freak out about kids eating unhealthy foods. Mind you, I'm not arguing that we should stuff little Jack and little Jill full of hamburgers and pizza every single day and expect them to be look like fashion models, but has anyone stopped to ask whether this really does much to help the problem?

Anyway, the story in the paper involved a healthy eating program in the Opelika school system. The point of the story was that Opelika has been doing this for 10 years now so other systems in the state can just copy what they've done. Well congratulations to Opelika, but why haven't they asked the obvious question: are there fewer obese children in the Opelika school system? Here we have a 10 year sample that could be used to test "now vs. then" or to compare Opelika to other school systems that have not changed their menus. There are MANY good research questions that could be answered using the Opelika system, but no one seems to be interested in asking them. Sigh, it's disappointing.

So, today's lesson: ask the right questions!

Okay, I realize I care way more about this than 99% of you (hey, I can hope for 1%, can't I?). Still, as I've said before -- it's my blog. I know I promised a book post this week; I'll get there.

P.S. Talk about spooky. I just saw this story from the BBC on the misuse of statistics. All I can say is, "Amen!"