Show Me
December 18, 2007 – 7:48 amIn an online discussion about why there are so few women in computer science, one student wrote that he thought it was because there are fundamental differences between the sexes. When asked what data he was basing that opinion on, he replied:
It’s more of an unrefuted hypothesis based on personal observation.
More recently, in a comment on an earlier post asking what data there was to back up the claim that Erlang and Haskell made parallel programming easier, Neil Bartlett said:
As a working programmer I favour my own personal experiences and experimentation over academic studies…
Put that way, it sounds very pragmatic, but what if you replace “academic studies” with the phrase “careful observation and analysis of a large number of people followed by critical peer review of the claims”? Or, if you like shorter sentences, “science”. I wouldn’t believe a drug company’s claims about a new headache remedy unless they’d been subjected to independent scrutiny—in fact, while the rules are bent all the time, the drug company wouldn’t actually be able to make any claims publicly until they were able to prove them. “My Uncle Vlad used it, and his migraines went away” isn’t good enough—there could be a thousand other reasons why Uncle Vlad’s headaches went away, and there’s no way of knowing whether the effects on one (self-selected) subject are representative of the population as a whole. “It’s made from mushrooms, which contain lots of mitochlorians, which are an effective vasodilator” isn’t good enough either—there are at least three leaps of logic in that particular “just so” story, any one of which could be invalidated by some factor not included in the argument.
Unfortunately, most of the claims about programming and related topics that I read online or hear in pubs are equally broken. Having worked with scientists and engineers for much of the last 25 years, though, I’ve learned that the good ones don’t believe or disbelieve something until there is evidence—not just anecdote, personal experience, or a plausible chain of reasoning, but evidence—to back it up. As a profession, we are finally starting to accumulate that kind of evidence: see Glass’s Facts and Fallacies of Software Engineering for an easy-to-read summary, or the journal Empirical Software Engineering
for more recent results. It’s hard work, but I suspect the real challenge will lie in persuading working programmers to say “evidence, please” more often.
6 Responses to “Show Me”
Great post Greg. I follow this blog: http://badscience.net/, and I’m often struck by how similar many software ‘engineering’ claims are to the claims of homeopathy, naturopathy, etc. Even an empirical study may have faults: selection biases, not controlled, not blind, etc.
I would say that such studies seem easier in medicine than software. How easy is it to do a double-blind controlled trial with UML class diagrams in an industrially relevant manner?
My other question is, to what extent are working programmers relying on anecdotal experience because they are subject to so much hype? E.g., .Net will do this better, use RUP, etc… Eventually I imagine one would just ignore it and use what works (Vi, C, what have you).
One difference between medicine and software is that medical devices/drugs cannot make unsubstantiated claims the way software vendors can.
By Neil on Dec 18, 2007
The problem with scientific studies, though, is they are notoriously hard to conduct. So when you don’t have any science on hand for a particular situation, you’ve got to base your opinions on _something_. At least, that’s what really irks me.
No doubt, though, programming community would be much better off if all programmers were as rational as the machines they work with.
By David Wolever on Dec 18, 2007
Speaking as a biologist / programmer, I couldn’t agree with you more. I’ve been thinking about moderating my gung ho rants about testing a bit, because I’m not really sure it works for everyone — even though I can tell you that it *should*, if done properly. (heh.)
Anecdotal evidence drives quite a bit of ‘net wrangling about languages, OSes, etc. and it does CS quite a bit of harm.
By Titus Brown on Dec 18, 2007
Greg, it’s not that I am disinterested in or would ignore objective evidence. But such evidence so rarely exists that working programmers have no choice but to generate our own evidence, by actually trying the language or framework in question.
If I try a technique and it works then that is the ultimate in hard evidence _for me_. It is the evidence of my own senses. I know that for you and for other programmers, it’s just another anecdote… so they have to go and generate their own evidence.
The pharmaceutical analogy is flawed because I am unlikely to be killed by a dangerous programming language. Personal experimentation in our field has no downside risk other than opportunity cost.
By Neil Bartlett on Dec 18, 2007
Don’t overlook IEEE Software, which now has an excellent column called Voice of Evidence, edited by Forrest Shull, which summarizes results of empirical software engineering studies in a format that’s more accessible to practitioners than the journals.
By Lorin Hochstein on Dec 19, 2007