I posted recently on why we should mistrust research. I’ll spare you the unrelated details, but this morning I listened to a podcast in which Stanford Prof Dr. Laura Carstensen, an expert on the psychological impact of aging, was asked whether elderly people in special homes were happier than elderly people in the mainstream. What she answered was:

"It’s a really interesting idea, people often ask this question, but you just can’t do the studies on this because there’s such a selection bias of who moves there. People who live in retirement homes say that they are very happy there, but nobody has randomly assigned them. We can’t answer the question."

It struck me, as I heard her answer, that "we can’t answer that question" is a very strong answer. We should use that more often in business. This academic respect for the underlying validity of research is important. In this case you can’t do valid research because you can’t randomize the subjects. When do we apply the same standard for market research? Isn’t it better to just realize when research won’t answer a question.

I watched a focus group in San Francisco in which most of the subjects regularly attended focus groups. Does researching what semi-pro focus group participants say and do lend us any insight into what the real target market thinks or feels? How would we know? What about the surveys that record opinions of people browsing on the web who want a free iPod?

I think it’s okay to say no.

— Tim