Research Design and Kool-Aid

I shouldn’t be posting blogs, but there are times when equalizing my level of marginal guilt across the too many activities I ought to be doing requires me to write one. And the topic de jour of research methodology is too irresistible.

Is strong research design important? Ultimately it depends on your objective and audience, I generally find those disciplines, fields, and research communities that pay little attention to proper research design to be uninteresting and certainly unpersuasive. There are exceptions; some disciplines that are primarily theoretical such as mathematics and philosophy have different notions of “proof” that are both elegant and convincing. However for those disciplines such as in the social sciences where we need to rely primarily on empirical evidence to identify the relationships that determine social phenomena, the alternative to proper research design seems to be to hope that your readers have drunk the same ideological Kool-Aid as you have. Good empirical research design doesn’t dissolve in Kool-Aid, it transcends prior beliefs and opinions and, if planned and executed properly, should convince honest skeptics of your results and conclusions.

The capacity to convince skeptics and identify the most persuasive evidence regarding some phenomenon of interest ought to be crucial both within academia and outside it. It is not always the case, unfortunately. Some fields of study within academia have become clubs of mutual admiration and citation, having abandoned any pretense of trying to be relevant to “outsiders”. I suppose that is fine, and if you want to be part of such a research community then clearly being trained in methods that will be recognizable to others is less essential for success. Similarly, as much as we might like well-conducted research to be the foundation of public policy, too often the divide between research and policy remains unbridged.  The current cynical comment around Ottawa is that instead of evidence-based policy-making we have policy-based evidence-making.

In my opinion, the use of good research design and the ability to recognize good research design are essential first steps for relevant and accessible research and for good policy-making. Good research design should transcend discipline, theory, and ideology. Someone skilled at research design should be able to examine papers in just about any field of study and determine whether it has been well-designed or not, and whether the findings should be more or less reliable. Some methods, especially quantitative methods, may be trickier to evaluate in terms of the details of the more sophisticated and advanced statistical techniques, but the overall approach should still be recognizable as either good or bad research. For researchers in an academic setting good research design is essential for making progress on our understanding of phenomena that have incited unresolved debates or have too many unexplained dimensions. Debates need to be addressed not through an appeal to prejudice, but by the carful assessment of relative research design strengths and the identification of further research approaches to settle outstanding points of dispute. If research disintegrates into a shouting match between those with irreconcilable predispositions, how are we to improve our understanding of important topics except through the application of good, ideologically neutral research design? Good research design is the elevation of means over ends. Good research should start with a question, not the answer; if I already know the answer or, more likely, will only accept the answer I want to get, then what is the point of asking the question? Good research design is the process that should get me from “question” to the best answer I can discover.

The same is true outside academia, where decisions makers from parents to Prime Ministers must make choices in the face of incomplete understanding. We must all use the evidence at hand to make the best inferences about causes and effects, actions and consequences. The better we use the evidence, the more likely we will be to make good choices or adopt better policies. The better we understand the research produced by others, including academics, the better we will be able to synthesize the associated lessons and hopefully make better decisions.

Of course these comments beg the one traditionally posed by students, which is “what’s the best research design?”. Usually this question revolves around the classic qualitative versus quantitative debate, and takes the form of “Are qualitative methods superior to quantitative methods, or vice versa?” Let me settle that debate authoritatively once and for all for NPSIA students: yes. The message we hope you are getting is that the best research method should be conditional on your research question, and a good research question should essentially identify your methodology. In general, however, you should hopefully have enough facility in the many different social science research methodologies to be able to establish a convincing research design based on the most suitable approach. Although I do a lot of large-sample statistical analysis, I have a great appreciation for the subtlety and nuance that can be obtained from more “qualitative” approaches. To be honest, I think the “best” research design has elements that span the large sample-small sample divide. Large sample work can establish empirical regularities and help describe general relationships, but it generally lacks detail, context, and often a convincing understanding of causality. Small-sample studies provide detail and context but it is difficult to determine how generalizable the results are unless the position of the cases can be properly situated in the larger population. Nested analysis, which works back and forth across these methodological approaches, seems to me to be a very appealing compromise.

Finally, I admit to having a “scientific” bias in that I think there are “right” answers to questions about social phenomena, even if the rules generating these relationships are only transitory. In fact the more transitory the better, as it means papers that use updated empirical evidence are likely to be as publishable as the original. I recognize that there are debates about whether there is a “truth”, about whether methodology or research design can be ideologically or class “neutral”, and about whether “scientific understanding” should be privileged over other forms of understanding. Let me be clear about my prejudices. While I am prepared to admit to some mild existential doubt, I am firmly in the camp that believes that there is truth, that methodology can be neutral, and that empirically-grounded understanding based on good research design is our best hope for progress (however defined). If you think I’m wrong, well, prove it.

Dane Rowlands


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s