Are randomized experiments the gold standard in causal analytics?

Editor’s note: Kevin Gray is president of Cannon Gray, a marketing science and analytics consultancy. 

In causal analysis, randomized experiments are generally seen as the methodological gold standard. "No causation without manipulation" is a statistical adage that implies that only through randomized experiments can we establish the existence of a causal effect. 

Random assignment of research participants to control or treatment groups reduces the risk that these groups were different in significant ways prior to administration of the treatment. The “treatment” could be a new medication, therapy, training program, digital advertisement or any number of interventions. 

British statistician R. A. Fisher was a pioneer in the design and analysis of experiments in the 1920s and ‘30s, and randomized experiments are now an important tool in many disciplines.

Note that research “participants” need not be human or even alive, machines being an example of the latter. In some randomized experiments there are multiple controls and treatments, two standard medications and three new pharmaceuticals, for instance. Participants can also be followed over time after being randomly assigned to two or more experimental groups. Latent growth curve analysis is one example and event history analysis another.

Experimentation is a complex subject and a lot more than tossing a coin. “Experimental Design: Procedures for the Behavioral Sciences” (Kirk) and “Design and Analysis of Experiments” (Montgomery) are two standard reference books I can recommend. 

One caveat is that randomization does not ensure that the treatment and control groups are adequately balanced with respect to important variables. In some cases, we can statistically adjust for imbalance after the fact so it does not bias our conclusions. When designing our experiment, if we have reason to believe certain variables are especially consequen...