My statistical training is rooted in mathematical statistics, and taking these methods classes in my M.S. are a bit of a shock at the moment; it is currently difficult for me to be able to understand some of these “applied” methods since I lack experience in the industry.
One of the topics that we’ve been talking about in my methods classes is the idea of experimental design.
Say, for example, I want to perform an experiment on the effectiveness on an educational program that claims to raise test scores of K-12 students.
In the methods classes, they’ve taught the following to pursue such a problem: make sure you have a good research question, a good data gathering method, a randomized experiment, homogeneous treatment groups (i.e., one treated with this program, one perhaps not) ideally of equal size, and then run a t-test (or some sort of nonparametric hypothesis test), and it’s all fine and dandy, right?
I have little faith that this is how it works in reality.
I’ve learned that, sure, you might have to do some convenience sampling. But other than that, I have no idea how to implement experimental design other than what I’ve learned from a textbook.
Are there any textbooks, readings, etc. that explore these issues in practice (and ideally, don’t gloss over the math – I don’t need detailed proofs of everything, but I don’t want to be told that everything is “obvious,” for example)?
There are two fields where randomized experiments are almost always impossible: they are social sciences and economics. In these instances you can only do “quasi experiments”. Try searching with keywords quasi experiments, observational studies and social sciences; you will get some good text books. I can recommend two excellent books on this subject: the second book by Shadish and Cook is a classic:
- Counterfactuals and Causal Inference: Methods and Principles for Social Research By Morgan and Winship
- Experimental and Quasi-Experimental Designs for Generalized Causal Inference by
by William R. Shadish and Thomas D. Cook
A classic paper that uses a technique called “propensity score matching” in non experimental setting for causal inference by Dehejia and Wahba is highly recommended as well.
- Design of Observational Studies by Paul R. Rosenbaum.
- Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction by Imbens and Rubin.
IF you are looking at time series quasi experiments, the above books have some chapters devoted to them, but a dedicated book is by Gene v. Glass Design and Analysis of Time-Series Experiments and I would check his article Interrupted time series.
Trivia: Gene V Glass coined the term “Meta Analysis“.