Textbooks/readings on what to do when you can’t create an ideal experiment?

My statistical training is rooted in mathematical statistics, and taking these methods classes in my M.S. are a bit of a shock at the moment; it is currently difficult for me to be able to understand some of these “applied” methods since I lack experience in the industry.

One of the topics that we’ve been talking about in my methods classes is the idea of experimental design.

Say, for example, I want to perform an experiment on the effectiveness on an educational program that claims to raise test scores of K-12 students.

In the methods classes, they’ve taught the following to pursue such a problem: make sure you have a good research question, a good data gathering method, a randomized experiment, homogeneous treatment groups (i.e., one treated with this program, one perhaps not) ideally of equal size, and then run a $t$-test (or some sort of nonparametric hypothesis test), and it’s all fine and dandy, right?

I have little faith that this is how it works in reality.

I’ve learned that, sure, you might have to do some convenience sampling. But other than that, I have no idea how to implement experimental design other than what I’ve learned from a textbook.

Are there any textbooks, readings, etc. that explore these issues in practice (and ideally, don’t gloss over the math – I don’t need detailed proofs of everything, but I don’t want to be told that everything is “obvious,” for example)?

There are two fields where randomized experiments are almost always impossible: they are social sciences and economics. In these instances you can only do “quasi experiments”. Try searching with keywords quasi experiments, observational studies and social sciences; you will get some good text books. I can recommend two excellent books on this subject: the second book by Shadish and Cook is a classic:

1. Counterfactuals and Causal Inference: Methods and Principles for Social Research By Morgan and Winship
2. Experimental and Quasi-Experimental Designs for Generalized Causal Inference by
by William R. Shadish and Thomas D. Cook

A classic paper that uses a technique called “propensity score matching” in non experimental setting for causal inference by Dehejia and Wahba is highly recommended as well.