Casey Mulligan writes the following in a New York Times blog post regard RCTs:
The problems with randomized trials cannot be dismissed as mere philosophical challenges, because people react to the poor treatment they get from experimenters. Why should a patient agree to let a dice or random number generator decide his fate?
By insisting on randomization, experimenters have troubles recruiting study participants, and their reluctance to take part prevents us from learning as much as we could about new treatments (I owe this point to my colleagues Tomas Philipson and Gary Becker).
One approach to this problem is to prevent participants from knowing that they are participating in experiments or that researchers are introducing randomness into their environment, as natural or “unframed” field experiments do (see this paper by Omar Al-Ubaydli and John List on the different kinds of experiments in economics). Professor Sachs’s approach is to economize on the randomness.
Randomization is not a necessarily even the best way to advance science. More than a few statisticians suggest that study samples should be chosen more deliberately, and less randomly (see “The Unprincipled Randomization Principle in Economics and Medicine” by the econometricians Stephen T. Ziliak and Edward Teather-Posadas).
Jessica Goldberg responds here. She writes:
We use RCTs to answer certain types of questions about the impact of a program or product. When the questions have been answered and we know whether, how, and for whom a product works, RCTs are neither necessary nor interesting. But at the time of evaluation, we didn’t know whether tutoring improves student learning (or, for that matter, whether Millennium Villages improve welfare for their residents).