Elements of External Validity: Framework, Design, and Analysis

Abstract

External validity of randomized experiments is a focus of long-standing debates in the social sciences. While the issue has been extensively studied at the conceptual level, unfortunately, in practice, few empirical studies have explicit design or analysis aimed towards externally valid inferences. In this article, we make three contributions to improve empirical approaches for external validity. First, we propose a formal framework that encompasses four dimensions of external validity; X-, T-, Y-, and C-validity (units, treatments, outcomes, and contexts/settings). We clarify how the proposed framework synthesizes diverse external validity concerns that arise both in the literature and in practice. We then distinguish two central goals of generalization and develop tailored methods. To conduct effect-generalization — generalizing the magnitude of causal effects, we develop three estimators of the target population causal effects. For sign-generalization — assessing whether the direction of causal effects is generalizable, we propose a novel multiple-testing procedure under weaker assumptions than those required for the effect-generalization. We illustrate our proposed methods through three applications covering field, survey, and lab experiments.

Erin Hartman
Erin Hartman
Assistant Professor of
Political Science and Statistics

Related