Building on a research seminar given at the Department of Methodology by Professor Judith Green (LSHTM) I am pleased to say that the latest paper out of the ‘On the buses’ study I worked on as a Research Fellow has now been published. Using already published papers on the findings of this study as a backdrop, as well as presenting some of this data in the course of the argument, this paper reflects on some theoretical and practical challenges of conducting a mixed-methods evaluation of a public policy and the approach we employed for meeting these challenges.

The paper has been published as an article in the 20th Anniversary Special Issue of the journal Evaluation: The International Journal of Theory, Research and Practice. In broad terms the article draws on our experience of integrating the analysis of quantitative data (travel-diary surveys conducted by Transport for London to measure travel mode usage; police records of road injury; and hospital records of admissions for assault injuries) and qualitative data (derived from individual and group interviews and observations on and around London buses) in the context of the ‘On the buses’ study to propose an approach for integrating the logic of quasi-experimental methods with inductive qualitative analysis in public policy-based evaluations of ‘natural experiments.’ We argue that this approach can be used to at least partly address challenges routinely faced in the conduct of such evaluations, not only those relating to issues of internal validity (making credible claims for causal inference) but also (and more unusually) external validity (being able to claims that can be generalised beyond the particular case at hand).

This proposed methodology for integrating quasi-experimental and inductive designs in evaluation operationalises the work of Nancy Cartwright and the LSE’s Eileen Munro (Cartwright, 2007, 2013; Cartwright and Munro, 2010) to advocate the importance of interrogating the capacity of an intervention to cause changes in social outcomes – to interrogate the capacity of a Mayoral policy to eliminate the point of use cost of bus travel for secondary school children in London on determinants of health in this instance. By doing so, we argue that the quality of data derived through evaluations of public policy ‘natural experiments’ (which is conventionally considered ‘weak’ evidence) can be strengthened, and that indeed there may be situations for which evaluations of natural experiments may provide a stronger design than those reliant on randomizing cases.

Green, J., Roberts, H., Petticrew, M., Steinbach, R., Goodman, A., Jones, A. and Edwards, P. (2015) ‘Integrating quasi-experimental and inductive designs in evaluation: a case study of the impact of free bus travel on public health,’ Evaluation: The International Journal of Theory, Research and Practice 21: 391-406, DOI: 10.1177/1356389015605205

Dr Alasdair Jones is Assistant Professor in the Department of Methodology at LSE.