MethodS matter: on the importance of relevant

Evidence-based policy – policy analysis – evaluation Evidence-based policy – public policy based on scientifically sound evidence
Policy analysis as the analysis for policymaking – providing policy makers with (evidence-based) advice on problems, causal mechanisms, instruments and potential consequences of the available options
Evaluation as a source of knowledge about what works, for whom and in what circumstances<br>

Policy analysis Ex ante evaluation Ongoing evaluation<br>

Policy analysis without evaluation is blind evaluation without policy analysis is powerless<br>

A Nobel Prize winner, James Heckman ON evaluation of public policies facing three main problems: “Evaluating the impact of historical interventions on outcomes including their impact in terms of the well-being of the treated and society at large.”
“Forecasting the impacts (constructing counterfactual states) of interventions implemented in one environment in other environments, including their impacts in terms of well-being.”
“Forecasting the impacts of interventions (constructing counterfactual states associated with interventions) never historically experienced to various environments, including their impacts in terms of well-being.” (Heckman, 2008, p. 8; see also: Heckman, 2005).<br>

Good Evidence Policy-relevant – justifying the choice of policy conduct and instruments
Trustworthy
Sound theory – causal claims
Proper scientific methodology – research design and measurement
Dependable and up-to-date data
Conclusive – “clearly speaks for or against the policy” “Will it work here? That is, will the policy that you are considering make a positive difference in the desired outcome if you implement it, bearing in mind how, where, and when you would do so? In the language of the standard literature, this is a call for a prediction of effectiveness.” – Cartwright & Hardie<br>

Maryland Evaluation Scale Source: Daniel Fujiwara, ‘Methodological Developments and Challenges in UK Policy Evaluation’ (presentation on VIII Evaluation Conference in Warsaw) based on: Farrington, D. P. (2002) Methodological Quality Standards for Evaluation Research. Paper Presented at the Third Annual Jerry Lee Crime Prevention Symposium, University of Maryland)<br>

Randomized control Trials are important, but not sufficient „They [RCT] cannot alone support the expectation that a policy will work for you. What they tell you is true—that this policy produced that result there.”
“They do not even tell you that a policy works. What they tell you is that a policy worked there, where the trial was carried out, in that population.”
“The fact that it worked there is indeed fact. But for that fact to be evidence that it will work here, it needs to be relevant to that conclusion.”

Nancy Cartwright and Jeremy Hardie, Evidence-Based Policy. A Practical Guide to Doing It Better”. RCT provides internally valid effect size and significance (works there) but without insight into a causal mechanism and certainty of external validity (will work here)<br>

Why causal mechanisms matter X – the randomized stimulus
Y – the measure of the outcome
η1 - the latent variable manipulated by the stimulus (the cause of the policy outcome)
η2 – latent outcome variable (policy outcome)
Problems:
How strong is the effect of the stimulus X on η1?
How good is Y as the measure of the outcome η2 ?
If above relations are weak, results obtained traditionally by ANOVA/regression would not reveal influence, even if it is a strong one Kenneth A. Bollen and Judea Perl (2012), Eight Myths About Causality and Structural Equation Models<br>

X causes another latent variable η3 which in turn causes η2
X i Y remain significantly associated Why causal mechanisms matter Kenneth A. Bollen and Judea Perl (2012), Eight Myths About Causality and Structural Equation Models<br>

X and Y are significantly associated
η1 is not true cause of η2
The stimulus X causes another variable - η4, which does not cause η2 but causes Y Why causal mechanisms matter Kenneth A. Bollen and Judea Perl (2012), Eight Myths About Causality and Structural Equation Models<br>

What matters for good evidence is both: proper causal theory  and adequate methods<br>

Source: http://www.dagitty.net/dags.html#<br>

Limitations of Evidence-based policy Policy decisions are based not only in evidence, but they are also prone to the influence of competing interests – politics are involved here
Institutional and cultural constraints matter too
Evidence may not reflect political priority or social desirability of what is being measured
Priorities and social desirability are usually differentiated
External validity of social research is more problematic than in medicine (“what works there might not work here”)<br>

Evidence bias (Parkhurst, 2017)
Technical bias: evidence can be misused or manipulated for political reasons
Issue bias: appeals to evidence serve to obscure key social values or impose political priority in unrepresentative ways
EBP is a significant component of the rational model of policy-making, but not as significant or having a different meaning in case of other models like: policy as a political game, a discourse, a garbage can or an institutional process (see: Enserink, Koppenjan & Mayer 2013) Limitations of Evidence-based policy<br>

Problem of policy scope, size and complexity: evidence based on causal research (what works) is often restricted to selected policy problems and is of limited use for complex reforms
Communication problems – decision-makers use stories rather than pure scientific reports; there is a need for translation from the language of science into policy narratives
Timing – politicians (like businessmen) have much shorter timescales than researchers
The job of policy makers is to anticipate, rather than explain past events and processes, whereas social scientists prefer the latter
“No prophet is acceptable in his own country” Limitations of Evidence-based policy<br>

What to do? Policy makers: commission methodologically excellent and causally conclusive policy research
Universities: train researchers more thoroughly
Experts: obtain and use sound evidence
Scientists: develop proper theories and methods<br>

References Bollen, K. A., & Perl, J. (2013). Eight myths about causality and structural equation models. In S. L. Morgan (Ed.), Handbook of Causal Analysis for Social Research (pp. 301-328). Dordrecht: Springer.
Cartwright, N., & Hardie, J. (2012). Evidence-Based Policy. A Practical Guide to Doing It Better. Oxford University Press.
Enserink, B., Koppenjan, J. F. M., & Mayer, I. S. (2012). A Policy Sciences View on Policy Analysis. In W. A. H. Thissen & W. E. Walker (Eds.), Public Policy Analysis. New Developments.
Heckman, J. J. (2005). The scientific model of causality. Sociological methodology, 35(1), 1-97.
Heckman, J. J. (2008). Econometric causality. International Statistical Review, 76(1), 1-27.
Murnane, R. J., & Willett, J. B. (2011). Methods Matter. Improving Causal Inference in Educational and Social Science Research. Oxford New York: Oxford University Press.
Parkhurst, J. (2016). The Politics of Evidence. From evidence-based policy to the good governance of evidence (Open Access). London and New York: Taylor & Francis.<br>

Thank You for your Attention! Jaroslaw.gorniak@uj.edu.pl<br>