Jeffrey A. Butts
October 18, 2015
Researching the effectiveness of social policies is like pointing a flashlight inside a dark room. You can can only see what passes through the beam of light.
Depending on the size of the room and the power of the beam, you may learn relatively little about the entire room. And, whoever holds the flashlight controls what you can learn.
Research evidence does not emerge from a pristine and impartial search for truth. The evidence we have today is the fruit of previous research investments made by policymakers and funding bodies with goals, beliefs, values, and preferences. Funding provides the flashlight and points it as well.
Strong evidence requires the sustained efforts of researchers working in collaboration with practitioners, and these efforts require the investment of resources —- sometimes substantial resources. There will never be enough funding to evaluate every single social policy and practice.
Moreover, there is no such thing as a perfect study in the social policy field. Human behavior is enormously complex and not completely measurable. In a technical sense, researchers never prove that social programs or practices “work.” Their goal is simply to reduce uncertainty.
To say that a program or practice is “evidence-based” means that the odds of success are reasonably good. Positive evaluation findings do not guarantee success every time for every person and in every situation. Practitioner judgment is necessary.
As long as this is the case, we must be cautious when we interpret and apply the evidence produced by evaluation research.
We all prefer evidence-based practices over those based purely on faith or anecdote, but research is not infallible. The findings of existing evaluations are like small beams of light in a dark room. They are not sufficient for making all the choices required to formulate and implement social policies.
Naive lawmakers who demand irrefutable evidence for every funding decision are simply afraid of the dark.
JeffreyButts.net 