Before you demand “data-driven” justice, make sure you know who is driving

Jeffrey Butts

Someone must pay for research on justice policy and practice, and reliable research can be expensive. Simple statistical work can be conducted by academics and analysts operating on their own, but community-centered, policy-relevant evaluations are often costly and complicated. Researchers often must observe program operations, interview participants, conduct surveys, and obtain ongoing access to administrative data systems. When researchers are funded to answer questions about the effectiveness of justice systems, the answers are the ones policymakers and funding agencies are willing to pay for.

What sort of questions are ignored, and how does this affect policy and our understanding of public safety and community well-being?

Data and research evidence are inevitably the results of investments made by policymakers, government agencies, and foundations. Their investments are not unbiased. They reflect the beliefs and values of funding sources and public officials, as shaped by cultural, class-based, and racial perceptions. And, of course, economic and cultural elites tend to favor non-structural explanations of crime. In other words, they instinctively prefer to locate the origins of crime in individual pathology rather than the effects of economic inequality, injustice, and centuries of community disadvantage. As a result, public safety research more often than not focuses on individual behavior instead of social structure and community context.

Key question: Are wealthy neighborhoods relatively crime-free because so many inherently law-abiding people decided to live there, or do structural and economic advantages in wealthier neighborhoods themselves lead to less crime? Reasonable investigations of crime prevention would suggest a range of evaluation strategies, but researchers often concentrate on just one — individual interventions to address anti-social behavior of known offenders. Basic prevention and the safety benefits of comprehensive community support are not tracked or measured as often.

Community-level interventions and primary prevention programs are more challenging from a research perspective. Measuring their effects is time-consuming, harder to control, and more likely to produce ambiguous findings. Sample sizes are inherently smaller, which means statistical rigor is more elusive. Testing the effect of interventions at the neighborhood or community level leads to less immediate and less exciting results. So, like their funding partners, local officials and evaluation researchers prefer to address public safety issues at the level of individual behavior rather than trying to improve community safety overall.

Researchers go along with these efforts because they are rational creatures. Their main goal is to publish — a lot. They prefer to evaluate interventions that can be tested quickly to turn around publications quickly, and it is less time-consuming to study programs and policies focused on individuals using pre-existing administrative data.

Funding agencies and foundations also prefer to spend as little as possible on the infrastructure of research. They are reluctant to support labor-intensive studies requiring contemporaneous surveys and interviews with community members. They favor studies that use pre-existing data from law enforcement agencies and justice systems. Of course, half of all crimes are excluded from these studies because about half of all criminal incidents are never reported to police. Studies of arrest trends are incomplete because police are often unable to identify perpetrators to “clear” their cases.

All this means that data produced by social agencies and criminal justice systems are not comprehensive measures of community safety. They are work products of law enforcement and other bureaucracies that reflect public safety but cannot address all crime concerns in all neighborhoods.

When someone tells you “what the data says” about reducing crime and violence, remember they’re describing a relevant, but quite incomplete base of information created by people and organizations with opinions, values, and self-interest.

Data and research findings don’t appear like wildflowers in a meadow. They are planted and watered by gardeners with intention.