Do you work in government, philanthropy, or the nonprofit sector? Do you know enough about evaluation research to know it’s important, that it’s a specialized skill, and that many people can’t tell the difference between good research and advocacy that includes numbers?
If you answered “yes” to these questions, you already know a lot, but I want to take you behind the curtain and tell you a bit more about how research works. Mainly, I want you to work effectively with research partners and to survive the experience.
What is Research?
First, you should know what researchers mean when we say “evaluation research.” And before that, you need to know what we mean by “research.” It is not simply using numbers to back up a story, and it’s not just dropping in a couple of data graphs to give context to what is otherwise a position paper or a statement of belief.
I once wrote a book chapter with my friend, John Roman, in which we tried to explain research to a non-academic audience. In our chapter, we said that research is:
“any systematic investigation that uses a predefined standard of quality to determine its methods and to govern the interpretation of evidence. The use of a predefined standard of quality is essential for distinguishing research from argument and speculation (or even fabrication). Ideally, a researcher will apply the highest possible standard in collecting and analyzing evidence to reach a conclusion. The appropriate standard of quality, however, may vary according to the type of research being conducted and the ways in which the findings are to be applied. Within the wide range of investigations that legitimately could be called research, quality standards might vary from sufficient, to persuasive, to exacting.”
From: Jeffrey A. Butts and John K. Roman (2011). Better Research for Better Policies, in Juvenile Justice: Advancing Research, Policy, and Practice. Sherman, Francine and Francine Jacobs (Editors). Hoboken, NJ: John Wiley & Sons.
Research is an attempt to answer empirical questions by assembling and analyzing relevant data and then drawing unbiased conclusions from that analysis using rigorous standards of evidence and inference. Good researchers don’t just accept the first answer that appears in their data. They check many times to see if their initial conclusions might be wrong, and they use the data to try to prove themselves wrong before they ever release any findings.
If you’ve ever worked with researchers or hired researchers for a project, you know this behavior very well. We always look for alternate data interpretations that cast doubt on a previously accepted fact. It can be quite annoying to non-researchers, but we actually enjoy it.
Next, what is evaluation research? When governments or other agencies try to reduce social problems or change some social condition through policy and practice interventions, evaluation researchers are called in to measure the effects of those efforts. Evaluation research is just the application of research methods to establish the effects of whatever solution was implemented to solve some practical problem. In other words, “did it work?” and “would it likely work again?”
Evaluation projects may involve primary (new) data or secondary (pre-existing) data, such as agency records, police statistics, government reports, etc. Whatever data are used, it is the job of the researcher to organize the information with a research design that allows for a fair, unbiased test of cause and effect. No test is ever perfectly free of bias, but the researcher’s job is to eliminate as many sources of bias as possible.
The most obvious choice of evaluation design is the randomized study, or experimental design, in which groups of people or places are divided at random into those getting an intervention and those not getting it. Then, outcomes are compared for both groups. When it is done well (and when it is even feasible), the randomized design allows researchers to have higher confidence in their results. But it is not the only method available for evaluation.
Last year, I wrote an article about this with my friend John Roman (hey, John Roman again!?, can he do anything without that guy?)… We suggested that the most important consideration in evaluation research is that researchers ask the right questions–i.e. “Good Questions” come first; methods second.
“Experimental studies (or randomized controlled trials, RCTs) can produce high-quality evidence, but often within a relatively narrow frame. Randomization may be the method of choice for evaluating some interventions and practices in criminal justice, but other methods are essential as well, especially when researchers need to measure the effects of policy. The choice of evaluation method should be governed by the nature of the questions being asked and by the complexity and nuance of the answers required to apply the results, not by a reflexive preference for randomization.”
From: Butts, Jeffrey A. and John K. Roman (2018). Good Questions: Building Evaluation Evidence in a Competitive Policy Environment. Justice Evaluation Journal, 1(1): 15-31.
A good researcher wants to know a lot about the program or policy they are evaluating before expressing a preference for a particular research design. If your research partner tries to convince you to support a particular evaluation design before they really understand your situation, you are probably talking to someone in sales, not research. Get a new partner.
Who Does Research?
People often assume that anyone who works at a university is a researcher. This is not true. Even someone with the title of “professor” may not be a researcher. They may be great teachers and writers, and even clever thinkers, but most of the stuff written by people at universities does not qualify as research, and certainly not evaluation research.
Not all researchers are academics and not all academics are researchers, or at least not the type of researchers who create actionable information that can be used to improve policies and programs in concrete ways. In the accompanying graphic, I divided the research community into overlapping but distinct groups: academic researchers who I call “scholarly,” researchers largely outside universities who I describe as “entrepreneurial”, and finally the governmental sector.
The overlap between the scholarly group and the entrepreneurial group consists of university-based researchers who are not instructional faculty, but who may hold administrative positions, research professorships, or grant-funded appointments. Their work may exhibit scholarly characteristics as they operate within the academic culture. (Full disclosure: this overlap is where I live.)
If you have ever hired or supervised the work of researchers, you probably know a bit about these variations in the larger research community and how the groups relate to one another. But, let me explain how I see them.
University-based researchers can be pretty self-absorbed. With apologies to my academic colleagues, we’re often not what you’d call socially adept. Academic or scholarly researchers focus a lot of energy on competing with one another for prestige and appointments. The main way to do that in a university setting is to be well-known for using innovative research methods and to publish as many articles as possible in just one or two specialized areas–to the point of dominating the knowledge base.
And, when they get really famous and successful, academics just want to write books and have people listen to them talk about what they wrote in their last book.
There are two problems with this if you’re in government or some other funding organization: 1) academic researchers are mainly helpful if what you need happens to align with their “interests”; and 2) their idea of success is to publish things that impress people who already know a lot about their specialty areas. Of course, this means that only a small number of people will ever read their work, but this doesn’t bother them—as long as “the right people” read their work.
Certainly, there are exceptions within academia (see the overlapping area of the Venn diagram). The exceptions are often people who had other careers before academia–i.e. “real jobs.” This would include people who were cops, lawyers, social workers etc. before going back to graduate school to get advanced training in research. When people have professional experience before becoming researchers, their research tends to focus on practical challenges. It’s not just their intellectual “interest”–it’s their mission. (Again, this might be self-serving since I’m one of those people).
The other large community of researchers you will encounter are people working for one of the many for-profit or non-profit research firms. This would include places like the RAND Corporation, the Urban Institute, Westat, RTI, MDRC, NORC, etc. These researchers may be just as well-trained and just as talented as university-based researchers, but their incentives are different. They are in the “business” of research and they have business interests to consider.
Entrepreneurial researchers have to think about more than their contributions to knowledge. They have to think about revenue, about how many staff will be supported (i.e. funded) by a new project, whether spending the time to bid on new projects could interfere with existing projects, and whether they can raise enough revenue after a project is completed to hold onto the staff hired for that project. A successful researcher with 5 staff members would be excited to land a new contract requiring 10 new people. When that project is over, however, they would need to raise enough funding to continue paying for 15 people–or implement layoffs, which scare people and sometimes cause the most talented staff to look for other jobs. Growth is terrific, but it has to be managed carefully.
Being a professional, non-academic researcher is like running a small business. You need customers to hire staff and you need staff to serve your customers. And, you need to pay yourself. Keeping all of that in balance can get tricky and stressful. Academic researchers–especially tenured faculty–are somewhat insulated from this dynamic. An interruption in funding for an academic often means they might have to “go back to teaching” (the horror!). An interruption in funding for an entrepreneurial researcher usually means they have to “go find a job.”
Entrepreneurial researchers are always busy managing current projects, launching new projects, and trying to “cover” their staff. They don’t often have time to write articles for academic journals. This is less problematic than it was just 20 years ago because of the impact and influence of the web-based publications created by research organizations. A well-designed and competently edited open-access report on a site like the Brookings Institution or the Urban Institute will be read by far more people (easily 10 times more) than anything in an academic journal (most of which are still hidden behind pay walls).
Some policymakers, of course, will always be more impressed when they hear about research findings from a prestigious academic journal. (Note: they’ll only hear about it, they won’t read it. Maybe they will ask their staff to summarize it).
Get to Know Your Research Partner
Researchers often find it hard to focus on the concrete, daily challenges facing their government or nonprofit partners. You have to be careful when signing up to work with researchers on something that is important to you or your agency. We all know how to appear as if we’re listening to you, but sometimes while we’re nodding and agreeing, we’re also thinking, “I wonder how much these guys are willing spend on this project?”
We’re like home-building contractors in that way. You may say, “I need my kitchen to be renovated.” But, if we think you’ll go for it, we’ll try to sell you a fancy new 2-bedroom addition because it’s what you “really” need. I advise all government and philanthropic funding agencies to focus on building a relationship with research partners over time. See how we do with your kitchen before asking how much a 2-bedroom addition would cost.
Next, you need to find researchers who value “translational science.” This means research done as rigorously as possible, but in a way that keeps you in the loop and that generates written products an average state legislator can read and absorb. You may not have taken a course on structural equation modeling or hierarchical linear regression, but you need to be a full partner. If your researcher uses technical jargon to exclude you or to control your conversations, get yourself a new researcher.
I always think of Jimmy Fallon’s character, “Nick Burns: Your Company’s Computer Guy” on Saturday Night Live nearly 20 years ago — the one with the greasy hair and pocket protector who would just hear a few words about someone’s computer problem and then blurt out “Move!,” as he grabbed their keyboard while insulting their intelligence and fixing their problem. Your researcher should not remind you of Nick Burns.
Good researchers derive real gratification from seeing you engage in their work. They take the time to help you develop your own understanding of it. If your research partner gets impatient with you asking questions, or is irritated when you don’t understand their jargon, get yourself a new researcher.
Early on in your work with a researcher, you need to have an honest conversation about your intended audience. In other words, why are you even conducting research? Who are you trying to reach? What decisions are you hoping to influence? Would you like to see your agency in Science Magazine as the leading developer of some wonderful new intervention, or are you just trying to retain the support of the Governor or the state legislature so they don’t cut your appropriation? Perhaps you want to generate public support or get a flattering profile in the local news media?
Researchers can help you do all these things, but you need them to be aware of and supportive of your needs up front. We will always want you to pay us to pursue Science-Magazine quality studies. Remember, we mainly care about burnishing our reputations within our own very, very small worlds. But, you probably do not need big science. You just want a researcher who wants to help you reach your goals and advance your work, not their own. The best arrangement is often to tell us that we are welcome to publish something in a fancy journal that nobody reads, but only AFTER we’ve completed all of our work for you.
Focus on time and cost. What is your timeline for designing, conducting, and disseminating the findings of the research? How much would it be worth to you if a study turns out perfectly? What if it doesn’t turn out perfectly? How much would you be willing to spend for less-than-perfect research?
Self-centered researchers advocate an approach that meets their needs. Then they try to sell you on it. Client-centered researchers ask: what are YOUR goals, when do YOU need the results, and how much can YOU afford to spend on it. Then, they work with you to come up with a plan you can afford and a production schedule that meets your needs. (And, FYI, they will probably still be late.)
Of course, a happy compromise is not possible every time. I remember a character in a movie or TV show I saw years ago. She was a contractor of some kind and speaking to a prospective client, advising them: “This project can be done good, cheap, or fast. I can always deliver on 2 of those, but never all 3.”
The same thing is true with research.
You just need to tell us about your values and priorities. If we don’t listen, get yourself a new research partner.