Do you work in government, philanthropy, or the nonprofit sector? Do you know enough about evaluation research to know it’s important, that it’s a specialized skill, and that many people can’t tell the difference between actual research and advocacy which happens to includes numbers?

If you answered “yes” to these questions, you already know a lot. Now I want to take you behind the curtain and tell you how research works. Mainly, I want you to work effectively with research partners and to survive the experience.

What is research?

First, you should know what researchers mean when we say “evaluation” and “research.” It is not simply using numbers to back up a story, and it’s not just dropping in a couple of data graphs to give context to what is otherwise a position paper or a statement of belief.

I once wrote a book chapter with my friend, John Roman, in which we tried to explain research to a non-academic audience. In our chapter, we said that research is:

“any systematic investigation that uses a predefined standard of quality to determine its methods and to govern the interpretation of evidence. The use of a predefined standard of quality is essential for distinguishing research from argument and speculation (or even fabrication). Ideally, a researcher will apply the highest possible standard in collecting and analyzing evidence to reach a conclusion. The appropriate standard of quality, however, may vary according to the type of research being conducted and the ways in which the findings are to be applied. Within the wide range of investigations that legitimately could be called research, quality standards might vary from sufficient, to persuasive, to exacting.”

From: Jeffrey A. Butts and John K. Roman (2011). Better Research for Better Policies, in Juvenile Justice: Advancing Research, Policy, and Practice. Sherman, Francine and Francine Jacobs (Editors). Hoboken, NJ: John Wiley & Sons.

Research is an attempt to answer empirical questions by assembling and analyzing relevant data and then drawing unbiased conclusions from that analysis using rigorous standards of evidence and causal inference. Good researchers don’t just accept the first answer appearing in their data. They check their conclusions repeatedly and use their data in an attempt to prove themselves wrong before releasing any findings.

If you’ve ever worked with researchers, you know this behavior. We always look for alternative data interpretations that cast doubt on previously accepted facts. It can be annoying to non-researchers, but we actually enjoy it.

What is evaluation research?

When governments or private agencies try to reduce social problems and improve community conditions, evaluation researchers measure the effects of the effort. Did it work? How did it work? Would it likely work again?

Evaluation projects may involve primary data (newly collected) or secondary data such as agency records, police statistics, government reports, etc.

Whatever data are used, the researcher’s job is to organize the information with a research design that allows for a fair, unbiased test of cause and effect. No test is ever perfectly free of bias, but researchers eliminate as many sources of bias as possible.

The most appealing evaluation strategy is the randomized study or experiment. Groups of people or places are divided randomly. Some get the intervention; others do not. Outcomes are then compared for both groups.

A randomized experiment gives researchers higher confidence in the results when done well (and if it is even feasible). But it is not the only method available for evaluation.

In another article with John Roman, we said the analytical design of a study is not even the first consideration in determining its quality and utility. The most important goal for evaluation research is asking the right question. “Good Questions” come first; methods second.

“Experimental studies (or randomized controlled trials, RCTs) can produce high-quality evidence, but often within a relatively narrow frame. Randomization may be the method of choice for evaluating some interventions and practices in criminal justice, but other methods are essential as well, especially when researchers need to measure the effects of policy. The choice of evaluation method should be governed by the nature of the questions being asked and by the complexity and nuance of the answers required to apply the results, not by a reflexive preference for randomization.”

From: Butts, Jeffrey A. and John K. Roman (2018). Good Questions: Building Evaluation Evidence in a Competitive Policy Environment. Justice Evaluation Journal, 1(1): 15-31.

To ask the right questions, researchers must know a lot about the program or policy they are evaluating before expressing a preference for a research design. If your research partner tries to convince you to support a particular evaluation design before they really understand your situation, you are probably talking to someone in sales, not research. Get a new partner.

Who does research?

People often assume that anyone who works at a university is a researcher. This is not true. Even someone with the title of “professor” may not be a researcher. They may be great teachers and writers, and even clever thinkers, but most of the stuff written by people at universities does not qualify as research, and certainly not evaluation research.

Not all researchers are academics, and not all academics are researchers, or at least not the type of researchers who create actionable information that can be used to improve policies and programs in concrete ways. In the accompanying graphic, I divided the research community into overlapping but distinct groups: academic researchers who I call “scholarly,” researchers largely outside universities who I describe as “entrepreneurial,” and finally the governmental sector.

The overlap between the scholarly and entrepreneurial groups consists of university-based researchers who are not instructional faculty but hold administrative positions, research professorships, or grant-funded appointments. Their work may exhibit scholarly characteristics as they operate within the academic culture. (Full disclosure: this overlap is where I live.)

If you have ever hired or supervised the work of researchers, you probably know a bit about these variations in the larger research community and how the groups relate to one another. But, let me explain how I see them.

Academics/scholarly researchers

University-based researchers can be pretty self-absorbed. With apologies to my academic colleagues, we’re often not what you’d call socially adept. Academic or scholarly researchers focus a lot of energy on competing with one another for prestige and appointments. The main way to do that in a university setting is to be well-known for using innovative research methods and to publish as many articles as possible in just one or two specialized areas to dominate the knowledge base. If they become famous and successful, academics just want to write books and have people listen to them talk about what they wrote in their last book.

There are two problems with this if you’re in government or some other funding organization: 1) academic researchers are mainly helpful if what you need happens to align with their “interests”; and 2) their idea of success is to publish things that impress people who already know a lot about their specialty areas. Of course, this means that few people will ever read their work, but this doesn’t bother them—as long as “the right people” read it.

Certainly, there are exceptions within academia (see the overlapping area of the Venn diagram). The exceptions are often people who had other careers before academia–i.e., “real jobs.” This would include people who were cops, lawyers, social workers, etc. before going back to graduate school to get advanced research training. When people have professional experience before becoming researchers, their research tends to focus on practical challenges. It’s not just their intellectual “interest”–it’s their mission. (Again, this might be self-serving since I’m one of those people).

Entrepreneurial researchers

The other large community of researchers you will encounter are people working for one of the many for-profit or non-profit research firms. This would include places like the RAND Corporation, the Urban Institute, Westat, RTI, MDRC, NORC, etc. These researchers may be as well-trained and talented as university-based researchers, but their incentives differ. They are in the “business” of research. They have business interests to consider.

Entrepreneurial researchers have to think about more than their contributions to knowledge. They have to think about revenue, how many staff will be supported (i.e., funded) by a new project, whether spending the time to bid on new projects could interfere with existing projects, and whether they can raise enough revenue after a project is completed to hold onto the staff hired for that project. A successful researcher with 5 staff members would be excited to land a new contract requiring 10 new people. When that project is over, however, they would need to raise enough funding to continue paying for 15 people–or implement layoffs, which scare people and sometimes cause the most talented staff to look for other jobs. Growth is terrific, but it has to be managed carefully.

Being a professional, non-academic researcher is like running a small business. You need customers to hire staff and staff to serve your customers. And, you need to pay yourself. Keeping all of that in balance can get tricky and stressful. Academic researchers–especially tenured faculty–are somewhat insulated from this dynamic. An interruption in funding for an academic often means they might have to “go back to teaching” (the horror!). An interruption in funding for an entrepreneurial researcher usually means they must find a “job.”

Entrepreneurial researchers are always busy managing current projects, launching new projects, and trying to “cover” their staff. They don’t often have time to write articles for academic journals. This is less problematic than it was just 20 years ago because of the impact and influence of the web-based publications created by research organizations. A well-designed and competently edited open-access report on a site like the Brookings Institution or the Urban Institute will be read by far more people (easily 10 times more) than anything in an academic journal (most of which are still hidden behind paywalls). Of course, some policymakers are more impressed when they hear about research findings in a prestigious academic journal. (Note: They’ll only hear about it. They won’t read it. Maybe they will ask their staff to summarize it).

Researchers from the entrepreneurial sector present one particular risk not found as often with academic researchers. Because they are in the business of research, entrepreneurial researchers are sometimes tempted (perhaps even forced) to fashion a study’s findings to suit clients’ expectations. A client may not see this as a risk, at least initially. When researchers adapt or distort the findings of a study to please the funder, the client may be pleased, thinking the cost of the research was money well spent. The danger is that reputations catch up with everyone at some point. A client pleased with a dubious evaluation will come to regret it when the work of the research organization is exposed, as it eventually will be. A study once used to buttress the client’s credibility may become a source of dishonor and shame.

Get to know your research partner

Researchers often struggle to focus on the daily challenges facing their government or nonprofit partners. When signing up to work with researchers on something important to you or your agency, you must be careful. We all know how to appear as if we’re listening to you, but sometimes while we’re nodding and agreeing, we’re thinking, “hmm, I wonder how much these guys are willing to spend on this project?”

We’re like home-building contractors in that way. You may say, “I need to renovate my kitchen.” But, if we think you’ll go for it, we’ll try to sell you a fancy new 2-bedroom addition because hey, it’s what you “really” need. I advise all government and philanthropic funding agencies to focus on building a relationship with research partners over time. See how we do with your kitchen before asking how much a 2-bedroom addition would cost.

Next, you need to find researchers who value “translational science.” This means research is done as rigorously as possible, but in a way that keeps you in the loop and that generates written products an average state legislator can read and absorb. You may not have taken a course on structural equation modeling or hierarchical linear regression, but you must be a full partner. If your researcher uses technical jargon to exclude you or to control your conversations, get yourself a new researcher.

I always think of Jimmy Fallon’s character, “Nick Burns: Your Company’s Computer Guy” on Saturday Night Live 20 years ago — the one with bad hair and a pocket protector who would just hear a few words about someone’s computer problem and then blurt out “Move!,” as he grabbed their keyboard while insulting their intelligence and fixing their problem. Your researcher should not remind you of Nick Burns.

Good researchers derive professional gratification from seeing you engage in their work. They take the time to help you develop your own understanding of it. If your research partner gets impatient with you asking questions, or is irritated when you don’t understand their jargon, get yourself a new researcher.

Early in your work with a researcher, you must have an honest conversation about your intended audience. In other words, why are you even conducting research? Who are you trying to reach? What decisions are you hoping to influence? Would you like to see your agency in Science Magazine as the leading developer of some wonderful new intervention, or are you just trying to retain the support of the Governor or the state legislature, so they don’t cut your appropriation? Perhaps you want to generate public support or get a flattering profile in the local news media?

Researchers can help you do all these things, but you need us to be aware of and supportive of your needs. Remember, we start out mainly caring about burnishing our reputations within our professional networks. We will always want you to pay us to pursue Science-Magazine quality studies, but you probably don’t need big science. You need researchers who want to help you reach your goals and advance your work, not their own. The best arrangement is often to tell us that we are welcome to publish something in a fancy journal that nobody reads, but only AFTER we’ve completed all of our work for you.

Final word

Focus on time and cost. What is your timeline for designing, conducting, and disseminating the research findings? How much would it be worth if a study turned out perfectly? What if it doesn’t turn out perfectly? How much would you be willing to spend for less-than-perfect research?

Self-centered researchers advocate an approach that meets their needs. Then they try to sell you on it. Client-centered researchers ask: what are YOUR goals, when do YOU need the results, and how much can YOU afford to spend on it. Then, they work with you to devise a plan you can afford and a production schedule that meets your needs. (And, FYI, they will probably still be late.)

Of course, a happy compromise is not possible every time. I remember a character in a movie or TV show I saw years ago. She was a contractor of some kind, speaking to a prospective client, and advising them: “This project can be done good, cheap, or fast. I can always deliver on 2 of those, but never all 3.”

The same thing is true with research.

You just need to tell us about your values and priorities. If we don’t listen, get yourself a new research partner.