US Education’s Dominant Research Method: Cherry Picking Evidence

We all know about gerrymandering—the process by which politicians in a majority party sit down with a map and carve out electoral boundaries to maximize their party’s electoral advantage. Gerrymandering upends the electoral process. Rather than allow voters to choose their representatives, incumbent politicians choose their voters. More than 90 percent of US congressional elections are now noncompetitive, largely due to gerrymandering.

Some scholars employ a similar process in their research. Rather than allow all of the research “literature”—the full expanse of all relevant evidence on a topic—to lead them to a research conclusion, they reference only that part that supports their preferred conclusions. 

I call these scholars “dismissive reviewers,” because they ignore or declare nonexistent (i.e., dismiss) relevant evidence. When a group of dismissive reviewers cooperate they form a “citation cartel”—citing only each other’s research and dismissing all the rest. 

Readers of this blog already know that much—perhaps most—mainstream US education research is cherry picked. In part, that is how easily disprovable education myths persist. Education journals publish study after study that purports to consider all the relevant evidence on a topic but, in fact, references only that part of the evidence supportive of the myth. That proportion may be tiny, but it is still “evidence.”

This is why I find little reassurance in the phrase “evidence-based research.” All research is “evidence-based.” But, some is based on only part of the evidence available. Moreover, some of that is fraudulent. Education hosts a gargantuan quantity of research evidence. But, much is of poor quality. And, much more than most people realize is simply dishonest, with fabricated or doctored data, surreptitiously altered definitions of terms, selective references, and dismissive literature reviews. 

Despite its reputation as the most trustworthy of US education research sub-fields, the research conducted in education testing, or “psychometrics,” is no different. Some of it is poorly done. Some is biased. And, some cherry-picks its evidence to reach preferred conclusions. 

To my observation, honest, objective scholars still run things in the more technical realms of education testing research. In the realm of education testing policy, however, cherry pickers have run the show for over three decades. Moreover, they have managed to “capture” the testing policy research function at the National Research Council, the National Academy of Education, the World Bank, and the National Council of Measurement in Education (NCME), the primary US professional association of testing and measurement scholars in education. 

Indeed, just recently, NCME announced the names of the scholars who will write the testing policy section—”Accountability in K-12 Assessment”—of the next edition of the organization’s primary reference publication, Educational Measurement. NCME appears to have chosen a group that will assure continuity with past versions that reliably use cherry-picked evidence to advantage authors’ citation cartel. Furthermore, all four current authors and “reviewer-collaborators” have participated in Common Core promotion efforts and done work for the Gates Foundation.

By avoiding mention of rival evidence, and profusely referencing each other, citation cartel members can boost their own professional profiles, at the expense of other scholars’.

I have long been a strong advocate for education testing in general and standardized testing in particular. Yet, I would agree with many readers of this blog that US education testing policy has been sub-optimal since 2001 and remains so today. I do not share the conclusion that testing itself is responsible, however. Rather, responsibility lies with our country’s policymakers, in both major parties, who continue to rely on the advice of a relatively small group of policy analysts who limit their perspective to a pinhole of the available research evidence. 

Un-Rig the Research!