US Education’s Dominant Research Method: Cherry Picking Evidence

We all know about gerrymandering—the process by which politicians in a majority party sit down with a map and carve out electoral boundaries to maximize their party’s electoral advantage. Gerrymandering upends the electoral process. Rather than allow voters to choose their representatives, incumbent politicians choose their voters. More than 90 percent of US congressional elections are now noncompetitive, largely due to gerrymandering.

Some scholars employ a similar process in their research. Rather than allow all of the research “literature”—the full expanse of all relevant evidence on a topic—to lead them to a research conclusion, they reference only that part that supports their preferred conclusions. 

I call these scholars “dismissive reviewers,” because they ignore or declare nonexistent (i.e., dismiss) relevant evidence. When a group of dismissive reviewers cooperate they form a “citation cartel”—citing only each other’s research and dismissing all the rest. 

Readers of this blog already know that much—perhaps most—mainstream US education research is cherry picked. In part, that is how easily disprovable education myths persist. Education journals publish study after study that purports to consider all the relevant evidence on a topic but, in fact, references only that part of the evidence supportive of the myth. That proportion may be tiny, but it is still “evidence.”

This is why I find little reassurance in the phrase “evidence-based research.” All research is “evidence-based.” But, some is based on only part of the evidence available. Moreover, some of that is fraudulent. Education hosts a gargantuan quantity of research evidence. But, much is of poor quality. And, much more than most people realize is simply dishonest, with fabricated or doctored data, surreptitiously altered definitions of terms, selective references, and dismissive literature reviews. 

Despite its reputation as the most trustworthy of US education research sub-fields, the research conducted in education testing, or “psychometrics,” is no different. Some of it is poorly done. Some is biased. And, some cherry-picks its evidence to reach preferred conclusions. 

To my observation, honest, objective scholars still run things in the more technical realms of education testing research. In the realm of education testing policy, however, cherry pickers have run the show for over three decades. Moreover, they have managed to “capture” the testing policy research function at the National Research Council, the National Academy of Education, the World Bank, and the National Council of Measurement in Education (NCME), the primary US professional association of testing and measurement scholars in education. 

Indeed, just recently, NCME announced the names of the scholars who will write the testing policy section—”Accountability in K-12 Assessment”—of the next edition of the organization’s primary reference publication, Educational Measurement. NCME appears to have chosen a group that will assure continuity with past versions that reliably use cherry-picked evidence to advantage authors’ citation cartel. Furthermore, all four current authors and “reviewer-collaborators” have participated in Common Core promotion efforts and done work for the Gates Foundation.

By avoiding mention of rival evidence, and profusely referencing each other, citation cartel members can boost their own professional profiles, at the expense of other scholars’.

I have long been a strong advocate for education testing in general and standardized testing in particular. Yet, I would agree with many readers of this blog that US education testing policy has been sub-optimal since 2001 and remains so today. I do not share the conclusion that testing itself is responsible, however. Rather, responsibility lies with our country’s policymakers, in both major parties, who continue to rely on the advice of a relatively small group of policy analysts who limit their perspective to a pinhole of the available research evidence. 

Un-Rig the Research!

What Are Gubernatorial Candidates Saying About Education?

The tan colored states represent gubernatorial elections in 2018.

There are 36 gubernatorial contests in 2018 with 269 declared candidates. What are they saying about education?

According to Rick Hess and Sofia Gallo at American Enterprise Institute, not so much.

They wrote on Wednesday at Real Clear Policy:

So, during the first half of February, we used the National Governors Association website and Ballotpedia to identify the 269 declared gubernatorial candidates and then visited the websites for each. There were 121 candidates who had no website (a tiny handful) or who offered no information regarding their education positions. For the 148 candidates who had something to say on education — including 63 Republicans and 85 Democrats — we examined their sites to see what topics addressed and what they had to say. What did this exercise reveal?

First, there’s been a marked shift from many of the concerns that predominated 4 or 8 years ago. Candidates devoted little attention to topics like school accountability (mentioned by just nine candidates), teacher evaluation (mentioned by just five), or the Common Core (mentioned by 17). When testing and standards do arise, candidates don’t have many good things to say. For instance, the mentions of academic standards and the Common Core are overwhelmingly negative — with more than 80 percent denouncing them. Similarly, just one candidate makes a positive reference to testing; the other 19 candidates who mention the topic all promise to reduce the number of tests.

Second, the only educational issue that registered support from a majority of candidates was career and technical education (CTE), which received enthusiastic bipartisan backing. More than 60 candidates — including 40 Democrats and 24 Republicans — endorsed expanding CTE.

He also noted that there was little attention paid to school choice either positive or negative. I can vouch for this in Iowa, beyond school spending, CTE was part of Iowa Governor Kim Reynolds’ Condition of the State Address. She also mentioned school choice, but through accessing 529 savings accounts used for parents to save for college, not ESAs or vouchers. She also discussed STEM.

He did note that when gubernatorial candidates talk about CTE they all don’t mean the same thing.

By “career and technical education,” some mean vocational schools while others mean apprenticeships; some are championing more high school programs while others are thinking about community college systems.

Reynolds pointed to a new program called Future Ready Iowa that will implement pre-apprenticeships for high school students.

For the most part, it’s been pretty quiet on the education front on matters of policy (beyond spending which is always an issue). In terms of trying to find candidates who will challenge top-down reform and repeal top-down standards, it is challenging.

As you look for a candidate to support you’ll have to take the initiative to get candidates to talk about standards, assessments, and data privacy. It’s much easier to ask your questions during the primary process than it will be the general election. If there are opportunities to get to meet candidates and ask them questions, be sure to take advantage of it. Of course, talk is cheap, be sure to check out their record if they’ve been in elected office as an incumbent governor or as a legislator.

I plan to highlight those who are speaking out against Common Core and top-down standards here.

“People’s Views on Education Policy are Quite Malleable”

Photo credit: Cubmundo (CC-By-SA 2.0)

“People’s views on education policy are quite malleable.”

That was the conclusion of a study that Stephen Sawchuk wrote about at Education Week. The piece is called “Here’s One Way to Dispel Misconceptions About Common Core.”

This article does not reveal anything new. Common Core proponents have asserted this ever since the standards were first released. “We just don’t understand them,” they said.

Every negative thing written about them is “simply misinformation” they claim.

Sawchuk writes:

The respondents were given a set of six true-and-false questions on the common core, including these. (The answer to all three is false.)

  • Common Core requires more testing than previous standards.
  • The federal government required states to adopt the common core.
  • The Common Core State Standards were developed by the Obama administration.

They were also asked about whether they approved or disapproved of the standards.

Then, half of the sample were given a short refutation text created by the researchers; the other half, a control group, were given Education Week‘s own explainer on the common core, though in a significantly altered format—the researchers cut it from 1,400 words down to 360.

After reading these, the panelists were asked to take the quiz again. They took it a third and final time after a week.

Initially, the respondents were neutral on the common core and held a number of misconceptions. (Just 16 percent got the first question, on testing, correct.)

But upon reviewing the refutation text, the treatment group had a significantly reduced number of misconceptions and more correct conceptions of the standards; they were also likely to support the standards than before. The effects declined somewhat after a week but were still statistically significant.

The control group also improved, but the treatment group outperformed the control group on four of the six questions—a function, the authors believe, of the refutation structure explicitly built into the treatment text, but not into the modified Education Week article.

Congratulations, what Stephen Sawchuk discovered here is push polling and that it can be useful (which is why campaigns and political groups use it). What an earthshattering discovery.

We’ve had to deal with misconceptions from those pushing Common Core. Here’s a list I would include if I did my own “study.”

  • Common Core was state-led.
  • Common Core State Standards are more rigorous standards.
  • Common Core is internationally benchmarked.

All of the above statements are false even though Common Core advocates will claim that they are correct.

Common Core was special interest-led. Last I checked the National Governors’ Association and Council of Chief State School Officers are special interest groups located in Washington, DC. They are not government entities even though it sounds that way. Governors did not develop these standards, workgroups selected by NGA and CCSSO did. Also, state legislatures, for the most part, did not vote on Common Core before the implementation of the new standards.

Common Core State Standards are more “rigorous”? First, what does that mean? Secondly, based on what? Certainly not California’s math standards or Massachusetts’ ELA standards.

As far as international benchmarking for Common Core is concerned, that talking point was later changed to “informed by international benchmarks,” and eight years later I’m still uncertain what country they compared themselves to.

Initial polling for Common Core was quite high until we and others started to challenge what proponents were putting out there.

Now have Common Core opponents provided inaccurate information? Yes, and I have challenged some of it at Truth in American Education. He gave two examples. The first being, “Kids won’t read fiction anymore.”

That’s hyperbole that points to a truth. Kids will still read fiction, but they will read much less fiction as the standards call for more informational text to be read. Also, how much of the fiction that kids read represents entire literary works or just excerpts?

By the time a student is a senior in high school, under Common Core, 70 percent of what they read is informational text. That is simply a fact.

The other example he gives “schools are scanning children’s irises.”

I agree with him that this is not Common Core because Common Core State Standards in and of themselves are just academic benchmarks. Common Core was part of a package of reforms that included aligned assessments and data collection. All of those things were requirements to be eligible for Race to the Top money and the federal government spent a lot of money on the assessment consortia and statewide longitudinal database systems.

As far as iris scans at schools, yeah, so unbelievable.

Beware of Experts in Education Policy

Frederick Hess, resident scholar and director of education policies at American Enterprise Institute, warned that we should beware of experts, especially those in education policy, in AEI’s latest In 60 Seconds video.

Watch below:

Hess is right.

Hess mentioned education policy experts brought us No Child Left Behind and School Improvement Grants that did more harm than good. Had he had more time, based on Hess’ writings, I’m sure Race to the Top and Common Core would make his list as well.

Education policy experts often think top-down when the best solutions typically come from the local level. Top-down “experts” will never know local schools as well as parents, teachers, and administrators involved in those schools and school districts do.

I have yet to see a top-down idea work. I wouldn’t put much stock into education policy “experts” unless they are pushing local control and local solutions for problems in public education.

President-Elect Donald Trump on Education


President-Elect Donald Trump… what will he mean for education? I wrote about his education policy before and I just wanted to refresh our memories. He said in a video on his website:

I am tremendous believer in education, but education has to be at a local level. We can not have the bureaucrats in Washington telling you how to manage your child’s education. So Common Core is a total disaster, we can’t let it continue. We are rated 28 in the world, the United States, think of it – 28 in the world, and frankly we spend far more per pupil than any other country in the world by far it’s not even a close second.

So here we are, we spend more money and we are rated 28. Third world countries are ahead of us. We are going to end Common Core. We are going to have education, an absolute priority.

He has had good instincts on Common Core. Ultimately that is something that will need to be stopped at the state level. Having a presidential administration that doesn’t push it will be helpful however.

I wrote at the time.

The primary problem I have with Donald Trump when it comes to education policy is the lack of specificity. Why does he believe Common Core is bad? Why does he believe education needs to be at the local level. I want to know he understands the issue and isn’t just spouting off talking points.

How will he end Common Core when states have adopted these? Does this mean he’ll end the carrot and stick approach to federal education funding? Will he work to repeal the Every Student Succeeds Act? Will he work to end the U.S. Department of Education?

I will also be curious how he will navigate a Congress who has passed several bad education bills of late. All that said I know Hillary Clinton would have been a disaster, and I suspect Donald Trump will be better if follows through on devolving education policy to the states – something that wouldn’t have happened under a Clinton administration.

Another ray of light with Tuesday’s results. Bill Evers is advising him on education policy. Secretary of Education Bill Evers has a nice ring to it.

Hoover Institution’s Bill Evers to Advise Donald Trump on Education

williamson_eversWilliamson (Bill) Evers, a research fellow for the Hoover Institution at Stanford University, has been tapped to serve on Donald Trump’s presidential transition team as an education advisor should he win the election. In terms of how a Trump administration would handle education this is a good sign. Bill has been a friend of Truth in American Education and an outspoken opponent of the Common Core State Standards.

Education Week reports:

Republican presidential nominee Donald Trump has picked Williamson M. Evers, a research fellow at the Hoover Institution at Stanford University, and Gerard Robinson, a resident fellow at the American Enterprise Institute, to be on his presidential transition team for education, according to multiple sources.

Evers served as an assistant secretary for policy at the U.S. Department of Education from 2007 to 2009, and also was an adviser to former U.S. Secretary of Education Margaret Spellings in 2007 under President George W. Bush. Robinson served as Florida’s education commissioner from 2011 to 2012, and has also served as Virginia’s education secretary and as the president of the Black Alliance for Educational Options.

However, I don’t know anything about Gerard Robinson. If anyone from Florida or Virginia read this I’d appreciate you leaving a comment with information.

Gary Johnson: Let’s Get Rid of the Department of Education

Gary Johnson

Former New Mexico Governor Gary Johnson as the Libertarian candidate for President unsurprisingly wants to get rid of the U.S. Department of Education.

How about getting rid of the Department of Education?” Johnson asks in a video on his education issue page. “Washington can’t educate our kids. We used to have the brightest kids in the world, and we can again, but the Department of Education stands in the way.”

His education page shares the following:

Governor Gary Johnson was one of the first governors in the nation to propose and advocate a universally available program of school choice.

He did so while governing with an overwhelmingly Democrat legislature and while facing a powerful teachers’ union. He was well aware that his proposal would not be enacted and would generate fierce opposition. However, he believed it was important to raise the issue and force the teachers’ unions to defend a clearly failing status quo.

More broadly, Gov. Johnson believes there is no role for the Federal Government in education. He would eliminate the federal Department of Education, and return control to the state and local levels. He opposes Common Core and any other attempts to impose national standards and requirements on local schools, believing the key to restoring education excellence in the U.S. lies in the innovation, freedom and flexibility that federal interference inherently discourages.

As Governor, he saw first-hand that the costs of federal education programs and mandates far outweigh any benefits, both educationally and financially.

Well we know Johnson wouldn’t make education a priority for the federal government. I can’t criticize his education policy position as it is where I’m at. The only problem is that the likelihood of a President Gary Johnson is very, very slim.

Common Core Reduces Juvenile Crime?

juvenile-crimeI was shocked yesterday when I came across news that the Tennessee Association of Chiefs of Police came out in support of the Common Core State Standards.  Why in the world would police officers inject themselves into an education policy debate?

I smell Gates money. reports:

According to a report released at the press conference titled, “Helping students succeed cuts crime,” only 24 percent of eighth graders in Tennessee are proficient in math, 27 percent are proficient in reading and 31 percent are proficient in science while 14 percent of Tennessee freshman do not graduate in four years.

Law enforcement officials have a few reasons for supporting the new common core standards. The biggest reason is to keep kids out of jail.

“We support Fight Crime: Invest in kids,” said Murfreesboro Police Chief, Glenn Chrisman. “When we have an educated population, number one: it makes society much more livable and number two: keeps people out of our prison system.”

Fight Crime: Invest in Kids is a national, non-profit organization. The organization consists of more than 5,000 chiefs, sheriffs, prosecutors, attorneys general and violence survivors.

Common Core state standards are supposed to provide a consistent, clear understanding of what students are expected to learn in English language arts and mathematics. The common core standards are expected to prepare students for higher education or for entry into the workforce. Tennessee was one of the 12 states that led the development of common core standards. Full implementation of common core standards for Tennessee is expected for the 2014-2015 school year.

All the police chiefs believe educating children leads to less crime later on. 

“Tennessee students need an education that prepares them for the postsecondary education and the workforce so they will be less likely to turn to a life of crime,” said Johnson City Chief of Police, Mark Sirois.

Having worked with juvenile offenders for over 13 years and with youth in general for 20, I know a little something about risk factors for youth.  It is nonsensical to think that centralized standards will prevent juvenile crime and adult crime down the road when there is no data that demonstrates it raises student achievement.

Getting a good education is one factor in helping steer high-risk kids on the right track, but it is nonsensical to think that the Common Core State Standards will help in this endeavor.  These police chiefs would be better off advocating and pushing for mentoring.  That has provided tangible results.  Centralized standards such as the Common Core hasn’t.

Photo credit: Lou Angeli via Flickr (CC-By-NC-SA 2.0)

Should High Schoolers Be Required to Read Executive Order 13423?

Ezra Klein asks this at WashPo’s Wonkblog:

Top story: Are we getting smart on education policy?

Common core standards become a lightning rod. ”The Common Core State Standards in English, which have been adopted in 46 states and the District, call for public schools to ramp up nonfiction so that by 12th grade students will be reading mostly ‘informational text’ instead of fictional literature. But as teachers excise poetry and classic works of fiction from their classrooms, those who designed the guidelines say it appears that educators have misunderstood them…Among the suggested non­fiction pieces for high school juniors and seniors are Alexis de Tocqueville’s “Democracy in America,” “FedViews,” by the Federal Reserve Bank of San Francisco (2009) and “Executive Order 13423: Strengthening Federal Environmental, Energy, and Transportation Management,” published by the General Services Administration.” Lyndsey Layton in The Washington Post.

No, no we are not, and no they shouldn’t for reasons that Stanley Kurtz pointed out in my earlier post today.