US Education’s Dominant Research Method: Cherry Picking Evidence

We all know about gerrymandering—the process by which politicians in a majority party sit down with a map and carve out electoral boundaries to maximize their party’s electoral advantage. Gerrymandering upends the electoral process. Rather than allow voters to choose their representatives, incumbent politicians choose their voters. More than 90 percent of US congressional elections are now noncompetitive, largely due to gerrymandering.

Some scholars employ a similar process in their research. Rather than allow all of the research “literature”—the full expanse of all relevant evidence on a topic—to lead them to a research conclusion, they reference only that part that supports their preferred conclusions. 

I call these scholars “dismissive reviewers,” because they ignore or declare nonexistent (i.e., dismiss) relevant evidence. When a group of dismissive reviewers cooperate they form a “citation cartel”—citing only each other’s research and dismissing all the rest. 

Readers of this blog already know that much—perhaps most—mainstream US education research is cherry picked. In part, that is how easily disprovable education myths persist. Education journals publish study after study that purports to consider all the relevant evidence on a topic but, in fact, references only that part of the evidence supportive of the myth. That proportion may be tiny, but it is still “evidence.”

This is why I find little reassurance in the phrase “evidence-based research.” All research is “evidence-based.” But, some is based on only part of the evidence available. Moreover, some of that is fraudulent. Education hosts a gargantuan quantity of research evidence. But, much is of poor quality. And, much more than most people realize is simply dishonest, with fabricated or doctored data, surreptitiously altered definitions of terms, selective references, and dismissive literature reviews. 

Despite its reputation as the most trustworthy of US education research sub-fields, the research conducted in education testing, or “psychometrics,” is no different. Some of it is poorly done. Some is biased. And, some cherry-picks its evidence to reach preferred conclusions. 

To my observation, honest, objective scholars still run things in the more technical realms of education testing research. In the realm of education testing policy, however, cherry pickers have run the show for over three decades. Moreover, they have managed to “capture” the testing policy research function at the National Research Council, the National Academy of Education, the World Bank, and the National Council of Measurement in Education (NCME), the primary US professional association of testing and measurement scholars in education. 

Indeed, just recently, NCME announced the names of the scholars who will write the testing policy section—”Accountability in K-12 Assessment”—of the next edition of the organization’s primary reference publication, Educational Measurement. NCME appears to have chosen a group that will assure continuity with past versions that reliably use cherry-picked evidence to advantage authors’ citation cartel. Furthermore, all four current authors and “reviewer-collaborators” have participated in Common Core promotion efforts and done work for the Gates Foundation.

By avoiding mention of rival evidence, and profusely referencing each other, citation cartel members can boost their own professional profiles, at the expense of other scholars’.

I have long been a strong advocate for education testing in general and standardized testing in particular. Yet, I would agree with many readers of this blog that US education testing policy has been sub-optimal since 2001 and remains so today. I do not share the conclusion that testing itself is responsible, however. Rather, responsibility lies with our country’s policymakers, in both major parties, who continue to rely on the advice of a relatively small group of policy analysts who limit their perspective to a pinhole of the available research evidence. 

Un-Rig the Research!

Newsday: New York Parents Must Have Students Take Assessments

New York State Education Building in Albany, NY
Photo credit: Matt H. Wade (CC-By-SA 3.0)

Newsday, a newspaper and news site in New York State, declared the war over various education reforms over in an editorial this week.

They wrote:

The war over Common Core standards that had gotten so heated it spawned a statewide political party actually ended fairly well by 2017. As students, teachers and parents got used to the new curricula and learning methods that had initially been enacted too fast and with too little training, the state replaced the name Common Core with “Next Generation English Language Arts and Mathematics Learning Standards.” It also allowed public comment on the standards, tweaking them but leaving them largely intact.

The fight to tie student test scores to teacher evaluations, though, is now dead. State law says the scores have to be part of the evaluations, but there is a moratorium on enforcing that rule which will almost certainly be extended until the law connecting student scores to teacher evaluations is repealed.

And any forceful attempt to make school districts push kids to sit for those tests appears to be dead, too. The state Board of Regents this week retreated on its plan to divert a portion of schools’ federal funds toward encouraging test participation at high opt-out schools, and to make those schools craft plans to reduce those rates.

Regarding the war over Common Core, unfortunately, I think too many, including members of the media, have bought into the rebrand. And that is what it is, a rebrand. Those who speak out against the standards will have to point out the specific problems within New York’s academic standards such as the standards of mathematical practice remain the same, and New York’s ELA standards still have an undue emphasis on informational text.

As for the other changes they mention, those are positive developments, and I hope they stay in place. That said, how Newsday finished the editorial irked me.

It’s good news that the state has managed to keep a set of rigorous standards to ensure students are ready for work or college when they graduate high school. But the unions and Regents who claim teachers can be properly and rigorously evaluated without tests scores must craft a plan to do so. And parents and teachers, having won the battle to decouple standardized tests and teacher evaluations, must have the kids take the tests. 

They buy into the same talking points that education reformers have foisted. No, New York’s standards are not rigorous. No, they will not ensure students are ready for work or college. That is propaganda. New York’s tweaked standards and Common Core does not have any data that backs up those claims.

Then the statement that parents “must have kids take the tests.” Must? No, the point is that parents, not the state, not the school district, and indeed not the editorial board of Newsday, decides what is best for their student. 

What hubris.

Are School Takeovers Racist Power Plays?

The Kentucky State Board of Education is considering a state takeover of Jefferson County Public Schools.

Last month I wrote about state takeovers of school districts. I think they centralize power at the state level thereby diminishing local control in education. I also don’t think they work.

Dr. Domingo Morel, Assistant Professor of Political Science at Rutgers University, ascribed a rather nefarious purpose behind them.

At The Conversation he wrote:

Why does there appear to be a sudden interest in state takeovers of local school districts in the South? Have Southern governors figured out the blueprint for improving education in urban communities, predominantly occupied by people of color? Hardly. I would argue that these governors have discovered a blueprint for the economic and political disempowerment of their urban centers.

Much like the state takeovers in northern cities in the 1990s and early 2000s, these state interventions are being led by Republican governors and majority-Republican state legislatures.

The school districts that are targeted for takeovers are in majority-Democratic cities with mostly black student populations and black political leadership. Once the states assume leadership of the school districts, the locally elected school board – which has historically served as the foundation for local political empowerment – is stripped of its power. Decision-making and control of the school budget is removed from the local community.

So in the Northern states, while ineffective, school takeovers done by Democratic and Republican governors alike were not racist. Now, in the South, schools are taken over because of racist political power grab?

Really? Or, could it be, that these are just the school districts that have a pattern of poor performance?

Dr. Morel must be able to read minds to ascertain the real motivation to take over schools.

Dr. Morel could have built a stronger consensus that school takeovers are, for the most part, ineffective and are an anathema to local control. Instead, he scored cheap political points. He certainly riled up the Democrat cheering section, but this does not provide an answer to the problem of failing schools.

Teacher Shortages Become More Acute

Teacher at Maxwell AFB Elementary/Middle School
(Air Force photo/Kelly Deichert)

Valarie Strauss at The Washington Post writes about the teacher shortages every state is facing this school year, a problem that has become acute in recent years.

An excerpt:

Teacher shortages are nothing new — most states have reported some since data started being kept more than 25 years ago — but the problem has grown more acute in recent years as the profession has been hit with low morale over low pay, unfair evaluation methods, assaults on due-process rights, high-stakes testing requirements, insufficient resources and other issues.

According to a 2016 report by the nonprofit Learning Policy Institute, teacher education enrollment dropped from 691,000 to 451,000, a 35 percent reduction, between 2009 and 2014, the latest year for which there is data. And there are high levels of attrition, with nearly 8 percent of the teaching workforce leaving every year, the majority before retirement age.

She lists the five key factors that the Learning Policy Institute cited in their report:

The Learning Policy Institute report found five key factors that influence whether a teacher decides to enter, remain in or leave the profession: salaries and other compensation; preparation and costs to entry; hiring and personnel management; induction and support for new teachers; and working conditions, including school leadership, professional collaboration and shared decision-making, accountability systems, and resources for teaching and learning.

Read her whole article here.

Like Strauss said this has been a problem for years, but it has become worse over the years. I can’t help notice the spike between 2009 through 2014. What was introduced? What changed?

While there isn’t empirical evidence to point to this as the cause I believe Common Core along with the accompanying assessments has been a factor. The Learning Policy Institute cites working conditions – I have heard from numerous teachers they no longer feel like they are in control of their classrooms. Also linking teacher evaluations to assessments impacts morale among teachers.

We’ve seen the 2014 teacher of the year call it quits over Common Core. Elementary school teachers have struggled with Common Core math. Another teacher in Colorado is just another example of teachers who have left the profession over Common Core.

We have no idea how many have decided they do not want to enter the profession because of top-down education reforms. I’m sure it isn’t an insignificant issue.

The Every Child Achieves Act Continues Test-based Accountability

U.S. Senator Lamar Alexander (R-TN)

U.S. Senator Lamar Alexander (R-TN)

The Senate bill to reauthorize No Child Left Behind (NCLB), titled the Every Child Achieves Act (ECCA), was heard on the Senate floor this past week. Senator Lamar Alexander, a sponsor of the bill, argued that passage of his bill would go a long way towards ending the pain inflicted under NCLB and provide relief from the testing mania that has permeated schools. Yet, the bill does very little to change the current test-based accountability at the state level.  It retains the 17 NCLB mandated annual tests in grades 3-12 and the inclusion of student test performance on state assessments in state accountability systems.

After the last round of annual testing took over 12 hours to administer, concerns over the federally mandated tests (preserved in ECAA) have reach an all-time-high.  In his remarks on the Senate floor, Sen. Alexander argued  that the current testing mania isn’t created by the federal requirement to take the tests, which he believes only take two hours out of the school year. The real culprit, he claimed, is NCLB’s requirement that student test scores are used as the basis of state accountability systems; it forces states and schools to over-test students in preparation for the high-stakes assessment. He continued that ECAA would end the test-driven culture caused by NCLB by allowing states to design their own accountability systems, thus relieving the pressure of a single test determining school performance.

The problem with this argument is that since the Obama Administration issued NCLB waivers in 2011, the NCLB accountability system referenced by Sen. Alexander hasn’t been used in 45 states. It is the new accountability system created under the waiver, not NCLB’s, that is currently exacerbating the amount of testing.

Under the waiver, states were allowed to develop “state-designed” accountability systems that included additional indicators, not just student test scores, in determining school performance. The hope was that this would stop the testing mania by relieving the pressure of having a single test score determine the school’s grade. Unfortunately, with these new accountability systems, the amount of instructional time lost to preparing students for state assessments has increased, not decreased, and states are experiencing the longest testing windows in history.

The “state-designed” accountability system prescribed by ECAA and touted by Sen. Alexander is very similar in structure to those required under the waivers; therefore, it is unclear why Sen. Alexander believes things will change.  Although ECAA allows states to use a variety of indicators in their overall state accountability system, such as school climate, attendance rates, etc., there is a catch — a big one.  When it comes to identifying low-performing schools — which is what schools fear the most and what drives test pressure — the state must use the same test-based indicators required under the waiver as a “substantial factor.” The additional indicators Sen. Alexander notes would be part of the state accountability system, but they wouldn’t be determining factors in identifying low-performing schools.

Below is a comparison of the high school indicators required under the waiver and the “substantial factors” required in ECAA:

HIGH SCHOOL INDICATORS UNDER NCLB WAIVERS (current)

  1. Student achievement on state assessments in math and English which includes a measure of student growth (based on state assessments)
  2. High school graduation rates
  3. A measure of College and Career Readiness — on track to graduate prepared for entry-level courses at the state’s public higher education institutions or the workforce with the need for remediation.

HIGH SCHOOL SUBSTANTIAL FACTORS UNDER ECAA (proposed)

  1. Student achievement on state assessments in math and English which includes a measure of student growth (based on state assessments)
  2. High school graduation rates
  3. Progress of students on state assessments necessary “to graduate from high school prepared for postsecondary education or the workforce without the need for postsecondary remediation.” (based on state assessments)

Sen. Alexander doesn’t get it. His bill doesn’t offer states more flexibility where it counts, only more of the same test-driven policies enforced by the USED waivers. Regardless of the additional indicators ECAA would allow states to include in the overall accountability system, student performance on the state assessments will continue to drive how schools are identified as low-performing. 

If this bill were to be implemented, schools, teachers, parents, and students would come to unfortunate realization that nothing has changed; schools would still be tethered to the test. The only difference would be that it is the “state designed” accountability systems- not the U.S Department of Education- that are responsible.  In other words, when the inevitable complaints arise about the poorly constructed accountability systems, the U.S. Department of Education will, disingenuously, disclaim responsibility and leave the states holding the bag.