Why Gates’ Big Data Experiment Assessing Teacher Performance Failed

Cathy O’Neil, a mathematician, hedge-fund analyst and data scientist, points out in a piece for Bloomberg yesterday why the Gates’ Foundation’s Intensive Partnerships for Effective Teaching experiment not only failed but did actual harm to public schools.

Gathering data from assessments, principal observations of teachers, and evaluations from students and teachers they used an algorithm to determine whether a teacher was adding value.

She writes that the goal was to reward the good teachers and root out the bad.

She writes:

Laudable as the intention may have been, it didn’t work. As the independent assessment, produced by the Rand Corporation, put it: “The initiative did not achieve its goals for student achievement or graduation,” particularly for low-income minority students. The report, however, stops short of drawing what I see as the more important conclusion: The approach that the Gates program epitomizes has actually done damage. It has unfairly ruined careers, driving teachers out of the profession amid a nationwide shortage. And its flawed use of metrics has undermined science.

The program’s underlying assumption, common in the world of “big data,” is that data is good and more data is better. To that end, genuine efforts were made to gather as much potentially relevant information as possible. As such programs go, this was the best-case scenario.

Still, to a statistician, the problems are apparent. Principals tend to give almost all teachers great scores — a flaw that the Rand report found to be increasingly true in the latest observational frameworks, even though some teachers found them useful. The value-added models used to rate teachers — typically black boxes whose inner workings are kept secret — are known to be little better than random number generators, and the ones used in the Gates program were no exception. The models’ best defense was that the addition of other measures could mitigate their flaws — a terrible recommendation for a supposedly scientific instrument. Those other measures, such as parent and student surveys, are also biased: As every pollster knows, the answer depends on how you frame the question.

Read the whole thing.

It must be awesome to get to purchase education policy. How many schools, teachers, and students will they experiment on when they finally learn that this is not the way to go about determining education policy?

238 Education Data Bills Hit State Capitols in 2018 So Far

Photo credit: Nick Youngson (CC BY-SA 3.0)

Data Quality Campaign (not our ally in the fight against data mining) provided a snapshot of the number of education data bills hitting state capitol buildings near you.

They report there are 238 bills related to education data this year so far, and less than a third (70) have anything to do with protecting student data privacy.

They highlight bills before state legislators this year where they are trying to “make data work for students.”

Addressing inequities and underserved students’ needs

Echoing national conversations about disciplinary disparities and the unique needs of traditionally underserved students, numerous state bills this year target the reporting of data to address education inequities. For example:

  • Tennessee is considering a bill (HB 2651) to establish a commission on the school-to-prison pipeline. The commission would submit a report to the legislature including school discipline data and policy recommendations to implement restorative justice practices.
  • Indiana has a new law (HB 1314) requiring a report on how the state’s homeless students and students in foster care fare in school and how these students could be better supported.

Informing policy decisions and meeting state goals

Nearly 100 bills considered so far in 2018 have focused on how state policymakers themselves can use aggregate data to make policy decisions or meet their state’s education goals. For example:

  • California has introduced a bill (SB 1224) to create a state longitudinal data system (SLDS) with student data from kindergarten enrollment to workforce entry—a system that could help inform education policies across the state.
  • Mississippi considered a bill (HB 405) to use the state’s education data system to better understand the state’s workforce needs.

Empowering the public with more information

Over 60 bills this year would require states to publicly report more, or more useful and accessible, information about their schools. For example:

  • New Jersey is considering a bill (A 2192) to include data on chronic absence and disciplinary suspensions on school report cards.
  • Arizona is considering a bill (SB 1411) to create a new dashboard as part of the state’s school achievement profiles with new data on academic progress and school quality.

Empowering educators and families with student data

In years past, legislators have not frequently used legislation to give educators and parents secure access to their own student’s data. This year is seeing some more legislative activity on this important priority. For example:

  • Louisiana is considering a bill (SB 107) to ensure that teachers receive student-level assessment results in a format that is easy to understand and includes longitudinal student data if possible.
  • Massachusetts is considering a bill (S 40) that would create an electronic data “backpack” program for foster youth. The backpack would contain a student’s education record and would be available to the adults authorized to make decisions for that student.

The best way to “make data work for students” is to not collect it without parental knowledge and consent and to keep it at the local school level with the teachers where it could possibly do some good. The problem is, evidenced by the Louisiana bill, when data gets collected it heads to the state (and the feds and who knows what other third parties) who don’t teach the kids and have no business having that data.

Jane Robbins Testifies to House Committee on Education & the Workforce

Jane Robbins, senior fellow, with American Principles Project was one of four witnesses for the House Committee on Education and the Workforce‘s hearing on “Evidence-Based Policymaking and the Future of Education.”

Robbins and Paul Ohm, a professor at the Georgetown University Law Center, were the only two witnesses who seemed to be concerned with student privacy. The other two witnesses, Dr. Casey Wright, Mississippi’s State Superintendent of Education, and Dr. Neal Franklin, the program director for Innovation Studies with WestEd, were touted the gains that could be made with student data.

You can watch the whole hearing below:

Here are the transcript and video of Robbins’ opening statement to the committee:

Madam Chairman and members of the committee:

My name is Jane Robbins, and I’m with the American Principles Project, which works to restore our nation’s founding principles. Thank you for letting me speak today about protecting privacy when evaluating government programs, especially in the area of education.

The Commission on Evidence-Based Policymaking was created to pursue a laudable goal: To help analyze the effectiveness of federal programs. We all certainly agree that public policy should be based on evidence, on facts, not on opinion or dogma. So unbiased scientific research, for example, is vital for policymaking.

But the problem arises when the subjects of the research and analysis are human beings. Each American citizen is endowed with personal dignity and autonomy and therefore deserves respect and deference concerning his or her own personal data. Allowing the government to vacuum up mountains of such data and employ it for whatever purposes it deems useful – without the citizen’s consent, or in many cases even his knowledge – conflicts deeply with this truth about the dignity of persons.

Bear in mind that the analyses contemplated by the Commission go further than merely sharing discrete data points among agencies. They involve creating new information about individuals, via matching data, drawing conclusions, and making predictions about those individuals. So, in essence, the government would have information about a citizen that even he or she doesn’t have.

Our founding principles, which enshrine the consent of the governed, dictate that a citizen’s data belongs to him rather than to the government. If the government or its allied researchers want to use it for purposes other than those for which it was submitted, they should get consent (in the case of pre-K-12 students, parental consent). That’s how things should work in a free society.

Let’s consider a few specific problems. The Commission’s recommendations to improve evidence-building, while well-intentioned and couched in reasonable language, fail to recognize that data turned over by citizens for one purpose can be misused for others. It is always assumed that the data will be used in benevolent ways for the good of the individual who provides it. But especially with respect to the enormous scope of pre-K through college education data, that simply isn’t true.

Literally everything can be linked to education. Data-analysis might study the connection between one’s education and his employment. Or his health. Or his housing choices. Or the number of children he has. Or his political activity. Or whether his suspension from school in 6th grade foreshadows a life of crime. Education technology innovators brag that predictive algorithms can be created, and those algorithms could be used to steer students along some paths or close off others.

And much of this education data is extraordinarily sensitive – for example, data about children’s attitudes, mindsets, and dispositions currently being compiled, unfortunately, as part of so-called “social-emotional learning.” Do we really want this kind of data to be made more easily accessible for “evidence-building” to which we as parents have not consented?

The Commission recommends that all this data be disclosed only with “approval” to “authorized persons.” But we should ask: Approval of whom? Authorized by whom? There are myriad examples of government employees’ violating statute or policy by misusing or wrongfully disclosing data. And even if the custodians have only good intentions, what they consider appropriate use or disclosure may conflict diametrically with what the affected citizen considers appropriate. Again, this illustrates the necessity for consent.

We should take care to recognize the difference between two concepts that are conflated in the Commission’s report. “Data security” means whether the government can keep data systems from being breached (which the federal government in too many cases has been unable to do). “Data privacy” refers to whether the government has any right to collect and maintain such data in the first place. The federal Privacy Act sets out the Fair Information Principle of data minimization, which is designed to increase security by increasing privacy. A hacker can’t steal what isn’t there.

Another problem with the “evidence-building” mindset is that it assumes an omniscient government will make better decisions than individuals can themselves. But what these analyses are likely to turn up are correlations between some facts and others. And correlations do not equal causation. So, for example, we might end up designing official government policy based on flawed assumptions to “nudge” students into pursuing studies or careers they wouldn’t choose for themselves.

Human beings are not interchangeable. Our country has thrived for centuries without this kind of social engineering, and it’s deeply dangerous to change that now.

In closing, I reiterate my respect for the value of unbiased research as the foundation for policymaking. But speaking for the millions of parents who feel that their concerns about education policy and data privacy have been shunted aside at various levels of government, I urge you to continue the protections that keep their children from being treated as research subjects – without their consent. This might happen in China, but it should not happen here. Thank you.

You can read her longer, written testimony here.

Different committee members asked questions of the panel. Congresswoman Virginia Foxx (R-NC), the committee chair, asked Robbins the first question.

Congressman Brett Guthrie (R-KY) asked Robbins whether there is a balance between gathering useful information and privacy protection that can be found.

She had an exchange with Congressman Rick Allen (R-GA) about data collection and student privacy.

Congressman Tom Garrett, Jr. (R-VA) asked her why couldn’t the federal government pull together all of the metadata already out there.

She discussed the College Transparency Act with Congressman Paul Mitchell (R-MI).

She also explained to Congressman Glenn Thompson (R-PA) why education reformers will ignore evidence when it is negative, like in the case of digital learning.

She discussed longitudinal data systems with Congressman Glenn Grothman (R-WI).

Congressman Lloyd Smucker (R-PA) asked what student data would be ok for schools to collect, and if there was any data ok to share.