Data Comes From Humans
A lesson from the recent college enrollment data debacle
In October, headlines screamed that freshman college enrollment was down 5 percent. Did the U.S. Department of Education’s tremendous delays and ongoing errors with releasing a new federal financial aid application mean fewer students got desperately needed dollars to pay for college? Was this a post-COVID reaction, or a change in perception of the value of college? Is this the beginning of the end for college as we know it?
I wasn’t shocked at the dip. It’s much harder for me to accrue 10,000 steps a day than it is to find a negative headline about education.
But now in January 2025, the big reveal: the data were wrong. In truth, there was a 5 percent increase in first-year college student enrollment.
So what happened, and what can we learn from this error?
What happened was human error. Some students who are 18-years-old are still in high school but taking some college level courses, so-called “dual enrollment” students. This led to accidentally attributing the dual-enrollment label to a bunch of other 18-year-olds. At some point in recent weeks, researchers realized the mistake. There’s no obfuscation — National Student Clearinghouse (NSC, which collects the data) executive director Doug Shapiro wrote in a statement, “We deeply regret this error and are conducting a thorough review to understand the root cause and implement measures to prevent such occurrences in the future.”
What’s important to take away from this significant error is the reminder that all data come from humans, and at some point are subject to human error. Data are powerful, can sell ideas and products and policies, can reveal weaknesses or portend battles ahead. But for most data, at least in education, there is a human touching the data somewhere. And humans make mistakes. Once I showed up at a birthday party at the end instead of the beginning because I read the invitation wrong. My then-six-year-old was not pleased. Once I missed a flight because I didn’t see the email that said the flight had been moved 20 minutes earlier. Once I bought a Costco-sized container of protein powder because the internet told me women in their 40s should eat more protein. All mistakes, all human.
I don’t want you to distrust data. But I think it’s important to see data as a tool, and not as an omnipotent, all-powerful TRUTH. You can get data to say almost anything you want. You want a positive story about math instruction even though math scores keep going down? You want to make a school’s progress seem minimal despite the school staff’s belief and agreement that both culture and learning outcomes are changing? I don’t even have to cook the books to do that. We live in this strange time where there seems to be no truth, and it can make data even more appealing. But there is always human touch in objective analysis. This is as true for the researchers labeling 18-year-olds as dual-enrollment students rather than college freshman as it is for the people who take those data and analyze them.
I used to joke about writing a book called “Data is a Four-Letter Word” because sometimes our almost reverence for data is too much even for me. I want proof of what works. But education is a field of humans providing a service to other humans. There’s going to be tremendous variation. More than that, I think the way people feel matters. We measure student outcomes as a proxy for what we really care about, which is that students grow up to be adults who are happy with their lives and feel they have had opportunity, however they define that. Test scores, school attendance, even college enrollment only matter in service of this broader goal. Susanna Loeb and Helen Ladd, two economists, teamed up with two philosophy professors, Adam Swift and Harry Brighouse, and wrote a great book about this. It’s called “Educational Goods” and is worth a read.
I want us to have good data on everything. My biggest frustration with the wave of private school choice programs happening now, for example, is that we won’t be able to tell whether it’s working because most students aren’t required to take any kind of test. But that limitation hinges on defining “working” as “measuring student learning with standardized test scores.” There’s an opportunity here to talk about other ways to define “working.” Is it more students feeling satisfied with their own education? Is it more students attending school every day? Is it more students going to college?
Data are a tool to inform your opinion, not a shortcut to a specific answer. And so we can’t relinquish the work of thinking about what we believe to true, of debating different ways to achieve our goals, of asking questions about what doesn’t sit right, of collaborating to make the improbable just a bit more possible. To the good folks at the NSC — we all make mistakes. Just look at how much protein powder I still have. Thanks for owning yours, and for continuing to work to give us valuable information to inform our thinking and opinions higher education and beyond.