Thinking about NAEP in Colorado and the Denver Public Schools

[this blog post is a follow-up response to the Twitter exchange with Van Schoales posted below]

Hi Van.

Although I’ve admired your work for years, you and I have never met, which means that we don’t have a relationship to lean into. Seth Godin reminds me that ‘if your audience isn’t listening, it’s your fault, not theirs.’ So please take this post as a very-public apology for whatever anger or defensiveness I sparked by my tweet. I’m sure that it could have been worded more artfully, and I regret not phrasing it in a way that maybe would have been received better. I offer this longer-form blog post as an attempt to bypass the lack of nuance available in 280 Twitter characters. I tend to do much of my thinking out loud in front of others, because their feedback makes me smarter.

I don’t know if you had a chance to read my previous blog post, Much ado about NAEP, but I tried to make two key points. The first was that the timing of the March 2022 NAEP tests should give us pause when interpreting the results. Most educators in America probably would tell us that, as difficult as the 2020-2021 school year was, the 2021-2022 year was even tougher. Kids and families still were dealing with incredible trauma, children showed us daily in classrooms (or by their absences) that they needed more support, and we didn’t do a great job of effectively serving large numbers of our young people. Then we had another big COVID spike right before the NAEP administration, and we have absolutely no idea how that impacted student test-takers except that probably a whole bunch of them (and their families) were ill, absent, struggling, etc. during the months that immediately preceded the test. We also know that these things were most true for our least-resourced children. All of this together feels like a reason to take the 2022 NAEP results not just with a grain but a giant boulder of salt. It’s difficult enough in normal times to help our students feel motivated for standardized tests. It’s even tougher given the March 2022 context and when NAEP results don’t mean anything to students personally.

The second point that I tried to make was that ‘Proficient’ on NAEP doesn’t mean what most folks think it does. To quote Tom Loveless, former director of the Brown Center on Education Policy at the Brookings Institution, in most states the Proficient label is “significantly above” what most state policymakers and assessments deem as ‘grade level,’ and the more accurate NAEP proficiency level for most states would be closer to Basic. This is rarely if ever mentioned in media coverage of NAEP results. Accordingly, most people tend to interpret ‘proficient’ in the ordinary sense of the word rather than in an aspirational sense, which is why I mention it whenever I think it is relevant (e.g., in my tweet). As I said in my blog post post, we can have some interesting discussions about whether Basic or Proficient is the right NAEP target for states, but we should at least recognize that Proficient is very aspirational in most parts of the country.

Here in Colorado, about 75% of 4th graders met the Basic NAEP standard in math for 2022, and about 36% met the Proficient standard. In 8th grade, about 63% of Colorado students met the Basic NAEP standard in math in 2022, and about 28% of students met the Proficient standard. Those results are essentially equivalent to the 2022 national NAEP averages for math. For reading, about 68% of Colorado 4th graders met the Basic NAEP standard in 2022, and about 38% met the Proficient standard. For 8th grade, about 73% of students met the Basic NAEP standard for 2022 in reading, and about 34% met the Proficient standard. Colorado students’ NAEP results in reading were a few percentage points higher than the 2022 national averages. As many have noted, all of the scores for both Colorado and the nation are down from 2019, which is to be expected.

In my blog post, I also quoted Loveless’ statement that, because the NAEP Proficient standard is so aspirational, “If high school students are required to meet NAEP proficient to graduate from high school, large numbers will fail. If middle and elementary school students are forced to repeat grades because they fall short of a standard anchored to NAEP proficient, vast numbers will repeat grades.” [emphasis added]. We also have evidence that similar percentages of students in nominally higher-performing countries also would have trouble meeting the NAEP Proficient mark. NCES has done the work of mapping Colorado’s state standards for proficiency to NAEP equivalent scores. For math, Colorado’s standard is well above NAEP Proficient in 4th grade and close to Proficient in 8th grade. For reading, Colorado’s standard is much closer to Proficient than Basic in both 4th and 8th grade. Colorado’s standards clearly are more aspirational than those of most other states. Accordingly, fewer Colorado students will be deemed ‘at grade level’ than if our benchmarks were set closer to those elsewhere.

All of which brings us to the concerns that you note in the Denver Public Schools (DPS). DPS is one of 26 urban districts that was sampled in 2022 and, as you stated at DPS Boardhawk, results were worse than for Colorado and for the nation as a whole. DPS’ 4th grade math, 8th grade math, and 8th grade reading results all were essentially equivalent to the large city averages. DPS’ 4th grade reading results were generally a few percentage points higher than the large city averages. Enormous equity gaps exist across student subcategories and, unfortunately, NAEP noted that the 2022 performance gaps are not significantly different from those of 2017. Here is a more-detailed breakdown for DPS in a few categories (click on the links to see these tables):

• 4th grade math
• White students: 92% Basic, 62% Proficient
• Black students: 49% Basic, 12% Proficient
• Hispanic students: 48% Basic, 13% Proficient
• School lunch not eligible: 77% Basic, 47% Proficient
• School lunch eligible: 48% Basic, 10% Proficient
• 8th grade math
• White students: 81% Basic, 53% Proficient
• Black students: 41% Basic, 11% Proficient
• Hispanic students: 40% Basic, 10% Proficient
• School lunch not eligible: 66% Basic, 35% Proficient
• School lunch eligible: 38% Basic, 10% Proficient
• 4th grade reading
• White students: 84% Basic, 63% Proficient
• Black students: 46% Basic, 14% Proficient
• Hispanic students: 37% Basic, 14% Proficient
• School lunch not eligible: 72% Basic, 48% Proficient
• School lunch eligible: 36% Basic, 11% Proficient
• 8th grade reading
• White students: 88% Basic, 58% Proficient
• Black students: 54% Basic, 15% Proficient
• Hispanic students: 52% Basic, 16% Proficient
• School lunch not eligible: 76% Basic, 42% Proficient
• School lunch eligible: 51% Basic, 16% Proficient

Yikes! Those performance gaps are both troubling and persistent! They’re also similar to the other large city districts that were sampled, better than some and worse than others. Denver basically is in the middle of the pack for the 2022 NAEP sample of large city school districts.

To quote my own tweet, these performance “divides continue to be of concern.” Like you, I believe that DPS should be extremely transparent about those performance gaps. Also like you, I hope that DPS identifies publicly some concrete plans and actions to remedy its existing equity issues. Additionally, I’m cognizant of the difference between NAEP performance gaps and NAEP performance levels (which is what I was trying to say, albeit artlessly, in my tweet). If Colorado makes it harder for students to be deemed ‘proficient’ than in most other states, of course we’re going to say, “look, fewer kids are proficient!” That’s how we set up the system in the first place (and, once again, we can have a rich discussion about where the line should be set for proficiency). We’re also probably going to say that fewer students are ‘proficient’ in a large urban school system because, sadly, that’s basically the pattern that we see in big city school districts all across the country. Equity gaps are large and persistent in America for students of color, who live in poverty, whose primary language isn’t English, or who have a disability, and the past couple of decades of school reform haven’t done much to alter those. DPS isn’t doing great on these fronts, but it’s not an outlier either.

You said in your tweet that I should be “outraged by Black and Latinx proficiency levels.” Am I – and, without speaking for them, probably my colleagues at the University of Colorado Denver – outraged about proficiency gaps? Yes, of course. Just like you, we also care about equity and we all are fighting for historically-marginalized children across a variety of fronts. Am I personally outraged about proficiency levels? Less so, given the fact that Colorado decided to set a much higher bar than most other states. ‘Proficiency’ is a politically-determined label, not a context-free indicator. If DPS was in most states in America, we would say that 41% of its Black students were ‘at grade level’ in 8th grade math instead of 11%. That number is still terrible, particularly compared to their White student peers, but it’s not “1 in 10” either. Again, if Colorado sets the bar higher, by definition fewer students will be proficient. For me, the gaps are much more alarming than whatever level we apply to children’s performance. I think that the concern is in the inequity, not the label?

Van, this is a long post. You may disagree with much of what I said here, and who knows if you even read through to the end or not. But if you did, let me close with this: I think that you and I both have a similar passion for equity in schools, and I also think that we both have a passion for making school different, particularly for historically-marginalized children and families. I might be wrong, but I don’t think so. Given your work with DSST and The Odyssey School of Denver and my work around instructional redesign and leadership for deeper learning, I think that we might have a really interesting and productive conversation together. Let me know if you ever want to have a meetup. I’m happy to join you for lunch or coffee at whatever location is easy for you.

Thanks in advance if you actually read through all of this. Hope we get a chance to talk sometime.

SCOTT