You can’t make good decisions with the wrong data

It happened – again!

Another education organization got a presentation on Kentucky’s supposed performance compared to other states on the National Assessment of Educational Progress (NAEP). The presentation didn’t follow NAEP’s own guidance on Page 32 in the 2009 NAEP Science Report Card regarding state-to-state comparisons. Worse, the organization involved was the Kentucky Board of Education (KBE).

The partial story the board got came from comparison slides like Figure 1, which compares Kentucky’s performance on the NAEP Grade 4 Math assessment in 2022 to results in other states. The slide is correct, but it doesn’t begin to tell the whole story. In fact, the slide is quite misleading when considered by itself.

Figure 1 – An incomplete and biased picture

Figure 1 makes it look like Kentucky places in the middle of states for fourth grade math, which would indicate improvement from where the commonwealth was three decades ago when the State NAEP first started.

But the performance picture created by Figure 1 doesn’t begin to tell the full story. The rest of the story starts with understanding there are achievement gaps between different groups of students. Just like on Kentucky’s state tests, the achievement gaps are wide on the NAEP, too.

For example, in 2022 NAEP Grade 4 Math, a disappointing 36% of Kentucky’s white students scored proficient; only 9% of the state’s Black students met similar standards.

Nationwide, on the same 2022 NAEP Grade 4 Math assessment, 47% of white students scored proficient or above while just 15% of Blacks reached the same level. Figure 1 certainly doesn’t show that Kentucky scores well behind for both white and Black student results. So, getting a clear picture of Kentucky’s NAEP performance requires looking deeper than the surface view offered by overall average scores.

As we recently explored in “While Kentucky Was Sleeping…, the percentage of white students in the overall public-school enrollment nationwide is now very different from Kentucky’s student demographics. Figure 2 shows some examples of the different racial makeups of public schools in various jurisdictions.

Figure 2

In 2022, Kentucky’s NAEP Grade 4 Math assessment cohort was 74% white compared to only 45% of students nationally.

Figure 2 shows how looking at states’ overall average scores, as the KBE did, shows scores from a lot of Kentucky’s white students are being compared to those for lots of kids of color in other states.

Actually, we’re comparing scores for 29 white students out of each group of 100 Kentucky students to scores for students of color elsewhere across the nation. If we compare to Florida, the white-to-students of-color disparity is even larger, with 38 Kentucky whites being compared to students of color in the Sunshine State. Given the large achievement gaps in play, only comparing overall average scores across states creates an inflated picture of Kentucky’s performance.

The KBE should have been shown more, such as the map in Figure 3, which provides an apples-to-apples comparison of scores for only white students in each state.

Figure 3 – Some of the rest of the story

Contrasting figures 1 and 3, there’s a great deal of difference between thinking you score statistically significantly above at least 12 states and learning you only outscore a single state. Regarding the rankings, consider Table 1, which shows rankings for white student performance in each jurisdiction.

Table 1

In Table 1, the “Cross-jurisdiction significant difference” column shows the difference in the NAEP Scale Scores between Kentucky and the other listed jurisdictions. For example, white students in the District of Columbia’s public schools outscored Kentucky’s by 30 NAEP Scale Score points. That’s roughly about a 3 year difference in performance! The difference in scores between Kentucky and the District of Columbia is statistically significant, so the cell is shaded dark blue and also has an up arrow to make this clearer.

Moving down the column, Hawaii scored 7 points higher than Kentucky, but due largely to the small population in Hawaii’s school system, this was not a statistically significant difference and the cell has a medium blue shade and a diamond symbol.

Only one state in the listing, West Virginia, definitely scored lower than Kentucky once the statistical sampling error present in all NAEP scores is considered. That gets the West Virginia cell light blue shading plus a down arrow.

The entire Kentucky row of data is also shaded light blue, indicating it was the focal state around which data for the other states is referenced.

Other columns show how many states scored higher, the same as, or lower than the state listed on the row after the sampling errors are considered.

Examination of Table 1 shows the commonwealth ranks No. 43 for white student results, much lower than 33rd place shown on the only comparison slide for fourth grade math data viewed by the KBE.

The importance of this difference becomes even more apparent when you consider 74% of Kentucky’s students are white.

Fortunately, the misleading information didn’t go unchallenged at the KBE meeting. Board member and former Boone County Superintendent Dr. Randy Poe questioned the presentation of data and pointed out there’s more to the NAEP story and it isn’t so favorable for Kentucky. Poe also stressed that providing inflated images of Kentucky’s performance undermines an important sense of urgency about addressing issues like fixing Kentucky’s abysmal reading performance. After all, if educators think the state now ranks in the middle after formerly ranking at the bottom, they can argue the Bluegrass State is makin acceptable progress with current policies. So, why change?

For sure, just doubling down on maintaining the status quo isn’t helping the 200,000 kids in Kentucky’s public education system facing real trouble for reading, as we point out in our report about What Milton Wright knew about reading instruction, but lots of teachers apparently don’t. Urgency, not complacency, is badly needed. Only looking at misleading data won’t get Kentucky where it needs to be.

(Tech Note: Data and tools used to create Figures 2 and 3 and Table 1 came from the NAEP Data Explorer.)

https://bipps.org/blog/you-cant-make-good-decisions-with-the-wrong-data

https://bit.ly/3Gy1ETO