Rating Schools

Last week I threw out a comment about suburban school districts whining about State Report cards that don't let them coast on their demographic profiles. Paul from Save the Hilliard Schools took some exception in the comments, in part as I seemed to have implied that Bexley's success was as meaningful as Dublin's (and Hilliard's) shortcomings. It was a fair point, and what I really meant to say was that Dublin's shortfall this year was a) much more meaningful than the previous 'excellent' rating (or Bexley's currrent 'excellent' rating for that matter), and b) that the mere idea that Dublin and Cleveland deserved the same rating was not at all as ludicrous as the Dispatch tried to make it out to be.

As if in response (that's been happening a lot lately...), a press release has come out describing work on school ratings done by two OSU researchers:

Currently, most people believe that it is obvious which schools are the best – the ones with the highest achievement scores. But using achievement scores to measure school quality assumes that all schools have students with equivalent backgrounds and opportunities that will give them equal opportunities to succeed in school. And that's obviously not true, von Hippel said.

...

By comparing test scores at the end of kindergarten and the beginning of first grade, the researchers could measure learning rates during summer vacation.

Comparing test scores from the beginning and end of first grade allowed the researchers to see how much children learn during the school year.

They then were able to calculate how much faster students learned during the first-grade school year compared to when they were on summer vacation. This was the "impact" score that showed how much schools were actually helping students learn.

"If we evaluate schools that way, things change quite a bit as far as which ones we would identify as failing," Downey said.

...

Based on achievement scores, failing schools tend to be in urban areas, serve a higher percentage of children who qualify for a free lunch, and have a high minority population.

But if you look at impact scores, failing schools are not as concentrated in poor, urban areas with high minority populations.

"When you shift the focus from achievement to impact, there are still schools that do very well and some that do poorly," von Hippel said. "But they are not necessarily where you think they are. There are high-impact schools in every kind of neighborhood, serving every kind of child. The same is true of low-impact schools."

The research appears in the current issue of the journal Sociology of Education (for those who wonder about such things, it has an ISI impact of 1.4, putting it in the top 15-20% of journals in either Sociology or Education). I went back to the actual journal article (subscription only, no link) to answer a question I had and was satisfied both with the overall methodology (those sociologists certainly know how to isolate variance), and with the answer to my question:

A final concern is that impact-based evaluation may penalize schools with high achievement. It may be difficult for any school, no matter how good, to accelerate learning during the school year for high-achievement children. Our study, however, did not find a negative correlation between impact and achievement; to the contrary, the correlation between achievement and impact was positive, although small (see Table 3). And among schools in the top quintile on achievement, 26 percent were also in the top quintile on impact (see Table 4), suggesting that it is possible for a high-achieving school to have high impact as well.

I would love for Franklin County to pilot an impact-based evaluation model, and it would certainly be a chance for Dublin and Hilliard to show that they really are doing more for their students than Southwest or Whitehall are doing for theirs.