Charles Wheelan, Ph.D. The Naked Economist

The country may be sweltering in the August heat, but that means autumn isn't far off. Vacations are winding down. The NFL has started training camp. Mattress stores everywhere are gearing up for their Labor Day sales.

And soon the kids will be back in school. This seems like a reasonable time to begin a series of columns on what we know about the economics of school reform -- or, more accurately in some cases, how surprisingly little we know.

Something We Didn't Learn in School

We know a lot of things about educational outcomes -- for instance, that schooling matters now more than ever before. The wage gap between college graduates and high school graduates is large and growing. And students who end up at the bottom of the skill spectrum, such as high school dropouts, have seen their paychecks shrink now that a personal computer or a guy in India with a phone can do the same work for less.

And we know that American students aren't doing particularly well by international standards. In eighth grade science, for example, U.S. students trail their peers in Singapore, China, Korea, Japan, Hong Kong, Hungary, and Estonia.

But school itself -- what happens inside the buildings between the first and last bells -- is still largely a black hole. We have a very limited understanding of what inputs -- teachers, books, curricula, memorization, reading, extracurricular activities -- map into the outcomes that we care about, whether it's test scores, or earnings, or just not dropping out of high school.

An Error in Logic

It's actually more frustrating than that. Here's the shocking secret of school reform research: We don't really even know which schools are good schools.

That seems impossible. Surely the local newspaper has just released the list of the "50 Best Schools" or "20 Communities with Great School Districts." And indeed, the students in those places are likely to be doing very well.

But that doesn't necessarily mean the schools are any good.

Why? Well, see if you can spot the flaw in this logic: "The pediatrics unit of the hospital is doing a far better job than the intensive care unit because fewer of the patients are dying."

Hint: The patients in the intensive care unit start out sicker. And that's exactly what happens with schools. Family background has a profound influence on student outcomes. Highly educated parents produce children who do well in school, for lots of fairly obvious reasons: Smart parents are more likely to have smart kids (though it's no sure thing), and they spend extensive time and money doing things that promote the academic success of their children.

What 'Good' and 'Bad' Really Mean

More than a decade ago, I wrote a magazine story on a very famous high school located in an affluent Chicago suburb. I remember an education researcher telling me at the time, "If you took these kids and put them in a closet in their freshman year, and then let them out three years later, they'd still do well on the SAT."

(Just to be clear, this was a hypothetical example. No students were actually put in a closet. Their parents would have gone ballistic.)

Sadly, the opposite is true in other communities: Students begin kindergarten woefully unprepared to learn, and they receive little support while they're in school. Because American communities tend to be highly segregated by income, we have a lot of schools with a disproportionate number of privileged or disadvantaged students.

We all know that -- and yet we routinely describe schools as "good" or "bad" based on things that have more to do with who walks in the front door than with what happens inside the building.

Measuring with a Broken Yardstick

Think of it this way: If a golf pro gives Tiger Woods a lesson, and a different golf pro gives me a lesson, can we conclude that Tiger's teacher is better than mine because Tiger beats me by three shots in a match after our lessons?

That's usually how newspapers and real estate agents pick the "best schools." The true measure of quality -- with golf pros or elementary schools -- is how much value they add. In education, that turns out to be extremely difficult to calculate.

You may be thinking that we could probably fix this with a simple pre- and post-test -- let's measure what kids know when they walk in the school door, and then measure how much they know when they leave, one or four or eight years later. The best schools will be the ones with the students who show the most improvement.

Not exactly. Gifted students don't merely begin at a higher level, they also learn at a faster rate. So, to stick with our athletic example, suppose that neither Tiger Woods nor Michael Moore has ever played tennis before. If we give them each a tennis coach, can we evaluate the quality of those coaches based on the subsequent outcome of a tennis match between Tiger Woods and Michael Moore? (Pause for a moment and consider what a great pay-per-view event that would be.)

Evaluate the Evaluation

So schools with high test scores may or may not be doing a great job; perhaps their students are capable of much more. And conversely, some schools with middling or poor test scores may be doing a terrific job educating students who would otherwise be failing abjectly.

Obviously, we can spot the outliers -- the school in the middle of Detroit that manages to send 95 percent of its students to college, say. If we give researchers enough time and enough data, they can try to answer the school-quality question using statistical techniques that take account of what kind of students are walking through the front door.

But even then the results are often equivocal. The bottom line is that it's hard to evaluate school quality, which is why it's even harder to make schools better.

The First Challenge

We're trying to encourage and replicate success without being able to tell with any degree of certainty which schools are succeeding, then. Imagine a pharmaceutical company trying to evaluate new cancer treatments without being able to determine which patients are getting better.

So that's the first big education challenge -- developing a more sophisticated way to identify "good schools." Only then will we be able to create more of them.