Editor’s note: Over the summer, the Research Alliance for New York City Schools at New York University released its first analysis of the Department of Education’s School of One program, a technology-enhanced, individualized math curriculum for middle school students. School of One is a privately funded program that started in 2009 under then chancellor Joel Klein as a summer pilot and then after-school program. It ran in three schools – M.S. 131 in Manhattan, I.S. 339 in the Bronx and I.S. 228 in Brooklyn — during the 2010-11 school year. According to the DOE, four schools will use the program this year. Here, the alliance’s executive director urges a careful reading of the findings after some critical press coverage including this recent article in The New York Daily News.

As we embark on a new school year, it would be refreshing if we adults made the same commitment to learning that we expect of our children. As it stands, investments in education reform are too rarely accompanied by commensurate investments in learning whether the reforms work, for whom and under what circumstances.

And when we do make such investments, the results are often vastly oversimplified — framed as a “gotcha” when the findings fall short of expectations or as a panacea on the rare occasions when results are positive. A case in point is our recent report on the first-year impact of School of One and some of the reaction that has been. These reactions, particularly to early assessments of education innovations, are far more disappointing and unproductive than any particular unflattering research finding.

School of One aims to provide middle school students with individualized, self-paced math instruction through extensive use of technology and formative assessment. The Research Alliance report presented a mixed picture of the program’s potential strengths and limitations.

The initial findings indicated that the program improved math achievement in one school, decreased achievement in another, and made virtually no difference in a third. The study also produced some surprising findings about which students benefited most and how those benefits seemed to be accruing — findings that may suggest needed refinements in School of One’s curriculum and approach.

This mix of findings is typical of what one should expect from a complex and highly innovative effort to change teaching and learning. Neither victory nor defeat should be claimed from a single early assessment of school reform initiatives, regardless of whether the findings are one-sided or mixed.

Effective, sustainable reform requires persistence and adaptation, which must be informed by the accumulation of evidence, over time, about what seems to be working and what doesn’t. The central message of our report, perhaps conveyed too subtly, is that the ongoing commitment to building and learning from evidence is far more important than findings of a single study, for a single year, of a very new program.

Regardless of one’s perspective on the New York City Department of Education, School of One stands out as a compelling commitment to teaching and learning on the part of the system’s adults. While the DOE and others have certainly hyped the program, the hype has been accompanied by a commitment to assessing its impact and making changes in response to teachers’ feedback as well as the emerging evidence about its potential effectiveness.

In fact, a requirement for receiving a prestigious Investing in Innovation grant from the U.S. Department of Education was that the DOE commission an independent and scientifically rigorous assessment of its effectiveness over several years. It is particularly unfortunate that two of the original School of One schools decided to discontinue the program before they had an opportunity to incorporate ongoing improvement efforts and to more fully assess its effectiveness.

Proclamations of defeat from the initial stages of innovation run the risk of pushing innovators and their supporters away from their commitment to building evidence in concert with their efforts to improve schools and risks throwing the baby out with the bathwater — getting rid of promising programs prematurely, instead of refining them. By the same token, claiming victory from early successes can run an even greater risk of garnering long-term funding for approaches that aren’t effective after all. As any student will tell us, learning is hard, and we rarely get it right the first time.