Score one for CMS on data accuracy

Last year's CMS seniors didn't log any math gains on the SAT, but the folks who run the district made a big step forward in demonstrating their own ability to report numbers.

When the College Board released its 2013 SAT report today,  I downloaded the school by school report from the N.C. Department of Public Instruction and looked at the Charlotte-Mecklenburg Schools news release.

The numbers for CMS didn't match.

Barnes

That's not the surprise.  As many of you know,  there have been instances in years past when CMS numbers didn't add up.  And too often,  the people who released those numbers weren't prepared to explain.

Today, Chief Accountability Officer Frank Barnes was ready.  Before the scores went public his staff had checked the data they got from the College Board against the state report and caught a problem:  The state report didn't include Garinger High.

That omission made CMS look better.  Without the 142 tests from Garinger,  the CMS average was 1480,  one point above the state average.  With them,  CMS fell to 1473  (still a 10-point gain over 2012).

The CMS news release used the lower and more accurate scores.  It discussed the possibility that lower participation might have contributed to gains.  It mentioned that a school was missing from the state report -- and when I asked Barnes what Garinger's score was he gave it to me, even though the score of 1218 out of a possible 2400 was second-lowest in CMS.  He said he's working with state officials to get the Garinger scores added to state calculations.

"We wanted to report what we knew to be true,  even if it was lower than the state report,"  Barnes said.

All of this adds up to a promising sign that Superintendent Heath Morrison is delivering on his promise to make sure numbers are correct before they're released and to be honest about strengths and shortcomings.

Morrison was hired in 2012, as CMS was grappling with the embarrassment of badly botched school progress reports.  The first time I talked to him,  he said he was going to create systems to avoid such errors and provide honest explanations if mistakes did happen.

The first few weeks brought stumbles.  When state exam results were released in August 2012,  CMS declined to release results for West Charlotte and Harding high schools,  which were not given a state rating because they didn't test enough students.  The result:  A front-page story when I got the numbers from the state and they turned out to be the two lowest-performing schools in the district.

Since then it's been hard to judge.  Changes in state testing have delayed the exam results that usually land in the summer.  CMS still hasn't launched its own school ratings,  trying to make sure they mesh with a state system that's in flux.

The big test will come in November,  when the state releases 2013 exam results. It looks like CMS may be ready.

CMS gets kudos on data

The folks who think Charlotte-Mecklenburg Schools makes up its numbers might want to skip this one, because a couple of interesting, complex national reports on CMS and data have come out recently.

The Michael and Susan Dell Foundation spent 2010-11studying the district's efforts to create a "data-driven culture."  Researchers saw the turmoil of school closings,  the data overload some teachers face and the battle over testing and performance pay,  but still came away impressed by much of what was going on.  The 18-page report is worth a read for anyone trying to get a handle on how CMS can move forward.

About the same time, an article in EdWeek cited CMS as a district making sophisticated use of data to identify which students are at risk of dropping out, using risk factors that show up as early as elementary school.

Given CMS' recent data woes,  particularly the erroneous public report that purported to track students on track for graduation, that may sound like a setup for saying EdWeek got duped.  But I don't think that's the case.  The irony of the simplistic graduation-track calculation is that CMS does use a complex set of factors to identify and work with at-risk kids.

What strikes me about both of these reports is the tangle of moving pieces that makes up the CMS data picture.  For every spreadsheet I peruse,  there are reams of internal reports.  Some are almost certainly helpful to kids and teachers;  other internal data systems are so flawed or cumbersome that they fuel employees' skepticism.

For better or worse,  the CMS accountability department has lost so many key players in the past year that rebuilding it is going to be a front-burner task for the new superintendent.  The CMS data machine needs some repairs,  but as these articles make clear,  "scrap the whole thing" isn't a realistic solution.

Dropouts vs. grad rate: More confusion

The recent state report on 2010-11 dropout rates may have added to confusion over how many Charlotte-Mecklenburg students are staying in school.  As one person who emailed me noted,  the CMS dropout rate of 3.57 percent seems way out of whack with a 2011 graduation rate of 73.5 percent.

It's not as crazy as it sounds.  The annual dropout rate tallied the percent of all high school students who left school in 2010-11.  The on-time graduation rate is the percent of students who started ninth grade in 2007-08 and graduated four years later.  During those four years,  CMS' cumulative dropout rate was 18.6 percent  (the rate has been dropping every year).

That's still short of the 26.5 percent who failed to graduate in four years.  Some of those no doubt came back this year,   neither dropouts or on-time grads.  In 2010-11,  454 CMS students who should have graduated with the Class of 2010 got their diplomas,  bumping up the total graduation rate for that class by 4.6 percentage points.

I can't swear that there are no mistakes in CMS' drop-out or on-time graduation rates,  but there has been no indication that either is faulty.  The error that created a stir and led to the resignation of a data administrator had to do with a completely separate measure created by CMS,  which tallied the percent of students who had never been retained and used that as a gauge of being on track to graduate.