Score one for CMS on data accuracy

Last year's CMS seniors didn't log any math gains on the SAT, but the folks who run the district made a big step forward in demonstrating their own ability to report numbers.

When the College Board released its 2013 SAT report today,  I downloaded the school by school report from the N.C. Department of Public Instruction and looked at the Charlotte-Mecklenburg Schools news release.

The numbers for CMS didn't match.

Barnes

That's not the surprise.  As many of you know,  there have been instances in years past when CMS numbers didn't add up.  And too often,  the people who released those numbers weren't prepared to explain.

Today, Chief Accountability Officer Frank Barnes was ready.  Before the scores went public his staff had checked the data they got from the College Board against the state report and caught a problem:  The state report didn't include Garinger High.

That omission made CMS look better.  Without the 142 tests from Garinger,  the CMS average was 1480,  one point above the state average.  With them,  CMS fell to 1473  (still a 10-point gain over 2012).

The CMS news release used the lower and more accurate scores.  It discussed the possibility that lower participation might have contributed to gains.  It mentioned that a school was missing from the state report -- and when I asked Barnes what Garinger's score was he gave it to me, even though the score of 1218 out of a possible 2400 was second-lowest in CMS.  He said he's working with state officials to get the Garinger scores added to state calculations.

"We wanted to report what we knew to be true,  even if it was lower than the state report,"  Barnes said.

All of this adds up to a promising sign that Superintendent Heath Morrison is delivering on his promise to make sure numbers are correct before they're released and to be honest about strengths and shortcomings.

Morrison was hired in 2012, as CMS was grappling with the embarrassment of badly botched school progress reports.  The first time I talked to him,  he said he was going to create systems to avoid such errors and provide honest explanations if mistakes did happen.

The first few weeks brought stumbles.  When state exam results were released in August 2012,  CMS declined to release results for West Charlotte and Harding high schools,  which were not given a state rating because they didn't test enough students.  The result:  A front-page story when I got the numbers from the state and they turned out to be the two lowest-performing schools in the district.

Since then it's been hard to judge.  Changes in state testing have delayed the exam results that usually land in the summer.  CMS still hasn't launched its own school ratings,  trying to make sure they mesh with a state system that's in flux.

The big test will come in November,  when the state releases 2013 exam results. It looks like CMS may be ready.

Lake Woebegon and grad rates

Astute readers looking at the graduation rates for Charlotte-Mecklenburg's 28 high schools may notice something odd:  24 of them list rates above the district average of 81 percent, including 13 above 90 percent.

That may call to mind Garrison Keillor's mythical Lake Woebegon, where all the children are above average.  Worse, it may raise questions about whether the numbers are valid.

In this case,  the seeming impossibility is due to the way school and district rates are calculated.  It's not a CMS thing;  I'm willing to bet you'll find the same pattern in virtually every district.

North Carolina,  like other states,  tracks the students who start high school and calculates the on-time graduation rate by the number who get a diploma four years later  (Superintendent Heath Morrison says our state is unusual in not including summer-school grads).  But students who switch schools after falling behind don't count toward school calculations.

Take,  for instance,  a student who starts ninth grade at School A but doesn't get enough credits to be promoted.  Sometime in his second year of high school he moves to School B.  School A doesn't get blame for his failure to graduate  (or credit if he does)  because he left.  But School B also isn't held responsible because he was behind when he got there.  As long as that student stays within one district,  though,  he still counts toward the district rate.

As you can imagine,  students who fall behind and move around are at special risk for failing to graduate.  So you see those students bringing down the district and state numbers without dragging down schools.

Morrison and his crew are well aware that plenty of people,  including me,  scrutinize CMS numbers closely  --  and with good reason.  The district has gotten black eyes for bad data,  including flawed graduation rates reported in 2006,  when the current tracking system debuted,  and error-filled school progress reports posted last year.  The skepticism started flowing as soon as we posted a story on this year's gains.

John King of Harrisburg quickly emailed me questioning why CMS couldn't quantify the impact of a change in graduation requirements,  from 28 credits in previous years to 24 starting with the Class of 2013.

"Given any level of competent statistical tracking,  it should require very little time or effort to produce dual graduation numbers, one assuming 28 required credits and one assuming 24,"  King wrote.  "I do something similar almost every day in my job!  It’s a key step in evaluating the effectiveness of the decision to make the change.  The failure to do so simply proves that there is more interest in managing the perception than the result and that there is no more transparency under this Superintendent than under the last!"

I requested exactly that calculation Tuesday afternoon,  when I heard that graduation rates would be released Thursday.  I agree with King that it's valuable information,  and I'm disappointed it wasn't available right away.  But Morrison and Chief Accountability Officer Frank Barnes have repeatedly said they're going to check and recheck anything they release,  rather than take another credibility hit by giving out something they have to retract.  They say they'll give me the numbers when they're confident of them,  and I'll report on that information when I get it. 

New tests to map path for students, CMS

If you've got kids in Charlotte-Mecklenburg Schools,  get ready for more tests and a new acronym:  MAP.

Measures of Academic Progress,  a computerized testing system created by the nonprofit Northwest Evaluation Association,  will be a cornerstone of the district's efforts to make sure all students are on track to graduate high school and succeed in college or careers,  Superintendent Heath Morrison and Chief Accountability Officer Frank Barnes told the school board this week  (see the presentation here).

MAP reading and math tests will be given to students in grades 3-8 in fall,  winter and spring,  Morrison said.  Some schools used the system last year,  with others starting in 2013-14.  By 2014-15,  Morrison and Barnes said,  all elementary and middle schools will be using MAP,  with results used to report on whether students are on track to graduate.

"More tests"  is not a phrase that's likely to generate applause among teachers and parents.  Board members and Morrison were quick to say they're concerned about overtesting and have urged the state to ease up on mandatory end-of-year exams  (which will be given in addition to the MAPs).

But Morrison said MAP is the right kind of testing:  It provides timely information about what kids know and what they need help with,  in a format that students and parents can understand.  "Most parents ask a simple question:  Is my child reading on grade level?"  Morrison told the board.  He said MAP lexile scores make it easy to see whether students are at,  above or below grade level, and what each student needs to do to improve. And the tests are designed to provide consistent year-to-year tracking,  something that's a challenge with ever-changing state exams.

Chapman

Northwest,  or NWEA,  defines its approach as based on  "kid-centric education,"  with test results used to help teachers design individual strategies for learning.  President and CEO Matt Chapman writes that he is seeing an increased use of student results to evaluate teachers  --  one of the driving forces behind new N.C. exams -- and supports that approach only if it's part of a balanced evaluation system that focuses on helping teachers improve.

"Teaching is a complex task with numerous factors involved,  and teachers deserve to be treated with respect rather than threatened with public retribution based on test results that almost certainly do not present a comprehensive view of how a teacher is performing,"  Chapman writes (read the full post here). "The national fixation on testing,  especially standardized accountability testing,  is unfortunate and can hurt student learning."

Of course,  the most meaningful reviews come from those who have seen any system in action.  So if you've experienced MAP already,  please weigh in and let the rest of us know what to expect.

PreK-8s, Bright Beginnings top CMS research plan

Measuring the impact of Bright Beginnings prekindergarten and figuring out whether the recent switch to preK-8 schools benefits students are among top research goals for Charlotte-Mecklenburg Schools, Chief Accountability Officer Frank Barnes told the school board Tuesday.

Barnes

Several board members said they're eager to see that data.

The 2010 decision to close three struggling middle schools and add grades 6-8 to eight elementaries raised questions in the affected neighborhoods.  Some parents still believe the combined elementary/middle schools endanger the young children and give older students fewer academic options than they'd get in larger middle schools.  Proponents say students benefit from the more personalized setting and avoid the academic slump that often comes with the switch to middle school.

In September,  Morrison presented a preliminary study of the first year of pre-K-8s that found mixed academic results. Barnes said Tuesday a follow-up is expected in late fall or winter,  after results of 2013 state exams are in.

Barnes said CMS is also working with researchers from UNC Charlotte to study the district's prekindergarten:  "Is it working, where is it working best and are there challenges?"  Most school board members say they support public prekindergarten,  but many have voiced frustration that CMS failed to deliver on early promises to monitor how Bright Beginnings students do as they move through school.  Former Superintendent Eric Smith created the program in 1998.  Tracking of student results vanished during ensuing leadership changes,  even as the program's cost has risen.

Other research plans include identifying signs that students are at risk of dropping out and studying the "instructional culture" of CMS schools, part of a research project with TNTP.  Barnes said CMS hopes to refine the district's annual surveys of principals,  teachers,  students and parents.  Results of those surveys used to be posted online;  Barnes said the 2012-13 results haven't been released yet but should be posted later this summer.

Barnes told the board that he and his staff are still working on a reliable way to report on school success.  In 2012,  CMS withdrew its school progress reports after the Observer uncovered errors in the data.  A subsequent CMS review found even more problems.

Board member Ericka Ellis-Stewart asked Barnes how he plans to restore community confidence that CMS can get the numbers right.  Barnes said he's working on a system that uses data verified by the state,  but new state exams and uncertainties about a proposed letter-grade system are delaying that effort.

"Part of what I recognize is that trust is earned,"  Barnes said.  "I think what we will have to do is season by season,  year by year re-earn that trust."

CMS teachers being polled -- again

Charlotte-Mecklenburg Schools is polling teachers on school climate in hopes that the data will help principals create a better workplace.

The survey was created by TNTP (The New Teacher Project),  a nonprofit that focuses on training and supporting excellent teachers.  Earlier this school year,  Superintendent Heath Morrison brought TNTP's president in to talk to principals about the group's report on "The Irreplaceables,"  which highlighted how districts fail to do enough to reward great teachers while making it too easy for bad ones to stay.

Barnes

CMS Accountability Chief Frank Barnes says the TNTP survey will be given this spring,  next fall and in spring of 2014. "Its focus is the instructional culture of a school, which can impact student performance and teacher retention,"  he writes.  "Our intent is to use this survey ... as a tool for continuous improvement.  The results of this survey will be shared with each school principal this spring in a formal,  yet readable,  report.  The reports are structured to give school leaders actionable feedback that can be tracked over time."

At the start of the school year,  CMS polled all employees on morale,  as part of a yearlong,  $140,250 polling and communication contract with K12 Insight.  Although the new survey is titled Insight,  it's not part of that contract  (I've asked about the cost, but haven't gotten an answer yet).

Teachers are also taking the annual CMS survey,  which is used in principal evaluations  (the TNTP survey will not be).  They also take the state Teacher Working Conditions Survey every two years,  but this is an off year.
 
Barnes acknowledges that the level of surveying is  "more than normal and more than there will be next school year."  The big question is whether it will make life better for teachers. Reaction I've heard has been mixed,  with some teachers skeptical and others thrilled that the new leadership team is listening.