Who's to blame for CMS data delay?

More than six months into the school year,  I still can't tell you the poverty level,  racial breakdown or school-by-school enrollment for Charlotte-Mecklenburg Schools.

The cause of that delay has become a point of contention between state and CMS officials.

Staff at the N.C. Department of Public Instruction agree that the troubled PowerSchool rollout delayed the principals' monthly reports that normally generate that information in September or October.  CMS officials say several schools were unable to run those reports for months.

But state officials say the hold-up is no longer on their end.  On Feb. 21, spokeswoman Vanessa Jeter said all the monthly reports had been run and the state was working with districts to clean up any final errors  (a clean-up that was needed in 0.0031 percent of all monthly reports, she said).

Last Friday,  she asked if I had gotten any numbers from CMS.  "I understand that their PMRs have been run, numbers double-checked with NCDPI-CMS staff and all came up correct,"  she emailed.  "That should clear the PMR issue in PowerSchool as far as I know."

Not so fast,  says CMS.  When I prodded again for the report,  spokeswoman Kathryn Block said Scott McCully,  executive director for student placement, had uncovered  some additional concerns with the reports.  "Scott is scrubbing the numbers one last time and, barring any additional issues, we will share the information early next week,"  she said.

"Also,"  she added,  "DPI confirmed that GRS report is not functioning statewide so there is no grade, race or sex data to share at this time for any NC school district."

Huh? That has always been part of the 20th-day report released in September or October;  each school and the district as a whole is broken down by grade level and race  (poverty levels come in a separate report tallied in October).

Jeter and Philip Price, chief financial officer for DPI, say it's wrong to say there's no race/ethnicity data.  There is a quirk having to do with reconciling end-of-month enrollment with average monthly enrollment,  they say.  That will be corrected during the next system maintenance weekend,  March 14.

And the state is looking into about 8,000 students across the state who were once classified as Hispanic but are not this year.  "That would represent .0053333 percent of the student body,"  Jeter reports.

"We do not believe this is an issue that distorts the data at a specific (school districts),"  Price wrote.  "The validation is important for federal reporting. We expect other corrections as we work to complete our federal reporting requirements. All this means, we have race and ethnicity data for all our students."

Bottom line:  We have a collision of two forces here.  There's little doubt that PowerSchool has created a battery of problems for local districts.  Meanwhile,  Heath Morrison was hired as superintendent in 2012 on the heels of a series of CMS data errors that embarrassed leaders and hurt the district's credibility.  He and his staff have been wary of releasing anything that isn't also available in state records,  forestalling errors but also making it unusually difficult to get data we've all gotten used to finding at the click of a mouse.  He says he doesn't want to post numbers only to have the state make a PowerSchool adjustment that requires CMS to retract its information.

So,  as soon as I know what the numbers looked like last fall,  I'll let the rest of you know.

Then maybe we can hope to see results of the 2014-15 magnet lottery, which should be sending notification letters about now.

Details on N.C. PowerSchool problems

The glitches in North Carolina's school data system are intensely interesting to some readers and deadly dull to others,  depending on how much they affect your life. This post is for the first group.

Teachers,  administrators and parents around the state have been complaining about the new system since the start of the school year.  Keung Hui with the News & Observer got state officials' attention with a story last week detailing some of the frustrations,  including those in Charlotte-Mecklenburg Schools.  The state is now asking Pearson to refund some of the agreed-on $7.1 million first-year fee because of the problems.

Hui and I had both been asking our districts for details on the ongoing problems.  Wake County Public Schools came through first. After the story ran,  Philip Price,  the Department of Public Instruction official in charge of PowerSchool,  provided this response to the Wake issues,  which are similar to those detailed by CMS this week.

For those trying to follow the twists and turns, here's Price's take on the issues and solutions,  followed by the CMS list provided by data analyst Jay Parker.

From Price: 

We have gone through the list of issues from Wake County and have responded to each of them below. We are happy to talk with you further about any of the items.

In short, there were no real surprises. We know that some issues exist with the system and some with the districts’ data. We are working with Pearson and with the districts to resolve all the known issues. While I see the progress of the system from planning , transition, and implementation as being very successful, the users are the individuals who rely on the system to accomplish their assignments.  Their frustrations are real and need to be addressed.  As I mentioned before, we know problems will continue to surface as we progress through this transition year.  The key is to minimize (or better eliminate) problems from recurring.  I can say that we are doing an excellent job in that area, and we are doing all of this within budget.

Athletic Eligibility – At this time, DPI has not provided the ability for a district to run eligibility for athletics for students in PowerSchool

·              The athletic eligibility requirements for North Carolina do require development work for our version of PowerSchool .  This was known when the contract was signed.  The scheduled delivery date is in March 2014.  The fall eligibility was based on data in NC Wise.  To accommodate the spring eligibility, 3 reports were created to review attendance, grades, and residency.  Only those determined to be ineligible are listed on the report.  The districts were able to review those students to determine if they also should be eligible.

·              The posted schedule for upcoming releases can be found here.

Inadequate Progress - DPI has not provided the ability to generate a progress report needed for the school district so they can submit the needed data to the NC DMV

·              This is a customized report for PowerSchool (to comply with North Carolina’s law) and is scheduled to be available in the April/May timeframe

·              Meanwhile, all data necessary for LEAs to certify driver’s ed eligibility is available in PowerSchool. It does require an individual to review the student status for the following 3 areas and to manually complete the DMV required form:

• Academics – passing 70% of courses
• Dropout – haven’t dropped out
• Discipline – haven’t participated in specific disciplinary infractions like bringing incendiary device to campus

Mid-year GPA - DPI has not provided a midyear GPA calculation and Transcript.  The current NC Weighted and NC Unweighted GPA Calculations in PowerSchool do not include Semester 1 averages for yearlong courses.  This causes inaccurate GPA information for any student who is schedule in a yearlong class.  The schools are also not able to produce an accurate class rank list.

·              The PowerSchool software does not allow grades to be included on the transcript until the course is completed.  They can  add the course to the transcript (through a specific procedure); but, the grade cannot be  included.  This is how the software works in all locations where it is used as an entities student information system.

·              School districts have requested that a grade be included before the course is complete.  There are no current requirements for mid-year GPA calculation in statute or Board policy; therefore, this was not included in the State’s required deliverables for PowerSchool.

·              Without a grade, an estimated GPA cannot be calculated.  Without an estimated GPA, you cannot create an estimated class rank.

·              DPI is researching and working with Pearson to identify the changes necessary to allow interim grades for year-long classes (not completed) and how they can be outlined on a transcript.

PMR - DPI has not been successful in providing the ability for a school district to accurately run and approve the Principals Monthly Reports

·              The PMR has been completed.  We are now working on allotment adjustments and projecting ADM for FY 2014-15.

Dropout - The Dropout Report does not exist in PowerSchool

·              The dropout report is missing two key fields from the Previous Enrollment Screen within PowerSchool.  These fields will be added during the March 14 maintenance weekend.  The fields are Dropout Reason and Dropout Verified indicating the LEA has validated the student is a dropout.

·              DPI and SAS are creating reports which include the appropriate Dropout data for review including prior year data elements that are required for the report. 

·              This all should be completed by the end of March.  This is a transition year issue and will not be the process in future years (because prior year data will be in PowerSchool).

RPG – (Retention, Promotion, Graduation) an RPG report does not exist in PowerSchool to date

·              This is a system issue.  This report compares where a student was the previous year with where they are this year. The historical data for prior years needs to be loaded into PowerSchool to make this report useful.  Significant conversion issues have been encountered with bring in this data into PowerSchool.  We are looking at ways of correcting conversion problems or how we can utilize eSis (NC Wise) to try and avoid requiring districts to enter the data directly into PowerSchool.  This is a transitional year issue and will not be an issue in future years.

SAR - The student Activity Report does not exist in PowerSchool

·              This report requires two system issues to be resolved before it is 100% operational

·              The software needs to be able to manage cross enrollment between school districts (taking courses in multiple LEAs).  This ‘fix’ is currently in DPI quality review and, if passed, will be in production soon.  This ‘fix’ will also enable virtual school enrollment through PowerSchool (currently the registration process continues as it was in previous years).

·              Extended course attributes (properly defines courses at a detailed level) are being added to PowerSchool by Pearson.

Gradebook - Teachers assigned to a large number of sections.  The gradebook will not load for these teachers.  If it does load, it does not always function as expected.  The grade scale may not be attached, the teachers may not be able to save the grades, and they may not be able to make changes to grades that are already entered.  Dropdown boxes may not populate with the choices.  In elementary schools, the standards may or may not be listed or populate as expected. Teachers with a large number of assignments experience issues similar to teachers with large number of sections. Teachers that have had a large number of students dropped from sections experience issues similar to teachers with large numbers of sections. Grade Scales are not always attached to the sections; however, the grade scale is set up for the course appropriately in PowerSchool.  Other teachers with the same course have the grade scale attached. Poor/Slow/sluggish performance.  Users must save frequently, the saving function is slow. Grade Scales are not always attached to the sections; however, the grade scale is set up for the course appropriately in PowerSchool.  Other teachers with the same course have the grade scale attached. Gradebook freezes so it requires the teacher to shut down and restart.  Therefore, data entered and not yet saved is lost.  Also, Gradebook freezes when saving grades, so data is lost. Gradebook will not launch.

·              Issue with gradebook not loading for teachers with large number of sections should be fixed in March maintenance weekend (March 11).

·              No one has reported the other issues that Wake mentions. However, we might assume that these issues will similarly go away after the March maintenance weekend. We are asking for additional information.

·              Exception – “gradebook won’t launch” issue has not been replicated outside of Wake

EOG/EOC - EOG/EOC test scores are not in PowerSchool

·              EOC historical test scores were made available to import before 12/31/13. EOGs – Feb/March.

·              12-13 test courses were never in eSIS and will be made available to import in Feb/March.

Historical Data - No historical data for any student exists in PowerSchool from the 2002-2003 school through the 2011-2012 school year.

·              Historical data for students active in 2012-13 have been imported into Powerschool.

·              Data for students active until before 2012-13 are currently being uploaded into ODS.

At-Risk Report - The At Risk Report does not function in PowerSchool

·              More information is being requested from the school district. It appears to be the discipline part of the report that requires a software adjustment to target incidents.

NC Diploma - Not all NC Diploma tracts are present in PowerSchool

·              Future Ready Core (FRC) diploma path is available in Powerschool

·              The non-FRC paths did fail testing, but we are expecting them to pass a second round of development/testing within the next couple weeks.

NC Transcript - The NC Transcript does not display the true graduation date, but rather displays an expected date

·              Student’s actual graduation date can be entered on “student academic screen”

Historical data for transfer students - When transferring a student into WCPSS, none of the historical data for that student get transferred

·              All immunization and demographic info has been transferred

·              Historical grades and related fields will be brought over in March maintenance release provided it passes QA

If you're still reading, you're a PowerSchool die-hard. So here's what Jay Parker with CMS reported:

Athletic Eligibility

We cannot determine eligibility with the PowerSchool application at this time. The functionality does not specifically meet requirements nor does the logic on work around processes defined by DPI meet the needs of LEA’s for eligibility. CMS created local reports to meet business needs at this time.

Inadequate Progress

Districts have not been provided a procedure or report within PowerSchool  for NC DMV eligibility of student drivers.

Mid-Year graduation/GPA

The functionality does not exist within PowerSchool to produce transcripts with this mid-term criteria reflecting .5 credit and GPA calculations. At the Symposium, DPI indicated this would be provided soon for use with schools. (no ETA)

PMR

PMR functionality did not work until February 7^th, 2014 maintenance weekend, creating a timeline for districts to resolve fatal exceptions to enrollment data with approval by February 20^th. We are still not comfortable with the warnings within PMR as rendering correct logic for enrollment counts; however, we continue to analyze this each day.

Drop out Report

This does not exist at this time. I am seeing communications of this arriving soon by Ken Gattis (DPI business owner) and functionality of reporting to follow.

RPG

Retention, Promotion, Graduation report does not work at this time in PowerSchool.

SAR

School Activity Report that determines both classroom to teacher ratios and highly qualified teachers for instruction is not functional at this time. CMS has work around reports and external custom built applications performing this action. DPI has not implemented the code for this to be active.

Electronic Gradebook

We do not have the same issues as Wake due to not using the standards based features with CMS gradebooks.  The grade scales, assignments, and report cards all work, even parent portals reflect the same information.

Class rank and GPA

Both work in CMS due to the effort of reviewing and modifying historical grades.

Transcripts

There is still a transcript issue. At the last maintenance weekend, many of the corrections to course history that were made by counselors were lost in the weekend.  This means we have not been able to run new GPAs in HSs. 

Third-grade reading: Less testing, more flexibility?

A panel of educators and parents studying North Carolina's Read to Achieve program will ask lawmakers for fewer tests,  more flexibility and a one-year reprieve in mandated penalties for kids who fail exams,  the state Department of Public Instruction announced today.

The Read to Achieve Advisory Group's recommendations will go to the state legislature when it convenes in May.  Meanwhile,  the state Board of Education will consider proposals to relieve the testing burden on many local school districts at Wednesday's meeting.

The program is designed to make sure third-graders can read well before moving up,  a goal almost everyone supports.  But the specifics have proven problematic,  especially a  "portfolio"  option designed as an alternative to passing a one-time state exam.  In its current version,  students who want to demonstrate reading proficiency through the portfolio have to pass 36 small reading assessments between now and year's end.

Charlotte-Mecklenburg, Cabarrus, Cleveland, Gaston, Hickory, Kannapolis, Mooresville and Union County schools are among the 16 districts seeking approval this week for alternative testing approaches  (read their proposals here; click item II-B2 for a link).

In the longer run,  the advisory panel is asking the legislature to scale back portfolio testing,  give school districts more flexibility in school calendars and summer reading camps,  and count the current year as a trial run for Read to Achieve.

Lynn Bonner at the News & Observer reports that Senate leader Phil Berger, the law’s champion, doesn’t like that last part.

“No,” Berger said in a statement. “It’s wrong to let another class of third graders slip through the cracks. Passing them along unprepared has not solved the problem before, and it is not a solution now.”

Read more here: http://www.newsobserver.com/2014/02/04/3590879/berger-pops-read-to-achieve-trial.html#storylink=cpy

Academic growth formula: Not secret, just complex

I recently referred to the EVAAS formulas used to calculate North Carolina's school growth and teacher effectiveness ratings as secret. Turns out I'm behind the times.

The Cary-based software company SAS,  which created the formulas and markets them across the country,  initially kept the specifics a proprietary secret.  That's probably why Charlotte-Mecklenburg Schools officials have voiced wariness about having teachers' careers and school reputations depend on a formula they can't review.

It's because of such concerns that SAS released the formulas,  which have been tested by groups such as RAND Corp. and UNC Chapel Hill,  says Jennifer Preston of the N.C. Department of Public Instruction.

But that doesn't mean most educators, citizens and journalists can run the numbers themselves. I'm comfortable with Excel spreadsheets, education data and basic calculations.  But when I see lines like  "KTb + MTu is BLUP of KT + MT provided KT is estimable,"  I'm out.

The calculations turn each student's performance on prior exams into a prediction about how they'll do on the next ones. The actual score is compared with the projection.  Teachers'  "value-added" ratings compare their students' progress to that of other teachers across the state. Those ratings form part of the state's teacher evaluation;  persistent low ratings jeopardize a teacher's job,  while strong ratings may someday lead to performance pay.

Schools are labeled as meeting, exceeding or falling short of growth targets based on how their students did compared with projections.  For many,  2013 growth ratings provided a counterpoint to the bleak picture painted by low proficiency rates on new exams.  In 2014, proficiency and growth will combine to create a state-issued letter grade for all public schools.  For charter schools,  growth ratings are a key factor in determining whether a low-scoring school stays open.

There are,  of course,  people who say no formula can turn student test scores into meaningful measures of school quality and teacher effectiveness.  But given that our state legislators and many national policymakers believe otherwise,  it's important to be able to check the validity of those ratings.

Anyone who works with data,  even on a much simpler scale,  knows how easy it is to make a mistake -- and for that mistake to be compounded as you run it through further calculations.  I've caught plenty of errors  (my own and those of institutions I cover)  by seeing that numbers don't jibe with what I know of reality.

It worries me that such crucial numbers aren't subject to an obvious  "smell test."  But Preston said the state is building in backstops.  For starters,  teachers get a chance to review the roster of students being used in their ratings,  to make sure they're getting credit or blame for the right kids.  Schools and districts review the raw data before it's sent to SAS.  And the state has been reviewing dozens of questions that came in after the release of ratings,  Preston said.

Preston,  a former high school teacher,  says the real value of EVAAS numbers comes from teachers who use student data to craft teaching strategies and principals who use them to make good use of their faculty.  She said her numbers showed she was helping low-scoring students make big gains,  while the students who came in strong stayed flat.  Her principal assigned her to a low-performing class the next year,  while a teacher who got better gains from higher-level students took that group.  "We were both teaching to our strong points,"  she said.

No second chance on 2013-14 exams

N.C. students won't get a second chance if they flunk state exams this year, Vanessa Jeter of the Department of Public Instruction told me this week.

Before the new exams rolled out in 2012-13,  students who fell short were required to take the tests again  (different questions on the same material). If they passed on the second try,  it counted toward school proficiency rates, a significant bump for many struggling schools.

This year,  the only students who will be retested are those who fail third-grade reading exams.  The state's new Read to Achieve program spells out a series of actions that are triggered by failure on those tests,  starting with retesting and potentially leading to summer school and retention. (See the process on page 4 of this guide.)   Based on this year's results,  a lot of children aren't clearing the bar at the end of third grade.  See elementary school proficiency rates,  including the percent who passed third-grade reading,  on this new results map.

A second year of one-shot testing could be bad news for schools facing state-issued letter grades in 2014 (see 2013 results for all Charlotte-Mecklenburg schools and Mecklenburg charter schools in the rail at right).  The same goes for CMS high school students,  who will see 25 percent of their final grades in math I, English II and biology shaped by their score on the state tests.

But Superintendent Heath Morrison argues it will be good news for people trying to monitor how much progress schools make this year.  Before the state made it clear whether retesting would be revived this year,  he argued against it for the sake of having two consecutive years of comparable results.  In 2014-15,  the state is slated to switch tests again,  introducing Smarter Balanced Assessment Consortium exams designed to reflect national Common Core standards and provide a consistent testing system among participating states.

"We would prefer to keep the testing as similar as possible,"  Morrison told the school board.

Brace yourself for new test scores

There's a wild week ahead for supporters of Charlotte-Mecklenburg Schools, with a bond vote, school board election and release of state test scores coming.

We'll get election results Tuesday night. My guess is that backers of the education bonds for CMS and Central Piedmont Community College will be celebrating.  I went to the North Regional Library in Huntersville Tuesday to talk to early voters,  then got pulled off for another story before I could flesh out a full report.  But that small sample was consistent:  The people I talked to knew little or nothing about the bonds going in,  but voted yes because they support education.

"I always support anything to do with the schools,"  said Tara McAlinn of Huntersville,  the mother of a 4-year-old.  It's a sentiment I heard repeatedly,  from young parents to retirees.

However the election goes,  Thursday morning will dash a bucket of cold water in people's faces when the N.C. Department of Public Instruction releases long-awaited results from 2013 end-of-grade exams.  State officials have made it clear that there's going to be a big drop in proficiency rates  --  not because kids got dumber or teachers got less effective,  but because there are more rigorous new tests, new  "cut scores"  for passing and no second try for students who fell below grade level.  Many schools will see hard-earned gains disappear.  If past changes in testing are any predictor,  gaps between the highest- and lowest-performing groups of students will widen.  (One interesting feature:  For the first time,  the state will break out performance for academically gifted kids.)

CMS Superintendent Heath Morrison said this week that he supports tougher tests that are designed to more accurately reflect students' readiness for college and careers.  "Every time we set the bar higher,  our kids respond, our teachers respond."  But he worries that critics who tout the failure of public education will use the new results to say  "See,  I told you it was broken."

There are actually three phases of the test-score release.  Teachers have already started getting  "value-added"  ratings based on the new exams,  which is bound to be a source of some stress.  "You get this information and it's really confusing,"  said one caller,  who had just gotten her report and declined to give her name.  "It's hard to read and it's hard to understand."

Proficiency and growth for schools and districts come out Thursday.  And within 30 days of that,  families are supposed to get reports on how their own children did last year.

If you want to get prepared for the testing data,  DPI has a background brief posted.

If you want to prepare for Tuesday's election,  you can find details about the bonds and the candidates on the Observer's voter guide.

And if you've still got energy left at the end of that crazy week, you can turn your gaze to the future at a Nov. 9 forum on "What's next for public education in Charlotte metro?"  From 10 a.m. to noon at UNC Charlotte Center City,  panelists from CMS,  charter schools,  the state legislature and higher education will talk about choices,  challenges,  changes and coordination.  The session is sponsored by Staying Ahead Carolina and UNCC.  There's no charge,  but registration is required.