CMS high school teachers hosed on ratings?

The irony was obvious last spring:  Charlotte-Mecklenburg Schools didn't trust the state's new final exams enough to count them toward high school students' grades.  But those tests were created to rate teachers,  so teachers not only had to give the exams but spend hours grading new open-ended questions.

Now the results of the teacher effectiveness ratings are in,  and they indicate something went awry in rating CMS high school teachers.

Across the state and in CMS,  more than three-quarters of all teachers met or exceeded the goal for student gains.  But when I broke that out by grade level,  more than 80 percent of teachers in CMS elementary,  middle and K-8 schools met or exceeded the goal,  compared with just over 60 percent of CMS high school teachers.

Erlene Lyde at West Charlotte

I also ran the numbers for more than 12,700 non-CMS high school teachers around the state,  and 78 percent of them met or exceeded the target.

It's possible that these numbers reveal a real shortcoming unique to CMS high school teachers.  But a handful of  teachers and principals I spoke with questioned the results on two grounds:  The validity of the tests and the fact that CMS teens knew they had no stake in scoring high.  Erlene Lyde, a West Charlotte High teacher and vice president of the Charlotte-Mecklenburg Association of Educators,  put it most bluntly:  "Flawed data generated from flawed tests administered in flawed conditions and graded using a flawed scoring mechanism."

Thus, the perpetual challenge:  I think it's important to analyze and report on education data.  But at the same time,  you have to question what the numbers really mean.

I'm still not sure how well the EVAAS formulas from SAS Institute turn student test scores into meaningful measures of school growth and teacher value. But the EVAAS site for looking up school growth ratings is one of the best public data presentations I've seen.  It's a simple matter to look up schools and make comparisons in a number of different ways  (fellow geeks, check out the scatterplot option under comparison reports).

School growth and teacher effectiveness are both based on students'  year-to-year progress on state exams.  As you'd expect,  schools that score well on one measure are likely to look good on the other.  But they're not identical for a number of reasons.  One of them is that school growth is based only on End-of-Grade and End-of-Course exams,  while the teacher ratings include more tests.

Some of you asked excellent detail questions when those ratings first came out. I asked Jennifer Preston of the N.C. Department of Public Instruction to clarify the tests used and the way student results were assigned to teachers.  Here's her report,  for those who are interested in diving deep:

• The Department of Public Instruction and the SAS Institute were able to provide teacher-level value-added data for a pretty expansive list of grades/subjects and courses.  They are: Reading/ELA in Grades 4 – 8, Mathematics in Grades 4 – 8, Science in Grades 5 – 8, Social Studies in Grades 5 – 8, Biology, Earth/Environmental Science, Chemistry, Physics, English I, English II, English III, Algebra I/Math I, Geometry, Algebra II/Integrated Math III, World History, Civics and Economics, United States History, American History I, and American History II.  These estimates are all based on the administration of End-of-Grade assessments, End-of-Course assessments, and NC Final Exams.  North Carolina has also had a well-established Career and Technical Education assessment program for many years; teachers of more than twenty-five Career and Technical Education courses received individual value-added scores.  
• In order to ensure that all value-added estimates are fair and valid, we do have some safeguards in place around minimum student counts.  For End-of-Grade Assessments in Science, End-of-Course Assessments. NC Final Exams, and the CTE State Assessments, teacher must be connected at least ten students and the equivalent of six "full students," defined as students with 100% instructional responsibility claimed by one teacher.  This point is most easily explained with examples.  Let's say that an Exceptional Children's teacher has claimed 20 students at 10% instructional responsibility for each one.  While the teacher is connected to ten students, he is only connected to the equivalent of two "full students" (20 students X 10% each = 2 full students). The teacher will not have a value-added score because he is connected to fewer than six "full students."  A different Exceptional Children's teacher has claimed 20 students at 50% instructional responsibility for each one.  This teacher is connected to at least ten students, and is connected to the equivalent of 10 "full students."  He will have a value-added score.  Each of the students must have at least three prior test scores (in any grade/subject or course) in order to be used in the analysis.  For End-of-Grade Assessments in English Language Arts and Mathematics (Grades 4 – 8), a teacher must be connected to six "full students," using the same terminology as described above.  These business rules are to ensure the quality of the value-added data – if a value-added estimate is calculated using a very small number of students, it's simply not valid.  While a bit complicated, these rules simply reflect the reality of teaching today – there are lots of cases in which teachers share instructional responsibility for students and work as a team to provide them with the services they need.

Teacher ratings: Best and worst in Charlotte region

The teacher effectiveness ratings released this week provide rich material for analysis and debate. I just got the spreadsheet from the N.C. Department of Public Instruction, and I'll be poring through it to see what trends emerge.

But I couldn't resist a quick search to see which schools fall at the top and the bottom for our area. First I looked for those with the highest percentage of teachers who exceeded the state's goals for student growth on state exams. I eliminated those with fewer than 20 teachers, where percentages are so dramatically swayed by one or two individuals. That shuts out most elementary schools, because state testing starts in third grade and only fourth- and fifth-grade teachers have students with a previous year's scores to base growth projections on.

Here's what came up:

1. Weddington Middle (Union County), 77.8 percent of 45 teachers.
2. Highland School of Technology (Gaston County magnet), 73.9 percent of 23 teachers.
3. South Point High (Gaston County), 68.9 percent of 45 teachers.

Porter Ridge High

4. Ridge Road Middle (CMS), 60.9 percent of 46 teachers.
5. Marvin Ridge High (Union County), 60.5 percent of 38 teachers.
6. Mount Holly Middle (Gaston County), 59.3 percent of 27 teachers.
7. Lake Norman High (Iredell-Statesville), 53.6 percent of 56 teachers.
8-9. Winkler Middle (Cabarrus County), 52.5 percent of 40 teachers.
8-9. Porter Ridge High (Union County), 52.5 percent of 40 teachers.
10. South Charlotte Middle (CMS), 51.4 percent of 35 teachers.

Statewide, 23 percent of teachers exceeded the target.

I also sorted for schools with the highest percentage of teachers who failed to meet the growth target. Again eliminating schools with fewer than 20 teachers, they are:

1. Hopewell High (CMS), 56.8 percent of 44 teachers.
2. Vance High (CMS), 54 percent of 50 teachers.
3. Harding High (CMS), 53.2 percent of 47 teachers.
4. Statesville High (Iredell-Statesville), 51.5 percent of 33 teachers.
5. North Meck High (CMS), 50 percent of 44 teachers.
6. West Meck High (CMS), 48.4 percent of 62 teachers.
7. Friday Middle (Gaston County), 47.8 percent of 23 teachers.
8. Grier Middle (Gaston County), 46.2 percent of 26 teachers.
9. Hunter Huss High (Gaston County), 45.9 percent of 37 teachers.
10. Independence High (CMS), 44.3 percent of 61 teachers.

Twenty-one percent of all N.C. teachers fell short of the growth target.

My search included district and charter schools in Mecklenburg, Union, Cabarrus, Iredell, Catawba, Lincoln and Gaston counties.

I'm intrigued by these numbers, but I want to be clear that this is not a definitive picture of academic quality at these schools.  It's worth noting that all schools on the "worst"  list had teachers with top ratings,  and most on the  "best"  list had teachers who fell short.  There's still plenty of room for debate on whether the value-added formula can really turn student test scores into a meaningful measure of how good a teacher is.  But these ratings are shaping education decisions and teachers' careers,  so they're worth exploring.

Testing boycott and teacher ratings

A snowstorm, an earthquake  --  and now we have Pamela Grundy and Pat McCrory on the same side of an education issue. The end times must be here.

Grundy, a founder of Mecklenburg ACTS and Parents Across America,  is urging parents to boycott state exams this spring,  part of a national  "Testing Resistance and Reform Spring"  protest against excessive testing.  Other sponsors are FairTest,  United Opt Out,  the Network for Public Education and Save Our Schools.

Grundy and Carol Sawyer of Meck ACTS

So far there aren't a lot of details about the boycott.  Charlotte-Mecklenburg parents have talked about keeping their kids out of testing in the past;  the official stand is that tests are part of the curriculum and parents aren't allowed to opt out.

McCrory,  the Republican governor,  probably doesn't support the boycott.  But he did get vocal about the hazards of overtesting at last week's Emerging Issues Forum on teachers.   (Speaking of which,  anyone interested in issues raised at that forum can sign up for a free online course on world-class teaching,  sponsored by N.C. State's Emerging Issues Institute.)

No one's arguing against kids taking tests to show what they know.  The controversy springs from the barrage of N.C. exams designed primarily to rate teachers and schools.  A lot of folks who want solid data about the quality of public schools say the state is going too far in the quest to generate numbers that may or may not capture teacher quality.

Meanwhile,  we just got a first look at how those test-generated ratings play out for N.C. schools and districts.  I'll be eager to hear what people are thinking as they check out data on their schools.  The state's site makes it easy to look up schools and districts,  but it's tough to do any kind of big-picture comparison and analysis looking up one data point at a time.  State officials say they'll send me a spreadsheet as early as today.  If you'd like a copy,  shoot me an email at ahelms@charlotteobserver.com with  "spreadsheet"  in the header.

I caught up with Julie Kowal,  executive director of CarolinaCAN,  after I'd filed the story on value-added ratings.  Her group is big on data and accountability,  so it was no surprise to hear her say that  "the wonk in me"  loves this report:  "It is so valuable for the state to make this publicly available in the way they have. We need this data to be able to make responsible decisions."  But she added a note of ambiguity:  "The parent in me thinks it's very difficult to know what it's good for."

Academic growth formula: Not secret, just complex

I recently referred to the EVAAS formulas used to calculate North Carolina's school growth and teacher effectiveness ratings as secret. Turns out I'm behind the times.

The Cary-based software company SAS,  which created the formulas and markets them across the country,  initially kept the specifics a proprietary secret.  That's probably why Charlotte-Mecklenburg Schools officials have voiced wariness about having teachers' careers and school reputations depend on a formula they can't review.

It's because of such concerns that SAS released the formulas,  which have been tested by groups such as RAND Corp. and UNC Chapel Hill,  says Jennifer Preston of the N.C. Department of Public Instruction.

But that doesn't mean most educators, citizens and journalists can run the numbers themselves. I'm comfortable with Excel spreadsheets, education data and basic calculations.  But when I see lines like  "KTb + MTu is BLUP of KT + MT provided KT is estimable,"  I'm out.

The calculations turn each student's performance on prior exams into a prediction about how they'll do on the next ones. The actual score is compared with the projection.  Teachers'  "value-added" ratings compare their students' progress to that of other teachers across the state. Those ratings form part of the state's teacher evaluation;  persistent low ratings jeopardize a teacher's job,  while strong ratings may someday lead to performance pay.

Schools are labeled as meeting, exceeding or falling short of growth targets based on how their students did compared with projections.  For many,  2013 growth ratings provided a counterpoint to the bleak picture painted by low proficiency rates on new exams.  In 2014, proficiency and growth will combine to create a state-issued letter grade for all public schools.  For charter schools,  growth ratings are a key factor in determining whether a low-scoring school stays open.

There are,  of course,  people who say no formula can turn student test scores into meaningful measures of school quality and teacher effectiveness.  But given that our state legislators and many national policymakers believe otherwise,  it's important to be able to check the validity of those ratings.

Anyone who works with data,  even on a much simpler scale,  knows how easy it is to make a mistake -- and for that mistake to be compounded as you run it through further calculations.  I've caught plenty of errors  (my own and those of institutions I cover)  by seeing that numbers don't jibe with what I know of reality.

It worries me that such crucial numbers aren't subject to an obvious  "smell test."  But Preston said the state is building in backstops.  For starters,  teachers get a chance to review the roster of students being used in their ratings,  to make sure they're getting credit or blame for the right kids.  Schools and districts review the raw data before it's sent to SAS.  And the state has been reviewing dozens of questions that came in after the release of ratings,  Preston said.

Preston,  a former high school teacher,  says the real value of EVAAS numbers comes from teachers who use student data to craft teaching strategies and principals who use them to make good use of their faculty.  She said her numbers showed she was helping low-scoring students make big gains,  while the students who came in strong stayed flat.  Her principal assigned her to a low-performing class the next year,  while a teacher who got better gains from higher-level students took that group.  "We were both teaching to our strong points,"  she said.

New tests bring twists, frustrations

It's hard to miss the irony:  Charlotte-Mecklenburg Schools didn't have enough confidence in new state exams to stake students'  grades on them,  yet student performance on those tests will be used to evaluate teachers.

That's not necessarily a sign of hypocrisy from the district.  Local officials had a choice about counting the tests toward student grades,  but the state has mandated that value-added ratings generated by the SAS Institute's EVAAS system be part of teacher evaluations.  Still unclear is whether lawmakers will use scores from the new exams to assign letter grades to schools this year.

CMS leaders aren't saying the new exams are bad.  They're just saying there are too many unknowns this year,  with teachers having little information about what would be on the new tests and how to prepare students.  (Those of us who have been around awhile know the state has a history of discovering glitches after kids take a new test,  and these have not been field-tested.)

Rather than risk a student failing a class,  which could potentially jeopardize or delay graduation,  CMS decided the state exams won't count toward grades this year.  That's frustrating to some teachers,  who believe students will put little effort into an exam that can only benefit or harm their instructor.  To top it off,  teachers have to spend unpaid time scoring new items on the tests.

CMS created a parent guide to explain the exams students are taking now  (some exams started earlier in May and some will run through June).  In addition to the familiar terms  -- end-of-grade exams in elementary and middle schools, end-of-course exams in high school  --  you'll now hear about  "common exams,"  sometimes called MSLs,  for measures of student learning.  The difference is that EOGs and EOCs will be used to grade schools,  while common exams will only be used for teacher evaluations.

I'll be curious to hear what parents and teachers think as the exam period plays out.  If there's one consolation for those who think this is too much testing,  it's that the new state program doesn't include the K-2 tests CMS tried in 2010,  requiring adults to administer the tests one student at a time.  However,  officials do expect an early-grades reading test in 2014.

Tests, teachers and zombies

The N.C. Board of Education is trying again to turn student test scores into a reliable measure of teacher effectiveness.  At today's meeting in Raleigh,  members will talk about using the new tests that debut this spring to calculate ratings for teacher and principal job evaluations.

According to the online presentation,  this year's teacher evaluations included a rating that combined  the progress their own students made on 2012 tests and the school's overall growth.  That approach was supposed to encourage teamwork and provide ratings for teachers whose students don't take state exams.  But in reality,  the school numbers pulled down about 2,850 effective teachers and bumped up about 2,650 teachers whose individual ratings were low,  the presentation says.

"The inclusion of school-wide growth hurt high-performing teachers working in low-performing schools and discourages high-performing teachers from working in the schools where they are needed most,"  the report says.  For 2013,  the Department of Public Instruction is urging the board to base teacher ratings only on student scores they're directly responsible for.

Value-added ratings created a backlash when Charlotte-Mecklenburg Schools tried them three years ago.  This time around,  they'll be based on new, more complicated exams.  And the teacher value is being calculated through a system known as EVAAS,  a proprietary private formula  that can't be double-checked by teachers or districts.  That's raising concerns among local educators,  including Superintendent Heath Morrison.

Meanwhile,  some of the folks who fought the CMS testing are reviving the effort.  Mecklenburg ACTS has launched a "No Testing Zombies" campaign,  with a rally planned before Tuesday's school board meeting. The slogan:  "How are standardized tests like zombies? They're mindless and they just keep coming."  Laurie Smithwick,  a Chantilly Montessori mom who runs a design firm,  created the logo of zombies in a school-crossing sign.  As of Wednesday evening, the group had 579 signatures on its electronic petition urging state officials to put the teacher-rating effort on hold.

There's been some back-and-forth on this blog and in the Observer's editorial section about the number of new tests being rolled out this year,  based partly on whether you define a new version of a previously-tested subject as "new."  I hope to get a more detailed breakdown on Morrison's 177-test tally next week.  In the meantime, MeckACTS has revised its petition to refer to  "more than 100"  new tests.

Trying to simplify value-added ratings

As debate continues over the Charlotte-Mecklenburg quest to crunch teacher-quality numbers, one of the district's officials has forwarded a link to an explanatory slide show on value-added ratings.

As you may recall, those ratings are a way of analyzing student test scores to tease out how much progress can be attributed to teachers. CMS started running a preliminary formula to calculate those ratings based on 2010 results. By 2014, the district hopes to have a more sophisticated value-added formula in place and use it, along with other measures of teacher quality, to base part of teachers' pay on student results.

Susan Norwood runs a performance-pay pilot in 20 high-poverty CMS schools. She shared the link to a presentation from the Value-Added Research Center (part of the University of Wisconson-Madison education school) that she uses to help explain her program. She thought readers might find the oak-tree analogy helpful.

"Based on some of the online comments, there's still a lot of confusion about the measures and what their corresponding numbers represent," she e-mailed.


I'd be skeptical that anyone would watch a 13-minute slide show on value-added ratings. But a memo from CMS performance pay director Andy Baxter trying to explain the CMS calculation has gotten more than 2,000 views since Sunday, so evidently interest is high.
 
Eventually I hope CMS will be able to explain its calcualtion as clearly as the hypothetical oak-tree example. I know students are more complicated than trees, and variables such as absences and learning disabilities may be harder to quantify than sunlight and soil. But as long as the calcualtions are so convoluted that they boil down to "trust us," I suspect a lot of teachers won't.

Value-added formula: It's complicated

While trying to understand and explain CMS' formula for value-added teacher ratings, I asked Andy Baxter, the district's performance-pay guru, if he could show me the calculation for one student. I was hoping for something simple enough to fit into a box to attach to the main article.

He said he'd give it a try. The result ... well, it wasn't short and simple.  But with so much riding on these numbers, I figured some of you would like to read the full document.

Teachers in trouble and school-board lines

Just over 200 Charlotte-Mecklenburg teachers have performance and/or misconduct problems bad enough to merit warning letters, and almost 700 more relatively new teachers don't meet the standards to earn tenure, according to a report Superintendent Peter Gorman sent the school board today.

The tally comes in response to school board questions about the standards being used to evaluate teachers and decide who's in line for performance-based layoffs.

The weekly report to the school board also contains this nugget from Chief Operating Officer Hugh Hattabaugh: 345 teachers from schools that will close next year applied for transfers to other CMS schools, and less than half of them -- 153 -- got offers.

To answer an anticipated question: No, I don't know what all those "special meetings" on Gorman's calendar are.

In an unrelated item, the school board decided earlier this week not to change the structure or terms of the board, which now has six district members and three at-large representatives serving staggered four-year terms. They tapped planner Mike Raible, the district's latest go-to guy on all things complex and controversial, to draw new lines for the electoral districts based on 2010 Census data. He'll bring back options for board discussion and approval.

The school board decided not to focus on creating boundaries that coincide with county commissioner districts. However, at the end of the process they will confer with county folks and, if the two plans look similar, see if they can make minor adjustments to keep the lines the same.

"It may be better than we go down parallel paths and find that our destination may not be all that different," Rhonda Lennon said.

That's a wrap for this week.  If you're reading a schools blog on Friday evening or the weekend, you're probably hard-core enough to look forward to Sunday's paper, when I'll have an in-depth look at CMS's quest to crunch teacher value into a number. There's a school-by-school list of how effective each faculty looked based on the 2010 CMS formula (no listings for individual teachers).

Will Bill Gates come courting me?

Parent activist Pam Grundy shared this link to a Washington Post education blog about the Gates Foundation's latest quest to "win over the public and the media to its market-driven approach to school reform."

According to Valerie Strauss's blog, a grant proposal outlines plans to build “strong ties to local journalists, opinion elites, and local/state policymakers and their staffs” and ensure "frequent placement ... in local media coverage" of stories about teacher effectiveness, performance pay and value-added ratings.

Well! I had a front-page article about Charlotte-Mecklenburg's push toward performance pay in last Sunday's paper, with another coming out this Sunday looking at value-added ratings. I'm tempted to post an address where the Gates folks can mail a big ol' check.

Seriously, this is just another glimpse of the tangled world of education reform. You've got a billionaire philanthropist not only paying to promote his vision for better schools, but sponsoring a group to drum up "grassroots" support. Meanwhile, a local mom who's moving onto the national stage is keeping an eye on such developments and alerting the local media about the plan to woo us.

I may get confused, but I will never get bored on this beat!