CMS high school teachers hosed on ratings?

The irony was obvious last spring:  Charlotte-Mecklenburg Schools didn't trust the state's new final exams enough to count them toward high school students' grades.  But those tests were created to rate teachers,  so teachers not only had to give the exams but spend hours grading new open-ended questions.

Now the results of the teacher effectiveness ratings are in,  and they indicate something went awry in rating CMS high school teachers.

Across the state and in CMS,  more than three-quarters of all teachers met or exceeded the goal for student gains.  But when I broke that out by grade level,  more than 80 percent of teachers in CMS elementary,  middle and K-8 schools met or exceeded the goal,  compared with just over 60 percent of CMS high school teachers.

Erlene Lyde at West Charlotte

I also ran the numbers for more than 12,700 non-CMS high school teachers around the state,  and 78 percent of them met or exceeded the target.

It's possible that these numbers reveal a real shortcoming unique to CMS high school teachers.  But a handful of  teachers and principals I spoke with questioned the results on two grounds:  The validity of the tests and the fact that CMS teens knew they had no stake in scoring high.  Erlene Lyde, a West Charlotte High teacher and vice president of the Charlotte-Mecklenburg Association of Educators,  put it most bluntly:  "Flawed data generated from flawed tests administered in flawed conditions and graded using a flawed scoring mechanism."

Thus, the perpetual challenge:  I think it's important to analyze and report on education data.  But at the same time,  you have to question what the numbers really mean.

I'm still not sure how well the EVAAS formulas from SAS Institute turn student test scores into meaningful measures of school growth and teacher value. But the EVAAS site for looking up school growth ratings is one of the best public data presentations I've seen.  It's a simple matter to look up schools and make comparisons in a number of different ways  (fellow geeks, check out the scatterplot option under comparison reports).

School growth and teacher effectiveness are both based on students'  year-to-year progress on state exams.  As you'd expect,  schools that score well on one measure are likely to look good on the other.  But they're not identical for a number of reasons.  One of them is that school growth is based only on End-of-Grade and End-of-Course exams,  while the teacher ratings include more tests.

Some of you asked excellent detail questions when those ratings first came out. I asked Jennifer Preston of the N.C. Department of Public Instruction to clarify the tests used and the way student results were assigned to teachers.  Here's her report,  for those who are interested in diving deep:

• The Department of Public Instruction and the SAS Institute were able to provide teacher-level value-added data for a pretty expansive list of grades/subjects and courses.  They are: Reading/ELA in Grades 4 – 8, Mathematics in Grades 4 – 8, Science in Grades 5 – 8, Social Studies in Grades 5 – 8, Biology, Earth/Environmental Science, Chemistry, Physics, English I, English II, English III, Algebra I/Math I, Geometry, Algebra II/Integrated Math III, World History, Civics and Economics, United States History, American History I, and American History II.  These estimates are all based on the administration of End-of-Grade assessments, End-of-Course assessments, and NC Final Exams.  North Carolina has also had a well-established Career and Technical Education assessment program for many years; teachers of more than twenty-five Career and Technical Education courses received individual value-added scores.  
• In order to ensure that all value-added estimates are fair and valid, we do have some safeguards in place around minimum student counts.  For End-of-Grade Assessments in Science, End-of-Course Assessments. NC Final Exams, and the CTE State Assessments, teacher must be connected at least ten students and the equivalent of six "full students," defined as students with 100% instructional responsibility claimed by one teacher.  This point is most easily explained with examples.  Let's say that an Exceptional Children's teacher has claimed 20 students at 10% instructional responsibility for each one.  While the teacher is connected to ten students, he is only connected to the equivalent of two "full students" (20 students X 10% each = 2 full students). The teacher will not have a value-added score because he is connected to fewer than six "full students."  A different Exceptional Children's teacher has claimed 20 students at 50% instructional responsibility for each one.  This teacher is connected to at least ten students, and is connected to the equivalent of 10 "full students."  He will have a value-added score.  Each of the students must have at least three prior test scores (in any grade/subject or course) in order to be used in the analysis.  For End-of-Grade Assessments in English Language Arts and Mathematics (Grades 4 – 8), a teacher must be connected to six "full students," using the same terminology as described above.  These business rules are to ensure the quality of the value-added data – if a value-added estimate is calculated using a very small number of students, it's simply not valid.  While a bit complicated, these rules simply reflect the reality of teaching today – there are lots of cases in which teachers share instructional responsibility for students and work as a team to provide them with the services they need.