Testing lessons and the coal-mine quiz

Charlotte-Mecklenburg schools are closed for spring break, but the testing folks are analyzing results of the district's recent field tests in hopes of having valid questions selected by the start of next week. As you probably recall, CMS is rolling out 52 new tests, designed partly to help officials rate teacher effectiveness as part of a move toward performance pay.

The one-on-one tests for grades K-2, which outraged some parents and teachers because they were so time-consuming, will survive with some revisions, said Chris Cobitz, the CMS official in charge of the new exams. Instead of asking children to do 15 tasks for each of four subjects (reading, math, science and social studies), the tasks will be limited to 10 to hold testing time to 15 minutes per subject. Cobitz said his staff is talking to the elementary schools that had the fewest problems with field testing to set up guidelines for all schools.

The "best practices sheet" isn't ready yet, but one thing has been decided: Only school staff will be allowed to administer the real tests in May. Parent volunteers, who did some of the field testing this month, will still be encouraged to monitor the testing, Cobitz said.

Despite some requests for a shift to multiple-choice answers for the youngest kids, Cobitz said CMS will continue with open-ended questions, which require the adult tester to judge whether the child's answer is worth full, partial or no credit. CMS is looking at technology to streamline the testing, but that won't be available in May, he said.

Cobitz's crew is also poring through feedback from the schools about bad questions in the field tests. People administering the tests are supposed to preserve testing security, but we've all heard reports of everything from typos and faulty numbering to questions that just don't make sense.

If you listened to Charlotte Talks last Monday, you heard host Mike Collins challenge CMS officials Ann Clark and Andy Baxter to answer a question that a listener had sent in after seeing it on a third-grade test. The question involved a coal-mining town, and none of the three options sounded sensible. Clark and Baxter didn't even try, and Collins acknowledged it was possible he didn't have the precise question (I was listening from home, and I was stumped, too).

Cobitz, who wasn't on the show, later looked up the question: In a town built around a coal mine, which is most likely to be true? The options: All women work in the mine, most men work in the mine, the mine never lays people off, or the mine is the safest place to work.

The correct answer is "Most men work in the mine," Cobitz said. It's not so much about gender roles but about the relationship between a community and its dominant industry, which students should have learned in third grade, he said. The question was solid, according to Cobitz -- but it won't be used because it has been made public.

He said he's also hearing complaints about the new high-school chemistry exam (it replaces a state exam that N.C. officials recently discontinued). But Cobitz, a former chemistry teacher, said so far the CMS test seems to match what students should be learning.

Greensboro, CMS differ on teacher pay cut question

As Charlotte-Mecklenburg Schools struggle to close an estimated $100 million budget shortfall, everybody's been lobbing suggestions at CMS officials about how they can prevent the planned layoffs of roughly 600 educators. One popular idea has been to give all teachers an across-the-board pay cut rather than laying so many off.

CMS has said that's a non-starter. Under state law, the district argues, cuts to the salaries of certified personnel (e.g., tenured teachers) are regarded as demotions, which would trigger appeal and hearing rights. With some 10,000 teachers and instructional support staff, that could obviously prove cumbersome. Plus, CMS leaders say teachers have had no pay raises for the past three years, and have had to take on extra work. Together, that's a pay cut in itself.

Commissioner Bill James recently pointed out in an e-mail to the newspaper that the Greensboro school system apparently sees the issue a little differently than CMS Superintendent Peter Gorman. Guilford County Schools Superintendent Maurice "Mo" Green, a former Gorman protege, recently submitted his own budget plan to his school board. It does include, as a last option, a pay cut equal to two days salary. It would apply only to employees making at least $35,000. Green's chief of staff, Nora Carr, told me in an e-mail: "Employees will work the same number of days and hours, just receive less pay to do so. We’re trying to save as many jobs as possible for our hard-working employees."

Which, of course, is the exact logic used by folks asking for a pay cut here in Charlotte. So, why is a pay cut a valid option for saving Greensboro teachers' jobs, but invalid for CMS? When I asked Greensboro's attorney, Jill Wilson, about all this, she said a pay cut would indeed be a demotion and a reduction-in-force for career (tenured) and contract employees, and would trigger the same procedures as layoffs. All pretty straight-forward, in her estimation.

I've asked CMS leaders to respond on this issue. I'll post the answer here when I get it. I'll also have some more in-depth questions and answers about Gorman's 2011-12 budget proposal in a story that's slated to run in Sunday's Observer.

Stay tuned...

Test week wrap-up

After a week of vigorous commentary and complaints about CMS's "field test" of new exams, I sat down today with Chris Cobitz, the CMS official in charge, to get his take on the issues. Despite a lot of resistance and some obvious glitches, Cobitz says he's "very confident" that CMS is on track to get the results it wants: A set of exams that will size up student knowledge and help measure teacher effectiveness.

 A story running Sunday will provide more info, but here are highlights:

*Yes, the K-2 tests took a lot of time. Cobitz says the field tests, which were supposed to run no more than 50 minutes per child, contained more items than the real test will, including some questions that were intended for the grade lower and/or higher. Because the kids were taking only one subject this week, CMS used the opportunity to try out a longer list of questions. By May, they'll be doing about 10 "tasks" per subject, with a target time of 15 minutes per subject (reading, math, social studies and science) per child.

Cobitz says faculty reports convinced him that asking young children to read a passage and write even a short response was too time-consuming. So those exercises will be eliminated for the math, science and social studies tests (they're essential for reading).

Is the time demand still going to be too much? Cobitz says he's hearing a lot of concern from principals. Superintendent Peter Gorman said today it's too early to say if CMS will look at any revisions based on this week's concerns.

*Yes, there were mistakes, though Cobitz says most were minor. He says out of about 3,500 questions, he's identified "three dozen" with mistakes that shouldn't have gotten past the CMS staff that screened them -- things like repeated words, numbering errors or answers that didn't match the instructions. Those will be corrected or eliminated by May, he said.

I skimmed some K-2 tests, and they didn't look as sloppy as I'd expected from reading the critiques. There was one second-grade math question that confused me; I approached the question about someone who "wants to make a prism" as a sort of engineering question (how is this guy making it?). It wasn't until I looked at the answer that I understood it was just asking for the two-dimensional shapes that formed the surface of the prism. I can't say if that was a bad question, or if it just wasn't aimed at middle-aged reporters.

The kindergarten social studies exam was heavy on holiday questions, as several commenters noted. That's because one of the N.C. goals for kindergarteners is to "explain celebrated holidays and special days in communities." The kids are supposed to be able to "explore how families express their cultures through celebrations, rituals and traditions; identify religious and secular symbols associated with famous people, holidays and special days of diverse cultures; and state reasons for observing special, religious and secular holidays of diverse cultures."

I saw the question about Christmas symbols; baby Jesus and the star of Bethlehem were listed as acceptable answers, but so were trees and ornaments. There was a question asking how Americans could celebrate Memorial Day, one asking for two holiday traditions "you enjoy with your family during winter," and one asking the name of the February holiday celebrating love. Cobitz says his staff scrapped several proposed questions about the Mexican Cinco de Mayo.

*The amount of paper that went into this is mind-boggling. I walked in to see a stack maybe two inches high of tests Cobitz had pulled for me to look at. That was one version of each of the K-2 tests, or 12 tests. There were actually four field versions of each test. Each test contained the version the student used and the guide for the person doing the test, with explanations of how to present the question and a key to the answers. So if a teacher gave 10 versions of the same test, she got 10 identical copies of the teacher guide. That will be streamlined in May, Cobitz said, when there will be individual tests for students and one or two copies for the adults giving the test.

Expect to hear more about this at Tuesday's school board meeting. Parents, teachers and students (including high-schoolers wearing T-shirts emblazoned with the number of tests they're taking) are planning to pack the chamber -- which will already be crowded with people interested in Gorman's budget proposal -- and let the board know what they think of the tests.

And WFAE's Charlotte Talks will feature a panel of parents and CMS officials talking about the tests at 9 a.m. Monday.

Results still patchy on CMS school turnaround plan

Charlotte-Mecklenburg Schools has completed its second study of the Strategic Staffing school turnaround plan, and the results aren't any rosier than they were when I reported on the program in December.

Strategic staffing is Superintendent Peter Gorman's effort to improve student success at low-performing, high-poverty schools by enticing top principals with bonuses and giving them money and freedom to bring in teams of teachers with a track record of success. He describes it as a three-year process. The first batch of schools -- Briarwood, Bruns, Devonshire, Reid Park, Sterling and Westerly Hills elementaries and Ranson Middle -- are in their third year now.

The latest report looks at test scores, student attendance and discipline, teacher surveys and other measures of progress for the two complete years that those schools have been in the program, along with the first year's results for a second group: Allenbrook, Ashley Park, Druid Hills, Paw Creek and Thomasboro elementaries and Albemarle Road and Spaugh middle schools.

Bottom line: There aren't a lot of clear-cut victories so far, though there are bright spots. In many cases, a comparison group of struggling high-poverty schools outperformed strategic staffing schools.

I give Gorman, Chief Accountability Officer Robert Avossa and the staff credit for being willing to scrutinize a pet project and release results, even when they're not glowing. Those who care about the schools in question or the strategic staffing plan in general may find it worth working through 78 pages of detailed analysis.

Resistance builds as CMS test week continues

At East Mecklenburg High last night, parents swapped strategies for resisting CMS's push for more testing, as kids take "field versions" of dozens of new exams. Take your kids late or keep them home altogether so they'll miss the test, some parents suggested. Tell them to sign their name and leave it blank in protest, said another.

One man, who didn't identify himself, noted that the exams for older kids use a multiple-choice "bubble-in" format and urged them to choose one consistent answer: "Let them bubble E for 'Enough' and send a message to Dr. Gorman."

Someone else said each question has only four possible answers.

"Mark B for 'Bull!' " another man called out.

That's the mood among some parents, who are outraged at the time kids are spending on tests designed to rate teachers. Students in kindergarten through second grade were tested Monday and Tuesday. In calls, emails and comments on yesterday's blog, numerous teachers and parent volunteers said the one-on-one tests that officials had described as taking 15 minutes per child were taking up to an hour.

Many also raised questions about the quality of the test items, noting subject matter that seems too advanced and questions with spelling and punctuation errors. Some asked who's getting paid to put these tests together.

Chris Cobitz, the CMS official who's overseeing the new tests, said this week's field tests showed that asking young children to read and answer some of the questions was taking too long. That's something CMS will take into account in creating the final version, to be given in May, he said.

Cobitz said Measurement Inc. of Durham is being paid to submit questions, which his staff reviews. Putting them in front of real kids this week helps CMS staff winnow out questions. If some questions seem too advanced, he said, that's intentional. It not only lets teachers see how high-fliers do, it may indicate a question that is too easy -- if, for instance, most first-graders end up correctly answering a question that was designed for second-graders.

"We pay for items we accept," Cobitz said. "One of three we fully intend to reject. We don't pay for the ones we reject."

He acknowledged there are typos, a result of the rush to get some 400 tests ready for this week. Yes, that's larger than the number of subjects (52) discussed earlier; there are four versions of each test, he said.

"Punctuation is something we can fix," he said.

Now it's time for the older kids to start field-testing, everything from third-grade social studies and science to a raft of high school electives. That poses its own challenges, especially when parents are urging their children to resist.

Courtney Kramer, a German teacher at Smith Language Academy, says she sent home letters with her sixth-grade home room explaining the field tests, which don't count toward student grades or teacher ratings.

"When I passed out the letters and we started to talk about the tests, one student raised his hand," Kramer wrote. "He said, 'Why don't we just fail it on purpose? It doesn't count for a grade and they will have to make the real tests easier.' Of course the rest of the students in the class agreed. If this reaction was immediate in my class, I can only imagine how many other students all over CMS have decided to purposefully fail."

CMS test week: One teacher's view

Charlotte-Mecklenburg Schools is doing a "field test" of its new exams this week. Parents and teachers are buzzing about whether this is a good use of time, and whether the tests themselves are any good.

An elementary school English as a Second Language teacher sent this "snapshot of what has happened in one day, at one school" Monday evening. She identified herself to me, but asked that I not publish her name for fear it will cause problems for her or her principal. I'm sharing it to start the discussion; if any of you who are seeing this first-hand are willing to share your observations for publication, please get in touch (ahelms@charlotteobserver.com or 704-358-5033).

Here's the teacher's report, edited slightly for length and clarity:

*Tests did not arrive at school until Friday, April 1. School administration did not have time to train test administrators enough to feel confident about giving the test.

*Special area classes (music, art, PE, etc) and ESL classes are cancelled this week so that those teachers may assist with testing. This is to ensure that classroom instruction can continue. However, students will miss those special area classes. Most teachers at my school have some planning time during the special area class time. They will not have planning time this week.

*I have 50 kindergarten students to test this week. That is about 20-30 minutes per test, times 50 students. It's mentally exhausting for me. I am wondering how much time the final summative tests will be. We have to administer those next month. We are looking forward to having to cancel instruction for a week then as well. 10 days of instruction lost, out of 180 instructional days. That's a lot.

*The second-grade test has been taking more than 50 minutes. 50 minutes is supposed to be the upper limit of the test. This is only for one portion of the test (like, just math, or just social studies).

*I could write about 5 pages about how poorly constructed the test itself was but I'm not sure how much that would fall into breaking test security. I can say anecdotally that I have administered many different types of tests and this is about the worst test I have ever seen, as a "standardized" test. I don't know how much CMS spent just getting this field test version, but it appears to have been a complete waste of money, at the same time we are decreasing services and planning to lay off hundreds of teachers. The wording of the questions, the graphics that go along with the questions, the instructions for assessing the student's answers... It's not good. That is worrisome since these will (perhaps) eventually be used for Pay for Performance. How can we respect a PfP model if it is built on faulty testing data?

*I am giving the kindergarten science test. It is 34 pages long, so 17 sheets of paper. That times an average of 25 students per kindergarten class at my school. If each K-2 test is about that long: There are 21 K-2 classes in my school. So, 17 sheets of paper x 25 students x 21 classes x the number of elementary schools in CMS. That's a lot of paper. We usually have to ration paper to make copies at school. We would love to have that amount of paper to use to support instruction.

*I work with a lot of highly-educated, very intelligent teachers. We are all terrified of these tests being used to judge our "performance" because they are not indicative of what goes on in the classroom. They do not represent the interesting and innovative teaching that goes on across CMS. They are wasting our instructional time and it will be wasted again in May. These tests will not be used to "drive instruction" unless we begin teaching to the tests. And it's sad because we are convinced that the CMS central office has decided that this is going to happen TO teachers and TO students, not matter what anyone on the ground, in the classroom has to say about. Downtown knows best and no teacher is going to be able to tell them otherwise. They will ram these tests down our throats until we give up and quit.

So many tests, so little time

Parents are asking so many questions about the latest batch of tests that Charlotte-Mecklenburg Schools this week released a new explainer on all the tests kids take.

Schools will give a trial version of the new "summative assessments" next week. Those are year-end exams designed by a CMS contractor to size up student mastery of academic subjects, as well as teachers' effectiveness. Students will take a full version of those tests at the end of this school year.

CMS hasn't finished creating all the new exams that will be a foundation for performance pay in 2014. Coming next school year are tests in music, physical education, dance, visual arts and other subjects that require more than pencil-and-paper multiple-choice questions. Teachers are helping with those, and the cost is included in the $1.9 million currently budgeted for test design.

Superintendent Peter Gorman used his weekly news briefing to try to assure parents that their children aren't spending excessive time on testing, even with the new ones. Kids in K-2 spend a total of 4 hours and 15 minutes on testing, including the new ones, he said. That's out of 1,035 hours of class time -- a number that jumps to 1,170 with the longer school day next year.

In grades 3, 4, 6 and 7, students take state reading and math End of Grade exams and will add CMS social studies and science tests. That will be a total of 17 hours of testing, Gorman said. In grades 5 and 8, students take the state science exam, which brings the total to 19.5 hours. That's because the state requires four hours for its tests, while CMS' take 90 minutes.

High school testing is harder to calculate. The new plan calls for all courses to have a final exam designed by the state or CMS. But students also take a variety of other tests, including Advanced Placement, IB, PSAT and SAT.

While the total hours may not be overwhelming for students (I suspect parents would say that's open to debate), K-2 teachers are grappling with the demands of testing their kids one at a time -- 15 minutes each for reading, math, science and social studies. Because teachers' effectiveness ratings, and eventually their pay, will ride on results of these tests, they're supposed to either have a proctor watching or swap students, so they're not testing their own kids.

Which brings me to a final point: If you're a parent or school volunteer, expect to be begged to help with proctoring. And if you're a retiree or someone else with time to spare, you'd endear yourself to local educators if you'd call your nearest school or get in touch with the CMS volunteer office to step up.