Monday, September 17, 2012

Will new exams make the grade?

Bubbling in multiple-choice answers doesn't give a sophisticated picture of students' skills, most would agree.

But North Carolina's new exams,  which include open-ended questions,  are raising plenty of questions.

Last week,  after the Charlotte-Mecklenburg school board got a briefing on the new "measures of student learning"  tests,  state officials announced they'd give districts a little more time to hash out answers.  The state will not require high school students to take the new tests at the end of first-semester courses,  Rebecca  Garland,  the state's chief academic officer,  told me Friday.

The multiple-choice exams North Carolina has relied on for years have one big advantage:  They're cheap and quick to score with computer scanners.  Items that require students to solve a math problem or write an answer require human eyes.  And that's where things get complicated.

Superintendent Heath Morrison and new Chief Accountability Officer Frank Barnes told the CMS board last week that teachers will be trained to score the new tests in a process similar to that used for grading Advanced Placement exams.  That requires getting skilled teachers to volunteer,  training them to do consistent scoring and then putting them to work on the actual exams.  Teachers will be paid for their after-hours work,  Morrison said,  but the cost is unknown because CMS leaders haven't seen the tests and don't know how many open-ended questions will be included.

Garland said Friday that local officials won't see the tests until it's time to give them to students.  That's standard testing security,  so no one can unfairly coach students to success.  But she said no test will have more than six open-ended,  or "constructed response,"  questions.  And she said those questions will require one-paragraph answers at most,  not a full essay.

In the next couple of weeks,  Garland said,  teachers will start getting more specifics about the tests.  And she said the state will provide training for teachers who will score the exams.

The tests are not just designed to gauge student skills but teacher effectiveness.  And that makes things a whole lot stickier.

CMS has decided to make the new exams count for 25 percent of a student's grade for that class  (for a list of classes involved,  read the CMS presentation).  That's to motivate students to take the exams seriously,  so teachers won't be penalized for kids who just aren't trying.  But Morrison says he's still wavering on whether that's fair to seniors,  whose graduation could be sidetracked by a test that's new and unproven.  North Carolina has a history of rolling out new tests that turn out to be flawed  (I suspect other states do, too).  The new state tests will take the place of teachers' finals to avoid overtesting.

Teachers who give any of the new state tests this year will get a  "growth score"  on their evaluation,  though it will be  "for information only,"  CMS Deputy Superintendent Ann Clark said.  She said the state has assured CMS  "they have the tools to make that calculation,"  even though the ratings will be based on brand-new tests.  After three years of test-score data pile up,  those ratings will start to count toward evaluations and could ultimately cost persistent low-performers their jobs.

Garland said the growth ratings, which are generated by the SAS analytics company headquartered in Cary, will be similar to what teachers are already seeing if they give end-of-grade and end-of-course exams.  Teachers will get a numerical score and a label of met, exceeded or did not meet expectations.

Garland noted that some states getting federal Race To The Top money are requiring each district to come up with its own program for rating teachers on student growth.  As CMS can testify,  that's expensive, complex and politically explosive.  Garland said North Carolina is taking on that task, but  "we have tried to be as flexible as we can with the districts."

Meanwhile,  no matter how the details are worked out,  some believe the whole testing-and-rating path is the wrong one.

"As a teacher,  I want to know what my weaknesses are so that I can improve,  but the value-added assessment currently in vogue doesn't do that.  No matter who claims it does,  no matter how inevitable it feels,"  S.C. teacher Kay McSpadden wrote in a recent guest column in the Observer.

38 comments:

Truth Seeker said...

With so much riding on the exams having teachers grade them seems risky, especially if there are any subjective sections. This whole thing doesn't seem well thought out or planned by the state since the cost isn't known or budgeted in this year district 's planning. Teachers have a right to be concerned. Let's hope they don't go "Chicago" on CMS.

BolynMcClung said...

DESTRUCTIVE TESTING….NCDPI STYLE

The following statement best describes what Dr. Morrison is trying to prevent for the senior class.

DESTRUCTIVE TESTING
“….In destructive testing, tests are carried out to the specimen's failure, in order to understand a specimen's structural performance or material behaviour under different loads.

Destructive testing is most suitable, and economic, for objects which will be mass produced, as the cost of destroying a small number of specimens is negligible…”

Bolyn McClung
Pineville

Anonymous said...

Now isn't it somewhat STOOPID to let teachers grade subjective answers to tests that will be used as part of teacher evaluation?

Is this like the CMS board "grading" themselves?

C'mon, folks, snap to it.

There's a reason people do not trust all the garbage statistics the educrats spit out.

Anonymous said...

You know- as a teacher, one of the things I was trained to do was - assess. Go figure. For decades, teachers have assessed their students. Now, the bottom line is - politicians do not trust the teacher so they they must now construct secret exams..the content of which teachers are not supposed to know. How crazy is that! We always had measures of student learning - they were called GRADES on report cards. Don't trust the teacher? Hire some you DO trust and stop this foolishness!

Anonymous said...

More and more information is showing that poverty and not teachers is responsible for kids failing. Tests merely measure poverty. Just think if all this money for race to the top, etc. had been used to improve kids' living conditions (neighborhoods and schools) instead of endlessly testing them. I thought everything was about making them feel good about themselves these days. How does repeatedly failing tests do that?

Anonymous said...

This is exactly what CMS wants then no kids will be held back. Its a goal of theirs for a long time not to hold back a kid not on grade level. With "subjective tests" they wont hold a kid back as long as he answers the questions. Current middle school grade weight 45% for coming to school doing homework, 45% weitght for testing and 10% if you parents give to PTA. Ridiculous and they kids dont learn anything.

Anonymous said...

Truth Seeker - the teachers here are unable to go "Chicago" on CMS because they are not unionized. Any teacher here who walked off the job would be immediately terminated and replaced. So no need to worry about that.

Wiley Coyote said...

Wasn't it just a few weeks ago in Charlotte that Democrats were touting their strong ties to unions?

Rahm now faces the same issues Walker did in Wisconsin, where 14 Democrats left the reservation in support of unions in that state and now Rahm is going to sue the teachers union in Chicago for illegally striking?

Chicago school deficit $700 million.

Hypocrisy much?

You can't make this stuff up.

Bill Stevens said...

To 7:28, you are entirely correct correct. However how do you explain so many graduating in past years with a diploma and unable to function in society or even when 1200 of 1600 CMS graduates last year could not pass the entrance tests of CPCC. And as you well know, when a teacher does fail a student, an administrator comes in and changes the grades so they can graduate.

Teachers grading the open ended questions to the test is actually easier to solve than you might think. For example, within CMS, you could have the Providence HS teachers grade the tests of West Charlotte HS. But you would not have the West Charlotte HS teachers grade the Providence HS tests just to protect from revenge grading of previous year. Instead you would randomize which set of HS teachers got other schools' test to grade. Of course the easiest thing to do would be to have some unique number identifier and then mix up where the tests went to be graded. For example, many professional certifications tests in NC are graded by folks outside of the state and they are done much the same way, majority multiple choice and a few open ended problems/questions.

Anonymous said...

Unfortunately, there is one word that explains why teachers cannot be "trusted" to grade tests for which teacher effectiveness is measured:

ATLANTA

Anonymous said...

Anon 8:34.

If "poverty" is the reason for failure, then why does the NAEP report on the performance gap between black and white kids show that poor white kids (as in Free Lunch recipients) do better than non-poor (as in full price lunch) blacks?

If poverty WAS the problem, you wouldn't see that would you?

Of course, the FRL black kids fare even worse, but "poverty" in itself is not the determining factor for those low scores.

Being black is at least as bad as being poor (if not worse).

But I guess that's not politically correct to say.

Anonymous said...

What is LOST in this discussion is how the STUDENTS will be adversely affected. It is not only seniors who will be affected by these tests. Unknown, un-vetted tests that count for 25% of a high school student's grade can destroy a GPA. Two years ago 50 tests were field tested in our schools; these tests were riddled with errors. How can we know that these new MSL tests won't have the same failings? My children (and every other CMS student) shouldn't be guinea pigs in this race to the top game adults are playing.

Anonymous said...

The only way to guarantee any degree of fairness in grading is to take the grading out of the hands of ANYONE who would benefit from falsifying grades.

That includes teachers from "other schools" because ALL teachers could just decide to grade EVERYONE higher to reduce chances of affecting anyone adversely.

Unfortunately, if there is a "performance" system in place, someone will try to figure out how to game that system to their favor if they can.

Anonymous said...

Those using Atlanta as an example of why teachers cannot be trusted....it was hyper-testing that brought that on. Again - if the teacher cannot be trusted - fire the teacher. You are talking about a FEW people - NOT the majority. This insane idea is illogical, expensive, and ridiculous!

Truth Seeker said...

Too many teachers in Chicago to fire and then replace. That is the real power behind unions. That is also the reason to fear unions in non union states. Heck, CMS had difficulty covering classes when they scheduled school on Good Friday one year. I think teachers are starting to find their voice again.

Anonymous said...

In a recent letter to the editor, Tom Caldwell nailed it. We have to raise the salaries of teachers in order to keep the best and brightest. Rooting out bad teachers is simple. Every kid at a school knows the good and bad teachers...ask them first and proceed from there.

A book I read recently gives great advice on what makes a student do well in school. The book is named "Freakonomics". Great book and the author did a follow up called "Super Freakonomics". Great reading and very informative.

Anonymous said...

It's not just Atlanta.

In you have people gaming every kind of performance measurement system ever devised (if they can).

It's a problem with everyone not just teachers and not just a few teachers.

All you can do is minimize this.

Letting people grade their own performance (or input into their performance measurements) will not work.

Wiley Coyote said...

Anon 12:58

So how much do we raise teacher salaries to?

Who determines - and how - which teachers are the best and brightest?

Chicago teachers average between $71,000 and $76,000 per year, depending on which number you want to use. Yet students on grade level and graduation rates are dismal, while over 90% of Chicago teachers are rated excellent to superior.

In 2008 researchers at the Illinois Education Research Council reported what thousands of veteran Chicago teachers had scored on their ACT college entrance exams. The astonishing answer: An average of 19.4 (out of 36). That's about a point below what experts say is the minimum score necessary for college readiness. And it was far below the scores generally required for enrollment at highly competitive universities and colleges.

http://www.chicagonewscoop.org/concern-over-changing-teacher-evaluations/


The current blame du jour for problems is poverty. Let's just lump it all together under that category.

Jeff WIse said...

First in a series of posts....sorry.

Wiley - there are other posts from Chicago sources that show Chicago city teachers averaging less than $50,000. I saw a post over the weekend that was purportedly the salary schedule for Chicago teachers and my quick estimating got to an average of about $60,000.

Regarding value-adds. Many, many researchers - you know the people who study these kinds of thing - have shown that value-adds for teacher assessments is wildly inaccurate for a wide variety of reasons.

In just about every study, researchers found that a teacher who scored in the Excellent category ended up in the Failing category the next year, and vice versa.

The best, most reliable, teacher evaluation systems have been, and continue to be, those that rely heavily on peer reviews.

Wiley Coyote said...

Jeff,

The teachers union stated the salary to be $71,000, disputing the $76,000 others are reporting.

Jeff Wise said...

Second in a series...

All these tests are missing the bigger picture, schools are *not* teaching students to be successful as adults.

A growing body of research shows that our educational system teaches cognitive skills almost exclusively. Yet good cognitive skills are not a predictor of future success.

Read up on Jim Heckman and then Paul Tough's recent book, "How Children Succeed".

We will see in 5-10 years that all this hubbub about testing, testing, testing will have increased student abilities by negligible amounts.

Our students need to be taught how to reason and think critically.

They need to create and do projects, not spend twice the time learning multiplication tables and memorizing dates of various battles.

More testing is never the answer.

Jeff Wise said...

Wiley - I'm just passing on what I read from various places over the weekend.

Anonymous said...

Why doesn't CMess pilot this on the LIFT schools first?

Wiley Coyote said...

How many more years and how many more dollars will it take before someone gets it?

Here's an excerpt from a piece Ann Doss Helms wrote almost 9 years ago to the day; September 14, 2003:

The link between poverty and academic failure takes root outside the classroom.Children from poor families enter school with smaller vocabularies, researchers have found. Their parents often have less education, their homes fewer books.

Poverty can bring instability that makes it difficult to focus on studies.

Disadvantaged children desperately need the best teachers, many experts say. CMS works at recruiting them, offering such incentives as bonuses, graduate-school tuition and smaller classes for teachers who sign on at high-poverty schools.

Still it's a constant struggle to find staff to tackle the task and stick with it.


http://sparkaction.org/node/25332

Nine years and tens of millions of dollars later and almost one generation of students who have gone through the system, we're still hashing out the same issues.

Invariably, it comes down to the root of the problem as Ann noted - starting in the home.

For all the poverty apologists, no one can tell me why at West Charlotte how 201 African Americans graduated and 142 did not. Same school, same teachers, same high poverty and coming from the same neighborhoods.

Again, that is the question we need to be looking at so we can target like root causes starting in kindergarten.

Anonymous said...

@1:58PM (and others) This is not a "CMess" decision. It comes from the state. Read the article and the other info. Set an example for your kids that you get the facts first, before complaining.

Anonymous said...

Anon 10:49
And continuing this incessant testing will solve any of this? You show me your stats and I will show you mine - I was merely saying that poverty matters, alot, in improving education. I have never been known for being PC. I was an adult with opinions long before that became the filter thru which we had to speak.

Just say your stats were correct, I still say let's put our money to helping the 16 million kids who go to bed hungry and frightened (black and white) and maybe they will do better in school. I know for sure that taking another test isn't going to accomplish that.

Jeff Wise said...

Third in a series...

Set aside any preconceived notions you might have about inherent biases that emanate from This American Life.

Their latest episode is on education:

http://www.thisamericanlife.org/radio-archives/episode/474/back-to-school

"Doctor Nadine Burke Harris weighs in to discuss studies that show how poverty-related stress can affect brain development, and inhibit the development of non-cognitive skills."

It does start in the home, but the education system (whether you do weighted student staffing, throw buckets of dollars or do nothing) is not configured to successfully reach these students.

Like Wiley said, nearly a generation of students have passed through the system and we're having the same discussions...

Anonymous said...

Anon: 3:08. Poverty does not equal poor education results. I've known people who were raised in poverty who graduated near the top of their high school class and then went on to get a college degree in a great field. Why would someone in "poverty" in the US go to bed hungry? Their parents get more than enough free food from the government. Perhaps, the parents aren't giving them the food. Just like they aren't stressing education enough. And you won't correct that unless you adopt the child yourself. Feel free to say how we, or you, can force parents to do what they are supposed to do.

Anonymous said...

STRIKE !!!!!!!!!!

YES, CMeS can terminate and replace us, but the KEY question is HOW?

They have at least 200 jobs now and every year that are unfilled or filled with subs. Try replacing THOUSANDS. We have lost at least $5,000 in salary and benefits these last several years. The 3% and market adjustments should feel like a slap in the face not a "Thank You". How many years were we told "Feel lucky that you have a job". This will not work anymore! Especially when we are doing twice the work with less resources and less compensation.

Volunteer without being told specifics or compensation. This is the standard CMeS way of ready fire aim.

UNITE NOW and LET THE VOICES RING

Anonymous said...

Why did 200 students at West Charlotte graduate and 142 did not? There are students at West Charlotte who come to school to learn. The majority of students are respectful & hard working. West Charlotte has a signifiant number of students who are behind grade level when entering highschool and it takes them a semester or more to graduate. Of course there are a few students who are not there to learn at all whose sole purpose is to disrupt others but they are not the majority. It is obvious that West Charlotte teachers are not giving out unearned grades as some posters suggest.

Susan said...

Ann, ask Heath Morrison to explain this, and if this had anything to do with him leaving WCSD less than three years into it:

http://washoecountyschools.net/pinnacle/?p=2852#more-2852

A couple of paragraphs:

Following two years of significant increases in graduation rates for students in the Washoe County School District, the rate leveled out in 2012. The District’s graduation rate dipped slightly to 69 percent in 2012, down from 70 percent in 2011.
In the class of 2011, 3,114 students graduated out of 4,455. In the class of 2012, 3,117 students graduated out of 4,509.


“We are making strong efforts to help every student graduate, and, as we continue on this path, we know there will be ups and downs,” said Superintendent Pedro Martinez. “Our goal is to help all students graduate from high school, ready for college and highly-skilled careers. We have also pledged to be transparent with the community and to have crucial conversations about what further efforts will be necessary to accomplish that goal. It will take years of work by all of us—the District, our community, and our state—to confront these challenges and succeed.”
____

It's a safe bet Pedro Martinez won't be up for AASA's Superintendent of the Year.

Anonymous said...

The slop in Chicago has nothing on the sludge that is CMeS. Teachers have been bullied to the point of numbness.It is a shame the union has no power and no guts to try and do anything.

Anonymous said...

"Volunteer"

I already volunteer at least 50 hours of uncompensated time at the job. I am sure this is the norm with most teachers. Nobody ever talks about the actual time worked that is not paid. This is on top of the hours and weekends grading papers.

Anonymous said...

@2:26 PM - Blow me (a kiss)

Than Nguyen said...

You can’t manage what you can’t measure. I find it interesting to see the relatively low use of data in student learning and faculty performance (well under 20% use this data proactively). That’s surprising, especially since faculty are by far the largest expense at most schools, and student learning is the primary mission of most institutions. In addition, leveraging explicit outcomes data during the semester is a critical component of student learning, and also beneficial in helping faculty track outcomes against specific program objectives.

Online Testing Software said...

The value to these computer based tests is helping teachers better tailor lesson plans to their students’ particular strengths and weaknesses. I fully support computer based testing because it will provide fair and precise evaluation of a student's competency.
Than Nguyen

Anonymous said...

I wish that everyone who comments on these blog posts would take the time to go and volunteer in a school near you. Maybe it's a high poverty school in the district or maybe it's a affluent school. It is up to you. If each of you donated a day a month, you would be amazed at all the hard work going on inside classrooms where teachers are doing more with less. Teachers who, for the most part, put in long hours of love and self sacrifice in order to shape the minds of the future generation. Long hours to complete the tasks the district asks them to complete, in addition to doing what is right for their students. Find a school, get involved. Politicians are going to continue doing what they do. You can't change that, but you can change the life of one child. One teacher. Go in, read a book, be a lunch buddy, tutor a child. Show the teachers in your life some support and help build morale.

Anonymous said...

Being a student who has just completed their first MSL today and who will be taking their second tomorrow, no one student or teacher was prepared for the new testing. The teachers were given very little information with little time to prepare causing absolute mayhem on test day. I am very disappointed with the public school system and believe a review about the manner should be in full affect.