As debate continues over the Charlotte-Mecklenburg quest to crunch teacher-quality numbers, one of the district's officials has forwarded a link to an explanatory slide show on value-added ratings.
As you may recall, those ratings are a way of analyzing student test scores to tease out how much progress can be attributed to teachers. CMS started running a preliminary formula to calculate those ratings based on 2010 results. By 2014, the district hopes to have a more sophisticated value-added formula in place and use it, along with other measures of teacher quality, to base part of teachers' pay on student results.
Susan Norwood runs a performance-pay pilot in 20 high-poverty CMS schools. She shared the link to a presentation from the Value-Added Research Center (part of the University of Wisconson-Madison education school) that she uses to help explain her program. She thought readers might find the oak-tree analogy helpful.
"Based on some of the online comments, there's still a lot of confusion about the measures and what their corresponding numbers represent," she e-mailed.
I'd be skeptical that anyone would watch a 13-minute slide show on value-added ratings. But a memo from CMS performance pay director Andy Baxter trying to explain the CMS calculation has gotten more than 2,000 views since Sunday, so evidently interest is high.
Eventually I hope CMS will be able to explain its calcualtion as clearly as the hypothetical oak-tree example. I know students are more complicated than trees, and variables such as absences and learning disabilities may be harder to quantify than sunlight and soil. But as long as the calcualtions are so convoluted that they boil down to "trust us," I suspect a lot of teachers won't.
Tuesday, March 15, 2011
Trying to simplify value-added ratings
Subscribe to:
Post Comments (Atom)
Posts
26 comments:
Ann here is my feedback on Susan Norwoods "help".
1. Seriously- she attempts to provide insight from the Oak Tree Model using 3 Variables for growth (I know it says that the model / Oak Tree example is not exactly the same but more of a theory). The fundamental flaw is that EACH TREE (child) has over 20 varibales (not 3) which makes the process and procedure for measuring growth so much more complicated. The reality is that if you sat down with Susan Norwood she would not be able to explain each variable (trust me, I did sit down with her and she can not HONESTLY explain each variable)
2. The basic premise behind this concept is that teachers will be rewarded ONCE for bringing value....to much value added does not mean more pay each year (I am not driven by pay....but if they are selling a program that is driven by "incentives to perform" why would the incentive disappear after one year?
3. THE VERY POINT MY WHOLE FAMILY AT THE HOLIDAY CONTINUES TO ASK ME... Why do you continue to work in an environment where the decision makers are driven by pay and promtion, while the teachers who actually teach are driven by success of students and the learning process rather than the the carrot of (cash and promotion). Teaching is a way of life, not a ladder /career that is climbed for personal gain.
At some point I would love to see an article that surveys 2,000 CMS teachers to see if they support this system AND why they do or do not. The very clowns on these design teams are the very clowns who are hoping to climb a ladder and who have very little what I call "true Value in any classroom"...
How far did this person have their head stuck in one of these oak trees counting little elves making cookies?
Here's my problem with this whole thing.
We all know that one of the big components of any measurement will be students who are designated FRL, low performing/high poverty school(s).
As with the oak tree anaology, we'll have the Jethro Bodine analysis of naughts and guzintas.
Jethro Bodine is a mathematician who is an assistant professor of mathematics at East Beverly Hills University. Professor Bodine is famous for having developed the ciphering method of multiplication and division and is thought to have developed Cauchy's Number. He pioneered the "times" and "guzinta" concepts the rudiments of which are explained in his seminal classic "Ciphering", a leading graduate-level textbook in number theory. Professor Bodine has been compared with Ramanujan, India's most famous folk mathematician.
Until CMS and other school districts have an honest assessment of just how many of the 135,000 CMS students are actually deemed low income and not the 74,000+ current so designated, any data using FRL numbers is totally useless.
To use a value added process to measure teacher quality is ridiculous, especially since the much bigger issues with education have nothing to do with teacher performance.
Sure, we have some bad teachers and there should be a process to reward teachers based on their performance, but in the 12 years my son has been in CMS, we've only had issues with maybe 4 in all that time. Most of his teachers were very dedicated.
As Uncle Jed would say "wellll dawgies!".....
Wiley thanks for the post....I wonder how many of those 4 bad experiences you had are still actually teaching in CMS. The last 3 years has really pared down the bad or low performing teachers in the system. The variables now are excessive class sizes ( I have 4 at 45 in a high school CORE graduation requirmeent class. When I look around my school I see very few bad teachers but more importantly I see many more OUTSTANDING teachers looking for just 1 MORE reason to jump ship. It is only a matter of time maybe tomorrow maybe next week before we start compromising the quality of educators in our classes due to top heavy educrats that could not get a child to sit down and take out a notebook let alone teach them a Unit Circle!
So what happens to the gardener who just happens to get stuck with a batch of genetic dwarf trees?
All nuts are not created equal.
My response to Susan Norwood
"Everything should be made as simple as possible and no simpler" Albert Einstein
This is what happens when you base policies on liberal BS.
Nothing about the differing abilities and motivation of students at all.
I don't see where they even control for that.
Not even in their "control factors" which are all external (race, gender, ESL, FRL).
Let's put aside obvious questions such as IQ (which they probably don't think exists), what about
about just lazy, uninterested, or spoiled students.
Are they assuming that all kids have the same "potential" (and motivation) to grow.
Seems that way.
Major fail.
How do they know that by "throwing out what they cannot control" in their measurements they aren't also throwing out what makes the biggest difference in student performance?
It's a bit like the "little moron" who looks for his lost watch under the streetlight.
He is not looking there because that's where he lost the watch, but because that's where brightest light is.
There just might be somewhere else that is a better place to look.
THIS EXPLANATION IS FROM AN EXPERT.
Here's just one of the explanations of a part of the Value Added formula. It is from the Rand Corp. report.
".....To understand this, consider the model for a student’s grade 2
and grade 3 scores. From Equation 4.5 they are given by:
y2 = m2 + T2 + a21T1 + e2 (4.6)
y3 = m3 + T3 + a32T2 + a31T1 + e3 . (4.7)
By subtracting Eq. 4.6 from Eq. 4.7, we obtain the following
equation for the gain score:
y3 − y2 =(m3 − m2 )+T3 + (a32 −1)T2 +(a31 − a21)T1 + e3 − e2 .
Thus, gains from grade 2 to grade 3 depend on the second grade
teacher through the term (a32 −1)T2 and the first grade teacher
through the term ( a31 − a21)T1. With the persistence model of
McCaffrey et al. (2003), teacher effects from grades 1 and 2 affect
both level scores at grade 3 and gains from grade 2 to grade 3. In particular,
when a 32 <1, if T 2 is positive then the second grade teacher
has a negative contribution on gains—and vice versa if T2 is negative...."
It goes on of course.
I hope this is the clear explanation some have been requesting.
Bolyn McClung
CMSdollars.com
Pineville
Needless to say, I have been pondering this issue about how we rate the effectiveness of teachers based upon the premise of the "value added" model. Clearly Wiley and Bolyn are onto something.
Bottom line, you have what educrats crave so dearly is a "politically correct" sophisticated model that is not meant to be understood by us the "unwashed". They would stipulate that it is something far beyond us and that only those with PhD ED's would understand.
So as we have seen from the outcomes for the schools in Sunday's paper, the rating system is clearly geared toward "awarding" teachers highly effective ratings if they teach in these certain schools and definitely not teach the high achieving students. Is not a teacher as challenged to keep high achieving students engaged? I say most definitley yes.
So bottom line what you have here is the foundation for the new pay system which will richly encourage teachers to come to the high poverty schools, yes a fraudulent classification as we know currently, and as expected, another weapon in the class envy/warfare barrage against the students who are really this country's best hope for the future.
As I mentioned yesterday, the rest of the world laughs at our education system watching us slide so far so fast. They find it most comedic to see us throw so much money at a group of students with so little to show for it.
What the school reform debate misses about teachers: http://www.washingtonpost.com/wp-dyn/content/article/2011/03/11/AR2011031105900.html
This value added does not take into consideration other variables. What about the adolescent who has decided that he is no longer interested in school? Young Cheech now is more interested in pot. This mind altering weed creates the altered state know as anhedonia. He just does not care anymore. What about the youth who not only come from a family that is poor, his mother is a crack head. The night before his EOC, momma decides she needs some rocks and sees the only way to get the funds is to invite over seven men to satisfy their sexual needs. Poor Tyrone didn't get no sleep that night....or on most nights when momma needs her fix. The problem with valued added being simple it that people just are not simple. There are too many variables that are clearly beyond a teachers control.
Dearest Ann: What the good citizens of Mecklenburg County need from you is a balanced approach. Start asking hard questions such as,
1. If value added method is so reliable why are parents from low performing schools such as Davidson IB not flocking to high performing schools in urban areas?
2. How can one accurately measure the quality of a human being?
3. Ask Andy Baxter how successful he has been explaining to teachers how VAM works.
4. Ask teachers if they even remotely understand how VAM works.
5. Ask Dr. Gorman why teaches are receiving evaluations and being moved from schools based solely on VAM.
6. Ask teachers how intimidated they feel by Principals who abuse and threaten people.
7. Ask teachers who plans on leaving the system when things get better.
8. Investiage massive cheating that may be going on at the "high performing schools" when scores on state tests jump from 48 percent to 95 percent in one year.
Ann, it ain't hard darling. If you ask questions will be answered. Teachers have no protections in this state from anyone at anytime. Sometimes, like it our not, you are our only chance at getting our voices heard.
To anonymous 7:40, you are an example of what a poor educational system will do for a person. You can't buy a good education and you can't be taught by someone who knows nothing. Is that what happened to you?
As a former high proverty public school teacher, I recall the
respect and affinity i received from the students I visited in their public housing homes. I am convinced that those visits had an
effect on their performance efforts. Before discarding the minority or experienced classroom teachers based on esoterically academic theory, CMS might try hiring teachers who would actually
not be afraid to visit the parents
in their homes.
One of the things that makes the biggest difference is the support, or lack thereof, that children have at home. A child who has been taught the importance of education at home, who has parents who are committed to helping them get the best education possible and are willing to do their part in making that happen, has the best chance for success. These are all things that are completely out of the control of teachers, and yet the teachers are held accountable for these things. The powers that be know this, but they also know that there is little that they can do about it. They also know that when the truth is told, they may be subjected to accusations of racial, cultural, and socio-economic prejudice. So they ignore it, blame and threaten the teachers, and throw money at the problem so that it will appear that they are actually doing something to help. It's the elephant in the room that nobody has the guts to point out, so the blame is laid at the feet of the ones who are least to blame: the teachers.
Everything has consequences. For years we were told by activist groups that if children were not learning it was because their teachers were not effective. Before that it was because their schools were underfunded or decrepit. And as 8:11 says, cultural considerations could not be added to the mix at the risk of the prejudice card being played.
So now, because there was such an outcry against teachers, we have to have complicated expensive programs to evaluate teachers and low teacher moral; because of the claims of extreme inequity we have schools built or renovated that are underpopulated; and we have a huge difference in how resources are allocated to schools. We also have angry parents and taxpayers because so many were (and still are) willing to play this as an us against them thing.
Value added is clearly the best way to evaluate teachers. The difficulty is figuring out what you measure. There aren't simple clear measures such as temperature, rain and soil to apply to students. One option, for those students who undergo standardized testing, is to measure the PERCENTAGE gain by the students in a teacher's class. This accounts for the different attainment levels of the students and different ambitions of the students. It measures what the teacher does with the kids in the class.
Bolyn, thanks for the clarity (smirk).
7:12 p.m., thanks for the link to the Joel Klein piece. It's great at putting some of the key issues into common-sense English. I'm intrigued by this suggestion: "We should look at how much student progress each teacher gets by comparing teachers with similar challenges - e.g., those who start mainly with low-performing kids would be compared only to one another."
7:36, I sort of agree with you. It's not hard to ask those questions. But it's a good bit tougher to answer them in articles that provide the nuance and detail that CMS employees want while being accessible to general readers.
The teacher voice question is huge. This blog provides one format for teachers to raise issues, questions and comments anonymously, and I do know that the superintendent and his top staff often read it.
More important, a small but growing number of teachers are stepping up to speak on the record. That's not easy, for them or for me. There's generally quite a bit of talk and trust-building that goes into that. It's a big thing for someone who would rather focus on his or her students to go public on controversial issues, exposing themselves not only to their bosses but to public critique. I salute the teachers who do it, even as I understand those who remain anonymous.
The current, largely unjustified villification of teachers has nothing to do with activists' concerns about teacher quality in high poverty schools. It has to do with conservative figures at universities and large, wealthy foundations, whose "business" approach to education has singled out teachers as the easiest and supposedly cheapest target for their efforts to upend public education.
Curious, why would they want to upend public education? Second, conservative figures at universities? Oxymoron!
Sen. John East (NC) ECU
Died June 29, 1986? How is he relevant to today's discussion?
Your request for conservatives in the academics specified no particular date or location. I believe you'd find a lot more conservatives in higher education than you think. You might start with engineering, economics, and chemistry departments. Based on your observation Gov. Jim Martin or NC History was never relevant. If Billy Mayes was still alive he'd be pitching Oxymoron as the Value Added Accountability model available only from CMS. Directions on container to be determined in three years after possible vote by end users previously fired.
How is it that the guy who is in charge of the US Dept. of Education is promoting the concept of PFP through "innovative reform in education" in the Race to the Top (we know it should be the bottom)has any ground to stand on, or better yet, actually provide Pete with support when his own PFP system in Chicago failed miserably.
http://voices.washingtonpost.com/answer-sheet/research/analysis-of-chicagos-teacher-p
10:45 PM My response was from Pam's comment about the upending of public education. While there are a few conservatives in universities, they do not have seats or offices, or department heads that deal with influence over policies about public education. They are as you pointed out, in other disciplines.
I do agree with Pam about the villification of teachers. The black community has been doing it for years when the case was many teachers comong out of college were assigned to mostly black schools. If you interviewed one, you would have seen their energy , their desire, their willingness to sacrifice to reach these kids but their light was quickly put out by disprespectful parents and kids. The administration would not back them up for discipline issues.
But worse, Dr. Gorman has villified the teachers after shortly coming here. If he believes the kids' upbringing and environment is any reason for the child to fail, he has been reported as having said so. He has laid all the ills of that community at the teachers' feet.
Yes also, there are issues with "celebrity benefactors" pushing their agenda into this field. But educrats are more thanhappy to do as suggested when a lot of money comes their work way. Kinda like a pros**tute.
Rest,
Total agreement on your post. After 25+ years, I'm tired of the depiction of many fine educators as bottom feeders. Some of the most liberal of educators have begun to move to the center on "entitlement" issues. I'm beginning to sense that even Pam has reached a point that common sense is the first need in returning the power of education back to the community. The traveling educrat roadshow assembled from Florida, California, and Arkansas, while financed by billionaire financier grants, have ravaged local control from our village BOE and political idiots. When our super and his minions in are in the Broad and Gates cesspool you can depend on where his loyalties will rest.
Post a Comment