Monday, December 16, 2013

Academic growth formula: Not secret, just complex

I recently referred to the EVAAS formulas used to calculate North Carolina's school growth and teacher effectiveness ratings as secret. Turns out I'm behind the times.

The Cary-based software company SAS,  which created the formulas and markets them across the country,  initially kept the specifics a proprietary secret.  That's probably why Charlotte-Mecklenburg Schools officials have voiced wariness about having teachers' careers and school reputations depend on a formula they can't review.

It's because of such concerns that SAS released the formulas,  which have been tested by groups such as RAND Corp. and UNC Chapel Hill,  says Jennifer Preston of the N.C. Department of Public Instruction.

But that doesn't mean most educators, citizens and journalists can run the numbers themselves. I'm comfortable with Excel spreadsheets, education data and basic calculations.  But when I see lines like  "KTb + MTu is BLUP of KT + MT provided KT is estimable,"  I'm out.

The calculations turn each student's performance on prior exams into a prediction about how they'll do on the next ones. The actual score is compared with the projection.  Teachers'  "value-added" ratings compare their students' progress to that of other teachers across the state. Those ratings form part of the state's teacher evaluation;  persistent low ratings jeopardize a teacher's job,  while strong ratings may someday lead to performance pay.

Schools are labeled as meeting, exceeding or falling short of growth targets based on how their students did compared with projections.  For many,  2013 growth ratings provided a counterpoint to the bleak picture painted by low proficiency rates on new exams.  In 2014, proficiency and growth will combine to create a state-issued letter grade for all public schools.  For charter schools,  growth ratings are a key factor in determining whether a low-scoring school stays open.

There are,  of course,  people who say no formula can turn student test scores into meaningful measures of school quality and teacher effectiveness.  But given that our state legislators and many national policymakers believe otherwise,  it's important to be able to check the validity of those ratings.

Anyone who works with data,  even on a much simpler scale,  knows how easy it is to make a mistake -- and for that mistake to be compounded as you run it through further calculations.  I've caught plenty of errors  (my own and those of institutions I cover)  by seeing that numbers don't jibe with what I know of reality.

It worries me that such crucial numbers aren't subject to an obvious  "smell test."  But Preston said the state is building in backstops.  For starters,  teachers get a chance to review the roster of students being used in their ratings,  to make sure they're getting credit or blame for the right kids.  Schools and districts review the raw data before it's sent to SAS.  And the state has been reviewing dozens of questions that came in after the release of ratings,  Preston said.

Preston,  a former high school teacher,  says the real value of EVAAS numbers comes from teachers who use student data to craft teaching strategies and principals who use them to make good use of their faculty.  She said her numbers showed she was helping low-scoring students make big gains,  while the students who came in strong stayed flat.  Her principal assigned her to a low-performing class the next year,  while a teacher who got better gains from higher-level students took that group.  "We were both teaching to our strong points,"  she said.

22 comments:

Anonymous said...

1. The private school that SAS "founded" and opened...Cary Academy, in Cary, NC...they don't do any of this testing nor do they measure student growth or teacher effectiveness... intereseting, isn't it? SAS wants all of the rest of the State of NC to use this "excellent tool for helping students learn through use of data", but they don't use it in the school they founded? Hmmmmmm.....

2. The formula was developed by an agricultural statistician. A guy who for many, many years worked on how to get greater yields out of cattle... last time I checked...kids don't have udders...though on occasion, depending on their grade level, they may moo. Does no one else find it insulting and weird that the formula equivalent for cow productivity is being used for thinking, reasoning, feeling, developing CHILDREN?

3. Given that last year was the FIRST year with the new tests...how is it even POSSIBLE to have an accurate "prediction"? And yet... that bleak picture of growth was a result of predictions...based on what exactly? How DID the cow people at SAS come up with what a kid would make on an untried test that they had never taken a version of previously? Oh, that's right...they used the info from the OLD tests, where students scored "higher" before the State BOE decided to "up the bar" for what it meant to pass. They essentially set the kids up for not being able to do well or "grow" from the very beginning because NUMBERS can't adjust...that's what PEOPLE do.

The "growth" model can't account for any number of individual needs...homelessness, parent dying 3 weeks before taking the test, boyfriend breaking up with a teen girl hours before the test, low income, hospitalization, wanting to take a nap instead of filling in bubbles...poorly written tests, the fact that the HS kids were, in fact, DOUBLE EXAMED in all of their classes...they were plain sick of taking tests by the time the State tests rolled around they knew those tests didn't count for their grades...period. How can the idea of "growth" this year be deemed anything OTHER than meaningless? How can SAS with their EVAAS system preach and lobby this is an excellent tool, but not use it in their own school? Exactly.

Anonymous said...

If parents knew what was really going on at CMS schools, especially the high schools, they would be shocked. CMS is good at keeping secrets.

Anonymous said...

This is crap. Merely designed to punish teachers who are seen as " the enemy". And, designed to reward corporate buddies who fund campaigns. NO formula can make fair a system that penalizes teachers for variables which they cannot control. It is driving away GOOD teachers. It is insane! And - taxpayers are paying MILLIONS for it.

Anonymous said...

School systems across the country are wasting millions evaluating teachers on formulas that nobody in the education system can understand, have access to or review for accuracy thanks to Arne Duncan's Race to the Top bribery. Once the grants expire, most states will probably toss the formulas out the window due to the complexity, expense and potential lawsuits involved in using them. Value added formulas served as a one time corporate welfare give away by the Obama Administration. Other than that, they are completely useless. If there really was a magical formula that could accurately predict student growth on standardized tests (which are also kept secret and contain flaws) then the same formula should be used nationally instead of every state having a different contractor and different formula. You can read more about VAM being implemented in Miami-Dade here http://kafkateach.wordpress.com/2013/12/02/vam-speaks/.

Anonymous said...

According to the logic above that SAS can't be a vendor to public schools when they don't use their system in their own academy.

How about Anthony Foxx and Patrick Cannon...they refused to send their kids to public school, but certainly had plenty to say about the public school system. Mayor Cannon's kids go to Charlotte Christian. Should be disqualify him from making any comments about our public school system?

Anonymous said...

@8:39 What does a former mayor or a current mayor sending their kids to private school or commenting on CMS (county schools) have to do with STATE testing - which is performed and designed by SAS - which neither of these leaders control or have commented on...Who cares what they say or have said. They don't make policy for state testing, nevermind CMS.

Anonymous said...

@7:19
"If parents knew what was really going on at CMS schools, especially the high schools, they would be shocked. CMS is good at keeping secrets."
CMS doesn't control over state testing or SAS. What does this have to do with CMS or CMS keeping secrets? Nothing.

Anonymous said...

@6:14

Statement: "The formula was developed by an agricultural statistician. A guy who for many, many years worked on how to get greater yields out of cattle..."

Conclusion: "Does no one else find it insulting and weird that the formula equivalent for cow productivity is being used for thinking, reasoning, feeling, developing CHILDREN?"

*** It a logical fallacy to assume a statistician with any particular experience cannot apply their knowledge to any other industry or to assume that expertise in one industry means they do not have experience in another. The person's primary skill is statistical analysis and modeling ...not cows.
But most importantly is that you claimed they used the "same FORMULA EQUIVALENT for cow productivity is being used for thinking, reasoning, feeling, developing CHILDREN."
Not only have you not established why the formula equivalent is inaccurate in either case, but you haven't provided a source to verify they are in fact the same formula equivalent.

Wiley Coyote said...

Maybe this will help understand the formula.

KTb + το MTu είναι BLUP της KT + MT που προβλέπονται KT είναι πρόσωπο μιας σειράς πολύ συμπαθών μελών

Greek is much easier, but it still means teachers are getting screwed.

Anonymous said...

Another bunch of absurd education bs. They would have us believe that teachers are above evaluation. All teachers are wonderful and smart and care about nothing but the little darlings. Truth is teachers spend 4 years in one of the least demanding college courses of study, pass a trivial test and feel they should be allowed to spend the rest of their working lives feeding at the public slop bucket in ever increasing amounts, free of any kind of responsibility or evaluation.

Some fool thinks that teachers should not be responsible if students "want to take a nap instead of filling in bubbles". Isn't it the responsibility of teachers to at least wake kids up? It should be a responsibility of teachers to make students want to excel, but I guess it is not.

The question becomes then what exactly is the teacher's responsibility? Seemingly not to learn. I guess it is to make sure they don't wear a tee shirt with a flag on it or other pc police functions.

Anonymous said...

Why is this hard?? Test the students in the beginning of the year. Use that data to focus on where your students struggle. What they already know and don’t know. Test in the middle of the year to see what they have mastered, who is falling behind and what needs re-taught. Give an assessment at the end of the year and compare the score to the first test. Comparing a project score from previous years is ridicules…. What if a student’s parent died, or lost a job and the family is struggling. What if a student started using drugs or became disinterested in school for whatever reason. Should we hold teachers personally reasonable for things out of their control? This is what happens when you let midlevel bureaucrats like Purdue, Tillis and Pat in charge. I don’t care what party you like. Incompetence is incompetence. My buddy’s wife's a teacher. They do not mind being evaluated. Most teachers I know are good God fearing Americans. This is not the north. No unions here, never have been. Everybody gets evaluated, but it should be things they directly control... It should make sense...

Anonymous said...

Cyber culture is turning us into a society of know-nothings.

Parents, please question CMS about the push for digital devices (BYOT)in the classrooms K-12.

Anonymous said...

I am a conservative and have lived and traveled all over this great land. I have seen greedy teacher unions and bolted teacher pay. We do not have that problem in NC. I have never seen a state government so disrespectful to its own teachers. I have told my children that they are not aloud to become teachers. 4-5 years of school, 30 thousand dollars in debt and after 10 years years not even making 40 thousand dollars a year. Every 4 years you have some new bureaucrat chaining your world upside down. your a constant political football between the feds and state government. Parents are both working and to busy to kick in. Your constantly between competing interest. Crazy rating system that only a bureaucrat with a tax dollar leaching company could create. All the responsibility in the world with no authority. Nope, mamma don't let your babies grow up to teach in NC. It' sad, I have been really happy with most my childrens teachers while in NC.

Anonymous said...

How about this 1:47 . . .I know it is NOT my resposibility to make sure my elementary students read at home and do their math homework. I know it is NOT my resposibiltiy to make sure your child has had breakfast everyday - wait now it is . . .I know it is NOT my responsiblity to make sure I have had a parent conference with you even when you ignore 4 notes and my repeated phone calls . .. wait it is - 100% Parent contact is expected. I know it is NOT my resposiblity to clean out your child's back pack of graded papers, unsigned progress reports, and school notices.
Teachers have more reposnsiblity than you can ever dream of having. Every time there is a school tragedy, who puts their life on the line for the kids - a teacher! Every day I am RESPONSIBLE for my students whether or not their own parent/gaurdian is or not.
How dare you say all I want to do is "feed at some public slop bucket". First of all I don't think I could afford to feed all day long at some public bucket on my $500.00 raise. You are ignorant and probably couldn't last 1 hour in a teacher's shoes.
We are leaving in droves! Nine mid-year resignations as of today at my school and counting . . . I am working my exit plan for June. I would leave now but I feel reponsible for my students. After all, who would feed them breakfast if I didn't show up?

Anonymous said...

A relationship between standardized testing and cow productivity?

Priceless.

Alicia

Anonymous said...

Kudos to the Cow Productivity specialists who are out standing in the field.

Said a little prayer before taking the Praxis 2 today. God certainly has sense of humor.

Alicia

Anonymous said...

Life Regrets:

Telling Peter Smolowitz the truth when he asked me if CMS' mascot was really a cow.

If only I could turn back time...

Alicia

Wiley Coyote said...

Alicia,

I did some research and found out who is behind the cow computations.

It's a company called I.M. Holstein LLC.

https://www.youtube.com/watch?feature=player_embedded&v=dMqDiLcncCA

Notice how he fills in all the circles.

Anonymous said...

Wiley,
Lol
Alicia

Anonymous said...

Starting my morning with cows in an elevator was a nice opening to the day.

Anonymous said...

1. I said "formula equivalent"...not same formula is used.
2. It's not a logical fallacy. The idea is that children and cows are two different things...Children's learning productivty and agricultural yields are two different things.
3. Teachers are not ALLOWED to wake up kids during these state tests...FYI.

Anonymous said...

Students wouldn't fall sleep during exams if school didn't start at 7:15am. Just saying.