All of us who work with numbers know how easy it is to make a mistake.
Charlotte-Mecklenburg Schools made one last week when it posted inaccurate (and still unexplained) results for a new on-time graduation-track calculation, then was slow to recognize the error. Late Friday, interim Superintendent Hugh Hattabaugh pulled the school progress reports offline, telling the school board in a memo that "several issues of data accuracy have come to light." The reports will be reposted Feb. 3 "after the data has been fully audited," he said.
The incident poses some serious questions for a district that prides itself on being data-driven on everything from education strategies to accounting for public money.
Some numbers lend themselves to a common-sense reality check. It's like stepping on a scale: If it's five pounds off, you might believe it. If it's 50 pounds off you know the scale is broken. For anyone familiar with high schools, numbers showing fewer than 2 percent of all students have ever flunked a grade are a clear signal that the scale is busted.
Hattabaugh and his officials are facing questions about why they didn't catch the problem. But I'm wondering about principals who got their school reports almost two weeks before the error went public. At many of those schools, the bogus numbers were wildly out of sync with reality. Did anyone say "Hey, this can't be right"? If not, are principals so overwhelmed by central-office data that they've stopped caring whether numbers are accurate and meaningful? If so, were they unable or unwilling to let their superiors know there was a mistake?
I asked Hattabaugh about that during the weekend school board retreat. "They were probably just thrilled that it was a good number," he said, smiling.
I don't know if he's right. But if his school leaders are happy with glowing but false data, the problem goes deeper than a central-office flub.
Monday, January 23, 2012
Questions from a data flub
Subscribe to:
Post Comments (Atom)
Posts
41 comments:
The problem is MUCH deeper as you are starting to see.
Agreed. If principals are willing to believe false lies about their progress, they cannot be trusted.
Didn't one principal even say they were on target to meet one of those ridiculously warped goals?
However, I must say that I have seen this just about anywhere performance "metrics" are used.
In addition, there are usually people who will deliberately try to game the system to their favor, treating it like a game of poker in which the player who can lie with the straightest face wins.
Statistical quality control had its start in manufacturing where you are dealing with physical items and variations in their physical measurements.
That's probably where it should stay.
When it comes to people, many people will just try to manipulate the numbers rather than the reality behind them.
It's not like there is an actual body count for every measure as there is for graduation rates.
And few people can actually tell the difference because most of these measures are NOT easily quantified.
There is also the problem that most people are not trained in statistics so these formulae look very scary and "official", and seem beyond questioning for normal human beings.
Ann,
C'mon, it's their culture.
CMS already sloughs off the School Lunch Program data because they basically get a pass from the USDA.
CMS really doesn't want to deal with it now do they?
Same with Bright Beginnings. It's a cash generator. They don't want negativity with that program either, yet can't produce an ounce of data to support it.
A new report from the Chicago Tribune Jan 13, 2012:
...Officials also blame the U.S. Department of Agriculture, which oversees the lunch program, and Congress for severely restricting how much school districts can investigate.
When CPS audited a sample of lunch program applications nearest the income limits last year, it found that more than 70 percent included false or unverifiable claims, records show. But federal law allows CPS to verify fewer than 1 percent of the roughly 120,000 paper applications it receives each year from families who provide information that qualifies their children for free or reduced-price meals.
Most criticism of the federal lunch program in recent years has centered on health and nutrition, but documented abuses in school districts in North Carolina, Ohio, New Jersey and elsewhere have prompted questions about accountability.
http://articles.chicagotribune.com/2012-01-13/news/ct-met-cps-lunch-fraud-20120113_1_free-lunches-reduced-price-lunches-lunch-applications
Note that 70% discrepancy and also North Carolina mentioned in the article.
But eh....why should we care if the data and those driving the data are correct?
CD Spangler will give them some more money and make everything all better.
Baixiong wrote: "Statistical quality control had its start in manufacturing where you are dealing with physical items and variations in their physical measurements.
That's probably where it should stay."
Performance still has to be evaluated somehow... or would you prefer to just "trust them"? I'm always amused when schools are failing and we blame the grading system instead of addressing the reasons they can't meet the standard!
Schools were already failing before the standardized tests were introduced... in fact, it was that recognition of failure that brought the reforms of "No Child Left Behind".
Too many blame everything on NCLB but have no suggestions on a better way to meet the needed performance goals. It's like blaming radar guns for speeding!
HOW MUCH CAN A PRINCIPAL DO IN A DAY?
Subtitle: You really expect a person to read that!
While the thought is that schools can’t be operated as a business, I believe that the people within then do work on the same principles.
One rule that I have found useful is that any task requires 5 minutes minimum. Let’s look at the Principal.
The job requires about 10hrs a day plus lunch. Based on a 5 day week:
Tasks=5days x (10hrs x 60min)/5min= 600 tasks/wk.
This either forces the Principal to work longer or make smart choices.
So to really screw-up a good movie line: “What we’ve got here is not a failure to communicate but having something valuable to say.”
Bolyn McClung
Pineville
To quote Ann's article:
Did anyone say "Hey, this can't be right"? If not, are principals so overwhelmed by central-office data that they've stopped caring whether numbers are accurate and meaningful?
Yes, there's so much "data" that comes from the Central Office that it fails to have meaning any longer. Parents, teachers, and yes - even administrators - can no longer care about the statistics that dribble out from central administration, there's just too much of it.
Further quoting: If so, were they unable or unwilling to let their superiors know there was a mistake?
Ann, you showed us last week that it doesn't matter how obvious the mistake, administrators with CMS defend their lousy statistics before turning the microscope inward and asking if they make sense. What principal in their right mind is going to tell their superior that the data doesn't make sense, and risk being on the wrong side of Hugh's ire?
Inaccurate data from CMS has been long known. Many of us, even outside of the FRL fiasco, have seen issues with numbers and questioned the source only to get stonewalled. At best, it looks like half truths they use to tell the stories. The latest fiasco is the allowance to have those that fail the EOG's to take them again and use the later, probably better score. Thus the achievement gap is "played with". If you are going to allow the low end to take the test over for a better grade, why not allow the high end to take the test over. Afraid to see the gap widen?
LIFT has associated itself with an untrustworthy institution. Anyone with sense will not trust any student performance numbers from here on.
However, the hammer is about to drop on NC students and teachers. When Common Core is introduced next year, our teachers and students will find themselves 1 to 1/2 years behind the standard.
I feel sorry for the teachers. CMS is and allows other groups to pound on the teachers for failing most of these students. I think you will not find a larger percentage group of more dedicated people truly desiring to help any and all students. I believe most of the problems you will find will be becuase adminstrators and downtown have not supported the teachers when they have classroom issues. The students then have come to run the schools.
The problem goes much deeper than simple manipulation of data. The scariest aspect of the new focus on graduation rate is how far principals will go to increase this number. The falsified numbers can be met by ambitious principals by placing pressure on teachers to pass students who don't meet the standard for passing a class. As a current teacher in a CMS high school I can attest that if you want to fail a senior you had better have massive amounts of back up and in many cases that still isn't enough. The culture of CMS has changed in the last few years from stressing EOC composites to improving graduation rates. While in the past principals would remove delinquent students from EOC courses to improve scores, now principals are forcing teachers to lower their standards to provide a grade of "D" for students who are on track or near track for graduation.
Ann, Ask your local high school teachers and I'm sure you will find cases of plagiarism and excessive truancy being ignored to increase graduation rates. What is even more troubling is that grades are being changed without the teacher’s knowledge.
The real problem is not the incompetence shown by the graduation rate projections but instead by the length some will go to meet these projections.
9:07, this may sound naive and even contradictory, but I think Chris Cobitz and Scott Muri would have loved for someone to catch that error before it went public. I've been flagging CMS mistakes for 10 years now, and previous officials have reacted with defensiveness and anger. Cobitz and Muri kept getting back to me and seemed to be genuinely trying to sort it out ... but obviously didn't do so quickly or effectively. However, I also realize a reporter may have better access (and less to lose) than a CMS employee.
9:32, the LIFT folks said they'll use independent third parties to track their results, But those parties will almost certainly have to get the basic data from CMS.
I'm off Monday, back in Tuesday....
See, we can't even believe Ann. Here she is working on her day off...
...hehe
Will those LIFT schools be removed from the schoolmates project? Or can they adopt a suburban school using their new windfall? Our school supports a schoolmates school and our teachers are now begging for school supplies because our PTA provides more money to that school! This is a program that needs to be stopped.
Maybe those LIFT schools can donate some furniture to AKHS!
Ann, I had looked briefly at schoolmates some time ago and it appeared the receiving schools never totaled up that benefit, manhours tutoring, donated clothing, supplies, equipment, etc. and the sending schools never got any credit for the generousity nor was there any public affirmation of this sacrifice to these schools.
Ann, also, I wondered if we could find out what the value of the material was Classroom Central provided?
Sure would look like a good story to dig into all the extra help and resources these schols had been getting over time. I know at one of these breakfast meetings in West Charlotte, that a lot of those people still think those schools get shorted on resources and money spent per pupil.
But like Tate and his ilk, you could never prove they got more to satisfy them.
10:39 I agree with you that far too little attention has been given to the generosity of suburban schools and of the entire community in general to many of our urban schools (and not just through Schoolmates). I don't believe that any of this volunteer time, money, supplies, etc. is done for "extra credit" or for publicity for the to the donors. It's done in response to the perceived need.
Unfortunately, because groups are not seeking publicity and because for so long common wisdom in Charlotte was that the suburbs got it all and that suburban residents were selfish too many people still believe that suburban schools receive favored treatment over urban schools. And some groups take advantage of that belief to foment discontent.
It would be very interesting if somehow all of the volunteer hours, money, supplies, etc. could be totaled--not to begrudge what has been given but to illustrate that there are an awful lot of caring people throughout our community. It would also be wonderful if someone would stand up at the breakfast club and say, "You know folks, let's not forget that our children do receive a lot of help from throughout the county."
Sharon Starks
Project LIFT document notes that they are concerned about the high FRL numbers in those "westside" schools that are doing poorly.
Maybe they could work toward reducing the FRL precentage by cutting down the fraud.
After all, if a parent is willing to lie to get a free lunch, how can they be trusted to help in the proper education of their children?
The whole process of quality improvement starts with integrity.
If you don't have it, don't even waste your time (or money).
I dare them to do this.
What good is the data if it is inaccurate or skewed to reflect desired results opposed to actual results?
Here's a very in-depth look at philanthropy in education...
It's worth reading in my opinion.
Be wary of who you get into bed with.
One note I read in the article was that for every 1 negative news article on philanthropy in education, there are 13 positive ones. It goes on to show how much the media is not willing to talk negatively about donors and how much these donors give to certain media outlets..
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Winter 2011
Got Dough? How Billionaires Rule Our Schools
By Joanne Barkan
THE COST of K–12 public schooling in the United States comes to well over $500 billion per year. So, how much influence could anyone in the private sector exert by controlling just a few billion dollars of that immense sum? Decisive influence, it turns out. A few billion dollars in private foundation money, strategically invested every year for a decade, has sufficed to define the national debate on education; sustain a crusade for a set of mostly ill-conceived reforms; and determine public policy at the local, state, and national levels. In the domain of venture philanthropy—where donors decide what social transformation they want to engineer and then design and fund projects to implement their vision—investing in education yields great bang for the buck.
http://dissentmagazine.org/article/?article=3781
John (8:56) - no that's not a bible quote, it's a post...
No, we have already seen with CMS that you cannot "trust" the data or the methods they use, so I don't see any difference between that and someone's "gut" opinion of a teacher.
In fact, I've seen studies out there that show that a principal's opinion is just about as useful as these statistical measures they throw around.
What good is a standardized test if it measures the wrong thing?
Standards aren't always sacred in and of themselves.
I'm not blaming the radar guns.
Or the use of radar guns provided those guns are proven accurate.
I'm saying these particular radar guns are broken and we need something else.
I also think a good principal should be able to tell what a good teacher is like.
The failure of them to do so is probably a part of our management culture as glorified in Dilbert cartoons.
Our system is broken and failing miserably.
In education, we are where our auto industry was in the 1970's.
I think we should broaden our scope a bit and look internationally at what the top schools in the world (such as Finland) are doing to evaluate teachers.
I doubt that it's the kind of silliness we do here.
Oh, FWIW, for those who don't know.
One of the pioneers of statistical quality control, Dr. Deming, was practically unknown in the US (where he lived), but was highly revered in Japan for years.
Japanese adopted a lot of his techniques in their auto manufacturing and came to dominate the world in quality.
Then, in order to catch up, we looked at what the Japanese were doing to reclaim some of that lost ground.
I think we need to do the same thing in education.
We need to stop comparing ourselves to ourselves and gazing at our navels worrying about "performance gaps" between one group and the other.
Those are bad enough for sure.
But some of the things I've been reading about our international standings are scary.
This is not a battle between GM, Ford, Chrysler and American Motors for world domination any longer.
Even our white suburban kids in some of the "best" schools are barely in the running internationally.
That's one reason I hate to see the best slighted in favor of improving our worst.
The global competition is not among the worst students in the world.
It's between the best.
The "LIFT" folks will generate their own data you know this. And it will be auditied by a minority firm of their choice. The qoute was from the LIFT folks " the money from the project will stay in the minority community" Tuesday Morning breakfast in September 2011by their leader. CMS should not get in bed with such a racists group with such high hatred. Muri is in way over his head on this one as many others. While they sleep nobody knows what the kids are actually learning.
Our teachers are too afraid to ask the principal or PTA for supplies. I no longer support the PTA at our school. They operate just like CMS! Ha Ha. I donate directly to my child's teacher. I refuse to give anything to classmates. Another joke.
OHHH wait the new superintendant that is lining up for the CMS job will fix it all no worries. Its very possible nobody has applied that is worthy of buying in to the CMS mentality (or lack their of).
If LIFT is audited by a "minority" firm, I hope they're Asian.
It certainly is looking like they are trying to keep all the money and probably forcing more taxpayer into this area despite other insistances. As you look into the plan document, you see longer school days proposed. Saturday school. Summer school programs.
Does CMS have to absorb higher utility costs? Does CMS have to absorb more busing costs? Do teachers get paid extra for longer days, Saturdays, and summer schools by LIFT or by CMS.
LIFT has specified 3 people for CMS hiring but LIFT reimbursement and you know the sole purpose of that is to latch onto retirement contributions and medical, dental and other health benefit plans on the taxpayers' back.
Project LIFT/Philanthropists hiring lobbyists....
That's enough to raise a boatload of red flags right there.
I don't care what all this is about, I am just happy things are going so well, and our Children are graduating and learning at CMS, as these numbers prove.
Let's see, how much money did they spend on that Data Dashboard???
Ann, the only reason by Muri and Cobitz tried to get back with you is to save what little credibility they have in this area. Principals at your schools probably saw that the numbers are high, but seeing that they also receive fat bonuses for these numbers they probably elected to stay silent. The principals currently running some schools in CMS are climbers and are unconcerned what the numbers look like as long as they favor the school.
Let's be honest, what these men did was incompetent as the least and shows a true lack of how to do their job at the most. I suppose if someone "ANN" or the "OBERVER" where to dig a little deeper what you might uncover would be pretty devastating to CMS and humiliate its leadership. I suppose we could just let this go by and blame the teachers like we tend to do in this district.
Anon 8:00....
That's why CMS will never do away with Bright Beginnings, even though Gorman said data wasn't there to support or keep it and wanted it gutted. There has been NO data presented since 2000 and that data is so-so at best.
Of course they wouldn't gut or eliminate it. To do so would be admitting failure and be a negative towards one of their own, Eric Smith. Plus, the More at Four program the state replicated off of Bright Beginnings would be suspect as well.
Throw in our clueless Board of Education and you get quite a cocktail of inept people who don't have the guts to pull the trigger and make the hard decisions, only the popular one.
One of CMS goals is to raise the graduation rate. CMS is known to take poetic liberty with numbers on a routine basis. This type of quality control, managerial oversight error caught by other than the 1.2 billion dollar education system we call CMS has convinced me those who regularly opine against CMS leadership are right. Data driven or driven to create self-serving data? I do not think we should assume the principals did not catch this. CMS is so creative with numbers they may have been waiting to hear the company line on how to explain them. There should not be a $40,000 a year patsy to take the blame either. The higher the pay the more open to alternate explanations. The Board needs to fire somebody or the County needs to investigate! Accountability?
I am aware of examples where specific metrics developed for scorecards were presented in pre-publication review using slightly different truncated language than when the scorecard metrics were developed. To catch a missing word in a column heading when your mission is to review the numbers in the column is tricky. A change or omission of a word or two in the metric column heading to save space can vastly change what the numeric result appears to state. People get pretty specific with metrics, not poetic but precise, and a one or two word change can completely change the meaning of the result presented. You do not need a UIUC degree to grasp this. Not making excuses, just sharing a simple thought.
I would submit something I believe a lot of folks are beating around the bush to not say. Clearly a culture exists in CMS, and probably most of the "urban" school systems that goals like closing the achievement gap, increasing graduation rate, decreasing dropout, etc. have become the altars of worship and policies that help make names for educrats who once failed in a school system, can earn a good living consulting around the circuit because of the lack of and the willingness to cherry coat past failures. Most school boards are not run by analytical, managerial experienced people who are willing to step out against community organizers, politicans cowtowing to community organizers, and left leaning news rags' editorial staffs.
The children of Mecklenburg County would be better served by turning all of CMS into charter schools if this way of thinking/culture is not stopped.
Here's a tool for comparing schools globally:
The Global Report Card:
http://globalreportcard.org/map.html#bottom-results
CMS RANKS:
State - 49% Math, 47% Reading
Nation - 53% Math, 45% Reading
International - 40% Math, 42% Reading
Canada - 31% Math, 32% Reading
Finland - 23% Math, 23% Reading
Singapore - 23% Math, 32% Reading
Clearly, our competition is not among ourselves any longer.
When Canada, Finland, and Singapore are that much better than we are, it's time for a closer look at what is wrong with our entire education system.
Not just our worst performing schools.
I would agree with much of what 9:27 has to say. Former superintendents, school board chair, etc. become paid consultants. From Charlotte,for instance, former CMS board chair Arthur Griffin--http://www.mheducation.com/uar/index.shtml (please note his involvement with the hated Broad Institute);Dr. James Pughsley, former superintendent--http://www.americanprogress.org/projects/education/about.html (scroll down to read his bio--didn't know he had done so much for CMS); Eric Smith, former superintendent--http://www.tampabay.com/blogs/gradebook/content/weekend-interview-former-florida-education-commissioner-eric-j-smith/. I'm not saying that these folks failed here, but all of them certainly had their detractors. But now they all make a nice living by pushing the various educational theories du jour . Arthur has even offered his services to the new school board. And yes, they all are into "urban education" and how to solve the achievement gap.
Cobitz...gone!
10:12, I think everyone who has a little history with CMS realizes that when Wilhemia and Arthur steps up to "assist" this BOE, you know where the next 4 years are going for this community.
12:10--Agreed!
Post a Comment