Monday, February 6, 2012

Dropout success: Hard to gauge

The headline on the national Communities in Schools press release last February was eye-catching: "Evaluation Ranks Communities In Schools as Most Effective Dropout Prevention Organization in America."  The ranking,  it said,  was based on  "the largest and most comprehensive evaluation of dropout prevention programs ever completed."

The attached 34-page report was much less dramatic. There was no independent entity doing long-term tracking of participants in various dropout-prevention programs.  Instead, CIS had commissioned ICF International to look at how well the schools participating in the program are carrying out its work and suggest ways to improve the system.  The group also pulled in some effectiveness data,  including randomized controlled trials in three states  (only two involved high schools)  and effects on dropout and graduation rates from four other groups listed on the federal What Works Clearinghouse.

For those of you who want to delve in and draw your own conclusions,  a description of the data sources in on pages 4-5 of the report.  The comparison with four other programs is on pages 12-13,  and data on  differences between CIS schools and comparison schools not using the program is on pages 22-23.

My conclusion:  There was nothing clear enough to justify a  "best in the nation"  headline,  or even to grab the attention of general readers.  I didn't loop back to this until Charlotte-Mecklenburg Schools repeated the claim as part of its 2012-13 budget pitch.  At that point,  I was surprised to discover that the What Works Clearinghouse does do effectiveness ratings on dropout prevention programs, but Communities in Schools isn't among them.

Talking to folks in the U.S. Department of Education,  which runs the clearinghouse,  led me to Mark Dynarski,  a former clearinghouse director who developed the dropout-prevention effectiveness tracking.  Now a consultant,  he was quick to tell me he has "a financial relationship"  with ICF and with the philanthropy that paid for the CIS study.  But he wasn't complimentary about what he saw:  Small numbers,  "cherry-picking" of findings and a marketing spin on the conclusion.  He said you might expect a group that started in 1974 to have extensive long-term studies:  "They sure don't study themselves a lot, do they?"

Not that it's unusual to find a dropout prevention group with sparse data.  Dynarski agrees with the journal  article by John Tyler and Magnus Loftstrum saying such data is "woefully inadequate,"  even though boosting graduation rates is a top priority for public schools across the country.  "The Dropout Prevention
Center/Network lists hundreds of dropout prevention programs in its online database of  'model programs.'  Only relatively few of these programs,  however,  have been rigorously evaluated for effectiveness," they write.

35 comments:

Wiley Coyote said...

Nothing new.

We have no data to support Bright Beginnings but we still spend $21 million per year keeping it.

You can make data say anything you want, whether you have reams of it or just a few numbers and sometimes whether you have no data at all.

Anonymous said...

I guess these clowns think no one is watching.

They're probably right.

At least not very closely in the last 50 years or so.

There were too many social agendas to consider.

Anonymous said...
This comment has been removed by the author.
Anonymous said...
This comment has been removed by the author.
Anonymous said...

Oh, yeah, forgot to quote the report:

Page 8 -

"While data on student dropout were only available for the RCTs in Austin..."

So, out of a "sampling" of only 3 school districts, only ONE had actual dropout data?

And WHAT were they measuring the effectiveness of?

Wasn't it something to do with DROPOUTS?

Classic FAIL in a study.

Anonymous said...

Wiley,

They don't have "no data at all".

After all, they have dropout data on the Austin schools.

Or rather, their random sample of 6 Austin schools,

Or rather, their RCT (Randomized Control Trials) which included 93 CIS students and 58 non-CIS STUDENTS in Austin.

Shouldn't a study of 150 kids be enough to determine the effectiveness of a dropout prevention program that claims to serve 1.3 million students in 3400 schools?

Anonymous said...

The best data that could be collected is to somehow identify these probable "birthers" early on and maybe even pay them not to have children.

Clearly, many of these births are a new form of child abuse.

Anonymous said...

As with all these social programs, it makes these politicans and such "feel good" and that is how they justify their continued practices even when no data will support them. Then when folks start objecting to them, they play the race card on you and start the characteer assasination efforts through the media.

Anonymous said...

This is the program Ericka Ellis Stewart was involved with so back off the mother HEN ! So it has not data or proven results ? Neither will LIFT , but we just became their partner and hired their mother HEN (back) ! This stuff is amazing if you watch the useless money go down the drain. Then we do a PR campaign with our new PR staff to throw everyone under the bus that we dont have any budget money poor CMS !!!!!! BOZO'S

Anonymous said...

Sounds like another jobs program for the typical worthless do-gooder class.

Anonymous said...

Oh Lord, let's not expect CMS and BOCC to figure out what money already being spent is less effective and simply move it, rather than going to the taxpayers for more!

Anonymous said...

LIFT and CIS will do the same "CHERRY PICKING" analysis going forward. They eliminate anyone from outside the circle that wont graduate on time or is pregnant. If a child has struggled , BUT made it to 12th grade odds are he will graduate. They will measure this number as a tool to show high graduation rates. Heck CMS cannot even get the correct number of kids that graduate in main stream today at schools so I doubt this will be on radar. Especially since we just hired a few minority folks to run LIFT/ CIS. DUMBYDOWN.

Anonymous said...

Ericka was invlolved with Right Moves For Youth.

Anonymous said...

I will guarantee anyone who posts on this site that CMS data next year will show a significant increase in the graduation rate. Students who do not deserve to graduate are being pushed through the system at an alarming rate. Everything that can be done is being done at this very moment to increase the CMS graduation rate.

Anonymous said...

Keep movin', movin', movin'
Though they're disapprovin'
Keep them dogies movin'
Rawhide!

Don't try to understand 'em
Just rope, throw, and brand 'em
Soon we'll be living high and wide.

Anonymous said...

5:33, from the regular article, not on the blog, it states LIFT is not looking for any improvements till this next class entering high school graduates in 2016.

Anonymous said...

Please lets hire more PR people to sell this. Andy Baxter needs more help. Please o please hire more data and PR people at a hundred grand apiece.

Anonymous said...

No Data!
NO Peace!

Anonymous said...

Ann, why are they promoting CIS who only take a self-chosen caseload of 100 and drops anyone who doesn't meet their standards that costs the same amount as a CMS social worker that works with the most neediest students and doesn't give up or drop students from their caseloads. CIS claims to be counselors and social workers and yet, they have no such qualifications. Why does this continue?

Wiley Coyote said...

Anon 6:09...

Here's the problem with Project LIFT, "having no improvements first year".

This is only a 5 year project so if you're in the 9th grade next year, you will be impacted for only four years, a tenth grader 3 years and so on, with a senior getting one year of benefit. Only 8th graders and younger will get the full impact of $1,100 per pupil per year for five years.

What do kids entering kindergarden do after five years when they are in the fourth grade and the program ends?

So who do they start looking at regarding data?

Next year's seniors will get whatever benefits at the high school level on LIFT for one year but do you attribute any academic increases to LIFT or would they have gotten the increases anyway?

Anonymous said...

First off, kudos to Ann for delving into this - say what you will about the Observer, but this is the kind of the story (along with the Cobitz/Muri disaster) that needs reporting and follow ups.

For a district that touts its data-driven decisions and all that, there's a lot holes and missing data.

While CIS doesn't fall into Muri's kingdom, he's still the CIO, right? How can we trust anything that comes out of his department?

But much like the school board stayed silent on the Cobitz mess, no doubt there won't be a peep from them on this either.

OnBeattiesFordRd said...

Ann
CIS calculates high school drop out rates base on students that make it to the 12 grade and drop out. The problem with their numbers is that students drop out in the ninth, tenth and eleventh grade so, CIS does not count them as drop out because they did not make it to the 12th grade. I have met many students from West Charlotte High who are in the 9th grade and will turn 17, you know they will not stick around to graduate. They told me they have never heard of CIS. We need a better value and accountability for taxpayers money.

Wiley Coyote said...

Anon 10:31..

They told me they have never heard of CIS. We need a better value and accountability for taxpayers money.

After complaining about CIS, then complain about the taxpayer's billions improperly paid out in the NSLP...

Historical Improper Payment Rates for National School Lunch Program

FY.....Improper Pmt. (Billions)
2007............$1.4
2008............$1.4
2009............$1.6
2010............$1.5
2011............$1.7

Anonymous said...

How can the school board NOT address the Cobitz issue? Does the district not know what the true data is that they supply the public and foundations such as Broad? It goes way deeper than this. Look into Muri, Baxter and the new staff that was hired. Follow the money!

Anonymous said...

Let's get Andy Baxter to figure out a formula to track the workings and "success" of CIS. He sure had one heck of a formula for tracking the minute and almost irrelevant details of a student's life and growth for the Pay for Performance nonsense CMS had going last year.

Ann Doss Helms said...

OnBeatties and others, some of the points you're making were addressed in the main story, which is still posted at www.charlotteobserver.com/education I probably should have cross-linked for convenience, but I wrote this blog before leaving for the weekend so didn't have an online link.

Anonymous said...

Ericka= CIS, LIFT what ever other failure program you want to list knucklehead ! Ask her where he son goes to school I will tell you its not CMS !!

Anonymous said...

Has anyone seen the memo yet…that CMS employees are supposed to add the LIFT logo to their email signature? If no data is required to substantiate education value, why not simply have a reverse auction for the underperforming schools? The lowest bidder can then run the schools and get their logo pushed on staff by Muri and the CMS spin masters. Baxter can run the FRL numbers into the many millions and Wiley will have something to rant about for years to come. Is CMS part of the Mecklenburg County family? It does not feel like it.

Anonymous said...

3:27 - if your kid could get into science and math in Durham, he'd go there too. Lay off.

Wiley Coyote said...

Anon 9:04...

I think the point being made is this:

Mayor Foxx doesn't have his kids in CMS, yet loves to run his mouth on how great it is and what CMS needs.

Mayor Pro Tem Patrick Cannon doesn't have his kids in CMS.

Chair of the BOE Ellis-Stewart doesn't have her child in CMS as the other poster stated.

We seem to have all these huge supporters of CMS yet they don't send their kids there, which is certainly their choice.

However, it seems these Democrats fit the very definition of hypocrisy....

At least Obama had the guts to call it like it is:

Posted on Monday, September 27, 2010 12:52:52 PM

President Obama said Monday that his daughters could not get the same level of education from D.C. public schools that they receive at the elite private school they attend.

In an appearance Monday morning on NBC's "Today" show, Obama was asked by a woman in a television audience whether a public school in his home city could measure up to the standards of his children's private school.

"I'll be blunt with you: The answer is no right now," the president replied. The D.C. public schools, he said "are struggling."

Jeff Wise said...

Point of clarification:

Erika Ellis-Stewart's son goes to the Science and Math school. Her daughter is in CMS.

One can make political hay about her son, but that's an opportunity you don't pass up.

You cannot legitimately lump her in with other politicians who have their children in non-CMS schools.

Ann Doss Helms said...

Wiley, I think Mayor Foxx does have his younger child in CMS.

Wiley Coyote said...

I stand corrected on Mayor Foxx having another child in CMS
....

My apologies for incorrect information.

Anonymous said...

For the folks who are bashing CIS based on this woefully fact lacking piece, research before so. Read the research yourself, but please, if you don't know how to read research, don't interpret something you don't understand. As for the effectiveness of CIS, they target at-risk youth to help alleviate the load on the schools. I don't any of you are actively going to houses of kids in gangs, who have been raped, or are evicted to help. The schools aren't equipped for that. For all the "liberal" and other such labeling, CIS embodies local, non-profit organizations stepping in and doing things government can't. I personally know of many students on the brink of disaster buoyed up by the folks of CIS in your area. You should be trying to make them more effective, talk speaking to something you know nothing about except what somebody has interpreted for you.

Anonymous said...

In response to February 8, 2012 10:36 AM. I work in a CMS school that has CIS workers there. The CMS social worker has a masters degree in social work and is an LCSW. The CIS workers have bachelor degrees. The CMS social worker works circles around the three CIS workers. The three are constantly seeking input from the one CMS social worker. The big advantage that the CIS workers have are financial resources to food, medical care and dental care. The CIS workers get to hand pick their case loads. The CMS worker sees everyone in the building. CIS workers are no where near the skill level of the CMS social worker. CIS is BS for the most part. They constantly get kids out of my class for field trips...taking them from the very place they should be, in the classroom. CIS has their role but they are far inferior to the CMS social worker at my school