A few weeks ago, I wrote about our schools’ “secret success.” Simply stated, poor and minority students are achieving at dramatically higher levels today than they were two decades ago—in some cases two or three grade levels higher. And while we can’t be sure what led to this academic acceleration, test-based accountability was probably the most important factor. Or so I argued.
But the plot thickens, because these national averages mask state-by-state differences that are quite instructive, too. See, for example, this chart from Matt Ladner, a longtime Goldwater Institute education analyst who now works for Jeb Bush’s Foundation for Excellence in Education. It examines NAEP gains over a shorter six-year period—2003 to 2009—for kids eligible for free or reduced-price lunch.
Combined NAEP Gains for Free and Reduced Price Lunch Eligible Children, 2003-2009
Those are huge differences, with Florida leading the pack. While this is hardly causal evidence, it sure seems likely that Florida has been doing something in the realm of education policy that is making a big difference. (Perhaps Governor Bush’s comprehensive and aggressive reforms?) And likewise, that West Virginia is doing something very wrong. (Twenty-first century skills, anyone?)
Skeptics, including the union-funded “National Education Policy Center” at the University of Colorado, wonder if Florida’s grade-retention policies explain the impressive results. (Florida students can’t move onto the fourth-grade until they are reading proficiently.) Sunshine State kids are scoring better, goes the argument, because they are older. But Ladner dismisses that argument thoroughly in the pages of Education Next—something the NEPC refuses to acknowledge.
There’s plenty of bad news to bemoan in education. How about all of us—reformers and skeptics alike—agree to make the most of the good news that falls in our lap every now and then? Poor kids in Florida and a few other states are making HUGE gains. Let’s figure out why.
—Mike Petrilli
Mike, this is amazing. And one thing I hope this data does is put to rest the tired argument that you can’t improve poor kids’ academic achievement until you cure their poverty disease. (Do we have to worry about cheating?)
–peter m.
Oy vey, Mike. You’re better than this.
My response posted here:
http://nepc.colorado.edu/blog/response-mike-petrilli
What I find refreshing in this exchange is respectful disagreement and two posts that encourage those of us interested in deepening student learning gains to read and talk further. That’s the best of what policy think tanks do. Thanks to both Kevin and Michael for great posts.
Thanks for the post Mike. Here is my response to Dr. Welner’s response:
http://jaypgreene.com/2011/08/04/nepcs-attempt-at-strategic-nihilism/
…and my reply to Dr. Ladner is posted here:
http://nepc.colorado.edu/blog/reply-matthew-ladner
Dr. Welner-
I’ll address your first two points.
1. There is no “clear and obvious” artifact of grade retention. Your reviewers have never dug very deep into the Florida situation, prefering to lean on what they think they know from studies of programs in other states. Your team has repeatedly ignored glaring flaws in the retention artifact thesis, and simply continued to assert that it exists. Neither you nor your reviewers have ever addressed the research specific the Florida retention policy, including two highly sophisticated regression discontinutity design models which show gains independent of mere aging. Your description of the Florida retention policy as “just flunking a kid” shows that you have an entirely inaccurate view of how the program works.
Your first reviewer included a chart in her appendix showing a substantial improvement in 3rd grade reading scores, which in turn led to a substantial decline in children qualifying for retention. Although your reviewer failed to recognize the significance of this, it is deeply damaging to your retention artifact theory, as Florida’s reading scores continued to rise even as retention fell.
In the end though, if you look at the chart Mike included in his post above, which is the gains for free and reduced lunch students on all four regular NAEP exams for the entire period which all 50 states took the NAEP, Florida still ranks first even if you exclude 4th grade reading from the analysis, which leads to your second point…
2. The chart above is a combination of all four NAEP exams that are given on a regular basis- 4th grade reading, 4th grade math, 8th grade reading and 8th grade math. The NAEP science test was reformatted and has no trend data. The only other test that has some trend data is 8th grade writing. Florida’s progress on 8th grade writing has more than doubled the national average since 1998 (the year before the reforms) with a 16 point gain in Florida and a 6 point gain nationally.
Hi Matt
I’ll try to get to the other points here tomorrow. But I do want to clarify. I wasn’t saying that FL’s grade retention plan involves “just flunking”. Quite the opposite: it includes interventions that likely have a positive effect. My point was/is that while these interventions may have such a positive effect, the research on grade retention itself (“just flunking”) strongly suggests that states would be wise to avoid that part and focus only on interventions that have been shown to be beneficial.
The other point, which I’ve made repeatedly now, is that what we’re calling the “retention artifact” may very well not account for the full FL NAEP improvement over these years, but to ignore it is to mislead. If I live in a house with high radon levels and I also smoke, my increased risk of lung cancer is explained by both factors. If I attribute that entire increased risk to only smoking, I’m being misleading.
Florida’s seniors scored below the national average in Reading and Math on the NAEP.
Only 3 of the 11 participating states met this criteria. This dubious distinction was earned following the students having undergone several years of Bush’s reforms, which Ladner so promotes while forgetting this fact. I recall for years reading Mr. Ladner’s enthusiastic praise of Florida’s grade 4 scores while neglecting to mention the retention policy. He did finally begin to mention it but it always deserved a place in the conversation and comparisons, imho.
I have long wondered why one should celebrate if a nine year old performs on grade level if that performance vanishes by the time one graduates and enters the workforce or attempts higher education. If I can swim as a ten year old but no longer as seventeen year old, drowning is still an option for me.
I have long held the opinion that Mr. Ladner’s background in politics comes to the surface in his work and I appreciate instead the background of Dr. Welner. The students of Florida need the able work of Dr. Welner and I thank him for discriminating between illusion and reality.
Dr. Welner-
Your reference to “the research on grade retention” ignores the research on this specific program in Florida. The Florida program, using objective data and targeting children during the developmentally critical period for literacy acquisition is far different than retention policies of the past.
In addition, the regression discontinutity analysis performed specifically on the Florida program is far more sophisticated than the vast majority of previous studies on programs in other states. The RAND Corporation performed a regression discontinutity analysis on the New York City retention program and also found statistically significant learning gains outside of mere aging as well:
http://www.rand.org/pubs/monographs/MG894.html
Far from ignoring Haney’s artifact theory, I addressed it long before your first review in the pages of Education Next. Your first reviewer cited neither the original work of Professor Haney nor the Education Next article addresssing it. She also included a table in her appendix which blew a very serious hole in Haney’s theory. No one on your team has yet to make any effort to explain how Florida’s NAEP scores have continued to climb while retentions dropped off substantially. Nor can Haney’s retention artifact theory explain why 3rd grade FCAT scores have improved so profoundly.
Your second reviewer ignores all of this and trumpets the first analysis as if it were valid.
Earlier communication with Mr ladner regarding misleading folks and retention…
http://jaypgreene.com/2008/08/22/demography-is-not-destiny/>
I invite Mr. ladner to entice me into celebrating hte wonderd of Florida’s flawed accountability program as it relates to end performance on the FCAT in Reading. The highest grade level of this test in this subject is grade 10. Look what has not happened…a miracle.
In 2002, the data for grade 10 students (all) in Reading on the FCAT went as this:
59% passed. 32% level 1, 33% level 2, 21% level 3, 8% level 4, and 7% level 5.
In 2010, the data for grade 10 students (all) in Reading on the FCAT went as this:
60% passed, 32% level 1. 29% level 2, 18% level 3, 8% level 4, and 14% level 5.
I feel no need to clap. (if I remember correctly there was a nine point gain needed to go from level four to level five.) I do feel a need to ask for honest information from folks like Mr. Ladner and again thank Mr. Welner for any help Florida students may get via his work.
Does Mr. Ladner have anything to say about the Florida grade 10 Reading data comparing distributions of scores from students who had minimal exposure to Bush policies and those who had extensive exposure?
Look what I found related to a reformer idol contest and this article.
OhioGadflyOhio Gadfly
You really should have waited to release this information until AFTER #edreformidol. Unfair advantage to FL? http://bit.ly/oIFoMd
Ms. Hanfmann-
Although I fear that I have bored Mike readers too much already, I have noted to you before that there is only a single data point for 12th grade NAEP scores from Florida. We have no trend data, so it very well could be the case that there has been substantial improvement in Florida’s 12th grade scores.
Other indicators, such as graduation rates, AP and college remediation rates have all improved.
Mr. Ladner,
I am only stating data which you fail to share and I believe belongs in the conversation as you peddle the reforms of a state which is producing seniors who score below the national average in both Reading and Math after exposure to the Bush flawed policies. Your ” very well could be the case” is exactly the wrong words for my ears and irresponsible as one who tries to influence what happens to Florida students, imho. Wasn’t the NAEP grade 12 study from 2009?
What are your words about the grade 10 missing miracle or am I blind to the wonders of the ironically titled A+ Plan? Please advise .
I have yet a bit more data for your review from the FCAT involving the total student performance of grades 3, 4, and 5 in 2002 and 2004 in both subjects of Reading and Math. When was the retention policy implemented? It was only based on reading performance but it seems math data also may have benfitted from the chnage in population making up the grades. Take a look at what I see.
2002 – 2004, Math, total students, % scoring at or above level 3
Grade 5 went from 48-52, a gain of 4
Grade 4 went from 51-64, a gain of 13, hmmm
Grade 5 went from 59 to 64m a gain of 5
The prize goes to grade 4.
2002-2004 Reading, total students, % scoring at or above level 3
Grade 5 went from 53-59, gain of 6
Grade 4 went from 55-70m gain of 15, hmmm
Grade 3 went from 60 to 66, gain of 6
The prize goes again to grade 4.
Imagine that. Could it be the removal of low scorers from the sample makes for better data and the data does not reflect instructional improvement? Let me
allow you the thrill you have in boasting grade 4 stats. Why would it be of a single benefit to any child to be proficient in grade 4 if it disappears as time goes on? Would you be thrilled if your child was “proficient” in grade 4 , yet unable to graduate with a meaningful diploma or future as a Senior?
The calculation of grad rates is dependent on the defintion and you certainly know this. Did you also know the ACLU filed a lawsuit against Florida for its dismal graduation rates?
Provide me with your data on Florida’s AP improvement, Are you speaking of participation or pass rates? Thank you.
Slicing the pie differently, I now follow the total student group of third graders in 2002 as they progress to grade four in 2003 and grade 5 in 2004.
Here is what happens to the percentage scoring at or above level three in Reading as the years go by: The group starts at grade 3 in 02 with 60% meeting that criteria.
next in grade 4 in 03 with 60%
then in grade 5 in 04 with 59 % I see no miracle.
Here is what happens to the percentage scoring at or above level three in Math as the years go by. The group starts at grade 3 in 02 with 59%
next in grade 4 in 03 with 54%
then in grade 5 in 94 with 52% I see no miracle.
I hope, rather than boring Mike’s readers, I am enlightening them to some realities that would have been missing had I not provided them. Should anyone challenge the accuracy of my state test data, I invite correction as I am not about misleading folks,
Umm… how would that work as an add on with all the wires and stuff? Also that would put a strain on the right hand, plus it’s just confusing with the position and all… I’m still trying to get used to using both the circle pad and the d pad on my 3DS…