That's an excellent point that I, too, have made many times in the past.
So I found the
actual study for this, and the methodology is not bad (especially compared to others I've seen), they spend time examining the performance of the students prior to entering the charter compared with the average for neighboring regular schools, and found the entering students tended to be of lower performance. They also looked at the problem of attrition (students leaving the school, often due to issues keeping up with the required work.) and found relatively low attrition. They also kept students who left the school in the pool of students, so even if attrition were weeding out weak students, they would still count in the school performance.
Finally, when making the comparisons which lead to the big gains headlines, the comparison was not between the charter students and all students in neighboring schools, but between the charter students and a sample of students from the neighboring schools who were identified as matching the charter school body based on pre-middle school performance and demographics.
This part I'm little less confident on, and I would have liked to have seen not only the comparison with the model-matched non-charter group, but the wider set of all students for the appropriate years enrolled at the neighboring schools. The fact that it is not included makes me wonder if that comparison was less favorable, since I find it hard to believe the authors would go through as much rigor as they did but not do that comparison at all.