Many short-term interventions in a child’s life (e.g. pre-school) can generate positive effects for the subsequent years. However, the problem is that the effects of many of these interventions is not persistent. The gains disappear 2-3 years after the program.

Does the same happen with charter school enrollment? We know that charter schools can increase test scores (though not always). But in order to identify long-term effects, we’d have to look at the college or labor market outcomes of former charter enrollees vs. students of traditional schools.

Let us first look at various tests and decisions related to high school. These in and of themselves do not have long-term effects, but how one performs on them can have long-run repercussions. Think of e.g. the SATs or the decision whether to go to college.

Angrist et al. (2013) (ungated) do such an analysis for six Boston charter high schools using a sample of around 3,700 students who applied between 2002 and 2009. They make use of lotteries (that oversubscribed charters use to allocate places) to estimate the effects of charter schools.

This is important, because if one just simply compares students in traditional schools and students in charter schools, problems may arise. This is because some hard-to-measure but important characteristics of the students (e.g. motivation) in the two types of schools could differ. Lotteries ensure that the sample is restricted to students who *want to* attend charters. Thus by comparing lottery losers to lottery winners, we can make sure that the two groups are not different on some “fundamental” trait like motivation.

First, in the 10th grade students take a state-wide standardized assessment test called the MCAS. Good performance in this test makes a student eligible for public university tuition waivers, and can therefore have an influence on long-term outcomes. The authors estimate that charter students perform about 30-40% of a standard deviation higher in “English Language Arts” and 50-55% of a standard deviation higher in math. The figure below shows this shift in score distribution for math.

The red curve represents students who participated in the lottery, won and now attend charters. The blue curve are students who did not win in the lottery, but had they won, they would have attended charters.

Second, Angrist et al. look at SAT scores, which are obviously very important for college admissions. Here the results are a little weaker. Altogether, the probability of scoring above the bottom quartile or the median (both measured within Massachusetts) increase by around 10-15 percentage points. The probability to score in the top quartile is not affected in any way. The only exception is math, but even there the effect is weak (5 percentage points, significant at 10% only).

The composite SAT score if one attends a charter is increased by about 100 points (about 40% of a standard deviation), the math SAT score is increased by 52 points on average (54% of a standard deviation).

The general picture is that low-performing and mediocre students gain a lot from attending charters, top students don’t (but neither do they lose out). This is shown for reading and math below. Notice how the distributions converge in the right tail (i.e. at top performers).

Third, let’s consider Advanced Placement (AP) courses. Charter students are 29% more likely to take AP exams. On average, charter students take one more AP exam than students at traditional schools. Scores on the exams are not affected to a very large extent: performance on science, history and English tests does not change. Performance on the calculus test improves: about the probability of scoring 2 or higher rises by 8.7 percentage points, 3 or higher by about 7.3 percentage points. Getting a score of 4 or 5, however, is not more likely for charter students.

AP exams can be converted into college credits with scores 3 or above (e.g. in California) or with scores 4 or above (e.g. in Massachusetts). Thus there are some positive effects on the scores, and there are definite positive effects on the probability of taking an exam. The former is mostly driven by math.

Fourth, what about high school graduation? The results here are interesting and go against the rest, sort of. Charter school students are actually 12.5 percentage points less likely to graduate on time. This is because they’re more likely to repeat 12th grade. However, they do tend to graduate in 5 years, so overall graduation rates don’t differ much. It seems though that charters are more inclined to postpone graduation for bad performers. Indeed, when the authors perform their analyses by subgroups, they find that it is at-risk students (e.g. boys, special ed students, below median students) that are most often held back in charters.

Let me also introduce another paper here. Booker et al. (2014) (policy brief) look at 3,500 charter students in Florida and 1,000 students in Chicago. They estimate the effect of charter high schools by comparing two groups: those that were in a charter in 8th grade but switched to a traditional school in 9th grade, and those that didn’t switch.

They find that high school graduation rates within five years (so including those who are held back for a year) are higher in charters by 7.4 (Chicago) to 10.9 (Florida) percentage points.

Now, for the second part of this post, let’s consider student outcomes outside high school. First, college attendance is not more likely for charter students in Massachusetts. But they are 17 percentage points more likely to go to a 4-year institution and 11 percentage point less likely to go to a 2-year institution.

Booker et al. find that charter students on average are 10 (Florida) to 11 percentage points (Chicago) more likely to attend a two- or four-year college within 6 years.

Second, it seems charter students are also more likely to stay in college. In Massachusetts, charter students are not more likely to persist in 2-year colleges, but they are about 17 percentage points more likely to spend one, 11-13 percentage points more likely to spend three semester(s) in 4-year colleges. Seemingly though, college persistence in two-year colleges may be lower for charter students (significance weak, not very robust), but altogether (i.e. considering both two- and four-year colleges) the effects are positive (but again not significant).

Booker et al. have more encouraging results for Florida: staying in college for two consecutive years is 12.6 percentage points more likely for charter students. For Chicago, the effects are not significant.

Finally, given that the Florida sample of Booker et al. is older, they can also look at labor market outcomes. Annual earnings in ages 23-25 are found to be about $2,350 higher for charter students, which is about 13% higher than the earnings of traditional students. This is significant at 5%.

So what can we conclude? The results as we can see are somewhat heterogeneous: charters are not always unambiguously better than traditional high schools on all accounts. But what’s clear is that few negative effects are found.

On the other hand, charters seem to positively affect lots of outcomes that correlate with later success in life. It seems that charter students make large gains relative to traditional students especially on math. And charters improve the performance of low-performing and mediocre students the most. There are no positive effects for top students (but neither are there negative effects).

One concern that may arise is that charters can be very heterogeneous: there can be good ones and bad ones. But along what dimensions do charters differ? Zimmer et al. (2013) find that who authorizes a charter does not matter much. Charters authorized by nonprofits have about 9% of a standard deviation lower math/reading scores, but the effect is not significant once we exclude new schools (i.e. those in their first/second years). So it may not be easy to identify weaker charters.

Also, Booker et al. mention that Florida and Chicago charters have been shown to have no advantage in test scores over traditional schools in these two locations. And yet they have these nice positive effects on long-term outcomes. This suggests that charters might not exert their positive effects by improving test scores, but by other means. What these are is perhaps a subject of further study: it could be better counseling, or more effectively instilling skills that are useful in college/work than traditional schools.

All in all, my opinion is that we can see that charters can have nice positive long-term effects, especially on at-risk students. But to me it seems that they’re not a silver bullet: the long-term positive impacts are small compared to the strong-ish gain in test scores. Also, charters need to be properly supervised to weed out the bad ones. For instance, Massachusetts has a very strict charter authorization process. And this can be instrumental to the success of Boston charters.

Pingback: Study: Charter high grads earn more as adults — Joanne Jacobs