Educational Vouchers: A Review of the Research
by
Alex Molnar
Center for Education Research, Analysis, and Innovation
School of Education
University of Wisconsin-Milwaukee
PO Box 413
Milwaukee WI 53201
414-229-2716
October, 1999
CERAI-99-21
Educational Vouchers: A Review of the Research
October 1999
CERAI-99-21Alex Molnar
Professor, Department of Curriculum and Instruction University of Wisconsin-Milwaukee
This document combines excerpts from two reports: "Smaller Classes -- Not Vouchers -- Increase Student Achievement" (Harrisburg, Pa.: Keystone Research Center, March 1998); and "Smaller Classes and Educational Vouchers: A Research Update" (Harrisburg, Pa.: Keystone Research Center, June 1999). Both documents are available on the website of the Center for Education Research, Analysis, and Innovation at http://www.uwm.edu/Dept/CERAI
Official Evaluation Results for Cleveland Scholarship and Tutoring Program
The legislatively mandated independent evaluation of the Cleveland Scholarship and Tutoring Program is being conducted by an Indiana University research team headed by Professor Kim Metcalf. This team published reports on the program’s first year (1996-97) in March 1998 and second year (1997-98) in November 1998.16
To evaluate the Cleveland voucher program, Metcalf’s team compared the test scores of third-grade voucher recipients with those of Cleveland Public School students, controlling for prior test scores and family characteristics. In 1996-97, the Metcalf evaluation examined third grade performance because that was the lowest grade for which usable test data (from second grade) existed to measure student ability prior to the voucher experiment.
The first-year official evaluation report found that, after controlling for background characteristics, third-graders participating in the voucher program in 1996-97 did not achieve at a higher level (on reading, language, mathematics, science, and social studies tests) than students who remained in the Cleveland Public Schools. The second-year report (1997-98) found that fourth-grade students in the voucher program achieved significantly better than their public school counterparts in science and language. When classroom variables (e.g., class size, teacher experience, and teacher level of education) are accounted for, the voucher students achieved significantly higher scores only in language.
The Peterson team criticized the Metcalf team’s first-year report for several reasons.17 PGH argued against the use of second grade test data as a control for student performance prior to the voucher program on the grounds that these test results "lack plausibility." PGH deemed these test scores implausible because the scores showed low-income, largely single-parent families performing close to the national average in the second grade and then scoring at substantially lower levels the next year. PGH also maintained that the second-grade test scores have implausibly weak correlations with family background characteristics. Leaving out the second-grade test scores, however, means that any comparison of voucher student achievement with that of public school students takes no account of differences in student performance prior to the program. Moreover, if the second-grade test scores were uniformly inflated for both voucher students and those who remained in the Cleveland Public Schools (e.g., because second-grade public schools "teach to the test"), they would still be a good control measure.
PGH also maintained that the Metcalf evaluation team should have included student scores from the Hope schools, since 25 percent of voucher students went to these newly created schools. Metcalf’s team had excluded the Hope schools because their students took a different test than the public school students and students at other voucher schools. An additional problem with including Hope students is that approximately 58 of the 155 Hope students tested in the spring of 1996 appear not to have been tested in the fall of 1997, an unusually high attrition rate. Without information on the characteristics of these students it cannot be known what impact their absence may have had on the results reported.
When PGH reanalyzed the official data excluding the second-grade test scores and including the Hope students with converted scores, they found that voucher students scored significantly higher in language and science, but not significantly higher in math, reading, or social studies. When the second-grade test scores were included, the Peterson team found results consistent with those of the official evaluation team: voucher students did not score significantly higher than their public school counterparts at conventional levels of statistical significance. Using a lower statistical significance threshold than conventional (the .10 level, a 10 percent chance that the results could have occurred by chance), PGH found that voucher students did better in language and science, but not in reading, math, and social studies.
The second-year (1997-98) Metcalf team evaluation also found that not all schools participating in the voucher program had similar achievement results. Students attending established private schools were responsible for the voucher student achievement advantage in science and language. Students in the newly established private Hope schools scored significantly lower than their public school counterparts in all tested areas.
The finding that student performance in the new voucher schools is significantly worse than student performance in public schools raises serious questions about the viability of voucher programs as a large-scale education reform. Existing private schools may produce benefits for low-income students by placing them with a majority of students from more privileged or more academically oriented backgrounds. The adoption of large-scale voucher programs may, however, alter the social context that produces whatever achievement benefit there may be for low-income minority students in attending private schools.18
Voucher programs supported by private sources provide another potential source of information on the educational consequences of vouchers. In 1998-99 there were 41 privately funded voucher programs in the United States, according to Troy Williamson of the CEO America Foundation (interview, March 29, 1999). There have been few systematic efforts to study the impact these programs are having on student achievement. This section describes those programs for which achievement data exist or for which an evaluation plan that will provide achievement information has been adopted.
Milwaukee: Partners Advancing Values in Education (PAVE)
Perhaps the country’s largest private program operates in Milwaukee. Partners Advancing Values in Education (PAVE) -- formerly the Milwaukee Archdiocesan Education Foundation -- was founded in 1992. PAVE provides low-income families with scholarships worth half of the tuition charged by a private religious or non-sectarian school up to a maximum of $1,000 for elementary and middle school students and $1,500 for high school students. PAVE’s major donors include the Lynde and Harry Bradley Foundation, TREK Corporation, CEO America, Johnson Controls, Northwestern Mutual Life Insurance Co., Siebert Lutheran Foundation, and Wisconsin Electric Power.
Of the five evaluations of the PAVE program, only the 1994 report made a serious effort to determine the program’s effect on student achievement.19 The 1994 evaluation suggested that students who attended private schools for their entire school career achieved at higher levels than students who transferred from a public school into a private school participating in the PAVE program. Further, the evaluation suggested that the longer transfer students stayed in participating private schools the greater their achievement.
Unfortunately, since the data gathered depended entirely on the voluntary cooperation of parents, the findings are suspect and no conclusion can be drawn from the evaluation’s results.
Indianapolis: The Educational Choice Charitable Trust
The Educational Choice Charitable Trust was established in 1991 with a $1.2 million grant from J. Patrick Rooney, Chairman and CEO of Golden Rule Insurance Company. The Trust provides educational vouchers worth half the cost of private school tuition up to a maximum of $800. Families with children who qualify for the free or reduced-price lunch program and live in the Indianapolis school district are eligible. Half the money in the program was reserved for families whose children were in private schools prior to the creation of the program.
In March 1996 the Hudson Institute issued a report by David Weinschrott and Sally Kilgore assessing the impact of the program.20 Public school students, but not voucher students, showed a drop-off in reading, language, and math scores in sixth and eighth grade.
Weinschrott and Kilgore described their evaluation framework as "informal." It was based on a small number of voucher students enrolled in a handful of voucher schools. The analysis did not control for differences in student characteristics, test scores prior to the voucher program, or other potentially significant variables that may have influenced the findings.
The New York School Choice Program
The New York City School Choice Scholarships Foundation (SCSF) was established in 1997 with $5 million of its $7 million commitment coming from New York businesspeople. SCSF offers tuition vouchers worth up to $1,400 to students whose family income makes them eligible for the free school lunch program. Eighty-five percent of the scholarships are reserved for public school students whose test scores are below the citywide median. In its first year (1997-98), the program offered scholarships for up to 1,300 students and actually placed about 1,200 students in private schools. In 1998-99, an additional 1,000 students participated in the program. SCSF has made four-year commitments to the current participants and will add more students as funding permits.
Of parents expressing interest in the program, a randomly selected group were interviewed to determine their eligibility, while their children (except for kindergartners) were administered the Iowa Test of Basic Skills in reading and math. A lottery determined which eligible students would be offered vouchers.
In the spring of 1997, Mathematica Policy Research and Paul Peterson of the Harvard Program on Education Policy and Governance began a three-year evaluation of the performance of students entering the New York SCSF Program in 1997-98.21 The evaluation examines two issues. (1) It compares the achievement of about 750 students who used vouchers with that of 960 students whose families sought but did not receive a scholarship. (Ten percent of the non-voucher students ultimately attended private school anyway.) (2) The evaluation also compares the achievement of 1,000 students offered a voucher, including some students that did not use one, with that of the same control group of 960 students.
A limitation of the first comparison is that although a random group of students received scholarship offers, a non-random group appears to be have accepted offers. According to Peterson, Myers, and Howell (PMH), families that used scholarships had higher incomes and more education than families that did not use scholarships.22 PMH used standard statistical procedures to control for differences between voucher users and students not offered a scholarship. However, they did not provide enough information about these procedures to permit a complete evaluation of them.
The second comparison gets around the non-random nature of the group that actually used scholarships by taking advantage of the "natural experiment" resulting from the use of a random lottery to select those offered vouchers. As a result of this lottery, the background characteristics of those offered scholarships and of those not offered scholarships may be assumed to be, on average, the same. Any differences between the two groups can be attributed to the "offer" of a scholarship. This comparison, however, is somewhat difficult to interpret. Why would the offer of a scholarship be expected to make a difference to the performance of students who do not actually accept the scholarship?
In November, 1998, PMH released first-year evaluation results. They found that being offered a voucher raised performance significantly in math in second, third, and fifth grades, and in reading in fifth grade. In third grade, being offered a voucher was negatively correlated with math and reading achievement but not significantly so. The effect on achievement of actually receiving a voucher was statistically significant in math in second, fourth, and fifth grade, and in reading in fifth grade. In third grade, receiving a voucher was negatively correlated with math and reading achievement but not significantly so. PMH increased the number of so-called significant results by using a statistical method that requires assuming vouchers can increase but not decrease student achievement.23
The conflicting results reported in the literature on vouchers and public versus private schools make this assumption questionable. Without this assumption, only the results for fourth-grade math, fifth-grade reading, and combined fourth- and fifth-grade math are significant. In addition, the differences between the results across grade levels are hard to interpret. This suggests that the results should be treated with caution until more data are available. Since the PMH evaluation of the New York SCSF program constructs comparison groups, it is more informative than the PGH analysis of the two Hope Schools in Cleveland. However, as PMH acknowledge, their SCSF evaluation involved a small number of students and the impact of a much larger program could have quite different program outcomes. A number of characteristics of the schools attended by voucher students in the New York experiment might not exist in a large-scale experiment. For example, compared to the schools attended by the control group, the voucher schools had small classes and were somewhat more racially integrated. Parents perceived that voucher schools had fewer problems with safety, fighting, cheating, missing classes, being late for school, and destroying property.
The frailty of positive findings from participation in voucher programs is suggested by the ad hoc and inconsistent ways that Peterson and co-authors explained findings from New York and from Milwaukee. In their analysis of the Milwaukee Parental Choice Program, Greene, Peterson, and Du found significant achievement effects only for students who had been in the program for three or four years.24 They hypothesized that participation in a voucher program has a cumulative effect, with positive results only appearing in the third and fourth years, after students have been socialized in their new setting. In discussing the New York program, Peterson, Myers, and Howell hypothesized that they found significant results only for fourth- and fifth-grade students because vouchers are a more potent intervention for older students. They added that smaller classes may be more potent for younger students -- an explanation at odds with the fact that students at voucher schools in the New York program attended smaller classes than students in the control group.
In discussing their first year New York results, PMH argued that the magnitudes of the positive achievement effects observed "do not differ materially from those observed in" the Tennessee class-size reduction program.25 This comparison is problematic because of the instability of most of the SCSF findings compared with the Tennessee results. Charles Achilles, one of the Tennessee experiment principal investigators, pointed out that since the students in the SCSF evaluation are about 95 percent minority, it might be more appropriate to compare SCSF effect sizes with the effect sizes observed for Tennessee minority students.26 When this comparison is made, the Tennessee effect sizes (between .30 and .40) are much larger and much more stable than the effect sizes reported by PMH (-.09 to .27).
Continue this Section Private School Vouchers
