Educational Vouchers: A Review of the Research

by
Alex Molnar

Center for Education Research, Analysis, and Innovation
School of Education
University of Wisconsin-Milwaukee
PO Box 413
Milwaukee WI 53201
414-229-2716

October, 1999

CERAI-99-21

Educational Vouchers: A Review of the Research
October 1999
CERAI-99-21

Alex Molnar
Professor, Department of Curriculum and Instruction University of Wisconsin-Milwaukee

This document combines excerpts from two reports: "Smaller Classes Ð Not Vouchers Ð Increase Student Achievement" (Harrisburg, Pa.: Keystone Research Center, March 1998); and "Smaller Classes and Educational Vouchers: A Research Update" (Harrisburg, Pa.: Keystone Research Center, June 1999). Both documents are available on the website of the Center for Education Research, Analysis, and Innovation at http://www.uwm.edu/Dept/CERAI

Table of Contents - Exercept 1 Historical Background Educational Choice Enters the Mainstream The Battle Over Vouchers Today The Milwaukee Parental Choice Voucher Program The Debate Over the Achievement Effect of the Milwaukee Voucher Program Box 3: Public vs. Private Schools Why Different Researchers Reach Different Conclusions The Witte Evaluations Box 4: Sorting through the Conflicting Voucher Results The Greene, Peterson, and Du Evaluation Box 5: When are Significant Results Not So Significant? The Rouse Evaluation Milwaukee’s Private Voucher Program -- PAVE Box 6 - A Case Example of the Relative Cost and Performance of Public and Private Schools The Cleveland Scholarship and Tutoring Program (CSTP) Vouchers, Values, and Educational Equity Box 7: Does Money Matter? School Spending and School Outcomes References

Table of Contents - Exercept 2 The Argument Over Vouchers The Milwaukee Parental Choice Voucher Program The Achievement Effects of the Milwaukee Voucher Program The Cleveland Scholarship and Tutoring Program (CSTP) Official Evaluation Results for CSTP Private Voucher Programs Private School Vouchers (Con't) Vouchers and Educational Equity References

Milwaukee’s Private Voucher Program -- PAVE

Voucher programs supported by private sources provide another potential source of information on the educational consequences of vouchers. Perhaps the country’s largest private program operates in Milwaukee. Partners Advancing Values in Education (PAVE), formerly the Milwaukee Archdiocesan Education Foundation, was founded in 1992. PAVE provides low-income families with scholarships worth half of the tuition charged by a private religious or non-sectarian school up to a maximum of $1,000 for elementary and middle school students and $1,500 for high school students. PAVE’s overhead is about 7 percent of its annual costs.⁶⁰

Box 6 - A Case Example of the Relative Cost and Performance of Public and Private Schools

Milwaukee provides a case example on both the relative performance and the relative cost of public vs. private schools. In 1991, the Catholic archdiocese of Milwaukee released the test scores of children in its schools. The results showed that when the performance of children from similar social and economic backgrounds were compared, the Catholic schools in the Milwaukee archdiocese did no better and perhaps a bit worse at educating minority children than the Milwaukee Public Schools.⁶¹

The picture looks about the same with the issue of cost. In 1994, when the archdiocese began closing its four central-city elementary schools, the Catholic school system had a per-pupil cost of approximately $4,000 at the four schools.⁶² By comparison, in the 1992-93 school year, when excluding centrally budgeted items such as fringe benefits and transportation, each elementary school in Milwaukee received, on average, $2,958. Even including all centrally budgeted items the public schools spent $4,645 per student.⁶³ The Milwaukee public schools also provide many more services and a more complete educational program than the private Catholic schools, according to an independent Milwaukee-based research institution.⁶⁴

PAVE awards about half of its scholarships to students who already attended private school. Approximately 95 percent of PAVE-supported students attend religious schools, with more than half (about 60 percent) enrolled in Catholic schools. Unlike the Milwaukee Parental Choice Program, PAVE enrolls a higher percentage of white students than the Milwaukee Public Schools. Also, unlike MPCP, schools participating in PAVE may reject applicants.⁶⁵

PAVE has for the most part shied away from assessing student achievement gains, preferring to focus on other issues such as parental satisfaction, parents’ reasons for participating in PAVE, and the extent to which they assist with their children’s school activities.⁶⁶ The most recent (1996) evaluation, for example, examined discipline in participating schools, the residential mobility of participating families, and the reasons eligible families did not participate.⁶⁷ The evaluations commissioned by PAVE have found that people who participate in the program are well satisfied and that there are relatively few serious discipline problems at PAVE schools.

Of the four evaluations of the PAVE program, only the 1994 report made a serious effort to determine the program’s effect on student achievement. The 1994 evaluation suggested that students who attended private schools for their entire school career achieved at higher levels than students who transferred from a public school into a private school participating in the PAVE program. Further, the evaluation suggested that the longer transfer students stayed in participating private schools the greater their achievement.

Unfortunately, since the data gathered depended entirely on the voluntary cooperation of parents, the findings are suspect and no conclusion can be drawn from the evaluation’s results.

The Cleveland Scholarship and Tutoring Program (CSTP)

Ohio enacted the Cleveland Scholarship and Tutoring Program (CSTP) legislation into law in March 1995.⁶⁸ It allowed the Ohio Superintendent of Public Instruction to create a pilot voucher program in Cleveland. It was expected that the $6.4 million appropriated for the program’s first year would be enough for 1,500 scholarships. The Cleveland program is largely supported by $5.25 million from Ohio’s Disadvantaged Pupil Impact Aid Program previously earmarked for the Cleveland Public Schools.

For families whose income is less than double the Federal poverty level, CSTP provides vouchers of up to 90 percent of a private school’s (including religious schools) tuition, up to a maximum of $2,250. If a family’s income is more than twice the Federal poverty level, the state pays up to 75 percent of a participating school’s tuition to a maximum of $1,875. Up to 25 percent of the new scholarships each year may be awarded to children previously enrolled in a private school.

Scholarship applicants are selected by lottery with priority going to applicants whose income is less than the Federal poverty level. Second priority goes to families whose income is less than twice the poverty level. Within these guidelines there is no income cap on participation.

The approximately 30,000 K-3 students who reside within the Cleveland School District are eligible to apply to the program. Once admitted to the program, students may receive scholarships through 8^th grade. In the first year, 6,246 applications were received for the 1,500 slots assigned by lottery in January 1996. Over the next several months, the state increased the number of vouchers that could be awarded to 1,801 because it more accurately calculated the actual tuition amounts involved. Ultimately, all public school applicants were offered a voucher. However, there was a waiting list of students previously enrolled in private schools. At the start of the 1997-98 school year, the total number of participants increased to 3,000.

In 1996-97, about 35 percent of the participants were kindergartners with no previous enrollment history, another 35 percent were formerly enrolled in the Cleveland Public Schools, and about 29 percent (up from 25 percent because of lower attrition among students already in a private school) were previously enrolled in private schools. Since some kindergarten students would have enrolled in private school even without the program, the 29 percent figure is probably a conservative estimate of the share of voucher recipients that would be in private schools anyway.

In 1996-97, about 77 percent of the scholarship students attended one of 46 religious schools, 35 of which are Catholic. The other 23 percent attended non-sectarian private schools, with over three quarters of them attending two schools. Although the law allows program participants to attend suburban public schools, none did. The vast majority of participants in the program are low-income African-Americans.

The actual cost of Cleveland scholarships to taxpayers is somewhat controversial. The Ohio Office of Management and Budget sets the average voucher payment for 1996-97 at $1,763. An analysis by the American Federation of Teachers estimates the cost of transportation at $629 per scholarship recipient, the cost of administering the program at $257 per student, and the additional state aid the program generates for each scholarship student enrolled in a private school at $543. Using these figures, the AFT estimates the total scholarship cost at $3,192 per recipient.⁶⁹

The Cleveland program is a scholarship and tutoring program. By law, the number of Cleveland public school tutorial-grant recipients may not exceed the number of students who receive vouchers. The value of tutorial grants is based on an income-related sliding scale up to a maximum of 20 percent of the average scholarship amount (i.e., the tutorial grant ceiling equals $450 for families with income below twice the poverty line and so on). In 1996-97, 542 students received tutorial assistance, and there was a waiting list of 201 students who were unable to find a qualified tutor.

Since the Cleveland voucher program has allowed religious schools to participate from its inception, its constitutionality was immediately challenged. On July 31, 1996, the Franklin County Court of Common Pleas held the program constitutional and allowed it to be implemented. On May 1, 1997, an Ohio appeals court ruled the program unconstitutional. The Ohio Supreme Court allowed the program to go forward pending an appeal. In May 1999, the Ohio Supreme Court struck down the program on procedural grounds, ruling that the program was unconstitutional for the way in which the Legislature approved the bill founding the program, not for First Amendment reasons. The program was subsequently reauthorized by the Legislature for two more years.

Soon after the reauthorization, a new challenge to the CSTP was filed by the Ohio Education Association and others on First Amendment grounds in federal court. On August 24, 1999, Federal Judge Solomon Oliver ruled that the program was unconstitutional and ordered the voucher students to return to public schools for the 1999-2000 school year. After public outcry regarding the timing of the ruling, Judge Oliver allowed the currently enrolled voucher students to remain in the program for the fall 1999 semester.   The case was immediately appealed to the 6th Circuit Court of Appeals, in Cincinnati, where it is being reviewed. A decision is expected sometime in 2000. Regardless of the appeals court’s ruling, the case is widely expected to reach the U.S. Supreme Court. 

The Cleveland Scholarship and Tutoring Program legislation requires the state superintendent to contract with an independent research entity to conduct an evaluation of the program’s impact on student performance, parental involvement, public schools, and the market supply of alternative education. The contract to evaluate the program was awarded to an Indiana University research team headed by Professor Kim Metcalf.

The legislatively mandated independent evaluation of the Cleveland Scholarship and Tutoring Program is being conducted by an Indiana University research team headed by Professor Kim Metcalf. This team published reports on the program’s first year (1996-97) in March 1998 and second year (1997-98) in November 1998.⁷⁰

To evaluate the Cleveland voucher program, Metcalf’s team compared the test scores of third-grade voucher recipients with those of Cleveland Public School students, controlling for prior test scores and family characteristics. In 1996-97, the Metcalf evaluation examined third-grade performance because that was the lowest grade for which usable test data (from second grade) existed to measure student ability prior to the voucher experiment. 

The first-year official evaluation report found that, after controlling for background characteristics, third-graders participating in the voucher program did not achieve at a higher level (on reading, language, mathematics, science, and social studies tests) than students who remained in the Cleveland Public Schools. The second-year report (1997-98) found that fourth-grade students in the voucher program achieved significantly better than their public school counterparts in science and language. When classroom variables (e.g., class size, teacher experience, and teacher level of education) are accounted for, the voucher students achieved significantly higher scores only in language.

There has been some confusion surrounding the Cleveland evaluation because of the publicity associated with the analysis of test score data from the two largest non-religious private schools in the program. On June 24, 1997, Professor Paul Peterson of Harvard issued a press release describing his team’s analysis of test results from these two schools and explaining that "a more extensive examination of the Cleveland School Choice Program is underway to determine if the gains witnessed here are being produced by the entire scholar-ship program. Results from this evaluation should be available by the fall." Professor Peterson’s press release was interpreted by some to mean that his research team was officially evaluating the Cleveland program.

In September 1997, the Harvard Program on Education Policy and Governance (PEPG) report, "An Evaluation of the Cleveland Scholarship Program" was released and drew wide publicity from a New York Times article and a Wall Street Journal article under Professor Peterson’s byline. The report itself was co-authored by Jay Greene (University of Texas, Austin), William Howell (Stanford University), and Paul Peterson (Harvard University).⁷¹

On December 27, 1997, a front page story in the New York Times reported that reading and math scores had improved in both the Cleveland and Milwaukee voucher programs. The only available source of information on test score results in Cleveland was the PEPG report. The Times story shows the degree to which the PEPG report is wrongly considered to be the official evaluation of the Cleveland program. In fact, the PEPG report is a privately funded effort that was not commissioned by the Ohio Department of Education.

Although it is titled "An Evaluation of the Cleveland Scholarship Program," the PEPG report describes test score results only from Hope Central Academy and Hope Ohio City Academy. The test results reported are expressed as percentile gains on fall-to-spring testing. It reports overall K-3 percentile gains of 5.6 (reading), -4.5 (language), 11.6 (math total), and 12.8 (math concepts).

The testing regimen whose results are described in the PEPG report was rejected as unsound practice years ago for Federal Chapter I evaluations.⁷² Most schools gain every spring and fall back the next autumn. For fall-to-spring changes in test scores to be meaningful, a carefully chosen comparison group must also be tested. The PEPG analysis has no such comparison group. Instead, it makes a comparison to low-income Milwaukee voucher applicants (whose results are not from the same test used by the Hope schools). Therefore, the results reported contribute little to an understanding of how voucher programs might affect student achievement.

Most of the PEPG report details the results of a telephone survey of program applicants. The survey results reported are generally consistent with Witte’s findings in Milwaukee that voucher program participants are well satisfied with the program. In the Cleveland survey, parents listed academic quality as their most important reason for participating.

Continue with the Next Section Vouchers, Values, and Educational Equity