Educational Vouchers: A Review of the Research

by
Alex Molnar

Center for Education Research, Analysis, and Innovation
School of Education
University of Wisconsin-Milwaukee
PO Box 413
Milwaukee WI 53201
414-229-2716

October, 1999

CERAI-99-21

Educational Vouchers: A Review of the Research 
October 1999
CERAI-99-21

Alex Molnar
Professor, Department of Curriculum and Instruction University of Wisconsin-Milwaukee 

This document combines excerpts from two reports: "Smaller Classes -- Not Vouchers -- Increase Student Achievement" (Harrisburg, Pa.: Keystone Research Center, March 1998); and "Smaller Classes and Educational Vouchers: A Research Update" (Harrisburg, Pa.: Keystone Research Center, June 1999). Both documents are available on the website of the Center for Education Research, Analysis, and Innovation at http://www.uwm.edu/Dept/CERAI

Table of Contents - Exercept 1 Historical Background Educational Choice Enters the Mainstream The Battle Over Vouchers Today The Milwaukee Parental Choice Voucher Program The Debate Over the Achievement Effect of the Milwaukee Voucher Program Box 3: Public vs. Private Schools Why Different Researchers Reach Different Conclusions The Witte Evaluations Box 4: Sorting through the Conflicting Voucher Results The Greene, Peterson, and Du Evaluation Box 5: When are Significant Results Not So Significant? The Rouse Evaluation Milwaukee’s Private Voucher Program -- PAVE Box 6 - A Case Example of the Relative Cost and Performance of Public and Private Schools The Cleveland Scholarship and Tutoring Program (CSTP) Vouchers, Values, and Educational Equity Box 7: Does Money Matter? School Spending and School Outcomes References

Table of Contents - Exercept 2 The Argument Over Vouchers The Milwaukee Parental Choice Voucher Program The Achievement Effects of the Milwaukee Voucher Program The Cleveland Scholarship and Tutoring Program (CSTP) Official Evaluation Results for CSTP Private Voucher Programs  Private School Vouchers (Con't) Vouchers and Educational Equity  References

Why Different Researchers Reach Different Conclusions

When researchers in ideologically polarized debates disagree, general readers who want to weigh the "facts" for themselves can end up confused and not knowing what to think. To avoid this problem, this section walks the reader through the findings of the three efforts to analyze the Milwaukee experience. It seeks to explain in everyday language how essentially the same underlying data can lead different analysts to different conclusions.⁴⁵

There is actually less disagreement than meets the eye among the findings of the three Milwaukee evaluations. When researchers of the MPCP program use similar methods, they come to the same basic conclusions.

Researchers of the Milwaukee voucher program arrive at conflicting results for two basic reasons: (1) they use different definitions of the reference or control group to which the performance of voucher program participants should be compared, and (2) they use different methods to control for family background and student ability. All of the researchers must contend with the relatively small samples of students in the data bases analyzed. All must address the shrinkage (or "attrition") of their sample due to student mobility and missing data. All of them also lack any model of what actually goes on in schools or of the educational features (such as small class size or an innovative curriculum) that may generate good outcomes.

Table 2 Findings of Three Studies of the Milwaukee Parental Choice Program
	Witte	Greene, Peterson, and Du	Rouse
Main Comparison^46	Compares voucher students’ achievement with that of a random sample of Milwaukee Public School (MPS) students, controlling for observed individual and family characteristics.	Compares voucher students’ achievement with that of unsuccessful applicants who returned to the Milwaukee Public Schools.	Compares achievement of successful applicants for vouchers with that of a random sample of Milwaukee Public School students, controlling for an estimate of innate ability and family influences.
Reading Findings	No significant difference between voucher students’ achievement and that of the MPS comparison group.	In their 1997 "main analysis": 2-3 percentile rank advantage for voucher students in year four. Conventional levels of statistical significance approached only when 3^rd and 4^th years are jointly estimated. When background characteristics are controlled for, voucher students’ advantage in 1^st and 3^rd years approaches significance.	Similar to Witte: no statistically significant difference between successful voucher applicants’ achievement and that of the MPS comparison group.
Math Findings	No significant difference between Choice students and MPS sample.	5-11 percentile rank advantage for voucher students over unsuccessful choice applicants in years 3 and 4. Conventional levels of statistical significance achieved in 4^th year and in joint estimate of 3^rd and 4^th years.	Similar to GPD: statistically significant advantage in years 3 and 4 for students selected for Choice schools. Effect size of 0.08-0.12 per year.
Main Statistical Limitations	? Does not control for unobserved individual differences. ? Voucher students who remain in program may be a non-random high-scoring group. ? Does not include school variables (e.g., class size, curricula).	? Control group of unsuccessful voucher applicants who return to MPS is a small and shrinking sample (26 in year 4). ? Control group may be a non-random, low-scoring group. ? Voucher students who remain in program may be a non-random, high-scoring group. ? Does not include school variables (e.g., class size, curricula) that may explain observed differences.	? Successful voucher applicants have more educated parents with high expectations: improvement in math scores over time might take place without voucher program. ? Does not include school variables (e.g., class size, curricula) that may explain observed differences.

The Witte Evaluations 

In his five evaluations of the Milwaukee program, Witte compares voucher students’ average test scores and changes in test scores to the same figures for two other groups: a random sample of Milwaukee Public School students and a random sample of low-income Milwaukee Public School students. Since neither of these two groups are genuine "control" groups for Choice students, Witte also combines the Choice and non-Choice students into a single sample and uses statistical controls to take account of the impact of family and individual differences (e.g., prior test performance, family income, race, and gender) on test scores. 

Continue with the Next Section Box 4: Sorting through the Conflicting Voucher Results