Introduction
Hill and Morris (2008) conducted a five-study meta-analysis of welfare program interventions on maternal employment and child development.
Specifically, the aim was to find out whether maternal participation in welfare-based employment and supplemental-income intervention had favorable effects on the development of very young children who were from 6 months to three years of age when the programs started.
All this is a contribution to the Next Generation Project, a well-intentioned effort to investigate the effect of welfare on the children of beneficiaries. The period under investigation is the middle the to late 1990s, at a time when pilot programs were underway to afford parents on welfare employment opportunities and therefore reduce dependence on dole-outs.
Research Design
Hill and Morris justify the “experiment” label in their report title since the five studies comprising the meta-analysis — Bloom et al. (a benchmark measure in 2000 and post-assessment of cognitive and behavioral outcomes two years later); Bos et al., a one-time “dipstick” measure in 1999; Gennetian and Miller (also a once-off assessment in 2000; Huston et al., (2003 only); Michalopoulos et al. (2002 only); Morris & Michalopoulos (also a one-time measure in 2000) – were about trials (“field experiments”) in varying locations around the U.S. and Canada of an intervention consisting of jobs, income disregards or supplements. This then is the independent variable in all five studies that Hill and Morris assembled for the meta-analysis.
For clarity, an “income disregard” is the term used in the U.K. and Canada for, respectively, total tax credits and the increase in annual income that does not count for tax purposes. This does not quite have the same potent effect on family wellbeing as the tax rebates that the U.S. Congress passed earlier this year. A tax rebate check amounts to “windfall” income (not non-taxed income) that, the administration hopes, tax-filing Americans will cheerfully spend and thus help resuscitate an economy moribund since the second half of 2007.
The dependent variables consisted of standardized scores which the authors calculated from data produced by follow-up measures of the cooperating researchers. Standardization of achievement/cognitive and behavioral outcomes could have gone a long way towards the reliability of the meta-analysis were it not for the fact that the five programs employed a varying range of survey instruments: the Academic subscale of the Social Skills Rating System; single-item parent ratings of how well a child was perceived to do in school; the Peabody Picture Vocabulary Test-Revised and a math skills test for measurement of cognitive achievement; the Behavior Problem Index; and either the short or 25-item version of the Positive Behavior Scale.
Reliability and Validity
A meta-analysis usually involves an effort to combine the results of several studies boasting precisely equivalent sampling methods and research designs so that regression analysis, for instance, might benefit from the relative stability of very large samples. However, Hill and Morris acknowledge that the studies, conducted in varying locations and boasting different features, differ in important respects.
The size and policy context of income disregards/supplements differed materially. For instance, the Florida Family Transition Program (FTP) exempted the first $200 and half of any remaining income from the tax owed by a family. This is not as generous as it sounds since the state provided families on welfare no more than $303 each month for a family of three. On the other hand, the Canadian counterpart Self-Sufficiency Project (SSP) paid out 50% of the difference between actual income and threshold annual income of $25,000 that the government considered just. In the first case, a family that survived solely on welfare would gross $3,636 for the year but have to report a taxable income of just $1,718. A Canadian family that eked out subsistence-level existence at C$3,636 annually could look forward to receiving an income supplement of C$10,682, theirs to spend.
In effect, one confronts an independent variable that differs so much in implementation details and impact on family income that validity is clearly sacrificed. Hill and Norris could not even control the key independent variable for consistency, much less manipulate the IV as standard experimental study designs call for.
Since welfare is clearly not enough to live on from month to month, four of the programs had mechanisms to encourage employment-seeking. One program implemented this by reducing benefits if the recipients did not search for work or at least participate in occupational training. Connecticut simply set a time limit for being on the dole. Two programs were thought of including child care for mothers who found work. But the fifth program simply made no provision for garnering employment and, again, this degrades the reliability of cross-comparison immeasurably.
Conclusion
Ultimately, Hill and Norris are chagrined to admit that their combinative study failed to reveal any favorable effects on child development. They could only point to isolated “domain- and age-specific effects” (2008, p. 1557) with respect to positive social behavior for one age group and improved scholastic achievement, solely for those who were 2 years of age when the interventions commenced. Neither theory nor the varying circumstances of each study permitted the authors to conclusively explain these unremarkable findings.
In fact, this whole undertaking is a non-experimental study because Hill and Morris were at least twice removed from, and therefore exercised no control over, both parents and children. Never having been in the research settings and hence, in no position to manipulate the IV’s, the researchers were relegated to processing the information secondhand. Such are the limitations of correlational, passive, naturalistic, and observational studies, widely-publicized examples of these being those reporting risk factors for lung or colon cancer. Such non-experimental studies can never make conclusions on cause and effect, the very same trap that Hill and Morris fell into.
References
Hill, H. D. & Morris, P. (2008). Welfare policies and very young children: Experimental data on stage–environment fit. Developmental Psychology, 44 (6): 1557–1571.