Highlights The research and evaluation literature used in this review refer to a very wide range of activities from sport, music and drama to remedial reading, homework assistance and more therapeutic or preventive interventions aimed at reducing drug use, teen pregnancy and criminal behaviour. Different jurisdictions shape OSS provision to focus on different outcomes. Some policies and programmes focus on outcomes for children and others on family or societal level outcomes. Outcomes can also have an educational, developmental or care focus. The outcomes do not need to be exclusive. While much international literature distinguishes between the concepts of care and education, these are not easily separated.

In this review the term ‘OSS’ includes before and after school care and holiday programmes for school-aged children, including children at or over the school leaving age.

This review does not include programmes where the care is incidental to the activity provided such as music or sports lessons, even though parents are likely to expect that their children’s care and safety are being attended to during these activities. Therefore, this review includes OSS organised programmes for school age children that include music and sport among a variety of activities, but not privately arranged instruction on any topic or involving any activity that is not part of an organised programme.

Each country has its own terminology and patterns of provision that reflect different emphases (Meagher Lundberg and Podmore, 1998). Therefore, the research and evaluation literature refers to a very wide range of activities from sport, music and drama to remedial reading, homework assistance and more therapeutic or preventive interventions aimed at reducing drug use, teen pregnancy and criminal behaviour.

International literature refers to ‘after-school programmes’, ‘after-school care’, after-school clubs’, ‘out of school programmes’, ‘out of school time (OST) programmes’, ‘school-age childcare’ and ‘out-of-school-hours care’ (OSHC). The literature tends to include most activities organised for children outside compulsory school hours that do not involve parental care or supervision.

As a general rule literature from the US refers to OST, literature from the UK refers to OSHC and literature from New Zealand to out of school care and recreation (OSCAR) or OSS. In this report, the terminology used reflects that used in the literature being discussed.

An issue that arises from the different types of OSS internationally is whether it is useful to distinguish between the concepts of ‘custodial care’ and ‘non-school education’ where school-age children are concerned. In New Zealand these two concepts have been merged in early childhood education (ECE) since the mid 1980s, following the recognition that despite parental intentions or preferences learning occurs whether the provision focuses on care or education. (For further discussion, see Minister of Education (2004)). This understanding underpins the obligation of providers of ECE to comply with quality standards set high enough to ensure good outcomes for children across all aspects of their participation in the different services.

Internationally, the distinction between care and education originates in part from parental preferences (shaped by the influences of class, cultural background and family style), which, until recently, have been potent factors in the design and provision of services for school-age children. In the US, funders of services are now increasingly demanding evidence of positive educational outcomes from services that were originally established to supervise children when their parents were at work.

While school-age childcare is not characterised as ‘education’ in the UK, neither is the emphasis on custodial care. In the UK the notion of ‘care’ tends to arise within research and evaluation literature on low-income or ‘problem’ children (who are more likely to be working-class children), whereas the notion of ‘study support’ or ‘extracurricular enrichment opportunities’ after school appears to be associated with social expectations for middle-class children. Contrary to the impression its name might imply, however, ‘study support’ activities offered by schools appear to include physical education and sports, music and drama, and information technology and computing, as well as homework, revision and study skills activities (Market Opinion Research International, 2004).

An increasing body of research shows that services must offer a range of enriching experiences, as well as meeting children’s basic physical and emotional needs, for children to experience positive outcomes. It appears that the needs of children, parents and wider society can be met simultaneously if services meet acceptable levels of quality.

Highlights Few research and evaluation studies provide statistically significant results about the outcomes for children and families from their use of OSS. While several evaluations demonstrate positive outcomes for children, the researchers express caution about inferring causality from the results. Jurisdictional differences in the literature Little New Zealand research and evaluation on OSS exists. However, the inclusion of New Zealand in an Organisation for Economic Co operation and Development (OECD) policy review (OECD, 2004a) has enabled New Zealand’s situation to be compared with that of other OECD countries (OECD, 2004b). There is a clear difference between the US and the UK in approach and the type of studies done. Most of the more experimental research and evaluation studies yielding statistically robust results originate in the US and focus on academic outcomes for participants. Conversely, much of the research and evaluation from the UK involves less experimental research and focuses on general family and community outcomes. Most evaluations in the US have also focused on OST programmes directed at disadvantaged children. Although the evaluations found positive outcomes for these children, these outcomes may not necessarily be experienced by more advantaged children.

The literature considered for this report included:

• refereed journal articles
• published research reports
• government reports
• statements and reviews of policy
• reports from international bodies such as the OECD
• statements from research and provider organisations
• academic reviews and commentaries on research, including evaluations by provider groups.
  

The most important websites searched are listed in Appendix A.
As well as published research, the literature surveyed included reviews of research and commentary. Particularly helpful were the literature reviews by Miller (2003), Peter (2002), and Malcolm, Wilson and Davidson (2002). Several publications provided smaller literature reviews and lists of research studies, their findings, and an assessment of their research quality (e.g. Bodilly and Beckett, 2005; Kane, 2004; Beckett, Hawken and Jacknowitz, 2002; Blau and Currie, 2003; Miller, 2003; Fashola, 1998).

There is little New Zealand research on OSS, apart from the two evaluations of Out of School Care and Recreation services (Saville-Smith et al., 1997; BRC Marketing, 2004). Meagher Lundberg and Podmore (1998) include New Zealand in their review of OSS in selected countries. Meagher Lundberg and Podmore (1998) is a major source of background information and gives an account of OSS in the late 1990s.

Since 2000, the inclusion of New Zealand in the OECD policy review Babies and Bosses, has updated the information and commentary on OSS in New Zealand, as well as providing a contrast with 12 other OECD countries (OECD, 2004a; OECD, 2004b). The main message from the review is that in New Zealand OSS need to be expanded through altered funding arrangements to enhance opportunities for parents, particularly mothers, to participate in employment. Although specific outcomes are not discussed in the review, it acknowledges that high-quality care is needed for school-age children, and that unregulated care is associated with concern about quality (OECD, 2004b). The implicit high-level outcome sought is increased labour market participation facilitated by a climate of enhanced family-friendly policies and practices at all levels, rather than specific outcomes for school-age children (except, presumably, for safety). Since the review is not research based, it has not been included in any further detail in this report.

The most noticeable feature of the literature is the difference in approach between the US research and evaluation literature and policy, research and evaluation documents from the UK, European Union and, to some extent, the OECD. Less material was available from Australia, Canada and Scandinavian countries. Most of the experimental research and evaluation studies originate in the US, where the desire to measure outcomes to a high level of statistical accuracy is the strongest. (See Appendix B for short definitions of experimental and quasi-experimental research designs.)

The most recent US research and evaluation literature reflects the changes caused by the strong policy direction from the US government enacted in the No Child Left Behind Act of 2001. This Act incorporated a direction that all government-funded services for children must be underpinned by science-based research (SBR). This requires that the ‘gold standard’ of experimental research (randomised controlled trials) be the aim of all programme evaluations funded by the 21st Century Community Learning Centers (21st CCLCs). The 21st CCLCs are an agglomeration of different OST programmes in the US originally funded in 1988, then reconfigured under the No Child Left Behind Act.

The programmes were initiated in response to continuing and heightened concern about:

• the well-being and educational attainment of children who had been poor performers at school (Huang et al., 2000) or were from low-income, disadvantaged backgrounds (Blau and Currie, 2003; Miller, 2003) and community pressure to promote youth development by situating youth in safe environments
• preventing youth from engaging in delinquent activities
• teaching youth general and specific skills, beliefs and behaviours
• providing youth with opportunities to develop positive relationships with peers and mentors.
  

The UK school-age childcare research and evaluation is described as ‘thin, mixed and inconclusive’ (Gammage, 2003: 5), including almost no systematic evaluation involving experimental or quasi-experimental research designs.

In the decade since the mid 1990s the UK government has developed and implemented a 10-year strategy aimed at helping parents to reconcile work and family responsibilities called Choice for Parents, the Best Start for Children. The strategy was announced in April 2004 and followed the early childhood strategy Every Child Matters (UK Government, 2004). One of the government’s aims is to ensure that good quality early years or school-age childcare from 8.00 am to 6.00 pm eventually becomes available for all children whose parents want to use it. This policy goal reflects a whole-of-government approach in which children’s interests are balanced with those of parents and the state. This high-level approach has more of a conceptual link with the situation in New Zealand than the current US focus on children’s academic improvement as a result of OST programmes.

Cummings et al (2005) evaluated the first year (2003/04) of a further UK government initiative, Full Service Extended Schools (FSES). FSES is part of the government’s overall vision for all schools to offer a core set of extended activities by 2010 and is intended to provide a comprehensive range of services on a single site. Such services might include health services, adult learning and community activities, study support, and 8 am to 6 pm childcare. In the first year, 61 projects were funded. They were all behaviour improvement programmes, suggesting a strong link between the behaviour problems in the area and the community response through FSES. In this initiative, associated with the Choice for Parents, the Best Start for Children strategy, the UK government is committed to integrated, wrap around services that inevitably blur the lines between schools and the community.

The review of OSHC literature by Malcolm, Wilson and Davidson (2002) for the Scottish Council for Research in Education contains no British examples of experimental research and evaluations of the type found in the US, although the National Foundation for Educational Research is conducting a longitudinal study of OSHC learning activities. The authors assert that study support, mostly provided by schools, is one branch of out of school care. They note that the impetus for developing out of school care services in Britain arises from the goals of the Out-of-school Childcare Grant Initiative of the mid-1990s. That initiative’s purpose was to improve the quantity and quality of OSHC to facilitate increased labour market participation among parents of school-age children who wanted to combine employment and family life. It is not surprising, then, that they found the major benefit of the care was enabling mothers to engage in the labour market. Apart from this study, however, few studies presented evidence of other types of benefits arising from school-age childcare, although some studies assumed such benefits exist.

Highlights Research and evaluations of OSS have looked at a range of outcomes for participants including: care and safety, educational or academic outcomes (particularly research from the US), prevention and youth development outcomes, and the link between participation in OSS and these outcomes. Almost none of the research and evaluation literature examined outcomes for the families of participants. Various studies have shown specific outcomes associated with participation in OSS including: improved academic outcomes (such as better attitudes towards school, better performance at school, higher school attendance, and less disciplinary action) linked to overall programme participation, but not with specific activities within programmes improved personal and social development (such as improved self-esteem, development of leadership potential) improved youth development outcomes (such as increased avoidance of drug and alcohol use, less delinquency and violent behaviour, increased skills for coping with peer pressure and improved social skills) a clear link between ongoing attendance in OSS and the above benefits. Increased labour market participation of parents was the only outcome for the wider family noted in the literature. However, most evaluations qualified results by saying it was not possible to detect the extent to which positive results were due to participation in OSS rather than other factors in a child’s life.

Nancy Peter (2002), in a review of US research literature, makes the case for including outcomes in the definition of a quality OST programme. ‘This means selecting strategies, techniques, approaches, and activities that are likely to produce results as well as engage and satisfy participants and their families’ (Peter, 2002: 5). She also points out that not all OST programmes can or should address the four outcomes she has identified: academic achievement; risk reduction; successful employment in the longer term; and improved life skills. The choice of outcomes will reflect the funding agency’s goals or provider organisation’s purpose.

In their critical review of OST programme evaluations in the US, Bodilly and Beckett (2005) found that claims of effectiveness fall into four outcome areas: school-age care; attitudes towards or changes in academic achievement and attainment; social and health behaviours; and social interactions. Nevertheless, their study found few well-designed studies from which firm conclusions could be drawn.

In the new science-based research (SBR) climate, many US programmes are placing increasing focus on academic achievement, access and accountability for funding – dubbed the ‘three A’s’ by Steve Gunderson in a Harvard Family Research Project publication (HFRP, 2003). It appears that many existing programmes are now aimed at providing remedial education services for low achievers, rather than for students of mixed abilities and backgrounds. Another set of programmes is clearly aimed at reducing crime, by attempting to deal with delinquent behaviour early in the young person’s life. Many of these programmes would not necessarily be considered as OSS in New Zealand.

The National Institute on Out-of-school Time (NIOST) makes the case for providing OST programmes on several evidential grounds: improved health and safety, particularly by decreasing levels of obesity; declining juvenile crime rate; and the benefits to children of participation (NIOST, 2005). The NIOST cites a 2004 study by Duffett and Johnson that found parental reasons for wanting OST programmes varied from giving children a break from academic study after school (54 percent) to providing an additional focus on academic skills (38 percent).

Some evaluations from the US examined the link between specific outcomes and specific activities within a programme. Other evaluations examined programmes that were aimed at achieving a range of outcomes rather than one specific one. However, almost all the outcomes identified in the US literature are child focused.

Outcomes researched include:

• care and safety (this can include, in a de facto manner, the ability for parents and caregivers to participate in the labour force)
• educational or academic outcomes (including an improved level of engagement with the school process and improved academic grades, sometimes in specific topics such as English or maths)
• prevention and youth development outcomes (including: increases in the number of supportive adult relationships in a child’s life, the development of leadership potential in the child, and fewer behavioural problems
• the link between participation in OSS and these outcomes.
  

The need to provide a safe environment for participants is a common theme across all studies. Much expert commentary stresses the need for childcare to be a ‘safe haven’ for children (Fashola, 1998).

No experimental research and evaluation findings have been reported on the level of physical safety experienced by children in OSS. However, a major driver of the development of services has been the provision of a safe environment for children outside school time. This is mentioned most often by those who acknowledge that parents want to take up opportunities for employment without leaving their children to care for themselves.

The only UK study Malcolm, Wilson and Davidson (2002) uncovered that found clear benefits of care and had a childcare focus (rather than a focus on parental labour market participation), was undertaken by Dench and O’Regan (1997, cited in Malcolm, Wilson and Davidson, 2002). In a survey of 282 parents, almost all said their children were happy attending an after-school club. Three-quarters reported social benefits for their children, especially in social interaction and development, and for those living in isolated areas.

Remedial education is a very strong theme in many OST programmes in the US. Huang and colleagues (2000) note in their synthesis of longitudinal research on the LA’s BEST (Los Angeles’ Better Educated Children for Tomorrow) programme that there was a realisation in the late 1980s that parents of children who often did poorly at school were increasingly likely to hold full-time jobs. This led to policy strategy to:

‘address comprehensively the problem of educational achievement and the increasing need for student supervision by creating and sustaining an after-school program for students. Instead of addressing only custodial care, this program … has been designed to provide activities to meet specific educational, social and motivational goals.’ (Huang et al., 2000: 1–2)

It is important to note that sites for the LA’s BEST programme are selected on the basis of identified needs that are related to: low achievement of children within the community; the community’s low economic status; and the neighbourhood’s high gang participation and crime rates. This is not a service for random groups of the population or for children from all schools.

Little and Harris (2003) reviewed the 25 OST programme evaluations in the Harvard Family Research Project (HFRP) database that used an experimental or quasi-experimental design to determine academic outcomes. (The HFRP database is at www.gse.harvard.edu).  Their results were statistically significant – academic outcomes were linked to overall programme participation, but not with specific activities within programmes.

Similar to the findings of Huang and colleagues (2000), the academic outcomes included:

• better attitudes towards school
• better performance at school
• higher school attendance
• less disciplinary action (e.g. fewer suspensions).
  

The After School Corporation’s (TASC’s) evaluation of their programme reported that participants showed gains in educational performance, attendance, personal skills and enjoyment of after-school learning, especially when the student participated regularly over two years (TASC, 2005). Among older students, whose school attendance rates tend to decline, the attendance rate for TASC students declined more slowly. The gains experienced by participants is greater than that made by comparable students who are not involved in the programme, but the authors caution that these results could be due to factors other than the programme.

Spielberger and Halpern’s (2002) survey of literacy practices in an OST programme in Chigago and Seattle suggests that while teaching literacy to school-age children is not the purpose of an OST programme, such programmes can nevertheless support and nurture the social and cultural dimensions of literacy in a way that is complementary to school provision. These researchers consider that OST homework time is an especially important source of support for children whose parents cannot help them because of language difficulties, as long as it does not crowd out other enriching activities.

More studies have been carried out into the benefits of study support, mostly for the Department for Education and Skills (DfES) in the UK. Most evaluations of pilot study support programmes reported benefits in personal and social development, increased motivation to read and do maths, improved self-esteem and improved social skills, although data was limited. One study reported improved attitudes towards school generally. Study support programmes that involve structured learning activities are also associated with good examination results.

Despite the general evidence of positive outcomes and improved examination results, Malcolm, Wilson and Davidson (2002) caution that it is not possible to detect the extent to which positive results reflect the study support programme, rather than other factors in a child’s life. Consistent monitoring and data collection on academic achievement as well as social and personal development has not been a strong feature of study support evaluations. The authors remark that if such benefits are real, then they are impressive.

Many of the claims are weakened by being unsupported by strong evidence and the studies are often subject to selection bias. Malcolm, Wilson and Davidson also note that many of the interventions were targeted in some way, thus leaving out the children ‘in the middle’ whose circumstances might also improve from participation in a service. For example, breakfast clubs in the UK are claimed by parents to be of great value and demand is high, but it is not known what the basis of the parental approval is (e.g. parental convenience, improved nutrition or something else). More systematic research is needed.

A qualitative study for the DfES on the impact of OSHC (Barker et al., 2003) involving site visits, a questionnaire-based survey and interviews, found that OSHC has ‘generally positive’ social and educational impacts on younger children (aged under nine), children from minority ethnic groups and children with special needs. However, older children found the services too oriented toward younger children and not interesting or challenging enough. These results question the extent to which clubs for a group of children aged four to 11 can respond to the needs of children aged over eight.

Little and Harris (2003) reviewed results from the 27 OST programme evaluations in the HFRP database. Twelve of these evaluations used an experimental or quasi experimental design to examine prevention outcomes such as changes in sexual behaviour, feelings of personal safety, use and abuse of drugs and alcohol, and overall improvements in physical health. Fewer programmes focus on these issues compared to educational issues, but from the 12 evaluations available, results indicate that participation in OST programmes is linked to:

• increased avoidance of drug and alcohol use
• less delinquent and violent behaviour
• increased knowledge of safe sex and how to avoid sexual activity and pregnancy
• increased skills for coping with peer pressure.
  

Hall and colleagues (2003) reviewed how OST programmes in Boston could most effectively promote positive youth development as a support to academic achievement. They outline key ways in which quality programmes with a positive youth development approach can help to overcome critical barriers to learning, thereby supporting academic achievement and well being.

The youth development approach emphasises the development of social, moral, emotional, physical and cognitive competence during adolescence. The review included a wide variety of programmes. The key elements of effective OST programmes were identified as:

• safe, stable places
• basic care and services
• caring relationships
• relevant, challenging experiences
• networks and connections
• high expectations and standards
• opportunities for voice, choice and contribution
• personalised, high-quality instruction.
  

Little and Harris’s (2003) review of OST programme evaluations from the HFRP database included 15 evaluations where evidence of youth outcomes had been sought using experimental or quasi-experimental research. Participation in an OST programme was linked to positive youth development results:

• fewer behavioural problems
• improved social and communication skills and relationships with others (peers, parents and teachers)
• more community involvement and a broader world view
• increased confidence and self-esteem.
  

In a different environment, schools in the UK evaluated the outcomes of full service extended schools. Effective multi-agency working arrangements were reported. Since local authorities in the UK administer education services, both the local communities and the schools involved see full service status as a means of addressing some of the out of school difficulties faced by pupils.

Full-service extended schools are intended to produce positive outcomes for three groups: school pupils, their families and the communities in which they live. Several difficult measurement issues arose, not the least of which was the lack of a single evaluation strategy. This has led to patchy and essentially anecdotal evidence. It is not possible to make claims about the overall effectiveness of FSES or that particular schools have made a fundamental difference to children, families and the community. However, almost all schools were able to produce evidence of positive effects across a range of outcomes including:

• improved behaviour and decreased vandalism and other crime
• improved attitudes to learning and increased academic standards
• fewer teenage pregnancies and improvements in sexual health
• less alcohol, cigarette and other drug abuse
• increased employment by parents (Cummings et al., 2005).
  

Posner and Vandell’s (1994) empirical research in the US provided a foundation of evidence for subsequent research into whether attending OSS makes a positive difference for children. They found that when maternal education, race and family income were controlled for, attending a formal after-school programme was associated with better academic achievement and social adjustment than attending three other types of after-school care (mother care, informal adult supervision and self-care).

A study of third-graders from almost twenty years ago (Vandell and Corasaniti, 1988) found significant differences for children who attended day-care centres after school. These children were less popular and did worse in standardised tests than did ‘latch-key’ (i.e. self-care) children, mother-care children or children who went to sitters. Their analysis of this finding proposed that children with problems may have been more likely to go to day-care, since their parents may have wished for a higher level of supervision. In addition, the quality of day-care was questionable and, in their view, evidence from teachers and parents revealed a high level of stigma associated with attendance. These results demonstrate the problems with research into OST programmes where causality cannot be determined. Subsequent analysis found other factors that would have influenced the result.

In another study of self-care and sibling-care arrangements, Berman and colleagues (1992) found that, despite public concern, children in self-care did not show lower levels of competence than other groups. Indeed, many reported high self-esteem derived from their independence and ability to manage the arrangement. However, self-care children were found to be more isolated socially. Children cared for after school by siblings reported lower competence scores, indicating a risk of low self-esteem. The quality of sibling care must be an important issue here.

These findings were countered by Goyette-Ewing (2000) whose cross-sectional study failed to find any benefits (in terms of adaptive behaviour or competence) of leaving children without adult supervision after school. Her study found that self-care children who ‘hang-out’ after school had lower school achievement than their peers in other care arrangements. The mother’s employment status was the major difference in the families of the two groups, with mothers of self care children much more likely to work full-time than other mothers.

Nine of the HFRP evaluations reviewed by Little and Harris (2003) examined their outcome findings in light of OST programme participation, with eight finding statistically significant positive relationships between participation in a programme and positive academic and youth development. Those who attended most consistently and for the longest time were more likely to have learned a specific skill and achieved a higher score on a developmental assets assessment than those who attended less consistently or for shorter periods of time.

Miller’s (2003) review of US research and evaluation evidence also found clear evidence that:

• students who lack adult supervision after school are at greater risk of poor outcomes than other students
• students who participate in extracurricular activities and community service have higher academic achievement than other students, and this holds true even when other factors that affect school success are considered
• students who attend after-school programmes are more engaged in learning than other students
•  increased engagement in learning can result in higher academic performance
• after-school programmes can play a role in reducing racial and income achievement gaps.
  

As part of the HFRP research, evidence about three aspects of attendance, intensity (number of days per period), duration (number of years) and breadth (number of activities), was reviewed to explore how participation in OST programmes can lead to positive benefits (HFRP, 2004). The authors stress that participation means ‘active’ participation, not merely attendance. They found that attendance intensity was not always related to outcomes, but that most studies suggest a significant relationship. Attending a programme for at least two years is positively related to positive youth outcomes. Few researchers have explored breadth, but evidence from those who have suggests elementary school children who participated in three or more different activities had higher grades and academic test scores than non-participants or children who participated in fewer activities.

The San Francisco Beacon Initiative was established in the mid-1990s to address concerns about the poor outcomes of low-income youth. It placed importance on articulating an underlying ‘theory of change’ to establish the logic by which an intervention will be expected to generate positive outcomes (Walker and Arbreton, 2002). Walker and Arbreton’s logic followed this path.

• If Beacon centres provide safe and welcoming settings that have high-quality activities, young people will participate.
• Youth who participate will have positive developmental experiences.
• Positive experiences will contribute to youth well-being and productivity.
  

The programme leaders were cautious about assuming that participation would lead to changes in academic outcomes, but collected data from early on to assess the relationship between participation and academic outcomes. Although this evidence did not show increased academic grades for Beacon participants compared with non-participants, there was plenty of evidence of other benefits, such as increased levels of adult support, leadership experiences, and an apparently protective influence deterring the typical decline in engagement with school among middle-school children. The researchers concluded that academic grade scores may not have increased because of a lack of specific academic content in some programmes and low average attendance rates (one to three times a week).

At a very general level, the evaluation of the Extended-Service Schools initiative in the US (Grossman et al., 2002) suggests that:

• participation in school-based, after-school programmes is associated with behaviour that could help youth stay out of trouble, and
• participation in after-school programmes is associated with positive effects on behaviour and attitudes towards school (although the study was too early to determine whether participation affected students’ grades and test scores).
  

These findings reinforce the message that development outcomes can result from participation, even though academic changes might not. They also reinforce the importance of identifying the programme’s desired outcomes in the programme’s early design phases to ensure all aspects of it will support the achievement of those outcomes.

Highlights Promoting positive outcomes for children as a result of participating in out of school services requires more than providing a safe environment.  Factors that can influence outcomes for participants cover aspects of both the quality and the focus of the service and include: the range and type of activities, including the variety of activities and specific activities the quality of adult/child interactions the attractiveness of the programme a focus on youth development, and a focus on enrichment. A high quality service promotes positive outcomes for participants. Conversely, a poor quality service can have a negative impact on outcomes. No factors that influence the outcomes for the family of participants were identified in the research and evaluation.

The Rand Corporation in the US (Rand Corporation, 2001), note that while research and evaluation is limited, evidence suggests that specific characteristics of OST programmes are associated with positive outcomes. These are flexible programming, a favourable emotional climate, and a sufficient variety of activities.

The main body of research and evaluation literature concurs with these. Factors highlighted in this section include, the quality of adult/child interactions and other quality indicators, the range and type of activities, the attractiveness of the programme, a focus on youth development, and a focus on enrichment.

Further to these, it is important to identify the desired outcomes at an early stage in the design of the OSS and the evaluation strategy.

Posner and Vandell’s (1994) research found that children in formal programmes spent more time in academic activities and enrichment lessons and less time watching television and playing outside unsupervised. They also spent more time doing activities with peers and adults and less time with siblings than did other children. This strongly suggests that the quality of the interactions with adults will make a difference to child outcomes, and that the more competent adults are in these settings, the more beneficial the effect on children will be. This finding is supported by the evidence of Berman and colleagues (1992) who found that children cared for by siblings had low self-competence scores and poor social development.

Halpern (2000) identifies issues associated with the lack of high-quality provision in some OST programmes. Issues include not just the choice of activity and the children’s experiences, but also the nature of the environment of the OST programme for the adults involved (such as pay rates (usually very low), reliance on volunteers (often untrained), the availability of training (often none), inadequate staff–child ratios, the need for resources such as encyclopaedias and Internet access for homework), and the financial support needed to maintain an assured level of service supply.

Bodilly and Beckett (2005) found that among the very few statistical correlations available on quality issues, Deborah Vandell and her colleagues have conducted the most influential research. In different samples of low-income children they found a strong and significant correlation between positive outcomes such as better adjustment in school and positive staff–child interactions, and factors like total enrolment, staff–child ratios and staff education. They cite Rosenthal and Vandell’s 1996 study that investigated the relationship between programme features and programme quality, and found that a higher number of observed negative staff–child interactions were associated with larger staff–child ratios, lower staff education and fewer programme activities. Children reported a poorer emotional climate in programmes with larger total enrolment and a higher number of observed negative staff–child interactions.

Nancy Peter (2002) noted that most quality OST programmes include varied combinations of similar activities, such as project-based learning; service learning and community service; mentoring; tutoring and homework help; cultural enrichment opportunities; technology training; connections to families, schools and the community; interpersonal development; on-the-job training; and career readiness. This list illustrates a focus far beyond the ‘safe haven’ concept that formed the basis of many early OST programmes.

Commentary from the US suggests that there is an increasing demand for research and evaluation that links specific programme activities with children’s outcomes, particularly academic outcomes. Bouffard and Little (2003) reviewed 27 OST programmes for which information about activities was available. Of these, 19 programmes offered multiple activities. A further eight focused on single activities (such as maths and science extension), where participation in the activity was mandatory. Six of these eight focused on academic activities, one on fostering positive adult relationships, and one on developing a positive self concept. It is reasonable, the authors assert, to assume most single-activity OST programmes are oriented towards increasing academic achievement. Non academic activities offered by single-focus providers included youth development skills such as leadership skills, social and communication skills, conflict management skills, and self-concept development.

Despite the focus on academic results from most of the single-activity programmes, academic achievement was also a specific focus of most multi activity programmes, with the majority offering some form of academic enrichment. Academic activities were likely to involve homework help, tutoring services, test skills tutoring, and other forms of academic enrichment. Among the 27 OST programmes reviewed, 21 offered academic activities, 14 offered arts, 14 offered sports or exercise, and 13 offered computer skills. Only four programmes involved unstructured play. Activities such as field trips, youth development, career skills, and community service were offered by eight or fewer services. Such a wide range of activities enables providers to target several goals. The findings demonstrate that many OST programmes serve a larger need than basic childcare or recreation, providing important services and skills training.

In 1999, Cooper and colleagues reported on a survey of 424 US students in grades six to 12 (ages 11 to 17) to measure their participation in specific activities outside school. Activities included homework, television viewing, extracurricular activities (e.g. academic clubs and sports), other structured after-school groups (e.g. church groups and scouts), and jobs. They found that, generally, more time in extracurricular activities and other structured groups and less time in jobs and television viewing were significantly associated with higher test scores and class grades. The results of a correlation analysis between homework completion and achievement test score results were confused by differences in reporting between the students and their parents. However, the data suggested that higher test scores were associated with more time spent on homework (as reported by the students, but not by the parents). Both television viewing and time spent in jobs were negatively related to achievement scores. The authors concluded that even though their cross-sectional data do not permit strong causal inferences, there is clear evidence that the relationships between activity types and achievement scored were not simply artefacts of student variables such as gender and ethnicity, family income and level of adult supervision after school.

These findings are consistent with the theory that forming a positive identification with school through extracurricular activities can positively influence school performance (Marsh, 1992 cited by Cooper et al. (1999). But further regression analysis showed that this assertion must be qualified, because there is an inverted U-shaped relationship between extracurricular activities and school performance. Cooper and colleagues found that activities may become detrimental to achievement if identification with the activity becomes so strong that it displaces the broader school identity, or if the time investment is so strong (e.g. long hours of employment) that it leaves little time outside school hours for academically related activities such as homework. Although their data do not demonstrate strong causality, ‘[it] seems safe to conclude that parents and educators can profitably focus on student after-school activities as a potentially important influence on achievement’ (Cooper et al., 1999).

The Positive Activities for Young People (PAYP) programme (CRG Research and evaluation Ltd, 2005) in the UK is a government initiative that has been evaluated since 2000. PAYP is a programme that works with people aged eight to 19 who are at most risk of social exclusion, committing crime and being the victims of crime. PAYP’s key objectives are to reduce crime and antisocial behaviour, support young people back into education and help them stay there, and support young people to move from primary to secondary school. Activities include high-quality arts, sports and cultural activities and opportunities for personal development. Although there are limitations on the conclusions that can be drawn from the data, the evidence is that they are having a positive impact on participants.

The goals of LA’s BEST programme reported on by Huang and colleagues (2000) are:

• a safe environment
• enhanced opportunities through an integrated educational support structure
• educational enrichment activities to supplement and deepen the regular programme
• recreational activities
• interpersonal skills and self-esteem development.
  

Associated activities include homework assistance, library activities and a range of other activities including field trips, sport, computing and the performing arts.

The results of Huang and colleagues’ synthesis of research showed that for students with long-term involvement (at least four years) in an OST programme, higher participation was related to better school attendance for all cohorts and significantly related to positive achievements in standardised tests of maths, reading and language arts. This was so even when the influences of gender, ethnicity, income and language status were controlled for. Six separate evaluation studies also contributed information, showing that:

• the children felt safer after school and their parents felt their children were safer
• the children said they liked school more since they joined the programme and were more engaged in school
• positive relationships between adults and children had been well established
• the children reported higher aspirations for the future
• the students improved academically while they were involved in the programme
• the enrichment activities of programmes can engender support from children, teachers and parents.
  

Huang and colleagues (2000) claim the message is clear: attendance predicts performance. Children will attend a programme that is attractive and relevant to school. A possible explanation is that coming to school regularly and attending the programme regularly are the reasons for good performance at school with the impact persisting after children leave the programme. In view of the importance of the educational outcomes sought, Huang and colleagues recommended that the programme be improved with more emphasis on homework time and more individualised assistance. But they clearly associate the ‘enrichment’ (rather than educational) activities with the children’s enthusiasm to participate.

Despite the emphasis in the US on the potential for OST programmes to contribute to academic remediation, Robert Halpern (2000: 186) makes a strong case for easing back on the academic focus:

‘My basic argument is that low-income children, as all children, need times and places in their lives where the adult agenda is modest, if not held at bay; where the emotional temperature is low, and acceptance is generous; where learning is self-directed, experiential, and structured to be enjoyable; where talents can be identified and nurtured; and where possible identities can be explored without risk of failure or ridicule.’

Halpern’s view is that school-related problems requiring remediation should be dealt with in the school system, rather than distracting after-school programmes from the tasks for which they are best suited: providing the opportunities and relationships critical to meeting children’s other developmental needs.

Halpern’s analysis of the Making the Most of Out-of-school Time programme in the US found structural variations within the same programme in different locations, as well as variations in philosophy, emphases and content, but found this unsurprising and unproblematic. He remarks that many services with a youth development philosophy had open enrolment, with youth dropping in when they want to. Other services who view their work primarily as childcare tend to have a more explicit agreement or contract with parents with respect to responsibility for the child during certain hours. Programmes in agencies that operate from a recreation or youth-work tradition tend to have relatively large groups, low adult–child ratios, and a whole-group orientation in activities. Those that operate from an early childhood or childcare tradition tend to have smaller groups, higher adult–child ratios, and provide a more individual focus. The message is that there is plenty of room for various styles of OST programme to suit different needs and interests.

The importance of enriching activities in the settings of OST programmes is supported by research involving data from the National Institute on Child Health and Development (NICHD) Study of Early Childcare. The institute found that after controlling for child and family factors and children’s prior functioning, children who consistently participated in extracurricular activities during kindergarten and first grade (i.e. children aged five to seven) obtained higher standardised test scores than those who did not. Participation in other types of out of school care was not associated with a child’s functioning in first grade when background factors were controlled for (NICHD Early Childhood Network, 2004).

The NIOST (2005) sees OST programmes as an opportunity for children to acquire twenty-first century skills or engage in opportunities they may not have access to at home or at school. These might include advanced computing skills and participation in multimedia projects. These activities are often highly attractive to teenagers, and can lead to success in high-order thinking and synthesising different points of view.

Posner and Vandell’s (1999) research on African-American and White children from low-income households who participated in an after-school programme, aimed to identify any relationship between activities and children’s adjustment over time. They found that girls were more likely to engage in academic activities and socialising, whereas boys were more likely to play coached sports. Children who were in OST programmes spent more time on academic and extracurricular activities, whereas children in informal settings after school spent more time watching television and hanging out.

Highlights Few research studies anywhere have been designed to provide statistically significant results on the outcomes of participation in out of school services. Although a number of evaluations have demonstrated positive outcomes for children, authors caution that it is difficult to establish causality. While some researchers point out that this does not have to prevent us from drawing conclusions from the studies, others consider that this has led to positive outcomes being overstated. Some researchers suggest that, as outcomes cannot be easily measured, a focus on inputs rather than outcomes may be useful.

Across all the literature reviewed, the pattern is patchy. A few highly detailed experimental research and evaluation studies provide evidence about mostly academic achievement at school among children who attend OST programmes in the US. This is accompanied by a body of quasi-experimental and correlational studies suggesting associations between attendance at OST programmes, the activities provided, and positive outcomes for participants. These studies are methodologically unable to establish causality at the level of statistical significance (Bodilly and Beckett, 2005; Kane, 2004; Blau and Currie, 2004; Beckett, Hawken and Jacknowitz, 2002).

Despite this, many of these studies point out that the lack of statistically significant findings does not, of itself, prevent conclusions from being drawn, and discuss how to reasonably interpret the results (e.g. Kane, 2004). Fashola (1998) includes detailed discussion of several research strategies to avoid selection bias in OST programme evaluations.

The remaining literature reviewed consisted of evaluations that focus on inputs, such as ensuring certain elements associated with service quality are present, rather than attempting to measure outcomes for the participants or any other interested party. These studies are thought to be the least compelling by a range of reviewers, especially those with a methodological focus, since many have been conducted by providers, rather than by independent researchers.

A different view from the US is provided by Hall and colleagues (2003). They consider (from a youth development perspective) that in this field where causality is so difficult to establish, defining quality from an inputs perspective (i.e. focusing on achieving excellence in programme elements and processes known to be associated with quality) can lighten the burden of expecting individual programmes to document their impact on youth outcomes. In their view, this approach would be a much better use of organisational and other resources.

This advice again raises the issue about the high costs that could be involved in administering complex research instruments to programmes that do not attract a commensurate level of funding. A dilemma could be reached where services that are not evaluated using the most rigorous research methodologies never get enough funding, so can never prove they produce positive outcomes. The early history of the provision of OSS in most countries shows that restrained access to resources usually results in poor-quality and unstable services.

In the competition for funding in the US, the new focus on SBR has resulted in the previous focus on developmental outcomes (such as increasing physical and emotional maturity and increasing social engagement to support future adult citizenship) losing ground to the more easily and precisely measured outcomes associated with academic progress (such as exam results in school). Much of the most recent literature now involves reviews of OST programme evaluations to rate them according to the gold standard of SBR. The clear implication is that cross-sectional evaluations and correlational studies that are ultimately descriptive do not carry sufficiently convincing authority.

In addition there is considerable commentary from various experts in the areas of child and youth development and education that says championing such an extreme standard of proof is inappropriate and unhelpful. The potential for selection bias is a fundamental and powerful problem with almost all OST programme evaluations, since no affordable evaluation can ever locate or measure the motivations of families whose children do not attend the programme or measure their academic outcomes to compare them with the outcomes for those who do attend (Bodilly and Beckett, 2005). Such research could easily cost more than the service itself.

Bodilly and Beckett (2005) found very few evaluations that provided the highest level of evidence of robust outcomes from OST programmes, since there were very few well-designed studies from which firm conclusions could be drawn. The only safe conclusion, in their view, is that the positive benefits of OST programmes are overstated. Nevertheless, they support the view of Kane (2004) who argues that the average hours of OST programme intervention are simply not high enough to produce the magnitude of effect that most of the studies have looked for. Moreover, none of the four evaluations in Kane’s review (The 21st CCLC, TASC (see TASC, 2006), the Extended-Service Schools Initiative (see Grossman et al., 2002); and the San Francisco Beacons Initiative (see Walker and Arbreton, 2002)). reported a statistically significant impact on achievement test scores for participants after one year of participation on the programme. Kane argues for more modest expectations of test scores, with a greater focus on more easily detected impacts, such as parental involvement in school-related activities, homework completion, school attendance and grades. Peter (2002) had stressed that popular programmes, unsupported by research and evaluation, are not automatically substandard or ineffective. In her view, the ability to attract participants, maintain enrolment, engage children, please families, secure community support, procure funding and stay in business are significant accomplishments.

An important issue when considering the results of any evaluation is that the results reflect the outcomes sought by the programme, but not necessarily those sought by another programme or strategy. This can lead to potentially unhelpful effects if interpretation is not made carefully. For example, Bodilly and Beckett (2005) criticise the results of an evaluation that showed no reduction in self-care – the implication being that the children now attending the particular OST programme were previously cared for at home by a responsible adult. The sum effect of the programme had been to substitute care at the centre for care at home. In terms of the evaluation, this result was deemed to be a failure of the service, since it had not attracted enough self-care children. A reduction in self care may not have been the aim of the programme. In the UK, where different goals are considered, it might be considered more successful since those adults would now be free to undertake employment or some other economically productive activity.

Highlights The United States’ strong focus on academic achievement in OST programmes is under scrutiny and there is a lot of debate, particularly in the US, around what an appropriate range of performance measures for services would be. Research-based strategies for achieving high quality services include, in broad categories, human relationships, programming, environment, partnerships with young people, families, schools and communities, and staff and administration. Experts from the youth development field emphasise the importance of children having supportive relationships and opportunities to belong. The factors identified by education experts focus on learning outcomes, which suggests class size and staff training are particularly important.

Halpern (2000) questions the need for OST programmes to focus on academic achievement to the detriment (in his opinion) of other goals associated with youth development. He explores the origins of this trend, noting that although OST programmes have been at the margins of social provision for many decades, much more attention is being given to it now. He concludes that, in part, it may simply be school-age children’s turn to be a source of public attention and worry.

But there are also distinct factors in the lives of today’s low-income children that may have heightened this attention, such as the sense that key child-rearing institutions and contexts are not now providing all the supports that these children need. The consequences include children’s failure to consolidate basic skills, negative self-perception of academic competence, psychological withdrawal from academic challenges and, by middle school, complete detachment from school as an institution (Voelkl, 1997 cited in Halpern, 2000).

In addition, inner-city neighbourhoods no longer provide a developmentally rich environment for children with surrogate parental supervision always available outside school hours, as was once the case.

Awareness of these wider issues may be the key factor in motivating politicians, public officials, child advocates, philanthropic foundations and others to take a fresh look at after-school programmes.

Another view suggests that increasing the focus on academic accountability in the US school system has filtered downwards into the OST sector (Beckett, Hawken and Jacknowitz, 2002).

In response to the ‘galvanising’ impact of the 2001 No Child Left Behind Act in the US (Bodilly and Beckett, 2005), researchers into OST programmes have had to respond by developing appropriate performance measures to provide the information required to meet funding criteria. The HFRP issued a discussion paper on performance measures (Little, Harris and Bouffard, 2004) to help OST programmes plan more effective evaluation processes. Little, Harris and Bouffard list appropriate performance measures for academic performance, youth development and prevention performance outcomes, as well as associated data sources including standardised text scores, school records, class grades, and surveys of parents, teachers and children, OST programme staff, and police records. In their view:

• the range of performance measures used to assess programme OST outcomes reflects the diversity of OST programmes
• performance measures are not the same as performance indicators
• the availability of data sources is a consideration when selecting and developing performance measures – using standardised academic and behavioural assessments requires training and is far more costly than conducting surveys to collect data from participants and others
• performance measures should, in part, be selected because they will yield useful information to improve programmes, as well as to fulfil accountability requirements.
  

A more technically detailed follow-up to this work was issued under the auspices of the HFRP in 2005 (Wimer, Bouffard and Little, 2005). This was a compilation of 58 established research instruments (to obtain standardised test scores) to use for measuring academic achievement, accompanied by citations from research reports where the instruments had been used. The topics covered are academic and educational attitudes and values; future orientation and future plans; life events and experiences; mental health and behaviour; relationships; self-perception and self-esteem; alcohol, tobacco and other drug use prevention; programme quality and programme environment; and multi-component scales and comprehensive surveys.

Many studies into OST programmes do not include much consideration of the children’s school environment, beyond protesting that it is not the role of an OST programme to fix educational problems (Miller, 2003). However, the education literature recognises that school is an important factor: as a very powerful influence in a child’s life – second after family; as the location of many OST services, sometimes in the form of extended-day schooling; as the community hub of OST programmes; and as the source of professional staff to run OST programmes (Dryfoos, 1999).

Dryfoos (1999) found that the proportion of public schools reporting extended day schooling in the US increased from 16 percent in 1988 to nearly 30 percent in 1994. The evolving role for schools has increasingly included locations for OST programmes, tutoring after school hours and supervised recreation in school buildings. Many communities regard schools as publicly owned community facilities. Schools feature as lead agencies in the federal funding arrangements for 21st CCLCs’ programmes. Evaluations of extended-day enrichment programmes have yielded some positive results, although, as ever, the most difficult aspect is attributing positive outcomes to the programme and not to other factors.

The New York Beacon Initiative, a school programme initially for disadvantaged children, is supported by youth development funding and is spread across all levels of school in New York. Most Beacon centres are open 13–14 hours a day, seven days a week in New York and other US locations.

Dryfoos (1999) is realistic about the extent of challenge created by using school facilities for after-school staff activities, pointing out that teachers can find the consequent disruption of their teaching space difficult and the issues of legal liability are complex. This issue was also recognised in the UK, but considerable progress has been made towards addressing it (Cummings et al., 2005).

The results from experimental and quasi-experimental research into OST programmes in the US, such as the research reviewed by Bodily and Beckett (2005) and Little and Harris (2003), may satisfy high standards of statistical accuracy, but should not be taken as suggesting those programmes would produce the same results in New Zealand. Without undertaking an exhaustive analysis of all the evaluations to identify the personal characteristics of all the participants in the programmes, it cannot be assumed that New Zealand school pupils would be sufficiently similar to their US counterparts. Furthermore, aspects of the US culture inherent in those programmes could not be assumed to lead to similar effects in New Zealand children as in US children.

Expert commentary after the reduction in funding for the 21st CCLCs’ programmes in the US included a helpful analysis of two alternative approaches to linking OSS to outcomes. It also helps to explain what went so wrong with the 2003 evaluation of 21st CCLCs.

The first approach starts with choosing the outcomes required of OSS, so programme activities align with those outcomes. The second approach starts with the activities the service wants to provide, so the choice of outcome goals can be aligned with what the research and evaluation evidence indicates is appropriate for those activities (HFRP 2003: 3).

In the evaluation of the 21st CCLCs, it appears that services taking the second approach were judged on the basis of the first approach. The first approach could be considered an approach informed by a dominant educational focus, whereas the second is more aligned with a youth development focus (and is favoured by commentators who consider it appropriate to hold programmes accountable simply for providing positive activities for young people while their parents are at work).

This second approach encourages a system of continuous learning for improvement, and appears to be more in line with the school-age childcare approach now established in the UK and in New Zealand.

The research indicates it is important for services to aim for high-quality standards. A discussion of the ‘handful of nationally acclaimed’ sets of OST standards to achieve high-quality (in the US), is provided by Nancy Peter (2002). In particular, the forty research-based strategies published by the National School Aged Childcare Alliance are in broad categories:

• human relationships
• programming
• environment
• partnerships with young people, families, schools and communities
• staff and administration.
  

Another set of standards discussed by Peter is the ‘40 Developmental Assets’ published by the SEARCH Institute. These include external assets (support, empowerment, boundaries and expectations) and internal assets (commitment to learning, positive values, social competencies and positive identity). Peter points out that some providers have amalgamated the Alliance’s categories and the 40 Developmental Assets.

Bodilly and Beckett (2005) reviewed the literature associated with expert opinion in the fields of school-age care, youth development and education. Among experts on school-age care, they found several expert panels that agreed about the importance of:

• fostering warm relationships between staff and students and between staff and parents
• encouraging and respecting students
• making children feel welcome, relaxed and safe
• fostering respect between staff and volunteers.
  

Experts from the youth development field emphasise the importance of children having supportive relationships and opportunities to belong (see RAND (2001)).  The factors identified by education experts focus on learning outcomes, which suggests class size and staff training are particularly important. Bodilly and Beckett (2005: 73) include a helpful table that demonstrates convergence of opinion from the three literatures (reproduced here as Table 1).

Table 1: Programme indicators supported by at least two of the three literatures, by literature

Youth development	School-age care	Education
Physical and psychological safety	Attention to safety and health	Safe, orderly school environment
Positive social norms		High expectations for student achievement
Supportive relationships	Supportive emotional climate
Opportunities for skill building	Age-appropriate activities	Opportunity to learn and strong instructional leadership
Appropriate structure
Integration of family, school and community efforts	Continuity and complementarity with day-school programmes	Constructive home and school relations
	Positive family involvement
	Use of volunteers
	Community partnerships
	Small total enrollment	Small classroom size
	Trained staff	Trained teachers
	Clear goals and evaluation of programme	Clear mission and frequent assessment of academic progress

Source: Bodilly and Beckett (2005: 73)

From this synthesis, Bodilly and Beckett (2005: 73) conclude that the programme components most likely (but not statistically proven) to produce more effective group-based OST programmes are programmes with:

• a clear mission
• high expectations and positive social norms
• a safe and healthy environment
• a supportive emotional climate
• a small total enrolment
• stable, trained personnel
• appropriate content and pedagogy relative to the children’s needs and the programme’s mission
• integrated family and community partners
• frequent assessment.
  

These authors support the conclusion of others that regular participation is essential for positive outcomes to develop, so more ways to encourage participation need to be developed.

In New Zealand, Meagher-Lundberg and Podmore (1998) expressed the strong opinion that adequate funding and quality standards are necessary to ensure good facilities, a reasonable standard of venue and appropriately trained staff to recognised standards. A flexible curriculum and the competence (i.e. training level, skills and knowledge) of the on-site adults are the important factors in providing a high-quality service. These authors expressed the strong wish to see mandatory national guidelines over all aspects of service provision to ensure agreed benchmarks of quality are met.

A fact sheet issued by the NIOST (2005) draws attention to the lack of a clear, professional identity for the workforce involved with OST programmes, and points out that the field lacks a national professional development system. In its view, the best programme and policy ideas are likely to be ineffective if they do not include proper staff training, have a well-developed infrastructure, have buy-in from parents and teenagers, and involve teenagers in programme development.

Walker and Arbreton (2002) list the organisational and staff practices they consider important for the San Fancisco Beacon Initiative:

• staff training – particularly in the area of interpersonal skills
• high staff to youth ratios
• dedicated space
• hiring staff with experience in community-based youth-serving organisations.
  

It is logical to presume that for more advantaged children to make more positive gains from attending OSS than from attending any other option available to them, the quality of OSS would have to be better than that of the other options.

In a study of the literature on the impact of parental involvement, parental support and family education on pupil achievement (i.e. a study not connected with OSS), the most important finding was that parental involvement in the form of good, at home parenting has a significant positive impact on children’s achievement and adjustment even after all other factors shaping attainment were taken into account:

‘In the primary age range the impact caused by different levels of parental involvement is much bigger than differences associated with variations in the quality of schools. The scale of impact is evident across all social classes and all ethnic groups … Differences between parents in their level of involvement are associated with social class, poverty, health, and also with parental perception of their role and their levels of confidence in fulfilling it.’ (Desforges and Abouchaar, 2003: 1)

The published research and evaluation about OSS usually focus on outcomes for children, without exploring outcomes for parents, communities and the economy. Academic and official commentary however, often includes consideration of the wider context.

The benefits to children of attending OSS are difficult to measure with the highest levels of statistical accuracy, since it is usually impossible to distinguish positive effects of attendance at OSS from other positive effects occurring elsewhere in the child’s life and it is unethical to deny a child participation in OSS just to measure the effect of their non-participation. Despite this issue a growing body of research and evaluation, particularly from the US, strongly suggests positive effects can indeed be associated with attendance at OSS. For example, for children with poor records of school attendance and achievement, particularly those from economically disadvantaged households, evidence exists that as long as children’s attendance at high-quality OSS is regular and frequent over several years, their engagement with school is likely to improve. If a child attends school more frequently the likelihood of their completing their homework is higher and it is more likely their levels of academic achievement will improve. These studies usually recommend action to improve frequency of participation in OSS, in order to activate this path of progress towards beneficial outcomes for the child.

Some evaluations have measured identifiable improvements in academic achievement among participants in OSS, using test scores in subjects like maths and reading. However, several researchers caution that since attendance at OSS is often brief compared with attendance at school and with family, it is inappropriate to rely on OSS as a direct remedial influence on academic achievement if a child is not doing well at school.

For children from more advantaged households the evidence is less clear, since most of the more rigorous studies have focused on outcomes for disadvantaged children (e.g. those with behavioural problems or from low-income households). Some studies have suggested negative effects resulting from poor quality services, emphasising the significance of providing high-quality services. There is a reasonable consensus among researchers that high-quality OSS have several similarities with high-quality ECE.

Even for children with poor educational achievement, just providing more education after school will not necessarily work. The LA’s BEST evaluation shows that enrichment activities are just as important in setting up the pathway towards the child’s improved engagement with school, which in turn is associated with their improved performance at school.

Fashola’s (1998) advice is that to run an effective programme that will generate a range of positive outcomes for children, the key requirements are to:

• train the staff
• create a structured programme
  
• evaluate the programme
• include families and children in the planning for the programme
• have an advisory board.

The emphasis in the US on justifying expenditure on OST programmes that increase academic achievement has its critics. An alternative view from Halpern (2000) is that one of the many pernicious effects of poverty in the US is the pressure it places on each individual institution in children’s lives to be and do more than is reasonable. He finds that it is dangerous to argue that after-school programmes can or should nurture the basic skills, sense of worth, competence and acceptance that family and school are primarily responsible for nurturing and that are so powerfully influenced by class, race and ethnicity. It is difficult for OST programmes to create the psychological space children need if there is too much pressure to meet other outcomes.

The trick in OST programmes, according to Halpern, is to design programmes that balance safety with a measure of guidance and enrichment, and have spaces children can feel they own.

Granger and Kane (2004) have four recommendations for policy-makers, evaluators and providers of OST programmes.

1. Programmes must increase attendance or they will not achieve their goals.
2. Evaluators need to be more realistic about what it takes to create discernable effects on achievement test scores.
3. Programmes need to reach vulnerable kids who would otherwise be unsupervised.
4. Programmes should build on examples that are demonstrably successful.

Department for Education and Skills, UK, www.dfes.gov.uk/research
Harvard Family Research Project, US, www.gse.harvard.edu/hfrp/
National Institute on Out of school Time, US, www.niost.org/research/index.html
Public/Private Ventures, US, www.ppv.org/ppv/publications/publications.asp?search_id=8
Rand Corporation’s Labour and Population Research Program, US, www.rand.org/pubs/research
The After-School Corporation, US, www.tascorp.org/publications/catalog/

‘Experimental design’ is a research design in which the researcher randomly assigns individuals to a treatment group or a control group. The researcher compares the two groups to determine the effects of the programme. It is also known as a randomised controlled trial.

‘Quasi-experimental design’ is a research design using individuals non-randomly selected to treatment and control groups or conditions and using controls to minimise threats to the validity of conclusions drawn. It is subject to selection bias.
Types of quasi-experimental designs include: comparison groups, pre-test and post-test design, time series and multiple time series designs, matched pairs, non-equivalent control group, and counterbalanced design.

• Barker, John, Smith, Fiona, Morrow, Virginia, Weller, Susie, Hey, Valerie, and Harwin, Judith (Out of School Research Unit and Child-Focused Research Centre, Brunel University) (2003) The Impact of Out of School Care: A qualitative study examining the views of children, families and playworkers. Research report RR446. UK: Department for Education and Skills, retrieved March 2006, from www.dfes.gov.uk/research/data/uploadfiles/RR446.pdf
  
• Beckett, Megan, Hawken, Angela, Jackowitz, Alison (2002) Accountability for After-School Care: Devising standards and measuring adherence to them. Santa Monica, California: Rand Labor and Population, retrieved 28 February 2006, from www.rand.org/pubs/monograph_reports/MR1411/.
• Berman, Brad, D., Winkleby, Marilyn, Chesterman, Elizabeth, and Boyce, Thomas, W. (1992) ‘After-school child care and self-esteem in school-age children’, Pediatrics 89(4): 654–659.
• Blau, David, and Currie, Janet (2003) Preschool, Day Care and After School Care: Who’s minding the kids? Retrieved 2 November, 2005, from http://www.econ.ucla.edu/people/papers/Currie/Currie291.pdf.
• Bodilly, Susan and Beckett, Megan K. (2005) Making Out of school Time Matter: Evidence for an action agenda. Santa Monica, California: Rand Education and Rand Labor and Population.
• Bouffard, Suzanne and Little, Priscilla (2003) A Review of Activity Implementation in Out of school Time Programs. Out of school time evaluation snapshot No. 2. Cambridge: Harvard Family Research Project, retrieved 23 March 2006, from www.gse.harvard.edu.
• BRC Marketing and Social Research (2004) OSCAR Programmes: Evaluation of the Ministry of Social Development’s package of assistance Wellington: Centre for Social Research and Evaluation (CSRE), Ministry of Social Development, retrieved March 2006, from www.msd.govt.nz.
• Cooper, Harris, Valentine, Jeffrey C., Nye, Barbara, and Lindsay, James J. (1999) ‘Relationships between five after-school activities and academic achievement’. Journal of Educational Psychology 91(2): 369–378.
• Cummings, Colleen, Dyson, Alan, Papps, Ivy, Pearson, Diana, Raffo, Carlo, and Todd, Liz (2005) Evaluation of the Full Service Extended Schools Project: End of First Year Report. Research report RR680. UK: Department for Education and Skills, retrieved March 2006, from www.dfes.gov.uk/research/data/uploadfiles/RR680.
• CRG Research Ltd (2005) Positive Activities for Young People: National evaluation: End of year 2 report. UK: Department for Education and Skills, retrieved 27 February 2006, from www.dfes.gov.uk/research/data/uploadfiles/RW46.pdf.
• DoL and NACEW (1999) The New Zealand Childcare Survey 1998: A survey of early childhood education and care arrangements for children. Wellington: Department of Labour and National Advisory Council on the Employment of Women.
• Desforges, Charles, Abouchaar, Alberto (2003) The Impact of Parental Involvement, Parental Support and Family Education on Pupil Achievement and Adjustment: A review of literature. Research brief 433. UK: Department for Education and Skills, retrieved 27 February 2006, from www.dfes.gov.uk/research/data/uploadfiles/RB433.pdf.
• Dryfoos, Joy G. (1999) ‘The role of the school in children’s out of school time’. When School is Out 9(2): 117-134.
• Fashola, Olatokunbo S. (1998) Review of Extended-Day and After-School Programs and Their Effectiveness. Report No. 24. Centre for Research on the Education of Students Placed at Risk, retrieved 15 February 2006, from www.csos.jhu.edu/crespar/techreports/Report24.pdf.
• Gammage, Philip (2003) ‘Outside school hours care: Does it matter?’ Keynote address to the School Age Care Symposium, Adelaide, 4 May. South Australian Department of Education and Children’s Services (DECS) and University of South Australia.
• Goyette-Ewing, Michele (2000), Children’s After School Arrangements: A study of self-care and developmental outcomes in diverse families, competent families, Haworth Press Inc.
• Granger, Robert C., and Kane, Thomas J. (2004) ‘Improving the quality of after-school programs’. Commentary submitted to Education Week by Robert C. Granger, William T. Grant Foundation and Thomas J. Kane, University of California, Los Angeles for the William T Grant Foundation, retrieved 7 March 2006, from www.wtgrantfoundation.org.
• Grossman, Jean Baldwin, Price, Marilyn L., Fellerath, Veronica, Jucovy, Linda Z., Kotloff, Lauren J., Raley, Rebecca, Walker, Karen E. (2002) Multiple Choices After School: Findings from the extended-service schools initiative. Public/Private Ventures, Philadelphia, retrieved 2 March 2006, from www.ppv.org/publications/.
• Hall, Georgia, Yohalem, Nicole, Tolman Joel, and Wilson, Alicia (2003) How Afterschool Programs Can Most Effectively Promote Youth Development as a Support to Academic Achievement. Boston: After-School for All Partnership, National Institute on Out of school Time, retrieved March 2006, from www.niost.org/research/index.html.
• Halpern, Robert (2000) ‘The promise of after-school programs for low-income children’. Early Childhood Research Quarterly 15(2): 185–214.
• HFRP (2003) Why, When and How to use Evaluation. Experts Speak out. Issues and Opportunities in Out of school Time Evaluation No. 5. Cambridge: Harvard Family Research Project.
• HFRP (2004) Understanding and Measuring Attendance in Out of school Time Programs. Issues and Opportunities in Out of school Time Evaluation No. 7. Cambridge: Harvard Family Research Project.
• Huang, Denise, Gribbons, Barry, Kim, Kyung Sung, Lee, Charlotte, and Baker, Eva L. (2000) A Decade of Results: The impact of the LA’s best after school enrichment program on subsequent achievement and performance. California: University of California, Los Angeles (UCLA), Center for the Study of Evaluation.
• Kane, Thomas J. (2004) The Impact of After-School Programs: Interpreting the results of four recent evaluations. Working paper, UCLA, retrieved 7 March 2006, from www.wtgrantfoundation.org.
• Little, Priscilla, and Harris, Erin (2003) A Review of Out of school Time Program Quasi-experimental and Experimental Evaluation Results. Out of school Time Evaluation Snapshot No. 1 Harvard Family Research Project, retrieved 13 March 2006, from www.gse.harvard.edu/.
• Little, Priscilla, Harris, Erin, and Bouffard, Suzanne (2004) Performance Measures in Out of school Time Evaluation. Out of school Time Evaluation Snapshot No. 3 Harvard Family Research Project, retrieved 2 March 2006, from www.gse.harvard.edu/.
• Malcolm, Heather, Wilson, Valerie, and Davidson, Julia (2002) Out of School Care: A brief review of the literature. Scotland: Scottish Council for Research in Education for the Scottish Executive Education Department, retrieved 25 January 1 2006, from www.scre.ac.uk/resreport/rr106.
• Market Opinion Research International (2004) Study Support Survey 2004. Research brief RB591. UK: Department for Education and Skills, retrieved 27 February 2006, from www.dfes.gov.uk/research/data/uploadfiles/RB591.pdf.
• Meagher-Lundberg, Patricia, and Podmore, Valerie N. (1998) Out of school Care. A review of quality provision. Wellington: New Zealand Council for Educational Research.
• Miller, Beth (2003) ‘Executive summary’ in Critical Hours: Afterschool programs and educational success. Nellie Mae Foundation, retrieved 16 February 2006, from www.nmefdn.org/uimages/documents/Critical_Hours_Summary.pdf.
• NICHD (National Institute of Child Health and Human Development Early Child Care Research Network) (2004) ‘Are child developmental outcomes related to before- and after-school care arrangements? Results from the NICHD study of early child care’. Child Development 75(1): 280–295.
• NIOST (2005) Making the Case: A Fact Sheet on Children and Youth in Out of school Time. National Institute on Out of school Time (NIOST), Centre for Research on Women at Wellesley College, retrieved February 2006, from www.niost.org.
• Minister of Education (2004) Pathways to the Future: Ngā Huaraki Arataki: A 10-year strategic plan for early childhood education. Wellington: Ministry of Education, retrieved 6 June 2006, from www.minedu.govt.nz/web/downloadable/dl7648_v1/english.plan.art.pdf.
• Organisation for Economic Co-operation and Development (OECD) OECD (2004a) Babies and Bosses: Reconciling work and family life volume 3: New Zealand, Portugal and Switzerland. Paris, France: Author.
• Organisation for Economic Co-operation and Development (OECD) (2004b). Reconciling earning and caring: social policies for working families, summary and conclusions.   Paris, France: Author.
• Peter, Nancy (2002) Outcomes and Research in Out of school Time Program Design. Philadelphia: Department of Human Services, Division of Community-Based Prevention Services, retrieved February 2006, from www.cbps.org.
• Posner, Jill K., and Vandell, Deborah Lowe (1994) ‘Low-income children’s after-school care: Are there beneficial effects of after-school programs?’ Child Development 65(2): 440–456.
• Posner, Jill K., and Vandell, Deborah Lowe (1999) ‘After-school activities and the development of low-income urban children: A longitudinal study.’ Developmental Psychology 35(3): 868–879.
• Rand Corporation (2001) Evaluating After-School Care. US: Labour and Population Program Research Brief, Rand Corporation, retrieved 28 February 2006, from www.rand.org/pubs/research_briefs/RB2505/index1.html.
• Saville-Smith, K., Parata-Blaine, A., & Mainey, A. (1997). An evaluation of the OSCAR pilot.  Wellington, New Zealand: CRESA
• Spielberger, Julie, and Halpern, Robert (2002) The Role of After-School Programs in Children’s Literacy Development. Chicago: Chapin Hall Centre for Children, University of Chicago.
• TASC (2004) Quality, Scale, and Effectiveness in After-School Programs. US: The After-School Corporation, retrieved 15 February 2006, from www.tascorp.org/publications/catalog/qseas/Binder1.pdf.
• UK Government (2004) Choice for Parents, the Best Start for Children: A ten-year strategy for childcare. London: HM Treasury, retrieved February 2006, from www.hm-treasury.gov.uk.
• Vandell, Deborah Lowe, and Corasaniti, Mary Anne (1988) ‘The relation between third graders’ after-school care and social, academic, and emotional `functioning.’ Child Development 59: 868–875.
• Walker, Karen E., Arbreton, Amy J.A., and Stanford University Research Team (2002) After-School Pursuits: An examination of outcomes in the San Francisco Beacon Initiative. US: Public/Private Ventures, retrieved 2 March 2006, from www.ppv.org/ppv/publications/assets/168_publication.pdf.
• Wimer, Christopher, Bouffard, Suzanne M., and Little, Priscilla M.D. (2005) Measurement Tools for Evaluating Out of school Time Programs: An evaluation resource. Out of school Time Evaluation Snapshot No. 6. Cambridge, MA: Harvard Family Research Project, retrieved 2 March 2006, from www.gse.harvard.edu/.