应用语言学论文

A systematic review and meta-analysis of randomised controlled trials evaluating interventions in adult literacy and numeracy Carole J. Torgerson and Jill Porthouse University of York Greg Brooks University of Sheffield This paper reports a systematic review of the trial literature in the field of adult literacy and numeracy. The review was undertaken to investigate the effectiveness of teaching strategies and pedagogies designed to increase adult literacy and numeracy. The objectives were to search for and locate, synthesise and quality appraise all the randomised controlled trials aiming to evaluate interventions in adult literacy and/or numeracy, published between 1980 and 2002. Fifty-nine papers were included in the descriptive map. Twelve papers were included that contained nine randomised controlled trials. All of the trials included in the review were of high quality in the sense that they had adopted an appropriate study design for assessing effectiveness. However, within that study design many of the studies had methodological problems, for example: small sample size and lack of justification of sample size calculation; unclear method of random allocation; high attrition rate and lack of ‘intention to teach’ analysis. There was evidence of publication bias. Pooling three studies that compared teaching against no teaching showed a strong, positive and statistically significant effect on outcome. Two other studies examined the use of computer- assisted instruction (CAI) on literacy among imprisoned adults. Pooling these two studies showed a modest but not statistically significant benefit. There is a dearth of rigorous RCTs in the field of adult literacy and numeracy. The evidence is suggestive of a benefit of adult literacy and numeracy interventions; however, because of the heterogeneity of studies, the precise role of any intervention is uncertain and this finding may be undermined by the presence of substantial publication bias. We recommend that a series of large, well-designed and well-conducted randomised trials should be undertaken in the field of adult literacy and numeracy. Poor adult literacy and numeracy skills are recognised as an important problem in society. Low literacy and numeracy are associated with higher unemployment and lower incomes for those in employment compared with those with higher levels of basic skills. Within the United Kingdom the publication of the Moser Report A Fresh Start (DfEE, Journal of Research in Reading, ISSN 0141-0423 Volume 26, Issue 3, 2003, pp 234–255 r United Kingdom Literacy Association 2003. Published by Blackwell Publishing Ltd., 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA 1999) drew attention to the large number of adults in England and Wales with poor basic skills, and gave this field greater prominence than ever before. An emphasis on related social factors was a feature of the report; it was specifically noted that there was much evidence connecting poor basic skills with issues such as unemployment and criminality. In response the British government published a White Paper setting out its strategy (Skills for Life, DfEE, 2001). This included the target that by 2004 a total of 750 000 adults would significantly improve their basic skills, as a result of a concerted approach to provision, teaching and learning. Ideally, before policy interventions are put into place they should be exposed to a rigorous evaluation by using a large well-designed randomised controlled trial. Failure to evaluate before implementation can lead to policies that are sub-optimal. To aid ‘evidence-based’ policy making in this area what is required is: first, a systematic review of all the available rigorous evidence. A recent overview of the literature in this field, by Brooks et al (2001b) concluded that few intervention studies had been carried out in the area of adult literacy and that no systematic review of intervention studies on adult literacy and numeracy had been undertaken. In particular, they found no randomised controlled trials (RCTs). They did find and review a number of national and other large- scale studies of progress in adult literacy and numeracy, drawing mainly on Beder (1999), Brooks et al (2001a) and Sticht and Armstrong (1994). The general conclusion from the studies was that progress was both slow and modest. This review was designed to remedy the current lack of a systematic review of all the randomised controlled trials (RCTs) in the area of adult literacy and numeracy. Such a review fulfilled two aims: first, to assess whether any rigorous evaluations have been undertaken; and second, to inform the research agenda for future RCTs that may be used to evaluate interventions in adult literacy and numeracy. The primary objective of this systematic review was to search for and locate, synthesise and quality appraise all the randomised controlled trials aiming to evaluate interventions in adult literacy and/or numeracy, published between 1980 and 2002. It was decided to restrict the search period to 1980 onwards because it was felt to be unlikely that policy-relevant studies would be identified prior to this date. Methods Protocol A protocol was developed for this systematic review in order to establish: the research question; the scope and limitations of the review; the methods for conducting the review; the inclusion and exclusion criteria; and the procedure for extraction of data and quality appraisal. The research question We undertook a systematic review to investigate the effectiveness of interventions or pedagogies for increasing literacy or numeracy in adults. A further focus for the search was studies that included a follow-up assessment of learners at some point after an intervention, in order to judge the extent to which gains while in provision were sustained afterwards. META-ANALYSIS IN ADULT LITERACY AND NUMERACY 235 r United Kingdom Literacy Association 2003 At each stage of the review, methods were informed by the NHS Centre for Reviews and Dissemination guidelines for undertaking systematic reviews (http//:york.ac.uk/inst/ crd/report4.htm). In addition, the Institute of Education’s Evidence-Informed Policy and Practice Co-ordinating Centre (EPPI) guidelines and tools for data extraction and quality appraisal of randomised controlled trials in educational research informed this review (http//:eppi.ioe.ac.uk/ED.Handbook/HBTitle.htm). Identification of studies Electronic searches. In collaboration with two members of the review team (GB and CJT), an independent information consultant wrote the search strategies for the electronic databases and carried out electronic searches for the review on PsycINFO, ERIC (Educational Resources Information Center), SSCI (Social Science Citation Index), C2- SPECTR (The Campbell Collaboration’s Social, Psychological, Educational and Criminological Trials Register), SIGLE (System for Information on Grey Literature in Europe) and Criminal Justice Abstracts (CJA). Search terms used included adult literacy; adult numeracy; adult basic education; worksite education; workplace education. Hand searches. All the bibliographies of included reviews were hand searched (CJT) for potentially relevant trials (randomised controlled trials and controlled trials) not already identified through the electronic searches. On completion of the searches the citations identified were imported into the reference manager EndNote and de-duplicated. Criteria for including and excluding studies The systematic review involved a two-stage process. The first stage comprised a descriptive ‘map’ of all the randomised controlled trials and the controlled trials in the field. The second stage comprised an in-depth review of the randomised controlled trials. Papers included in the mapping section of the review had to be evaluations of interventions aimed at increasing the literacy or numeracy in study populations of adults. Papers were included if they were one of the following study types: randomised controlled trial (RCT), controlled trial (CT) or review of RCTs and/or CTs. Studies of this type were included on the basis that randomised trials and controlled trials can be used to help to answer questions about the effectiveness of interventions designed to increase aspects of literacy and numeracy. In addition, studies were included if they were undertaken in English speaking countries and were written in the English language. Finally, papers were included if they were published or unpublished in the years 1980–2002. Studies were excluded if they were interventions aimed at increasing the literacy or numeracy of adult English Speakers of Other Languages (ESOL) students studying in English-speaking countries. Studies were also excluded on the basis of study type if they were interventions of a pre- and post-test design, or if they were non- interventions. Finally, studies were excluded if all or some of the participants were below the age of 18 years. Screening Double-screening of the titles and abstracts of the studies identified by the four main electronic searches (PsycInfo, ERIC, CJA and SIGLE) was undertaken by two researchers working independently (CJT and JP), using the inclusion/exclusion criteria 236 TORGERSON, PORTHOUSE and BROOKS r United Kingdom Literacy Association 2003 outlined above. A measure of agreement between the reviewers was calculated using the Cohen’s Kappa. Any disagreements about inclusion or exclusion of papers were discussed and resolved. All papers thus identified were sent for through library inter- lending. The remaining two small databases (SSCI and C2-SPECTR) and the bibliographies of identified reviews were screened by one of the reviewers (CJT). Exclusion at second stage All received papers were independently double-screened on the basis of the full papers. Papers were then included or excluded at second stage. Coding All received included papers were then coded (CJT) using one of the following codes: ‘RCT’ (randomised controlled trial), ‘CT’ (controlled trial), ‘Review’. Controlled trials and reviews were included in the mapping section of the report, but, as stated in the protocol, no data were extracted from them. Double data extraction and quality appraisal All included RCTs were then double data extracted and quality appraised by two reviewers (CJT and JP) working independently, and using pre-established standardised data extraction tables and quality appraisal tables based on the Consolidated Standards for Reporting Trials (CONSORT) guidelines (Begg et al, 1996). The two reviewers extracted data on the development and content of the intervention evaluated, the design and results of the RCT and data on the methodological quality of the RCT. For each paper, data were extracted about the identification and aims of the study; study design and content; use of allocation (random or otherwise) to the different groups; participants; pre- and post-intervention data; attrition rate. Full agreement was established through discussion. In addition, both reviewers independently calculated effect sizes for all studies using the computer programme Arcus Quickstat. Full agreement was established. Follow-up assessment In trials that included a follow-up assessment of learners at some point after an intervention in order to judge the extent to which gains while in provision were sustained afterwards (e.g. post-test 1, post-test 2) a comparison was made between the effect sizes at each post-test. Synthesis Synthesis took the form of a qualitative overview: a qualitative description of the main findings of the trials in terms of literacy and numeracy; and a description of the methodological strengths and weaknesses of the trials. Where it was appropriate to conduct a meta-analysis to synthesise data from two or more studies because they were homogeneous in terms of intervention and setting this was undertaken. Publication bias It is important to assess whether the results of our review were affected by publication bias. We looked at this issue in two ways. First, we plotted the effect sizes of identified META-ANALYSIS IN ADULT LITERACY AND NUMERACY 237 r United Kingdom Literacy Association 2003 trials against their sample size (funnel plot). Second, we compared the mean effect size of published studies with unpublished studies. Undertaking a funnel plot enabled us to assess whether there was a systematic publication bias. Comparing the means of published trials with unpublished studies enabled us to ascertain whether positive studies are more likely to be published than negative studies. If this is shown to be happening then the review will give an inflated estimate of the effectiveness of adult literacy and numeracy. Quality of randomised trials Although RCTs are widely seen as the ‘gold-standard’ of evaluative research, clearly their results are more reliable when the trials are of high quality. In the field of health care there has been considerable disquiet over many years about the quality of design, conduct and reporting of RCTs. Indeed, major medical journals insist that reports of randomised trials now conform to internationally agreed guidelines (the CONSORT guidelines, Begg et al, 1996). In this report we used a modified version of the CONSORT guidance to appraise the quality of identified trials rather than developed quality assurance instruments de novo. Results Identification of studies A total of 4555 potentially relevant papers were identified using the electronic and hand searches as described in the methods section. The 4555 papers identified were double-screened, using the inclusion/exclusion criteria established in the protocol. At this first stage 4387 papers were excluded and 168 papers were sent for. Of these, 19 papers were not received or were unavailable. All papers were re-screened on the basis of a reading of the full paper. At this (second) stage 90 papers were excluded, leaving a total of 59 papers to be included in the review. Table 1 presents a summary of the origin of all the included studies, by database or other method of retrieval. The 59 included papers were coded and the database was fully annotated with the codes. Ten reviews were identified (and included) through the electronic searches and through a contact. One RCT was identified from a hand search of the bibliographies of these included reviews. Quality assurance Screening. The electronic search strategy was very sensitive, but not very specific. Because of this a relatively few number of studies was included and a large number of studies was excluded. Total agreement of the four main databases was 4405 out of 4524 (85 included and 4320 excluded) or 97.37%. Therefore, disagreement occurred in only 2.63% of the total number of screened studies (119 studies). The two reviewers discussed these 119 studies and came to an agreement about whether or not to include them. The inter-rater reliability score between the two reviewers undertaking initial screening of titles and abstracts was 0.57 (moderate), because of the large size of the database relative to the number of included studies. The actual percentage disagreement was very small and not thought to be worrying. Full agreement between reviewers on exclusion at second stage was established (i.e. there was 100% agreement to include and exclude). 238 TORGERSON, PORTHOUSE and BROOKS r United Kingdom Literacy Association 2003 Data extraction. Data were double entered onto pre-established data extraction tables by two reviewers working independently. In cases where there was initial disagreement about data extraction or quality appraisal this was discussed and resolved. Agreement was generally excellent. Full agreement on calculation of effect sizes was established. Descriptive map of all included studies Table 2 presents the descriptive map of all the included studies. Of the 59 studies that were included in the review, 12 papers were included that reported on 9 RCTs. A further three RCTs were included that had no data or incomplete data on adult literacy and numeracy. Some of them were preliminary studies; others presented data on other outcomes. Ten reviews were included (Table 4), and 34 controlled trials (Table 3). A total of nine RCTs were extracted, quality appraised and reported on for this systematic review (Batchelder & Rachal, 2000a; Bean & Wilson, 1989; Cheek & Lindsey, 1994; Martinson & Friedlander, 1994; McKane & Greene, 1996; Nicol & Anderson, 2000; Rich & Shepherd, 1993; Schrum, 1985; St. Pierre et al, 1995). The remainder of this paper reports the review of the nine RCTs. Description of included RCTs Of the nine included RCTs, five evaluated interventions in literacy, two evaluated interventions in numeracy and two evaluated interventions in both. Methodological quality of studies The quality of all the included studies was high in the sense that they were all randomised controlled trials, and therefore of an appropriate study design for judging the effectiveness of an intervention. Also, in four trials the authors reported effect sizes (Batchelder & Rachal, 2000a; Martinson & Friedlander, 1994; Rich & Shepherd, 1993; St. Pierre et al, 1995), and in all other trials the reviewers were able to calculate the effect sizes from the statistics presented in the studies. Table 5 shows whether or not the studies fulfilled our modified CONSORT checklist of quality items. As the table shows the quality of the trials tended to vary, with no trial concealing random allocation, and only one trial using intention to teach analysis. In the following, we consider the quality of each trial in depth. Table 1. Summary of origin of included studies. Found Included ERIC 2628 39 PsycINFO 971 3 CJA 736 5 SSCI 15 2 C2-SPECTR 8 1 SIGLE 172 1 Website 11 3 Handsearch 13 4 Contact 1 1 TOTAL 4555 59 META-ANALYSIS IN ADULT LITERACY AND NUMERACY 239 r United Kingdom Literacy Association 2003 Batchelder & Rachal (2000a, 2000b) The aim of this study was to examine the effectiveness of using computer-assisted- instruction (CAI) for literacy and numeracy compared with using traditional instruction. Seventy-five male inmates in a maximum-security prison (mean age 30.5 years) were Table 2. Mapping of relevant RCTs, CTs, and reviews. RCTs 12 papers containing 9 trials RCTs (no results) 3 (Argento, 1980; Kemple et al, 1993; McCrossan, McDowell and Cooper, 1998) CTs 34 Reviews 10 Table 3. Included controlled trials. CTs with effect size Diem & Fairweather, 1980 Meyer et al, 1983 Gretes & Green, 1994 Nurss, 1989 Lavery & Townsend, 1998 Roberts & Cheek, 1994 Maclay & Askov, 1988 Schrader, 1984 CTs no effect size (incomplete data) Askov et al, 1986 Macmurdo, 1988 Mid Continent Regional Educational Laboratory, 1991 Broughton, 1994 Morrow et al, 1993 Broussard, 1983 Rio Salado Community College, 1991 Burtoff, 1985 Smith & Dalheim, 1990 Culclasure, 1982 Thuy, 1992 Dietrich, 1994 Wadsworth & Frazier, 1982 Dirx & Crawford, 1993 Indiana OIC, 1990 Wilson, 1992Irby et al, 1992 Wisher & O’Hara, 1981Lehigh Community College, 1993 CTs no results Conti & Giles, 1980 Newsom & Foxworth, 1980 Cox & Lane, 1981 Ramsey & Ramsey, 1983 Enriquez, 1990 Tewksbury, 1994 Johnston, 1980 Table 4. Included reviews. Adult Literacy & Basic Skills Unit, 1994 Beder, 1999 Bouffard, Mackenzie and Hickman, 2000 Jancic, 1998 Kruidenier, 2002 Kulik et al, 1986 Rachal, 1995 Scanlon et al, 1998 Sticht, 1995 Wilson, Gallagher and Coggleshall, 1999 240 TORGERSON, PORTHOUSE and BROOKS r United Kingdom Literacy Association 2003 T a b le 5 . Q u al it y o f in cl u d ed ra n d o m is ed tr ia ls . B at ch el d er & R ac h al