Are Learning Technologies Effective?
The research reported in this Research Brief is based on MERC's study which addressed the question, "Are learning technologies effective?" This study utilized a meta analytic technique to review comparative studies of differing technologies under different conditions (Pisapia & Perlman, 1992). The 184 studies selected for this review represent a diverse array of district and vendor evaluations, independent research projects published in scholarly journals, and dissertation studies. A study had to meet the following criteria to be included in this analysis: it 1) took place in a classroom; 2) had a control and treatment group structure; 3) was free of major methodological flaws; and 4) provided either an effect size or the data needed to calculate one.
A typology of three learning technology applications, computer assisted instruction (CAI), computer managed instruction (CMI), and computer enriched instruction (CEI), was created to categorize the studies. Specific applications for each category were also used to describe study results, i.e., writing to read (WTR), integrated learning systems (ILS), and multimedia (MM).
FINDINGS
Do learning technologies effect student performance?
The analysis and synthesis of 184 studies point to an educationally significant enhancement of learning by learning technologies. Specifically, students taught with computer-based learning technologies scored .32 standard deviations higher than students taught by traditional instruction. The study suggests that on average, a student performing at the 50th percentile will perform at the 62nd percentile of the standard normal curve when taught with computer-based learning technologies.
The following points illustrate the impact of technology on student performance:
- Of the 184 studies, 32% had a negligible effect, 19% percent had a moderate effect and 49% percent had a substantial effect on student performance. Each type of technology reviewed demonstrated this variability of results. (See chart 1)
- Of the CAI studies, 58% demonstrated substantial results. Of ILS studies, 54 % demonstrated substantial effects, 15% had moderate effects, and 31% produced negligible effects. (See chart 1 on hard copy)
- Newer technology applications are more effective than older applications. The findings support the notion that recent improvements produce more effective and adaptable learning technology applications, courseware and instructional design. Such advancement allows decision-makers to target their purchase decisions to specific learning outcomes.
- CAI and ILS applications are effective for teaching mathematics and language arts. There is preliminary evidence that multimedia may produce similar results in science.
- The use of technology in mathematics, language arts, and science proved to be educationally significant in terms of results.
- The manner in which a learning technology is assessed effects the results. Learning technologies raised scores 1) substantially on locally developed teacher/researcher district developed examinations; 2) moderately on state/regionally developed criteria referenced tests; and 3) moderately on standardized norm referenced tests.
- The write to read results were negligible and not educationally significant for reading. However, they were substantial and educationally significant for writing.
- The effect of technology on at-risk students is promising.
- ILSs demonstrated educationally significant results on standardized tests, and with at-risk students.
- There were few studies found that examined the effectiveness of the use of technology as a tool. In fact, these applications require that new assessment strategies be developed.
Does performance vary?
The overall analysis indicates technology applications are effective. However, it can also be concluded that performance varies. For example, Pisapia & Perlman (1993), found that 1) 32% of 184 studies had a negligible effect, 2) 19% had a moderate effect, and 3) 49% had a substantial effect on student learning. It also illustrates a wide range of effects (from -.07 to .61) across the 184 studies they reviewed indicating that factors other than the learning technology cause variability.
It is clearly evident that performance varies. The implication is that strengthening the way technology is implemented, the skill n which teachers use them, and improving purchase decisions will produce substantially better results. For example, the fluctuation of results by type of achievement measure, noted below, indicates that teachers must address alignment and assessment issues prior to assessing the results of their instruction.
Performance variability can also be controlled by making better purchase decisions. Purchase decisions can be improved when based on a clear description of the educational problem the user is trying to solve; or, the opportunity they are trying to provide students through learning technologies.
RECOMMENDATIONS
Decision-makers should target a learning technology to its intended purpose. Prior to purchase decisions, adopters should clearly identify the learning problem or opportunity they are trying to address or provide. Then, they should analyze the results vendors provide in terms of alignment of courseware, assessment strategies, and ability to improve student performance against the results expected by the school division.
Decision makers should also recognize that infusing technology into schools does not insure optimum results. To get consistent results, hardware and software must be accompanied by appropriate program support and staff development.
John Pisapia
Answers to questions found in this research brief have been synthesized from the following MERC publications. Copies can be purchased using the online order form on the publications page.
Pisapia, J., & Perlman, S. (1992, December). Learning technologies in the classroom: A study of results.