Learning Technologies and Student Performance

Society currently has some very specific measures for the effectiveness of its educational system: student achievement, attitudes, dropout rate, learning time. (Robyler et al. 1988)

The research reported in this paper focuses primarily on the short term questions: Do learning technologies effect student learning? Is effectiveness related to grade levels? Types of students? Recency of the technology? The way it is assessed?

Are there any caveats we should consider when assessing the research on the effectiveness of learning technologies?

The seemingly simple approach of comparing student performance after being taught with or without learning technologies presents several problems.

The most difficult problem is the impossibility of creating a comparable control group. More rigorous controls by researchers are required to produce reliable results. However, most reviewers who point to this weakness also go on to use the results as evidence of effectiveness (Clark, 1985; Roblyer et al., 1988; Jurkat et al., 1992; Power on!, 1988).

On the other hand, Becker and Henig (1987) concluded that comparison studies can provide valuable and reliable guidelines for policy decisions when a) the effect of the learning technology is isolated as much as possible, b) it is supplemented by evaluations focusing on the process and learning situations, and c) the results are used as interpretative trends.

The second most cited problem is that traditional achievement measures do not account for actual conditions occurring during implementation of learning technologies. While current testing techniques are relatively advanced in assessing whether or not students have learned basic content knowledge, they are immature in assessing more complex thinking skills (Power on!, 1988).

Teachers mention problem-solving, global awareness, motivation, writing and cooperative learning as positive benefits of using learning technologies. Effective measures of these skills must be developed. In fact, the International Society of Technology Education (ISTE) declared that new assessments must reflect the new curricular focus, new kinds of learning and new environments (Vision: Test, 1990).

While alternative measures are in the process of development, they either are not yet available, or not yet accepted as large scale measures for accountability purposes. Until the availability and accountability issues are decided, decision makers can be guided by the assertion of the National School Boards Association, "[N]o measure is the worst of all possible worlds." (Perlman, 1987, p. ES-14).

Do learning technologies effect student performance?

In general, student performance can be improved through the use of learning technologies.

Evidence on the effectiveness of using computer-based technologies in instruction has accumulated for over 30 years. Effect size tends to vary from study to study. However, Computer Based Learning (CBL) appears to have a rather consistent positive effect on achievement. Many reviews found increases in achievements from .27 to .56 standard deviation for computer-based technologies when compared to traditional approaches (Pisapia & Perlman, 1992; Becker,1991; Bennett, 1991; McNeil & Nelson, 1991; Bialo & Sivin, 1990; Fletcher, 1990; Kulik & Kulik, 1989; Debloois, 1988; Roblyer et al., 1988; Kulik & others, 1986; Samson et al., 1986; Kulik et al., 1985; Bangert-Drowns et al., 1985; Hartley, 1978; Visonhaler & Bass, 1972.

The Office of Technology Assessment concluded that the results suggest that certain configurations of hardware and software, used with particular populations of children and under the supervision of competent teachers, contribute to the achievement of specific instructional objectives. Computer Assisted Instruction (CAI) has been researched the most and has proven to be an effective supplement to traditional instruction (Power on!, 1988).

Pisapia and Perlman (1992) found in a meta analysis of 184 studies of learning technologies that, on the average, a student using CAI and performing at the 50th percentile will perform at the 62nd percentile on the standard normal curve.

The following points illustrate the impact of technology on performance:

• Newer technology applications are more effective than older applications.

• CAI and Integrated Learning Systems (ILS) applications are effective for teaching mathematics and language arts. There is preliminary evidence that multimedia may produce similar results in science.

• The use of technology in mathematics, language arts, and science proved to be educationally significant in terms of results.

• The effect of technology on at-risk students is promising.

• Learning technologies raised scores; 1) substantially on locally developed teacher/researcher district developed examinations, 2) moderately on state/regionally developed criteria referenced tests, and 3) moderately on standardized norm referenced tests.

• The Write to Read (WTR) results were negligible and not educationally significant for reading; yet substantial and educationally significant for writing.

• ILSs demonstrated educationally significant results on standardized tests, especially with at-risk students.

• There were few studies found that examined the effectiveness of the use of technology as a tool. In fact, these applications require that new assessment strategies be developed.

Does performance vary?

The overall analysis indicates technology applications are effective. However, it can also be concluded that performance varies. For example, Pisapia & Perlman (1992) found that 1) 32% of 184 studies had a negligible effect, 2) 19% had a moderate effect and 3) 49% had a substantial effect on student learning. It also illustrates a wide range of effects (from -.07 to .61) across the 184 studies they reviewed, indicating that factors other than learning technology cause variability.

It is clearly evident that performance varies. The implication is that strengthening the way technology is implemented, the skill with which teachers use technology, and in purchase decisions produces substantially better results. For example, the fluctuation of results by type of achievement measure, noted below, indicates that teachers must address alignment and assessment issues prior to assessing the results of their instruction.

Performance variability can also be controlled by making better purchase decisions. Purchase decisions can be improved when based on a clear description of the educational problem the user is trying to solve, or the opportunity educators are trying to provide students through learning technologies.

Performance by Timespan. Bangert-Drowns, Kulik & Kulik (1985) projected that differences between earlier mainframe-age studies and later microcomputer-age studies, may be due to improvements in instructional technology. Niemiec & Walberg (1987) in their review of reviews, reported an average improvement of .38 standard deviations in achievement for mainframe-based studies. This compared to an average improvement of 1.12 standard deviation for microcomputer-based studies.

A tendency for more recent studies to produce stronger results was found in the Pisapia & Perlman (1992) study. The years 1978 and 1985 were selected as benchmark dates because they marked approximate periods when new technology applications were introduced, such as videodiscs in 1978. They found that the average effect of studies prior to 1978 was .28, .32 between 1978 and 1985 and .35 post-1985. However, much of this timespan increase was found in ILS and WTR applications.

Pisapia & Perlman's research also pointed out the changes occurring in use of learning technologies. For example, 62% of the CAI results, and 62% of the Computer Managed Instruction (CMI) results and 100% of the Computer Enhanced Instruction (CEI) results were recorded prior to 1978. Whereas, 61% of the WTR, 88% of the ILS and 100% of the Multimedia (MM) results were recorded since 1985.

This change in courseware complements the increases found in the three time frames examined. It lends further support that the improvements in the instructional design of courseware, and the move to more adaptable learning technologies, produce more effective technology applications.

The practical significance of this time scale discovery is that more recent learning technology applications have demonstrated more substantial effects, which support current efforts to reframe instruction to utilize more learning technologies.

Performance by Grade Level. The Pisapia & Perlman (1992) results lend modest support to the claim existing in the literature that younger students seem to profit more than older students from the highly structured materials (small steps and immediate feedback) supplied in drill and practice, tutorial, and managed instruction. In this case, CAI proved effective at all grade levels. It had similar effects at each system level; grades K - 4 (Effect Size .49), grades 5 - 8 (ES .36), and at grades 9 - 12 (ES .41).

Overall, 55% of the learning technology applications were used in grades K-4. ILS and WTR were the predominate applications used in these grades. CAI, on the other hand, was utilized at each grade level - heavier in grades K-4, lighter in grades 9-12 - with substantial results at each grade level.

Generally, in elementary and middle grades, CAI and ILS produced better results. At the high school level, CAI was less effective and CMI was more effective. The data demonstrates that CEI and MM applications, with their emphasis on higher order skills, are being used primarily at the middle and high school levels; the basic skill approaches of ILS are primarily used at elementary and middle school levels.

Performance by Subject. The literature provides strong support for the effectiveness of CAI in mathematics, some support in language arts and science, and negligible support in other subjects.

One hundred and sixty-eight studies in the Pisapia & Perlman (1992) data base could be categorized by subject area. Of the total studies, 38% describe mathematics results, 50% language arts results including reading and writing, and 11% science results. No studies were located in history or geography. Sixty-five percent of the CAI studies were in math, 33% in language arts and 2% in science. The CMI studies were evenly dispersed across math, language arts and science. The ILS applications were evenly divided between math (53%) and language arts (47%). CEI studies were found in math (65%) and science (33%). The newer MM applications were used primarily in science. Studies of data base use in social studies were found in the literature, but did not meet the criteria of inclusion for this study.

In general, the effect sizes found in mathematics, language arts and science were educationally significant. In fact, the mathematics effect sizes for CAI (ES .49) and ILS (ES .40) were substantial. The language arts effect sizes for CAI (ES .32) and CMI (ES .36) and WTR (ES .31) were substantial, as were the science effect sizes for CMI (ES .36) and MM (ES .50). Although the number of multimedia studies is small, the results lend some credence to the increased use of MM in science.

Wise's (1989) recent meta-analysis of the use of computers in science found ESs ranging from -.62 to 1.21, with a mean of .34, indicating that students receiving CBL exhibited superior achievement. For example, videodisc-based applications in the laboratory had an ES of .40. Microcomputer-based laboratory lessons had an ES of .76. In biological science laboratories the ES was .22.

Performance by Student Characteristics. The effects of computer-based teaching seemed especially clear in studies of disadvantaged and low aptitude students; for example, effects appeared to be much smaller in studies of talented students (Bangert-Drowns et al., 1985; Burns & Bozeman, 1981 each cited in Pisapia & Perlman, 1992, and; Niemiec & Walberg, 1988; Robyler et al., 1988; Olson & Krendl, 1990).

Research on the effect with at-risk student learning is promising. These students often show significant gains in achievement levels in basic content areas. Their attitudes toward learning and toward the instructional content tend to improve according to parents and students, and absenteeism and drop out rates decline (Olson & Krendl, 1990).

The results of the Pisapia & Perlman (1992) study clearly indicate that ILS is a powerful application for at-risk, disadvantaged and low achieving student populations. In 34 ILS studies, the ES was substantial (ES .41) for low achieving students. In 41 ILS studies, the ES was substantial (ES .39) for at-risk students. The practical significance of these findings lies in the fact that a teacher of low achieving or at-risk children could expect the 50th percentile student to move up to the 63rd percentile.

Supportive of this finding is Olson & Krendl's (1990) disclosure that the research on the effect of technology on at-risk students is promising. These students often show significant gains in achievement levels in basic skills content areas.

Pisapia & Perlman (1992) found that ILSs focusing on basic skills proved to be effective in teaching reading, math and language achievement to low achieving students. The ILSs also produced similar results for regular students on basic skills. While the number of studies reviewed is small, the results for gifted students (ES .03) are consistent with reports indicating that high achieving students reach the ceiling level of the ILS. Therefore, the ability of an ILS with gifted students is limited.

Performance by Effectiveness Measure. Student learning in each of the 184 studies used by Pisapia & Perlman (1992) was measured by achievement documented at the end of a program of instruction. The data indicate that learning technologies raised scores: 1) substantially on locally developed teacher and researcher developed examinations; 2) moderately on state-regionally developed criterion-referenced examinations; and 3) moderately on standardized norm-referenced tests. The most powerful effects were demonstrated when local teacher or researcher assessments were utilized.

In particular, ILSs demonstrated higher ESs on standardized norm-referenced tests than on state level criterion-referenced tests. This finding was not unexpected, since ILS courseware is primarily developed for a curriculum supported by national standardized tests, and not by local or state assessments. Of course, for additional fees ILS vendors will customize to state or local curricula.

WTR applications were more effective when judged with local school division, teacher or researcher developed assessments than standardized tests of any type. In the WTR studies, the standardized tests were generally used to measure reading gains. The WTR results for reading were negligible and not educationally significant. The local and researcher developed assessments used to measure gains in writing produced educationally significant results.

In general, their findings lend themselves to two interpretations. First, it appears that the closer the test development is to the teacher and to the learner, the more significant the results. The implication for future studies and the development of cost-effective models is that different outcome measures and assessment techniques should be utilized when testing for basic skills and higher order skills.

Secondly, the fluctuation of results by the type of achievement measure indicates that teachers must be sure to address alignment and assessment issues prior to assessing the results of instruction. For example, as indicated by the results on norm and criterion-referenced tests in ILS, alignment problems may exist between instructional objectives, computer courseware and the tests used to measure achievement. These alignment problems possibly mask significant differences in student achievement which were not measured in a particular experiment.

Basic skills acquisition. Several meta-analyses indicate that computer-assisted instruction (CAI) produces small but significant increases in achievement test scores (Power on!, 1988). The improvement rate varies according to the grade range studied and the application, but, at a minimum, achievement increases from the 50th to the 60th percentile on such tests (Bangert-Drowns, 1985). Other meta-analyses find improvements from 50th to the 61st percentile (Kulik & Kulik, 1989), to the 63rd percentile (Samson et al., 1986), and to the 68th percentile (Kulik et al., 1985).

Higher order skills attainment. Higher-order skills can be demonstrably enhanced with the aid of technology. For instance, the average achievement level of junior high school general mathematics students on a standardized test of problem-solving rose from the 33rd to the 68th percentile over four years of television instruction (Chu & Schramm, 1989). The Higher Order Thinking Skills (HOTS) program developed by Stanley Pogrow of the University of Arizona shows gains in thinking skills and social interaction that continue beyond the experimental experience. The HOTS program is used to develop the thinking skills of metacognition, interference, deconceptualizations and synthesis in at-risk students (Pogrow, 1987).

Pogrow (1987) cautions that simply using software, word processors or the computer as a general tool, primarily benefits high-performing students who have already internalized procedures for generalizing and linking ideas. In the case of at-risk students, techniques need to be developed to promote internalization of key basic thinking processes before they can benefit measurably from sophisticated applications integrated into content.

Achievement results need to be understood in context. Standardized tests may be insensitive to the types of changes pursued in a tool orientation to computer use in the area of higher level thinking. Future technology evaluation may wish to consider developing a more detailed, longer range instrument that is more sensitive to the long-term, creative writing process students use with technology as a tool (Herman et. al., 1992).

Attitudes. There is ample evidence collected from surveys that students' attention span is greater when working with the computer than when working with drill cards (Robyler et al., 1988).

Only two of the field studies used scientific measures of student attitude outcomes. Both Cornelius (1986) and Rawitsch (1987, 1988) found more positive attitudes toward use of computer databases in problem-solving among students in computer-using classes, although in the Cornelius study there was a difference in only one of the two comparisons.

Herman et al. (1992) reported that the program does appear to have had a substantial impact on the students' attitudes toward school and learning in general, on their interest and proficiency in technology use, and on their attitudes and motivation while working with technology.

Motivational effect. Arguments about the motivational effects of media, the 'strong' media theory, suggest that certain media are more 'motivating' than other media. The 'weak' media theory suggests that the independent variable in motivation studies is not media, but is our beliefs of values related to media. For example, studies which have shown increases in motivation (or learning) with decreases in attitude toward a specific medium are now predictable given the self efficacy theory.

While the literature reports that teachers do not stress the use of the computer for motivating students, a substantial number of teachers interviewed for this study have called the computer "a motivator" (Clark & Salomon, 1986). Herman et al. (1992) also indicates that parents report that their childrens' interest and attitude towards learning and their language skills were more positive. Specifically, interactive video has improved learner motivation and attitudes toward instructions (Dalton, 1986).

Novelty effect. While it is possible that higher attention spans can be attributed to novelty effect, even after the initial novelty wears off, the level of interest in the automated workbook is still greater than that in the regular workbook. The increased attention by students sometimes results in increased effort or persistence, which yields achievement gains. If they are due to a novelty effect, these gains tend to diminish as students become more familiar with the new medium. This was the case in reviews of computer-assisted instruction at the secondary school level, grades 6 to 12 (Clark & Sugrue, 1988).

Time. Time pressure impacts on teachers and students alike. Closely related to time pressure was the extent of integration of the computer-based problem-solving into the curriculum. Where the integration was high, the time pressure seemed not much of a problem. But where the problem-solving was "tacked on" to the regular curriculum, the prevailing approach seemed to be one of hurrying on to the next task, and then to the end of the unit, rather than focusing on problem-solving outcomes.

Furthermore, there is a general consensus that students need less time to learn a given amount of material with the use of computer-based learning technologies than without (Kulik & Kulik, 1987), but not in all applications. For example, when using data bases, students demonstrated greater achievement, but it took them longer to solve problems (Rawitsch, 1987).

On the other hand, few current studies measure learning time. Roblyer et. al., (1988) feel this may be due to the fact that most studies are in public schools which have a set time for grade levels and courses. Therefore, decreasing learning time may not be important for decision makers since the entire system would need redesigning to take advantage of accelerating learning.

Retention. The effects on retention are basically positive, but not as clear as initial performance, because retention is difficult to measure. Five studies with follow-up examinations investigated retention over intervals ranging from 2 to 6 months. In 4 of the studies, retention examination scores were higher in the CBI class, but none of these 4 retention effects was large enough to be considered statistically significant. In the remaining study, retention examination scores were significantly higher in the control class (from Pisapia & Perlman, 1992 notes).

Duration. Studies that were shorter in duration produced stronger effects than did studies of longer duration. Although the small effects reported in longer studies may actually show that experimental effects decrease in potency with extended use - it is also possible that shorter studies are better controlled and more likely to estimate true effects (from Pisapia & Perlman, 1992 notes).

Writing to Read (WTR). Pisapia and Perlman (1992) reviewed the WTR studies. They reported negligible effects in 39% of the studies and substantial effects in 39% of the studies. Examination of the studies utilized for this review indicate that WTR is more effective than traditional methods in teaching writing in kindergarten and less so in the first grade. WTR's effect on reading is less pronounced at either level. Critics suggest that these results are to be expected, since writing is not a strong component of traditional kindergarten and first grade curricula (Pisapia & Perlman, 1992).

Integrated Learning Systems (ILSs). In fifty-one (51) ILS studies, Pisapia & Perlman (1992) found that 54% had substantial effects, 15% had moderate effects and 31% had negligible effects. Overall, the achievement effect of 69% of the ILS applications constituted a conventional measure of practical educational significance. In fact, in 54% of the cases, teachers could expect the 50th percentile student in their class to move to the 72nd percentile, a 22% gain in achievement.

Interactive Video Instruction. Barbara McNeil and Karyn Nelson, in a meta-analysis of 63 Interactive Video Instruction studies conducted in the last ten years, found a substantial effect size of .53 (McNeil & Nelson, 1991).

Databases. Rawitsch (1988) studied database use of 339 students in 16 different classes in suburban districts. He found:

• Students solved a greater percentage of the problem correctly when using a computer.
• Students took more time to solve problems.
• There was no difference in efficiency (combination of accuracy & speed) between using a computer and not using it.
• Students liked using a computer.
• Students with a more structured work style were more efficient in using the computer than those with an unstructured style.
• Students performed more efficiently or as efficiently on real life exercises as on academic exercises.
• Students who practice using data bases periodically during school will be significantly more efficient at solving problems.
• Instructional strategies need to be matched to work style.

Other studies found that students who used data bases increased their ability to classify and solve certain kinds of problems (White, 1985; Rawitsch, 1987; Rawitsch and Bart, 1987; Underwood, 1985; and Ehnamr and Glenn, 1987).

Distance learning. Distance learning is defined as recognized learning that takes place at a site remote from the point of origination. It has the potential to address equity/opportunity and is a strategy for improving education in rural schools. Examples of such uses are interactive TV, audio conferencing, video text (data base access from home), electronic pens and electronic mail.

Not all students benefit from telecommunication learning as much as conventional courses. On the one hand, students like distance programs as well as conventional programs. However, many prefer conventional classes, if available. On the other hand, distance programs are most effective with students who work well independently, are self-disciplined and well-motivated, and score above average range for most of their courses (from Pisapia & Perlman, 1992 notes).

John Pisapia

Answers to questions found in this research brief have been synthesized from the following MERC publications. Copies can be purchased using the online order form on the publications page.

Pisapia, J., & Perlman, S. (1992, December). Learning technologies in the classroom: A study of results.

Pisapia, J., Schlesinger, J., & Parks, A. (1993, February). Learning technologies in the classroom: Review of the literature.

Pisapia, J. (1993, April). Learning technologies in the classroom: Case studies of technology intensive schools.

top of page | return to research list