Electronic Journal for the Integration of Technology in Education

A View of the Research on the Efficacy of CAI

Michael S. Jenks
Yuanpei University of Science and Technology

John M. Springer
National Information Assurance Training and Education Center
Idaho State University

Abstract
Recent increases in computing power of affordable personal computers have resulted in an increase in the development and use of various forms of computer-delivered instruction. Understanding the effects of this instruction on learning is important to any future implementations of computers for instruction. This review of the literature on the effectiveness of computer-assisted instruction (CAI) is an attempt to show that CAI can be an effective mode of instruction in the education environment. It is also intended to address three major issues within the body of research on the efficacy of CAI: (a) an aging body of literature; (b) the greater effectiveness of using CAI as a supplement to conventional instruction; and (c) the alleged superiority of CAI over conventional instruction. The authors conclude that, due to potential shortcomings in some past research comparing CAI to conventional instruction, CAI should be considered to be at least as effective as conventional instruction. Next, they conclude that new studies are needed to clarify the effects of CAI in contemporary student/computer environments. Finally, they conclude that how CAI is delivered can impact its effectiveness.

The Association for Education Communications and Technology (1977) has defined computer-assisted instruction (CAI) as a method of instruction in which the computer is used to instruct the student and where the computer contains the instruction which is designed to teach, guide, and test the student until a desired level of proficiency is attained. In this paper the authors attempt to present a contemporary view of a body of research on the effectiveness of CAI as a teaching tool, especially as it impacts academic performance. Following the definition for CAI above, this review addresses the question of whether or not CAI can be effectively utilized to teach, guide, and test students. It is presented in two parts: A general view of the research on the efficacy of CAI and an overview of issues surrounding that research.

There has been a dramatic increase in the capabilities of computers, along with reduced cost, that has influenced an increase in the various forms of computer-delivered instruction (Brown, 2001). This increase has been seen in education as well as in other disciplines (Passerini, 2000). Throughout the 1980's and 90's computers have been generally heralded as being an effective teaching methodology (Christmann & Badgett, 2000). However, this "heralding" may still lack adequate research. Coffland (1999), in discussing the status of technology use in mathematics education, noted there is ample justification for research into how computers are used in education. In their focus on the status of research on the efficacy of CAI, Christmann and Badgett (2000) also suggest a need for further research by arguing that, "despite the accolades heralding CAI as the effective teaching methodology, there is still no documented evidence verifying its perceived superiority" (p. 92). While the authors of this current paper also feel more research is needed concerning the effectiveness of computers in education, the research below is felt to adequately show that CAI can be an effective mode of instruction in the education environment and is not offered to "prove" the superiority of CAI (Christmann & Badgett, 2000; Christmann, Badgett, & Lucking, 1997).

Research on the Efficacy of CAI

Dalton and Hannafin (1986, January), in a study involving junior high students, found that CAI alone tended to be the most effective instructional delivery system compared to video alone and interactive video. In a large curriculum integration project involving 1st through 8th grades, Lore and Chamberlain (1988) found that a CAI integrated curriculum was effective (e.g., reached a pre-established learning benchmark) when averaged over all grades. However, some grades performed at or above the benchmark while other grades performed below. Specifically, they found that grades 3, 4, 6, and 7 met or exceeded the goal level of academic achievement; grades 1, 2, 5, and 8 did not reach the goal level (p. 4). Price (1989) conducted an attitude survey and observed student progress in a middle school science project where CAI was used as a tutorial and research tool. It was concluded that the use of CAI in this way encouraged an overall improvement in motivation and interest in the science research project. Christmann, et al. (1997) conducted a meta-analysis of the effect of CAI in secondary education for the years of 1984 to 1995. They selected only studies that were correlative, quasi-experimental, or experimental in design and concluded that CAI had a greater effect size in the 1980s than it did in the 1990s (through 1995). Their research indicated that for the 12 year period (1984 to 1995), secondary students exposed to CAI showed "higher academic achievement than 57.2% of those students exposed to traditional instruction" (p. 328).

Roberts and Madhere (1990), in a study involving elementary and junior high schools, stated that their "findings indicate marginal successes in academic gains in reading and mathematics and an overwhelming positive student attitude toward the computer assisted medium of instruction and learning" (p. 45). In a report on the academic progress of mathematics and physics students taking CAI-based advanced placement courses (middle school through early high school), Ravaglia, Suppes, Stillinger, and Alper (1995) argued that such courses were shown to be effective for the targeted students. As evidence, they suggested that these students, upon completion of the CAI courses, scored especially high on Advanced Placement (AP) exams from the test years 1991-1993. Matejczyk (1996) implemented commercial software on creative writing and concluded that the software was a helpful tool in improving students' writing. In a study of CAI in a secondary science classroom, Brophy (1999) felt that the results indicated that CAI is effective in science classroom settings. Tseng (1999) found that a mathematics CAI was useful in teaching first grade students; results indicated that most students advanced in knowledge. In a similar study involving elementary students, Chang (2000) reported a significant increase in scores on a measure of academic achievement when CAI on arithmetic was used to teach addition and subtraction. Two final studies indicate that CAI can have a positive effect on skills and achievement for at risk secondary students. In the first study, Stern and Repa (2000) show that CAI was successfully used to teach social skills to teens enrolled in a behavior modification program. In the second study, Dunn (2002) found that at risk high school freshmen in the CAI treatment group scored significantly higher than the control group on a measure of reading comprehension.

Post-Secondary Students

Research from the 1980's suggests that (CAI) has had a positive effect on a variety of students (age) in the formal education environment. Wesley, Krockover and Hicks (1985, Winter), while examining the effects locus of control has on the acquisition of computer literacy, found that externally controlled preservice teachers learned better in the CAI mode than in the comparison text mode, while internally controlled preservice teachers learned equally in both modes of instruction. Danley and Baker (1988) conducted a study involving the teaching of special education 'mainstreaming' to preservice teachers using CAI and traditional lecture modes of instruction. They found that neither method proved superior to the other in a measure of content learning. In another study involving preservice teachers, Rowland (1988, April) argued that the results of the study indicate tutorial CAI should be used for teaching basic concepts and simulation CAI would be better suited to teaching the application of concepts. Further, it was suggested that if CAI is to be effective, educators should match the CAI type to the intended learning goal/s (Rowland, 1988, April).

Research from the 1990's also suggests that CAI has had a positive effect on a variety of students in post-secondary settings. Sasser (1990-91, Winter) found that the mathematics achievement of those preservice teachers who received computer tutorial assignments was significantly higher than those who received the textbook assignments. Tirosh, Tirosh, Graeber, & Wilson, (1990-91) reported on a study designed to use CAI to correct preservice teachers misconceptions about division story problems. It was found that CAI was effective in improving performance in writing expressions for division word problems, increasing awareness of a tendency to divide by the smaller number, and helping teachers explicitly recognize and correct their misconception about the relative size of the dividend and the divisor. Ivers and Barron (1998) reported significant learning increases when preservice teachers worked in a paired condition using computer-based instruction that was designed for individual learning. Glickman (2000) conducted a non-equivalent control group design study and found that the CAI treatment group did significantly better than the control group on concept understanding. However, there were no significant differences between the treatment and control groups on a measure of achievement. Huxford (1999), in a study comparing traditional instruction modes with a CAI mode, argued that the results suggest CAI is not as useful for instruction as previously believed. In the study, college students from the different groups did not perform significantly different on measures of cognitive and affective learning.
Two recent studies further support the efficacy of CAI. Springer (2002) studied academic performance improvements in delivering metric CAI (as a proctored tutorial) to preservice teachers and found that those individuals who received the CAI performed significantly better on a metric posttest than those students who received no instruction. Jenks (2002), in comparing paging text and scrolling text versions of a linear metric CAI module, found with both the paging and scrolling treatments, preservice teachers who scored low on the pretest had a much larger mean improvement on the posttest than did the students who scored high on pretest. This could indicate the linear metric CAI was effective as a remedial tool.

Overall, the review of individual studies involving college age students suggests that CAI is an effective mode of instruction. The meta-analyses below indicate the same conclusion.

Meta-Analytic Studies

Research from the 1980s and 1990s generally supports the efficacy of CAI in formal education environments. The above studies, while not representing an exhaustive listing of research, are offered in support of this assertion. The following meta-analyses (and a single review of past meta-analyses) also indicate that CAI is generally effective in education environments for a broad range of student ages.

Kulik, Kulik, and Bangert-Drowns (1985) analyzed 28 studies that examined the effectiveness of CAI with elementary students. Of these studies, 27 examined terminal-mainframe configurations, and only one looked at microcomputer configuration. CAI related instruction appeared to improve student achievement by 0.47 standard deviations, on average, over students receiving conventional instruction. The authors calculated that a typical student scoring in the 50th percentile with conventional instruction would score in the 68th percentile with CAI.
A meta-analysis of achievement effects on microcomputer applications in elementary schools was conducted by Ryan (1991). The results of 40 independent studies that looked at the effects of CAI in reading and mathematics were analyzed. Ryan found that CAI raised academic achievement, on average, by 0.309 standard deviations. The author concluded that a typical student's score would be raised from the 50th percentile to the 62nd percentile when exposed to CAI (Ryan, 1991).

Research examining the effects of computer-assisted instruction has shown generally positive results on secondary students. Kulik (1983) found that computer-assisted instruction has the potential for improving student achievement scores in pre-college classes. The results of the meta-analysis showed that the average effect of CAI was to raise student achievement by approximately 0.4 standard deviations.

In 1983, Kulik, Bangert, and Williams analyzed 48 studies on the effects of computer-based teaching on secondary students in mathematics and science. Thirty-nine of these studies reported those students with computer-based teaching scored better on final examinations than did students in conventionally taught classes. The other nine studies reported students in conventionally taught classes scored better on final examinations.

Further support for computer-assisted instruction at the secondary level is provided by Snowman (1995). The results of this study showed positive effects of computer-based education on secondary students. The typical student in a computer-based class scored at the 60th percentile while the typical student in a traditional class scored at the 50th percentile on final examinations.
Fletcher-Flinn and Gravatt (1995) examined studies on CAI from 1987 to 1992 and found that there is an overall favorable effect size at all grade levels for CAI versus traditional classroom settings. Included in the meta-analysis were several studies in which the same teacher taught both the CAI and "traditional" versions of a course; the researchers isolated the results of these studies and found no significant differences between CAI and traditional instruction. Further, Fletcher-Flinn and Gravatt (1995) found that, for those studies using paper and pencil equivalents of the CAI, there were no significant achievement differences between treatment and control groups.
Christmann et al. (1997) examined research from 1984 through 1995 that studied CAI effects on academic achievement of secondary students (grades 6-12). They found a small positive effect size of 0.187 for CAI - students exposed to CAI showed "higher academic achievement than 57.2% of those students exposed to traditional instruction" (p. 328). Christmann et al. (1997) noted that the effect of CAI, based on the studies used in the meta-analysis, declined between 1984 and 1995.

Christmann and Badgett (2000) compared the academic achievement levels of college students who had classes that used traditional methodology with those of college students who had classes in which CAI was used as a supplement to traditional methodology. The authors compiled data from 26 studies and calculated an overall mean effect size of 0.127. When exposed to CAI, typical student achievement moved from the 50th percentile to the 55th percentile (Christmann & Badgett, 2000).

Lowe (2001) reviewed several other meta-analyses from the 1980s and 1990s and concluded each of the reviewed meta-analyses showed a small positive effect size for computer-based education (CBE; the author describes CBE as a term that includes CAI) over conventional instruction. However, Lowe (2001) states that research indicates that, where CBE and conventional instruction is delivered by the same instructor, the CBE advantage is reduced to insignificant levels; further, simulation and tutorials as supplements to conventional instruction appear to be the most effective.

Issues within the Research

In reviewing the literature on CAI, several issues came to light. First, a preponderance of the research occurred in the 1980s. Even recent meta-analyses (e.g., Christmann & Badgett, 2000; Christmann & Badgett, 1997; Fletcher-Flinn and Gravatt, 1995) included studies from the 1980s and early to middle 1990s. Second, some research indicates that how CAI is used will affect its effectiveness (e.g., Kulik & Bangert-Drowns, 1983; Kulik, Kulik & Shwalb, 1985; 1986; Lowe, 2001). Third, some researchers have argued that the methodology of many studies comparing CAI to conventional instruction has unfairly favored CAI and that this is reflected in the results of meta-analytic studies (e.g., Fletcher-Flinn and Gravatt, 1995; Lowe, 2001).

The Aging Research Base

The bulk of research reported above occurred in the 1980s. Further, much of the research utilized in more recent meta-analyses also came from the 1980s. This fact may be especially important when considering differences between computer environments and computer users (learners in instructional environments) of the 1970s and 1980s and those of the late 1990s through 2002. For example, Kulik, Kulik, and Bangert-Drowns (1985) reported in their meta-analysis that the bulk of elementary students in the examined studies used mainframe-terminal configurations, rather than microcomputers. Much of the computing was text-based and non-graphical. In considering the trend of lower effect sizes for the early 1990s reported by Christmann et al. (1997), more contemporary studies, including meta-analyses, are needed to examine whether or not this trend has been maintained and to explore possible causes of such a trend.

Using CAI as a Supplement

Using CAI as a complete replacement for conventional teaching may seriously weaken its effectiveness (Kulik & Bangert-Drowns, 1983). CAI is not often used in this way, but Hartley (as cited in Kulik & Bangert-Drowns, 1983) managed to locate a few studies in which CAI totally replaced conventional teaching. The results of these studies were considered to be unimpressive. Based on Hartley's results, Kulik and Bangert-Drowns (1983) warn that, "Total reliance on the computer as teacher therefore seems to be one thing that school systems should avoid" (p. 154).
Other studies also support the use of CAI primarily as a supplement, or enhancement, to conventional instruction. Edwards, Norton, Taylor, Weiss, and Dusseldorp (1975) reviewed studies at various educational levels and in a range of subjects, and they found the effectiveness of CAI was most apparent when it was used as a supplement to conventional teaching. However, findings were less clear when CAI was substituted, in whole or in part, for traditional instruction. In looking at how computers were used in teaching situations, Kulik, Kulik and Shwalb (1985; 1986) found that Computer-Enriched Instruction (CEI) raised achievement by an average of 1.13 standard deviations (Note: The authors of the current article feel that CEI is often, but not always, CAI used as a supplement within conventional teaching; the distinction is not always clear in the literature); Computer-Managed Instruction (CMI) raised achievement by an average of 0.72 standard deviations; and Computer-Assisted Instruction (CAI) raised achievement by an average of 0.29 standard deviations. Lowe (2001), in reviewing meta-analyses, stated that computer-based education should be used to enhance conventional teaching methods. The computer may provide a new form of presentation and allow for some learning without the presence of an instructor; however, the computer alone may not be as effective as the computer with an instructor (Lowe, 2001).

Critique of the Research

Clark (as cited in Fletcher-Flinn and Gravatt, 1995), argued that much of the apparent success of CAI in the research literature may be related to the uncontrolled effects of instructional method and the novelty of the medium. Clark, according to Fletcher-Flinn and Gravatt (1995), further argued that when looking at studies that utilized the same teacher and a longer than typical study duration, the differences between CAI and conventional instruction were minimized. Fletcher-Flinn and Gravatt (1995) attempted to directly address this critique in their meta-analysis. In isolating studies of long duration and that utilized the same materials and teacher in the treatment and control conditions, they found no beneficial effect in favor of CAI (Fletcher-Flinn & Gravatt, 1995). Lowe (2001), in reviewing various meta-analyses, supports this idea by stating that, "When instruction of CBE and conventional classroom is delivered by the same person, the learning advantage for CBE is reduced to insignificant levels" (p. 170). Fletcher-Flinn and Gravatt (1995) point out that isolating these conditions may be highlighting a potential difference between the CAI and conventional instruction of many studies. This difference is that much CAI is carefully designed with sequenced instruction following clear instructional objectives. This generally high quality CAI is then compared to conventional instruction that is often not as carefully designed. Fletcher-Flinn and Gravatt (1995) suggest that this may be a part of the difference between CAI and conventional instruction in many studies.

Conclusions

In view of the age of the research, and in examining the potential shortfalls of much of that research, more research should be conducted on what makes CAI effective. Does CAI fair as well in contemporary settings with contemporary learners? What subjects are most conducive to CAI? What is an effective relationship between CAI and conventional instruction?

Most of the research literature can be used to claim a superiority of CAI over conventional instruction. However, discussions about the methodologies of much of the research reduce the significance of such claims. When studies control for internal validity issues such as instructional equivalency and instructor equivalency, differences between CAI and conventional instruction appear to be insignificant.

Considering this still allows one to conclude that CAI appears to be at least as effective as conventional instruction. In answering the question, "Is CAI an effective teaching tool?" the answer should be, "Yes. But don't throw out conventional instruction for CAI." This is because the literature further suggests CAI is best delivered as a supplement to conventional instruction and effective instruction appears to be more a matter of quality of design rather than a matter of medium (see Lowe, 2001).

Contributors

Michael S. Jenks, Ed.D., is Assistant Professor at Yuanpei University of Science and Technologyin Hsinchu, Taiwan. Also a teacher of English, Dr. Jenks prepares instructional multimedia training and instructional design materials for delivery to faculty. He is currently directing the development of the Yuanpei Instructional Multimedia Resource Center.

John M. Springer, Ed.D., is a Post-Doctoral Fellow at the National Information Assurance Training and Education Center at Idaho State University in Pocatello, Idaho. Dr. Springer serves as the center's instructional designer and his research interests include the examining the effects of using instructional systems design models on improving multimedia instructional systems.

References

Association for Educational Communications and Technology. (1977). Educational technology: Definition and glossary of terms (Vol. 1). Washington, D.C.: Association for Educational Communications and Technology.

Brophy, K. A. (1999). Is computer-assisted instruction effective in the science classroom? Masters Abstracts International, 37 (5), 1302. (UMI No. 1393859)

Brown, K. G. (2001). Using computers to deliver training: Which employees learn and why? Personnel Psychology, 54, 271-296.

Chang, J. C. (2000). A field test of CAI software: MagicTree. Masters Abstracts International, 38 (6), 1438. (UMI No. 1399856)

Christmann, E., Badgett, J., & Lucking, R. (1997). Progressive comparison of the effects of computer-assisted instruction on the academic achievement of secondary students. Journal of Research on Computing in Education, 29(4), 325-337.

Christmann, E. P., & Badgett, J. L. (2000). The comparative effectiveness of CAI on collegiate academic performance. Journal of Computing in Higher Education, 11(2), 91-103.

Coffland, D. A. (1999). Factors related to technology use by secondary geometry teachers. Unpublished Dissertation, Idaho State University, Pocatello, ID.

Dalton, D. W., & Hannafin, M. J. (1986, January). The effects of video-only, CAI only, and interactive video instructional systems on learner performance and attitude: An exploratory study. Paper presented at the Annual Convention of the Association for Educational Communications and Technology, Las Vegas, NV.

Danley, W. E., & Baker, C. (1988). Comparing a pre-service mainstreaming class taught by traditional methods with a similar class taught by computer assisted instruction. Computers in the Schools, 5(1-2), 151-255.

Dunn, C. A. (2002). An investigation of the effects of computer assisted reading instruction versus traditional reading instruction on selected high school freshmen. Unpublished Dissertation, Loyola University, Chicago.

Edwards, J., Norton, S., Taylor, S., Weiss, M., & Dusseldorp, R. (1975). How effective is CAI? A review of the research. Educational Leadership, 33(11), 147-153

Fletcher-Flinn, C. M., & Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI): A meta-analysis. Journal of Educational Computing Research, 12(3), 219-242.

Glickman, C. L. (2000). The effects of computerized instruction in intermediate algebra. Dissertation Abstracts International, 61 (5), 1773A. (UMI No. 9973965)

Huxford, D. E. (1999). The Relative Efficacy of Computer Assisted Instruction. Unpublished Thesis, West Virginia University.

Ivers, K. S., & Barron, A. E. (1998). Using paired learning conditions with computer-based instruction to teach preservice teachers about telecommunications. Journal of Technology and Teacher Education, 6(2-3), 183-191.

Jenks, M. S. (2002). An examination of the effect of scrolled and paged text fields on academic performance in the delivery of a computer-assisted instruction module for teaching metrics to preservice teacher education students. Unpublished Dissertation, Idaho State University, Pocatello, ID.

Kulik, J. A. (1983, October). Effects of computer-based teaching on learners. Paper presented at the National Forum of the College Board Symposium on Computer Competency and the Curriculum, Dallas, Texas. (ERIC Document Reproduction Service No. ED 246 877)

Kulik, J. A., Bangert, R. L., & Williams, G. W. (1983). Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 75(1), 19-26

Kulik, J. A., & Bangert-Drowns, R. L. (1983). Effectiveness of technology in precollege mathematics and science teaching. Journal of Educational Technology Research, 12(2), 137-158

Kulik, J. A., Kulik, C. L. C., & Bangert-Drowns, R. L. (1985). Effectiveness of computer-based education in elementary schools. Computers in Human Behavior, 1(1), 59-74

Kulik, C. L. C., Kulik, J. A., & Shwalb, B. J. (1985, April). Effectiveness of computer-based adult education. Paper presented at the 69th Annual Meeting of the American Educational Research Association, Chicago, IL. (ERIC Document Reproduction Service No. ED 263 888)

Kulik, C. L. C., Kulik, J. A., & Shwalb, B. J. (1986). The effectiveness of computer-based adult education: A meta-analysis. Journal of Educational Computing Research, 2(2), 235-252

Lore, R., & Chamberlain, E. (1988). Language development component, compensatory language experiences and reading program 1987-88. Final evaluation report. Final evaluation report for Columbus Public Schools, OH, Department of Evaluation Services.

Lowe, J. (2001). Computer-based education: Is it a panacea? Journal of Research on Technology in Education, 34(2), 163-171.

Matejczyk, M. (1996). Using multimedia technology to teach 7th grade process. Curriculum Administrator, 31(2), 6-9.

Passerini, K. (2000). A comparative analysis of performance and behavioral outcomes in different modes of technology-based learning. Unpublished Dissertation, George Washington University, Washington D.C.

Price, S. M. (1989). Using CAI to improve participation and achievement in science research projects in middle school science. Unpublished M.S. Practicum Paper, Nova University.

Ravaglia, R., Suppes, P., Stillinger, C., & Alper, T. M. (1995). Computer-based mathematics and physics for gifted students. Gifted Child Quarterly, 39(1), 7-13.

Roberts, V. A., & Madhere, S. (1990). Chapter I resource laboratory program for computer assisted instruction (CAI) 1989-1990. Evaluation report. Evaluation report for the District of Columbia Public Schools, Washington, DC.

Rowland, P. M. (1988, April). The effect of mode of CAI and individual learning differences on the understanding of concept relationships. Paper presented at the Annual Meeting of the International Association for Computing in Education, New Orleans, LA.

Ryan, A. W. (1991). Meta-analysis of achievement effects in microcomputer applications in elementary schools. Educational Administration Quarterly, 27(2), 161-184

Sasser, J. E. (1990-91, Winter). The effect of using computer tutorials as homework assignements on the mathematics achievement of elementary education majors. Journal of Computers in Mathematics and Science Teaching, 10(2), 95-102.

Springer, J. M. (2002). The formative evaluation of a computer-assisted instruction module for metric area instruction, for preservice teachers: Its effect on student achievement and its congruence with the ADDIE instructional design model. Unpublished Dissertation, Idaho State University, Pocatello, ID.

Snowman, J. (1995). Computer-based education: More hype than help? Mid-Western Educational Researcher, 8(1), 32-35

Stern, R., & Repa, J. T. (2000). A study of the efficacy of computerized skill building for adolescents: Reducing aggression and increasing pro-social behavior. Research Report available from EDRS (ERIC) (http://orders.edrs.com/members/sp.cfm?AN=ED447792): New York City Board of Education.

Tirosh, D., Tirosh, C., Graeber, A., & Wilson, J. (1990-91). Computer-based intervention to correct preservice teachers' misconceptions about the operations of division. Journal of Computers in Math and Science Teaching, 10(2), 71-78.

Tseng, H.-C. (1999). Computer-assisted instruction in the Math Family. Masters Abstracts International, 37 (5), 1303. (UMI No. 1394146)

Wesley, B. E., Krockover, G. H., & Hicks, C. R. (1985, Winter). Locus of control and acquisition of computer literacy. Journal of Computer-Based Instruction, 12(1), 12-16.