How to Conduct a Cost-Effectiveness AnalysisMetropolitan Educational Research Consortium

The purpose of this research brief is to describe how to conduct a cost-effectiveness analysis.

How do you conduct a cost-effectiveness analysis?

Three tasks must be completed to complete a cost-effectiveness analysis. First, the costs must be determined for alternative ways to deliver the program. Next, the effectiveness must be determined. Finally, the cost-effectiveness between the program alternatives is established.

TASK I: Are there any differences in the cost of operating program alternatives?

Every cost analyses begins by describing the program, determining the costs and cost savings of the program alternative, and then analyzing the costs in a decision framework.

Step 1: Describe each program alternative and its components and potential benefits. The description acts as a guide to determining costs and intended effectiveness of the project.

Step 2: Determine the costs of program alternatives by identifying cost factors, deciding on the data entry method, entering the cost data and calculating program costs.

  1. Identify cost factors. Cost analysis begins with identifying the cost factors required for a program, project, alternative or intervention. Cost factors include computer hardware, software, program development, direct personnel, facilities, training and evaluation.

    Direct personnel refers to personnel required to operate the program. In addition, it allows you to identify personnel required in each of the other cost factors (indirect personnel). An example would be, personnel costs related to developing software, personnel required to deliver training.

  2. Decide on the Method to Enter Data. Once cost factors have been identified and stipulated, it is necessary to ascertain their costs. Program costs are generally measured by three approaches: the estimated (aggregate) approach, the detailed (ingredients) approach, and the project evaluation approach. Only the first approach will be discussed.

    Estimation Method. In general, most estimates focus on costs associated with hardware and assume other costs are the same as normal school operations. This practice tends to underestimate the cost of instructional use of technology by focusing on what might be the least expensive item, the hardware. However, estimates of cost data can be reasonably accurate if internal data are good and estimates are made carefully.

    The unstated assumptions of the cost estimation method usually accounts for wide variance in cost estimates. Inaccuracies can also be attributed to faulty estimates in the following cost factors: 1) operating costs, telecommunications and courseware development, 2) rate of technology use, 3) number of students served per day, week, etc., 4) the length of the school day and year, 5) amount of time and number of sites at which the system will be used, and 6) the life span of system hardware (3-5 years) and courseware (10 years for basic skills software and less for other software).

    There are several ways to approach cost estimation. Equipment can be estimated by price of equipment currently available. Non-equipment costs can be estimated at three times the investment in equipment. The value of facilities can be determined by estimating their lease value. The annual value of facilities and equipment can be estimated through a relatively simple approach that takes into account depreciation and interest foregone by remaining capital investment.

    A typical estimation scenario is: costs may be estimated by examining the prices of high quality equipment currently available. For example, in 1992, the cost of an IBM compatible computer with monochrome monitor, 140 megabyte hard disk drive, a modem and a printer was $______ from vendors on a state contract. To provide one work station with this equipment for every 25 students would cost roughly $________, not including software, furniture, local area network, staff training, etc.

    In general, the following rules of thumb can be used in cost estimation:

    Market Value. The value of a cost factor is its market value and/or its market price. Market price is particularly useful when costing out equipment, materials and utilities. In the case of personnel, market value may be ascertained by determining what the costs would be for hiring a particular type of person. Such costs must include not only salary, but also fringe benefits and other employment costs paid by the employer.

    Lifetime costs. Lifetime costs are deferred costs spread out over a period of time. As true cost declines, the more they can be differed over time. Lifetime costing accounts for the expected useful life of each cost factor. For example, lifetime costs of computer hardware (8 years) and software (4 years) were obtained by surveying experts in the field of educational microcomputing.

    1. Lease Value. The value of facilities can be determined by estimating their lease value.

    2. Detailed Method. The detailed approach is the most accepted approach to determining costs. Even though it is the most time consuming method, the detailed approach is also the most accurate method because it requires that each cost factor be broken into its separate components, it provides cost data on each (determining annual and deferred costs), and sum these to a total.

    3. Project Evaluation Method. The project evaluation method provides information to continue the project, adjust the project, or replicate the project. It permits you to: 1) evaluate the actual costs against the projected costs used when justifying the purchase decision, 2) appraise actual effectiveness of the project on a number of indicators for which data was collected during the evaluation period, and 3) make future decisions regarding the cost-effectiveness of the project on a solid foundation.

  3. Enter Cost Data. Once the data entry method has been decided, each factor is costed, and summed to obtain a total cost. This step is most important in developing accurate cost analyses. It is strongly affected by the availability and reliability of cost data. Three principal sources of information - reports, observations and interviews - can be used to assure the accuracy of the data by comparing findings from each source and reconciling differences.

    Levin (1981) admonishes educators to be systematic rather than casual. Data from accounting and budgetary reports must be used selectively and appropriately. This data cannot be relied upon for all cost factors. For example, they omit completely or understate cost savings such as: the cost of volunteers and other donated resources, or capital improvements charged during the year of their purchase when they have a life of 20-30 years.

Step 3: Determine Cost Savings. Are there cost savings associated with the project? Cost savings can be: 1) predicted prior to adoption of an alternative, 2) collected during and after completion of an adopted alternative, or 3) analyzed and evaluated prior to replication.

The person who is designing the project determines what cost savings are expected to occur as the result of a specific expenditure. For example, is equipment or personnel being donated? What tax savings can be generated? How much fuel savings occurs?

Step 4: Calculate and Analyze Program Costs. The total cost of a project is obtained by summing the costs for each factor and subtracting the cost savings.

Once cost data are generated, the problem of appropriate analysis must be faced. The first level of analysis is cost feasibility. If an alternative exceeds the available resources it cannot be considered. Simply, are the costs feasible? If no, adjust costs, or discard the project and consider an alternative.

However, don't be hasty. Can trade-offs be made to make the costs feasible? For example, can you increase class size in order to hire a computer coordinator? Can planning time be used for staff development time? Can more software be substituted for more computers? Can more costs be deferred? Can you upgrade memory rather than buy more powerful computers?

TASK II: Are there any differences in the effectiveness and cost-effectiveness of feasible program alternatives?

Cost-effectiveness takes information on the effectiveness of a program and compares it to the cost of the program in a systematic way. The basic technique has been to

1) determine costs per unit of measure - i.e., cost per pupil, cost per work station, 2) derive effectiveness results, and 3) combine these results with cost per unit data to determine the cost-effectiveness ratio of each alternative.

Cost-effectiveness analysis assumes that: 1) the evaluation of effectiveness can be made separate from the evaluation of costs, 2) programs with similar or identical goals can be compared, and 3) a common measure of effectiveness can be used to assess them.

Step 5: Determine the effectiveness of program alternatives.

Choosing Appropriate Effectiveness Measures. Cost analyst's results and decision maker premises can only converge through increased use of multiple indicators of effectiveness. The use of single dimensions of effectiveness is inappropriate when one considers that many decision makers have multiple goals, and that all educational activities have multiple stakeholders (Windham, 1990).

Therefore, before conducting an analysis the cost analyst should identify appropriate measures of effectiveness held as critical by key decision makers. This improves the relevance of the quantitative data.

At its simplest level, cost-effectiveness assesses total costs against unit costs. At its most sophisticated level, cost-effectiveness assesses outcomes in educational terms (e.g., student achievement, drop outs, etc.) in relationship to their costs. Two types of measures are emphasized in this paper because of their frequency of use, general availability from standard data sources, and meaningfulness for policy analysis: unit costs and achievement costs.

Unit costs. There is policy insight to be gained from the analysis of cost, even when effectiveness measures are not available. Analyzing by unit cost allows one to identify areas of potential inefficiencies by studying specific opportunities to improve costs per unit for a project. Often unit cost analysis is all the cost or effectiveness data permits. The first step in determining unit cost is to determine the unit or units of analysis (i.e. per student, classroom, school, bus, work station). Then determine the number of units affected by the program, for example, 1,000 pupils, 55 buses, 75 work stations.

Achievement costs. The purpose of the achievement cost analysis is to identify whether a change in expenditure can lead to a change in achievement. The desired measure of outcomes is selected based on the value judgments of the key decision makers. If decision makers are interested in student opinions, they may find little value in information on enhanced mathematical skills. When decision makers have multiple subject areas of interest, they should analyze them individually rather than create an index. This permits decision makers to impose their values on interpretation of the results rather than the values of the presenter of the analysis. Once the achievement measures are selected, one simply compares the cost variations on the achievement measures.

Decide on Data Entry Method and Enter Effectiveness Data. Gains on an effectiveness measure can be:

Predicted prior to adoption. Where predictions are being made, the person who is designing the project determines what change is expected to occur in the unit of measurement. For example, how many points of improvement are expected in a test score because of the anticipated expenditure; or how many students will not drop out?

Collected during the program. Cost-effectiveness can also be determined with the use of actual data that is collected during the program and/or after its completion. This method of collection allows decision makers to analyze, evaluate and adjust the project prior to replication.

TASK III: Calculate and Analyze Cost-Effectiveness Ratios

Calculation. The cost-effectiveness ratio for each alternative identifies how much it costs per unit to achieve a unit of gain for each program alternative. It is calculated by dividing the cost by the effectiveness for each achievement outcome, thereby yielding the cost to accomplish one unit of gain (for instance, one month of grade placement gain) per student. For example, for the dropout rate category a cost-effectiveness ratio of $3,500/1 indicates that preventing one student from dropping out would cost $3,500. Or, in the category of test scores, a ratio of $280/1 indicates that achieving a test score gain of 10 points costs $2,800 ($280 x 10).

Analysis. Caution should be used in interpretation of ratios. It is generally assumed that cost-effectiveness increases the more cost units are affected, but not always. For example, when any cost factor is fixed (it does not increase in amount with enrollment), then the unit cost (per pupil) declines as enrollment increases. It is also possible that more effective strategies cost more (i.e. school size increases may cause costs per student to decrease and educational effectiveness to decline.)

Step 6: Making Decisions with Cost-Analysis Results.

The fact that computer-related activity is frequently "belief driven" rather than goal driven, argues for incorporating cost utility/feasibility analyses into the decision making process. These analyses evaluate outcomes in terms of their subjective value to the decision maker, or in terms of resources available for acquisition.

Of course, alternatives with the lowest costs relative to effects should have the highest priority for decisions. While it is the responsibility of decision makers to be informed of the quantitative findings, the final decision almost always comes down to a question of applying their own values and experiences in interpreting the available data.

In the final analysis, all educational decisions, from the student-teacher interaction to the formulation of national policy, are cost utility decisions.

Even when the decision makers accept the analysis data, they still must interpret the data for themselves in terms of the larger social or political systems within which they operate, and their own values. The best objective data will not eliminate the need for subjective judgments by decision makers.

In addition to decision maker beliefs, there are four other issues that must be raised before making a final decision.

  1. Are the results representative for the level of scale of the intervention that is appropriate? Some interventions have very different average costs per student when used for 30 students than when used for 1,000. For example, a Computer-Assisted Instruction (CAI) system that uses an expensive minicomputer might be very costly for 30 students relative to stand-alone microcomputers. Thus, it is appropriate to make sure that the scale of use on which the cost-effectiveness ratio is based is pertinent to the decision making context.

  2. Can the results be extrapolated for effects that are beyond those found in the interventions. For example, if $18 per student for alternative C generates an additional unit of effectiveness, would $180 generate 10 additional units of effectiveness? The answer to that question is that we do not know without further analysis of effectiveness under different intensities of resource use. Any generalization beyond the range covered by the analysis must be based on additional evidence rather than being extrapolated.

  3. Can the magnitude of differences among alternatives be correctly interpreted through cost-effectiveness ratios? Decisions should not be based upon results that suggest only small differences in cost-effectiveness. For instance, when the differences are small, within a 10% - 20% range, it is probably best to use other criteria to make the decision. When estimated cost-effectiveness results are not markedly different, given the attainable precision in results, other considerations such as the ease of implementation, acceptability by staff, and experience with the intervention should be used to make the decision (Levin, 1943). However, in other cases, differences in cost-effectiveness among alternatives may be as large as 4, 5, or 10 to 1. In these cases, there are likely to be important gains in using resources for the most cost-effective of the alternatives.

  4. When applying cost-effectiveness analysis, how does its inherent bias toward short-term results affect the overall conclusions? This problem is pronounced with respect to decisions about new educational technologies. Evidence that CAI can be cost-effective in raising test scores is an important finding. But for many educators, the computer's greatest promise is its potential to push back the traditional frontiers of cognition and learning and make possible the kinds of intellectual endeavor for which measures of achievement have not yet been invented.

John Pisapia

Answers to questions found in this research brief have been synthesized from the following MERC publications. Copies can be purchased using the online order form on the publications page.

Pisapia, J., Schlesinger, J. & Perlman, S. (1993). Cost analysis program (CAP), software and user manual.

Pisapia, J. (1994). Cost analysis and learning technologies.

top of page | return to research list