Evaluability Assesment: Is Your Program Ready to Be Fully Evaluated?

Evaluator's corner

By Julia Kohn, Evaluation Manager

Evaluating the effectiveness of your sexuality education program is a crucial step. It can also require a good deal of resources, such as money and staff time. Often, an organization will decide to undertake an evaluation simply because money for evaluation happens to become available. However, before you spend time and money conducting an outcome evaluation of your program, it is important to know whether your program is ready to evaluate. If a program is not ready for evaluation, or if it is not ready for a particular type of evaluation, it may not yield meaningful results, and therefore it would be better not to evaluate at this time, or to conduct a different type of evaluation.

Evaluability assessment is a systematic way to determine whether a full evaluation of program outcomes is warranted. There are many benefits to evaluability assessment. Undergoing this process can

• prevent waste of organizational resources on program evaluations of dubious value
• enable program stakeholders to determine if a full evaluation is necessary help stakeholders to formulate meaningful evaluation questions
• build consensus among stakeholders regarding the evaluation, and stakeholders' commitments to make appropriate use of the findings

Researchers in the field of program evaluation tend to agree that a program must satisfy three basic criteria in order to be ready for evaluation:

1.   The program should be operating as intended;

2.   The program should be relatively stable and;

3.   The program should seem to be achieving positive results. Evaluability assessment is a systematic way to determine whether or not these criteria are met.

There are three main stages of evaluability assessment. During the first stage, one must clarify the series of assumptions linking what the program does to what it hopes to accomplish. The evaluator may have to work with key stakeholders of the program to construct the program theory if it has not already been explicitly stated. One must then examine the program theory or logic to determine whether stakeholders' assumptions about how the program works are reasonable. For example, Program A is designed to reduce teen pregnancy in a specific community by offering a series of three presentations in local schools covering safer sex practices and contraception. The theory behind Program A is that three hours of education will lead community teens to delay intercourse or practice safe sex, thus significantly reducing the teen pregnancy rate in that community. While this would be great, it is not likely that such a program alone will achieve this outcome. If a program's logic is not reasonable, the program, or its goals and objectives, should be revised before evaluating.

During the second stage, one determines whether the program has been implemented well enough to produce its intended outcomes. This stage examines whether the program has been implemented as intended. One might look at whether program facilities are in place; staff have been trained; program activities are being implemented as planned; the intended target audience is actually being reached; and whether the program has been operating long enough to have a chance of influencing its intended outcomes, assuming its effectiveness. This stage is very similar to process evaluation, and often involves document review (e.g., proposals, program records, logic models, etc.), site visits, and staff interviews. If this stage reveals major discrepancies between how the program was intended and how it is actually being implemented, these discrepancies should be corrected before proceeding with an evaluation.

The third stage of evaluability assessment requires one to examine how key stakeholders plan to use the results of the evaluation. The purpose of this step is to identify and prioritize the main evaluation questions, in order to determine if they make sense in light of the current stage of knowledge about the program, and to establish research methods that are practical and feasible for addressing them.

As long as the program is stable and is being implemented reasonably with some indication of positive results, the evaluator can move on to the next phase of evaluation. A preliminary study should then be designed to discover what kinds of positive outcomes appear, for which participants, under which program conditions.

Evaluability assessment can screen out programs that are not yet ready for program evaluation, but it can also act as a preliminary evaluation. If neither program theory nor program data indicate successful outcomes, the assessment can tell the funding parties and program staff about potential obstacles and weaknesses of the program. It can direct attention to the areas where the program needs improvement. These improvements must be made if the program stands a chance of achieving positive outcomes. Evaluability assessment also provides an opportunity for program funding providers and managers to determine how they will use the results from further evaluation. If your program is not "evaluation-ready," spend your time and effort on getting it ready, and save your money for an evaluation that will yield more meaningful results when the program is ready to be evaluated.

A Resource on Evaluability Assessment from Sociometrics:

Evaluation Readiness Assessment Guide: Is Your Program Ready To Evaluate Its Effectiveness?
by Valery Sedivy. Available at: http://www.socio.com/srch/summary/misc/evapub22.htm.

If you have questions about evaluation, would like assistance, or would like to request an Evaluator's Corner edition on a particular topic, please do not hesitate to contact me: Julia Kohn, Planned Parenthood Federation of America, 434 W. 33rd St., New York, NY 10001. Phone: 212-261-4629. Fax: 212-247-6269. Email: julia.kohn@ppfa.org