Process Evaluation

Evaluator's Corner

Process Evaluation

By Julia Kohn

What is Process Evaluation and Why is it Useful?

When we talk about program evaluation, most often we are referring to outcome or impact evaluation — evaluating a program to determine whether it has achieved the desired effects on program participants or the broader target community. When someone says, "did the program work?" they are asking for an outcome evaluation.

Before you can adequately measure the effects of a program, it is sometimes necessary to ensure that the program or service is being delivered as intended. This is the purpose of conducting a process evaluation. When paired with outcome evaluation, process evaluation can be very helpful in understanding why a program has the effects it has — or why it doesn't have the effects that were expected.

Process evaluation can be used to describe a program, to determine if a program is ready for outcome evaluation, and/or to enable program replication. It can be used to unlock the fundamental core of a program and to determine exactly what is taking place and how a program is being implemented at different sites or locations.

Where to Focus

When conducting this type of evaluation there are many aspects of the process that you can assess. For example, you probably want to pay attention to whether the program is being delivered in its entirety. If your program is intended to have ten two-hour sessions, were all ten sessions delivered and did each session last approximately two hours? Whether or not all of the sessions were delivered for the appropriate amount of time can have an effect on the program outcomes.

Similarly, you may also focus on whether each participant has access to the program in completion. For example, did each participant attend all ten sessions? Did some participants attend only seven sessions, or three sessions? Attendance and attrition can be expected to impact individual results. It is also useful to delve even deeper into these issues. For example, did participants who attended more sessions differ initially from those who attended fewer (in terms of socio-economic status, age, or geographic location)? Did participants who dropped out of the program have any commonalities? This is important to determine whether your program may be more accessible to some groups than others.

You might also choose to focus on whether the program content or curriculum was delivered as intended. Did the program facilitator stick to the curriculum? What percentage of the intended content was covered? Instructors should keep records and notes adequate to determine the answers to these questions.

Programs can be difficult to replicate due to the many factors that may differ from one implementation to the next. For this reason, it can be helpful to pay attention to the variable aspects of learning environments. Were participants fully engaged in the program, or merely present? Did the facilitator or instructor form a strong rapport with participants or merely deliver the material? You might use focus groups, in-class observation, interviews, or open-ended questionnaires to answer these questions.

Process evaluation, sometimes called monitoring, may address issues such as:

• How many people did the program serve?
• How many hours of staff time and/or what kinds of resources were required by the program?
• Do people return to the program or recommend the program to others?
• Which parts of the programs did participants find most useful? Which parts did they find least useful?

Explaining Your Findings

When conducting process evaluation, it is not sufficient to monitor the previously mentioned issues exclusively. The next step, whenever possible, is to determine why the circumstances existed as such. For example, why were all ten sessions not delivered? Why did some participants attend more sessions than others? Why was the program content not delivered as intended? Through the process of asking these questions it may be possible to identify barriers to effective program implementation, and potential strategies for overcoming those barriers.

It is likely that you are already incorporating aspects of process evaluation into your work if you are keeping detailed program records. It can be even more useful, however, to carefully decide what aspects of the process you want to monitor and how you plan to do so. Focusing your process evaluation in this way will enable you to identify the ways that variation in the current process might affect the program and its outcomes. This can only serve to improve your program in the future.

Process evaluation can be an indispensable tool in conjunction with outcome evaluation in determining program effectiveness. Conducting an outcome evaluation and finding that your program was successful, without a detailed understanding of how the program was implemented, prevents you from knowing exactly how or why it worked. This makes it difficult to replicate the results of a successful program. Process evaluation can take some of the speculation out of successful program implementation, and is worth the effort to ensure that programs are being delivered as intended in order to achieve their goals and objectives.

If you have questions about evaluation, would like assistance, or would like to request an Evaluator's Corner edition on a particular topic, please contact: Julia Kohn, Planned Parenthood Federation of America, 434 W. 33rd St., New York, NY 10001. Phone: 212-261-4629. Fax: 212-247-6269. Email: julia.kohn@ppfa.org