Impact Assessment Techniques and Methodologies

The following steps are involved in performing a successful impact evaluation of a value chain project (the steps presume two study rounds — a baseline and follow-up):

Step 1: Select the Project(s) to be Assessed

Effect analyses are carried out because someone wants to know what effects are being accomplished by various programs or intervention methods. This demand for information is most likely to come from higher levels of donor organizations (those preparing and allocating money for aid projects), but can also come from heads of missions, program officers and implementers, such as NGOs. Especially good candidates for an impact evaluation are large projects or those which take creative and promising approaches. Finally, it is obviously important to have funds available to do the impact assessment.

Step 2: Conduct an Evaluability Assessment

An evaluability assessment is an initial assessment of whether the project can undergo an impact assessment and, if so, what is the most suitable approach to do so. Sitting down with project personnel to work on a causal model for all the project tasks to be covered in the impact evaluation is an important aspect of the evaluability assessment. This means deciding what the project is doing or will do exactly, during what time span, and with what outputs, effects, and impacts are planned. If this discussion reveals that the hypothesized relationships between project activities and impacts are impractical, or if the time period of the project is inadequate (as a general rule, sustainable impacts require a minimum of two years), then the impact evaluation is not worthwhile.

The evaluability evaluation should also take into account the goals served by the impact assessment, its potential audience, its cost-effectiveness and potential reputation, along with the best timing for the impact assessment to be carried out.

Prior to performing the baseline impact assessment, the evaluability assessment is performed. However, prior to performing the follow-up impact evaluation, it is also necessary to perform an updated evaluability assessment. In this scenario, the aim is to decide if it is worth engaging in the follow-up assessment at all or whether the scope of the impact assessment should be decreased in light of the events that have occurred since the baseline.

Step 3: Prepare a Research Plan

The causal model of the impact evaluation and a realistic strategy for implementing the analysis should be included in the research plan. The causal model is used to produce a set of findings and impact hypotheses which will be tested in the analysis. Impacts of many different forms will usually be expected at three levels:

·         Including commodity markets and often supporting markets for inputs, business services, and/or finance in the supply chains and markets involved,

·         Among the MSEs participating

·         In the households affiliated with the MSEs concerned

The next step is to identify quantitative metrics, once testable theories have been established, which can be used to assess if the effect has been achieved. After that, it is important to identify sources of information for calculating the indicators. The longitudinal survey serves as a significant source of knowledge in the quasi-experimental method (see definitions below) to assess if there is an effect at the MSE and household level. This involves selecting a sample of project participants (explicitly identified in a way that is consistent with the structure and methodology of the project) and matching it with a sample of non-participants who, in all relevant features (control group), are as close as possible to the project participants. This must be done carefully to mitigate the effect of selection bias, the propensity for individuals who would have done well to become project participants anyway, which contributes to the impact of the project being overstated. The two groups of survey respondents form a jury that will be interviewed at least twice, with a minimum period of two years between survey rounds, in a quasi-experimental impact assessment. Over-sampling is needed in the baseline round to allow for depletion in the sample between rounds. The two groups are chosen at random in an experimental evaluation and then interviewed only once at the end of the analysis.

The survey is the quantitative aspect of the evaluation of the effect. To get a richer view of the effect at the MSE and household levels, it can be combined with qualitative analysis, as well as to get an understanding of what the influence of the project is at the supply chain and market levels. It is likely to be difficult, if not impossible, to find a suitable control group at these higher levels, so the effect cannot be demonstrated as clearly as at MSE and household levels. Qualitative research consists of a formal interview project, focus group interviews, and other qualitative approaches with individuals who are interested in the related value chains and markets in different ways. In an effort to get a consistent image of the dynamics of the markets involved and shifts over time that could be due to project activities, their views and observations are then triangulated.

Detailed specifications for the questions to be asked on the survey questionnaire and recommendations for the interviews and focus group discussions should also be included in the study plan.

Step 4: Contract and Staff the Impact Assessment

The next step, once a research plan is drawn up, is to make plans for field research to be carried out. Usually, under the guidance of the sponsoring company and external advisors (often foreign experts) it might have employed, a local research firm is contracted to carry out the field research. A competitive bidding process is desirable in this instance. An offer based on a scope of work (SOW0 prepared by the sponsoring organization or its external advisers) is submitted by potential local research partners, which clearly specifies the tasks, time period, and budget for field research. Selection of the local research partner would consider many factors, including previous experience, technological knowledge, proposal quality, recommendations.

Step 5: Carry out the Field Research and Analyse Results

The local research partner performs the baseline field research under the supervision of the sponsoring organization and its external advisors, which includes the impact assessment survey and complementary qualitative data collection activities, primarily key informant interviews and focus group discussions. Next, the local research partner organizes the details and summarizes the conclusions. This includes entering the survey responses into a data collection for the impact survey, cleaning up the data, and tabulating the results into clear descriptions and frequencies. This includes sorting the raw answers for qualitative analysis, summarizing them, and noticing general patterns.

Determining who performs the in-depth data review is up to the sponsoring agency. It could be the local research partner or the sponsor's employed external consultant. The sponsor should confirm the data analysis ability of the partner, its familiarity with the project and its causal model (relevant for interpreting the results), and its language skills (relevant for preparing the final report) before recruiting the local research partner to conduct an in-depth data analysis. In certain situations, both data processing and report writing may be necessary for the local partner to do. In other situations, it may be reasonable to employ a local research partner to carry out data analysis and report writing, and to employ an external consultant to review the data analysis and final report and (if necessary) review it. This choice is especially important where, in the language of the final report, the local research partner does not have a native-speaking capability. In such situations, it may be necessary to employ an external consultant to conduct the review of the data and to prepare the final report.

The analysis of the baseline outcomes is primarily descriptive. Its goal is to build an accurate image of the conditions within the project and among participants in the treatment and control group at or near the start of the project to serve as a basis on which improvements can be assessed after the conclusion of the follow-up round of study.

The goal of the follow-up (or end line) data analysis, on the other hand, is to record the changes in performance and impacts that have taken place since the baseline and to relate the changes found to project activities. The effect of the project is measured using the 'difference in difference' approach, in which improvements in the care category of project beneficiaries are contrasted with improvements in the non-beneficiary control group. Effect is concluded if the treatment group changes are substantially more beneficial than the control group changes. The study must also take into account "mediating variables" that might influence this comparison, such as variations between the two samples in income, age, gender, or educational achievement.

Step 6: Disseminate the Impact Assessment Findings

Since the impact evaluation is likely to produce data that has relevance beyond the analysed specific project, it is important that the lessons learned are efficiently disseminated to all those who are in a position to use them. Web articles, seminar or conference presentations, seminars, and published papers provide potential means of dissemination.