The Global Impact Evaluation Forum (2024), co-hosted by UNICEF and the World Food Program (WFP) at the UN Headquarters in New York provided a platform for discussing critical themes in the use of impact evaluation to inform decisions and policies. A standout session on Capacity Development for Practitioners in impact evaluation brought together experts from UNICEF, Princeton University, the Network of Impact Evaluation Researchers in Africa (NIERA), the Somalia Government, the Partnership for Economic Policies (PEP) and Si3, a Social Impact Investment Initiative. This session unearthed strategies, challenges and innovations in building capacity of practitioners to conduct and apply impact evaluations.
A 2023 study by the Center for Global Development established that there were 181 organisations with capacity to conduct impact evaluations in Africa. Thirty nine percent of these organisations are NGOs or think tanks, 37% are universities and 24% are government entities. These findings suggest a limitation of impact evaluation capacity amongst practitioners - policy makers, funders and program implementers.
Anecdotal statements from participants in the Global Impact Evaluation Forum (2024) noted that most practitioners do not apply Impact Evaluations in their work. Consequently, they lose the opportunity to understand the effectiveness of their programs and make more informed decisions, fail to understand contextual factors that influence their projects and miss on learnings that could inform scaling. Most strikingly, they fail to focus their effort and limited resources to the interventions that have the greatest impact on their beneficiaries.
Mohamud Nur, the Director in charge of Monitoring and Evaluation in the Ministry of Planning in Somalia who has overseen the development of monitoring and evaluation frameworks in Somalia acknowledges the practitioners knowledge gap in Impact Evaluation saying “we do not have capacity to implement IE studies, the government has limited resources and IEs in Somalia are mostly funded by the donors. While we have formed M&E frameworks, IE is lagging behind.”
Similar sentiments were echoed by Michael Craft, the Regional Evaluation Advisor at UNICEF who giving his experience at UNICEF noted that the knowledge gap hinders effective Impact Evaluations. Luckily, UNICEF has an elaborate strategy that is entrenching impact evaluations in its projects.
In her experience Dr. Tigist Mekkonen, a Development Economist at Si3 observed that most of the Impact Evaluation studies that she has conducted on poverty analysis, trade, regional integration and agriculture face hurdles in shaping policy partly due to the inability of the policy makers to discern the findings and demand evidence that improve the policies that they are responsible for.
What Capacity Building and Knowledge Transfer Efforts are Available?
There are efforts that are being made in building capacity of practitioners in the impact evaluation space to not only enrich the studies but also to leverage on their expertise as they are the ones who better understand the contexts in which programs work and feel the direct effects of a working or (a struggling) intervention.
NIERA for example, has programs that support the training of basic principles of impact evaluation to practitioners. In the last five years NIERA has trained and mentored over 200 practitioners in Africa and made the training materials available on open source. While this is a great beginning, it is by far inadequate.
Moreover, NIERA supports a Demand Driven Research approach where questions that are answered by use of Impact Evaluations originate from the users of the research. Practitioners work with established researchers to co-create interventions and implement impact evaluation studies. This has enriched the understanding and appreciation of impact evaluation amongst practitioners. Through this Demand Driven Approach, NIERA researchers have worked with practitioners to to devise solutions for preventing and mitigating risky gambling behaviour in Kenya and Uganda, evaluated the impact of agri-food value chain interventions on the welfare of vulnerable urban poor in Kampala, evaluated the effectiveness of targeted subsidies and information provision on Community Health Based Insurance uptake and healthcare utilisation in Ethiopia, are assessing the impact of switching from 10-dose to 5-dose measles-rubella vaccine in Kenya, evaluating the effectiveness of HIV self-testing programs in Kenya and Tanzania and the determination on what works in creating decent jobs in the Kenyan creative sector.
Recently, NIERA completed a project dubbed Contextualizing Impact Evaluation Pedagogy in Africa (CIPA) - an initiative that has institutionalised IE in eight African Universities and two research institutions. This effort seeks to broaden the reach of impact evaluation to stakeholders in the overall ecosystem.
In a synergetic training model, Prof Jane Mariara the Executive Director at PEP stressed on their empowerment model that strengthens capacity and promotes researchers in the global south to collaborate with policy makers - creating a space for learning and knowledge sharing.
The African School of Economics (ASE) has a unique model that has supported the development of 5 research hubs in Africa. ASE has supported deliberative policy making in Nigeria, trained city managers in Zanzibar and Benin and is promoting evidence-based policy making with systems and agency in Niger, Burkina Faso, Benin and Nigeria.
What does it take to build capacity of Practitioners?
Practitioners need simple executive courses that demonstrate the value for Impact Evaluations - says Mr. Muhammad Nur. They have tradeoffs for acceptance of impact evaluations that include: cost benefit analysis, time that the studies take and the political costs of implementing interventions. To get them on board researchers must appreciate their needs, support them to pick out the benefits of Impact Evaluations, walk the journey with them and support them with cost analyses of the different projects. Patience is key says, Michael Craft, the Director of Evaluation at UNICEF.
Key Takeaways from the Panel Discussion
Capacity development must consider diverse operational contexts of practitioners. For example, UNICEF and Somalia Government must support evaluators working in fragile contexts and adapt impact evaluation methodologies to the environment.
There is need to bridge the academic-practitioner knowledge gap. Rigorous academic research must be connected with practical implementation. Universities such as Princeton, are critical in providing capacity development models that scale across countries and regions.
Local expertise is critical in empowering practitioners to contribute to impact evaluations. Organisations like NIERA invest in relationship building between researchers and practitioners for knowledge generation and sharing. Local experts understand the cultural, social and economic nuances of a project.
Technology is critical for learning. The Partnership for Economic Policies and Princeton University are leading in using e-learning platforms and virtual workshops to scale training and mentorship.
Integrating social investment is critical. Practitioners are conscious of social return on investment. Si3 presented a unique perspective on aligning social investment with impact evaluation. Evaluation findings can inform decision making in funding and resource allocation.
Challenges and Opportunities
There was general consensus amongst panelists that limited funding for capacity building initiatives, resources allocated for learning in development programs, use of ethical and inclusive approaches in evaluation and the need for harmonised impact evaluation standards hinders effective capacity development for practitioners in impact evaluation. However, there are opportunities to leverage on partnerships to overcome these challenges.
The Way Forward
The session concluded with a call to action for development agencies, academic institutions and practitioners to collaborate more effectively and avoid duplication. Capacity development for practitioners in impact evaluation is not just about knowledge - it is about translating of the knowledge to action through better programs that transform lives.
We acknowledge the support provided by Dr. Francesco Lacoela, Impact Evaluation Analyst at UNICEF, Geeta Dey and Gulgun Koozan.
By Prof Amos Njuguna (Network Chair, NIERA)

0 Comments