Document 6589616
Transcription
Document 6589616
Sep/Oct 2014 No. 12 Impact Evaluation Round Table As part of its support to impact evaluation in the ILO, EVAL is creating community of practice for those who are technically involved in and implementing impact evaluations. Beginning in October, EVAL organize several informal round table discussions of ongoing ILO evaluations or impact evaluation methodologies currently being developed. Anyone working on such activities is welcome to contact join the group and/or to present their work for a round table discussion. i-eval EVAL highlights Making Impact Evaluation Work Organizations and their donors are increasingly preoccupied with capturing knowledge about “what works” in international development. The ever-expanding numbers of authoritative evaluations, research reports and studies make this a very complicated task. ILO EVAL is expanding its efforts to filter through the available array of research and resources to distil the most pertinent and credible insights and thereby enhance its contribution to identifying the impact of ILOs work. Additionally, EVAL is dedicated to identifying knowledge gaps to better determine what further research, studies, evaluations or impact assessments are required. Planned collaboration between the ILO Evaluation Office and the World Bank is aimed to help us provide clarity on what impact assessments are (or should be) related to the world of work, labour and jobs -- and how these differ from and complement other processes of monitoring and evaluation. This collaboration, in conjunction with discussion with technical experts inside and outside the ILO, will lay the basis for a renewed effort by EVAL to support and coordinate impact evaluation in the ILO in a more rigorous manner. EVAL has stepped up its efforts to support the proper use of impact evaluation in the organization, through EVAL Guidance Note 13: Impact evaluation, peer review and exchange sessions and - more recently - a stock-taking review of current impact evaluation practices in the ILO, see EVAL’s latest Think Piece. The Think Piece shows that impact evaluation in the ILO has several strengths, but rigorous standards need to be applied, including in-depth needs analyses, better initial preparation, and evaluability analysis before conducting an impact study. This is necessary to ensure a high standard effort and also to justify the exorbitant cost which is associated with these complex and longer duration evaluations. EVAL looks forward to our next roundtable on impact evaluation where we will report on progress made. Guy Thijs, Director ILO, Evaluation Office Visit our website: http://www.ilo.ch/eval/lang--en/index.htm navigation Regional Article: Europe – Departmental Article: Bureau of Employers' Activities Innovation & Research – Learning Activities and Events Blogs - Evaluation Newsletters EVAL to Flash news EVAL is pleased to share the twelfth edition of i-eval Flash news with you. Through this quarterly electronic bulletin we provide readers with updates, news and information on publications and upcoming events related to evaluation. You are invited to alert us about any news item that you wish to include in the next issue at EVAL@ilo.org. QUICK a designing will impact i-eval Flash news – Sep/Oct 2014, page 2 Innovation & Research Completed Studies Think Piece No. 6 – Impact Evaluation in the ILO: Stock-taking of current practice, Oct. 2014. Written by Dr. Achim Engelhardt, this Think Piece presents information Dr. Engelhardt collected in a stock-taking exercise aimed to lay the groundwork for future EVAL involvement in this area. The study provides an overview of past and existing impact evaluation work in the ILO, and examines standards followed in the UN development community. Methodologies are also reviewed, and the piece concludes with some recommendations for future action. Think Piece No. 7 - Delivering decent work results: Implications of a meta-analysis of ILO Decent Work Country Programmes, Oct. 2014 – This Think Piece was written by Mr John Martin and aims to synthesize the findings of his 2014 meta-analysis of internal Decent Work Country Programme reviews (CPRs), as well as findings from his 2011 meta-analysis of independent high-level evaluations of country programmes. Recommendations are put forward for how better to approach internal country programme reviews and make these useful to broader DWCP planning and implementation. to conflicts, disasters and other crisis situations and requested the Office to make the necessary organizational arrangements to strengthen its capacity to deal with such situations. Soon after, the ILO’s Evaluation Advisory Committee held its 16th meeting and expressed its desire for EVAL to carry out an evaluation of TC in fragile states in conflict and post conflict areas. (forthcoming 2015). Synthesis review of labour protection: As in previous years, EVAL will manage a synthesis review of this recurrent discussion topic to be conducted by an external research team (forthcoming 2015). New Books on Evaluation Qualitative research and evaluation Methods 4th ed. – Michael Quinn Patton, Sage, November 2014. “This book contains hundreds of examples and stories illuminating all aspects of qualitative inquiry. Patton has created the most comprehensive, systematic review of qualitative methods available. Key Features: Explores, compares and contrasts 16 different theoretical and philosophical approaches to qualitative inquiry; offers strategies for enhancing quality and credibility of qualitative findings; unravels the complexities of mixed methods and triangulation; and explains the issues and approaches to fieldwork.” On-going and planned studies External quality assessment of independent project evaluation reports 2012-13: This annual study will use the scoring instrument as in previous years and will review a sample of ILO-managed, independent project evaluations from the reporting period. Due to be published in November 2014, the study is conducted by the International Organizational Development, Ltd, UK and is part of the regular externally-generated quality controls EVAL commissions on independent project evaluations. Thematic evaluation on fragile states: In March 2014, during its 320th session, the Governing Body discussed ILO technical cooperation in response Agricultural Policy Monitoring and Evaluation 2014 Organization for Economic Development (OECD), 2014. “This edition covers OECD member countries up-to-date estimates of support to agriculture. It is complemented by country profiles on agricultural policy. A number of new country-level frameworks for agricultural policies will become operational in 2014, and multilateral trade negotiations may have future bearing on agricultural trade.” i-eval Flash news – Sep/Oct 2014, page 3 News from the Departments Bureau for Employers’ Activities (ACT/EMP) Evaluation of Outcome 9: Employers have strong, independent and representative organizations ILO’s Bureau of Employers' Activities (ACT/EMP) received funding from partnerships with both Sweden (SP) and Norway (NP) to support the implementation of the Outcome 9 strategy. The funding has been used to support Outcome 9 targets, or Decent Work Country Programme Outcomes (CPOs), focussed on strengthening the capacity of employers’ organizations in 19 countries across all regions. The development of Outcome 9 Global Products (GPs) was also supported through NP and SP funding. The evaluations, both the Swedish (final evaluation) and Norwegian programmes (mid-term evaluation), covered the period from April 2012 to March 2014. The purpose was to examine the CPOs achieved and GPs produced under Outcome 9 and to assess the contribution of the programmes towards achieving Outcome 9’s strategy. The focus of the Norwegian funded programme was to strengthen the institutional capacity of employers’ organisations in policy development and dialogue at national, regional and interregional levels, and to assist them to develop new and improved services to strengthen membership recruitment and retention. The Swedish funded programme sought to enhance the capacity of employers’ organizations to participate in the policy dialogue on sustainable enterprise development for job-rich growth and provide opportunities to do more in-depth policy work that could contribute to more effective national policy dialogues. NP and SP funding also contributed to the development of Outcome 9 Global Products (GPs) which supported activities at the country level. A key product developed under NP was a global survey on women in business and management and creation of a resource network of employers to share good practices and advocacy on advancing women in business and gender equality. The SP contributed to further improvements to the Enabling Environment for Sustainable Enterprise (EESE) toolkit, which helps employers’ organizations identify constraints in the development of sustainable enterprises and formulate advocacy and reform proposals for dialogue with government. Main Findings The outcome evaluation findings confirm the relevance of the NP and SP funded interventions as well as those of the resulting CPOs and GPs. Strong mutual linkages were observed between Outcome 9 and the DWCPs. A fruitful collaboration environment had been established with other ILO Outcomes in the development of the Outcome 9 Global Products. The interventions reviewed were found to be logically coherent and had realistic objectives in terms of CPO (target) achievement. NP and SP funding contributed substantially to the overall achievement of CPO targets. In addition, the NP and SP funded global products reached most of their set objectives. Both SP and NP funded interventions contributed to the potential for development change. Outcome 9 indicators and related measurement statements were identified as a useful tool for aggregating results achieved at a departmental level. However, the indicators are not suited for monitoring gradual increases in the capacity of a partner EO over a longer period of time. The gender dimension was an integral part of NP funded activities. Under SP funding, while the gender dimension EESE tool had been strengthened as part of the support to GPs, challenges in mainstreaming gender equality in EESE processes at the country level still exist. In several cases the gender dimension was absent in business agendas. The NP and SP funded interventions indicated that continued support from the ILO to sustain results at the country level will be needed. In regard to the GP’s, the tools for assessing various dimensions of EO capacity and advocacy efforts are excellent. There is potential for these GP elements to benefit a much wider spectrum of agencies within the development community in the future. Lessons learned The specific country context in which an EO partner operates must be taken into account in the design of support in order to be able to respond flexibly in a dynamic and changing environment. It is important to work with EOs which are representative of the i-eval Flash news – Sep/Oct 2014, page 4 private sector and can apply a broad perspective on the needs of business environment reform. More focus is needed to support the sustainability of capacity building activities with EOs. EO membership is a critical factor in strengthening the financial capacity and representativeness of EOs. Additionally, it was emphasized that initiatives to promote gender equality need to be strengthened as they often remain as fairly “isolated islands” within EOs. Recommendations: The following recommendations from the outcome evaluation summarize future action to be taken: On donor support: Continue to support the creation of strong, independent and representative EOs. Give priority to building up the membership base and strengthening EO capacity for analysis and advocacy related to business environment reform and gender equality. On capacity development: Experiment with the methods aimed at capacity development. Learn from others through partnerships between EOs in different countries. Consider partnerships with various local institutions, including local economic think-tanks. Project evaluations from ACT/EMP 2010-2015 (summaries are available through the hyperlink, full reports from EVAL@ilo.org GLO/11/59/SID GLO/12/56/NOR INT/06/62/NET INT/06/63/NET INT/06/55/NOR GLO/14/59/NOR RAB/12/50/NOR Sweden-ILO Partnership Programme Phase II 2012-18 Outcome 18 – Maritime Labour Convention (Global Product) - Final Evaluation (2014) ILO Norway/Sweden Partnership Programmes: Outcome 9: Employers have strong, independent and representative organization Strengthening the capacity of employers' and workers' organizations to be effective partners in social dialogue - Final Evaluation (2010) Mainstreaming tripartism across the Netherlands/ILO Cooperation Programme (NICP) and product development for employers' and workers' organizations - Final Evaluation (2010) ILO-Norway Framework Agreement: Child Labour Component - Final Evaluation (2009) Outcome 9: Employers' organizations thematic funding for 2014-15 Norway-ILO Partnership Final evaluation (Forthcoming 2015) Developing the capacity of employers' organizations in the Arab Region through effective policy and social dialogue - Final Evaluation (Forthcoming 2015) Monitoring and evaluation: Improve the monitoring and evaluation of EO capacity development over a longer timeframe, including the use of data on EO membership development. Mainstreaming of gender equality: Identify opportunities to create successful gender initiatives that can be replicated in the various organisational structures of EOs and through EO advocacy processes such as the national business agendas. Evaluation Office Our named changed with Office Directive IGDS No. 74, V.2, 6/2014 i-eval Flash news – Sep/Oct 2014, page 5 Regional News Lessons Learned from Evaluation in Europe & Central Asia TC project evaluations generate a wealth of findings and knowledge, including useful perspectives that can improve organizational processes and practices. In the Europe and Central Asia Region some findings and practical lessons have emerged covering employment, social protection, labour migration, social dialogue and other topics which might be relevant to broader contexts and useful to various evaluation stakeholders and teams across the ILO. The following selected highlights of such findings come from 2012-2013 project evaluations. Quality of project Logical Frameworks: Certain shortcomings in project design have been noted on several occasions as posing a challenge to the evaluation exercise. Logical frameworks, or logframes, are typically mentioned as an area for improvement in EVAL meta-analyses of project evaluations as well as some internal studies EVAL has undertaken. Some of these specific deficits are related to the definition and expression of the objectives and the appropriateness and usefulness of the indicators. The most common drawbacks are linked to the formulation of indicators and result statements at different levels of the project results chain. There is sometimes confusion between the level of results expected at the outcome, output and activity levels. Indicators are in some cases formulated as outcomes, or even as activities, thus not being appropriate to the nature of the change they aim to capture. In a few cases, the evaluators reported that they had to develop their own “additional” indicators during the desk research in order to be able to start the active research phase and interviews. Asking the so-called “If – Then” question is considered to be a good way to test the logic behind the chain of results underlying the project approach at the design stage. Since elaboration of new projects is a time-consuming and multiple-stakeholder process, new project proposals are often being circulated for inputs in more than two rounds over an extended period of time. Technical perspectives may be prevailing over more formal programming perspectives in terms of formulation of results. It is also often not possible to effectively trace and harmonize all the inputs coming from various stakeholders, which is further complicated by time deficits weighing on officials in charge of project formulation. Having a single final reviewing entity for quality control might be helpful in this regard. It was felt that PARDEV’s role could be strengthened to better streamline internal capacities, and to offer real-time advice and support to the project design teams. Performance against the main evaluation criteria Over the two-year period under review, relevance and effectiveness of projects was assessed as satisfactory or good. All the projects evaluated in 2012-2013 were considered as being highly relevant to the national development goals, corresponding accurately to national development frameworks, and the ILO’s strategic framework and agreed-upon country priorities. Efficiency was also assessed satisfactorily with a particular acknowledgment for the high quality of expertise and technical resources that the ILO makes available to the projects and through the projects - to the constituents, including quality research, analytical products and training. Sustainability, however, was typically assessed less positively. This might be successfully addressed if project design teams introduced some standard sustainability components that would take effect during the last stages of implementation. Some elements of follow-up on outcomes (e.g., tracing the use of the most significant results of past interventions) could be built into ILO cooperation frameworks in the countries (DWCPs or Programmes of Cooperation) which often have longer timeframes and are based on long-term commitments, thus offering a good basis for continuity. Tripartite constituent involvement and engagement Constituents’ engagement in the evaluation process is being ensured at all stages and this reflects the tripartite nature of the organization, its values and methods of work. Such engagement is not only useful for obtaining feedback on project performance, but it also serves other important objectives. Having constituents participate in evaluation emphasizes equality of status among all the different parties and stakeholders. It also ensures that constituents take i-eval Flash news – Sep/Oct 2014, page 6 greater responsibility both for the process and the results: the stronger the involvement, the stronger the uptake of results and ownership of project outcomes. Also very important in Europe and Central Asia is the language challenge. In order to facilitate communication during the evaluation process between the ILO, the evaluators and constituents it is necessary to deal effectively with language barriers. Translation of essential evaluation documents from ILO is typically needed in everyday communication. This aspect should be accounted for in evaluation planning as it may have an effect on the timelines needed to complete an evaluation. Average time needed for consultations on the TOR, for example, could be longer than in other regions, so making relevant and adequate provisions for this in planning and implementation would improve evaluation processes at the national level. covered by such projects can vary from two to ten in some cases. There are specific aspects to consider in evaluations for these larger and more complex projects. They require a special set of higher level evaluation skills that reflect the versatility and technical knowledge required for multi-country and themed projects. Additionally, evaluations of this type may also require more time, which would have specific implications for evaluation budgets that need to be estimated at the project design phase. When these complex regional projects are implemented it is well worth emphasizing that each country will probably have slightly different outputs, means of action and varying activities, all of which make designing a coherent and comprehensive evaluation plan very difficult. At the project design stages these concerns should be considered. In conclusion, it is very important to underline that the evaluation function is closely linked to learning and knowledge management, so revisiting the findings periodically can improve utilization of monitoring and evaluation processes and findings. Project evaluations from Europe in 2012-2013 (summaries are available through the hyperlink, full reports from EVAL@ilo.org ALB/11/01/EEC RER/07/08/AUT RER/08/05/EEC Finally, because the ILO does not have a country office in many countries of the region, where the bulk of technical cooperation is taking place, the role of the ILO National Coordinators (NCs) in ensuring smooth communication at the country level should be recognized and emphasized. It should be noted in this context that a move towards stronger ILO presence in the countries would correspond to a number of evaluation recommendations (including the High-level Independent evaluation of DWCP for Kyrgyzstan 2010). Evaluation of multi-country projects There is an increasing trend to design multi-country projects, which can be additionally grouped by themes, such as migration, youth employment, and social dialogue, etc. The number of countries RER/11/04/EEC RER/11/51/FRG RER/09/04/EEC RER/09/05/FIN SRB/10/01/UND RBSA Funded Human resources in Albania project Midterm Evaluation Consolidating the legal and institutional foundations of social dialogue in the Western Balkans– Final Evaluation Increasing protection of migrant workers in the Russian Federation and South Caucasus - Final Evaluation Promoting integration of migrant domestic workers in Europe - Final Evaluation Combating child labour in Central Asia: Commitment becomes action (PROACT CAR Phase III) - Final Evaluation Effective governance of labour migration and its skills dimensions Final Evaluation From the crisis towards decent and safe jobs in Central Asia – Midterm and Final Evaluation Strengthening capacity for inclusive local development in Serbia - Final Joint Evaluation Review of RBSA projects - Yugoslav Republic of Macedonia 2010-2011 Final Evaluation i-eval Flash news – Sep/Oct 2014, page 7 ILO Evaluation Guidance i-eval Resource Kit - ILO policy guidelines, 2nd ed. Revision plans Guidance Note 13: Impact evaluation Work is now being planned for a revision of the impact guidance note. A new EVAL Think Piece on impact evaluation, see page three, undertook a stocktaking exercise on impact evaluation and as discussions continue, the guidance on this will be updated and revised, and reported in the newsletter. EVAL is also currently planning to revise guidance on how to conduct Decent Work Country Programme internal reviews, following discussions within the Evaluation Network,and findings and recommendations from the new Think Piece on DWCP internal reviews discussed on page three. EVAL welcomes your input for improvements on these two topics. Please send to EVAL@ilo.org. Evaluation Learning Activities & Events ILO Evaluation Learning Activities in Turin Independent evaluation of the International Training Center of the ILO Academies The Governing Board of the ITC-ILO requested an evaluation of the ITC’s Academies, ten of which took place between 2011 and 2013 and represented two recurring series of five academies. EVAL managed the evaluation in collaboration with the ITC. The evaluation was welcomed by ITC management and reflected well on this aspect of ITC’s activities. The evaluation demonstrated that the Academies have strong relevance to the ILO’s Strategic Programme Framework. Impact evaluation of technical cooperation and development projects - Nov 3-7, 2014 Target group: Technical specialists in charge of monitoring and evaluation of development projects and programmes. Description: The participants will be guided to perform an impact evaluation for a selected project (case study) based on the learning contents pursued in the course. Training course to certify evaluation managers - The fourth training session for EVAL’s Evaluation Manager Certification took place in Bangkok in October 2014. Both the Asia and the Africa regional training sessions which took place, in addition to the Turin trainings, were considered successful events and EVAL will continue to offer these courses to build the skills of evaluation managers. Once an official completes the course, then a pilot evaluation experience (practicum) is undertaken. After this has been successfully completed, the official will then be considered for certification as a recognized ILO Evaluation Manager. Click here to read the last quarterly report on the EMCP training. Registration information can be found here. EVAL congratulates the newly certified Evaluation Managers Rose Anang Maria Borsos Matthieu Cognac Darryl Crossman Gugsa Farice Sergio Iriarte Oktav Pasaribu Anne Schalper Eszter Szabo Andrés Yuren New UNEG publication: Integrating Human Rights and Gender Equality in Evaluations – August 2014 - This report was produced by UNEG as an in-depth guidance handbook to serve as a field guide to improve human rights and gender equality responsive evaluation throughout the UN system. It is aimed at increasing knowledge on the application of these two approaches in evaluation processes but also at raising awareness on their specific relevance and significance for UN work. It complements the UNEG's Handbook "Integrating Human Rights and Gender Equality in Evaluation: Towards UNEG Guidance," an abridged version that outlines practical steps on how to prepare, conduct and use HR & GE responsive evaluations. The present document deepens each of these aspects, and provides additional theoretical and applied information, tools and suggestions. i-eval Flash news – Sep/Oct 2014, page 8 External Knowledge Sharing, Courses and Webinars Advocacy planning, Monitoring and Evaluation course – International NGO Training and Research Centre (INTRAC), Oxford, UK – December 1 thru December 3 2014. Building on INTRAC’s extensive experience in both monitoring and evaluation and in advocacy, this course will support practitioners with how to ensure M&E into advocacy initiatives and into organizational learning and accountability. The course content draws on real-life case studies from INTRAC’s work in supporting organizations to develop advocacy strategies and from carrying out evaluations of advocacy initiatives. It will provide a series of practical tools which can help participants to develop and improve the advocacy M&E systems in accordance with their own capacity and needs. Claremont Graduate University Partners with UNICEF and the Rockefeller Foundation in partnership with IOCE and DevInfo, offer a series of live webinars on "Emerging Practices in Development Evaluation". This includes a range of guest speakers with experience in development evaluation. Please see the full program here. These webcasts are free and open to the general public. e-Learning programme on Development Evaluation – UNICEF and IOCE, under the EvalPartners initiative, offer an introductory. It is composed of an introductory class, independent units and a create-your-own-course option, as well as fixed courses such as: Introductory class on Development Evaluation; Equity-Focused and GenderResponsive Evaluations; National Evaluation Capacity Development for Country-led M&E Systems; and Emerging Practices in Development Evaluations, for example. The next cycle begins 27 January 2015 - 18 May 2015 and interested participants can register here. 2nd International Conference on Evaluating Climate Change and Development – Nov 4-6, Washington D.C. The Climate-Eval Community of Practice, hosted by the Global Environment Facility Independent Evaluation Office (GEF IEO) and its partners are pleased to announce the 2nd International Conference on Evaluating Climate Change and Development, scheduled for November 4-6, 2014, in Washington D.C. The conference will bring together evaluation professionals, academics, policymakers and other stakeholders from developed and developing countries with the purpose of fostering the interdisciplinary exchange of ideas and methods to evaluate climate change and sustainable development. Blogs on evaluation World Bank Blog on Impact Evaluation American Evaluation Association Blog Evidence matters blog – Impact 3iE (new) Foundation Strategy Group (FSG) Blog Better Evaluation Blog John Gargani’s EVAL Blog Genuine Evaluation Evaluation Capacity Development Group Intelligent measurement Design, Monitoring and Evaluation (new) IDB Development effectiveness blog (new) African Development Bank eval- blog (new) Other evaluation newsletters OECD DAC Evaluation News OIOS Inspection & Evaluation Evaluation Office (EVAL) International Labour Office CH-1211 Geneva 22, Switzerland Email: eval@ilo.org Editor-in-Chief: Guy Thijs, Director Executive Editor: Janet Neubecker UN Women Newsletter IFAD Evaluation News EVAL Partners Newsletter