Consists of objective assessment of a project, programme or policy at all of its stages, i.e. planning, implementation and measurement of gender mainstreaming outcomes.
What is gender-responsive evaluation?
Gender-responsive evaluation is an assessment aimed at:
- independently measuring progress towards achieving intended gender-related objectives and goals set out in policies, programmes and projects;
- evaluating from a gender perspective the relevant processes, activities, outputs, outcomes and impacts of an intervention.
Gender-responsive evaluations assess to what extent an intervention has resulted in progress (or the lack thereof) towards intended and/or unintended results regarding gender equality[1]. It assesses any changes related to gender equality – such as changes to cultural values, norms, attitudes, social behaviours and power relations, the participation and representation of women and men in all their diversity, the access and control over opportunities and resources, and shifts in policies, legislation and organisational rules.
‘Lived realities’ of women and men ‘in all their diversity’
‘Lived realities’ of women and men ‘in all their diversity’ It is important to consider the lived realities of women and men in all their diversity. While the term ‘lived realities’ is meant to recognise differences between women and men, the addition of ‘in all their diversity’ recognises how other characteristics such as age, socioeconomic situation, disability, race, ethnicity, religion and rural or urban location also affect women and men in their lived realities (e.g. the lived reality of a young woman living in a rural area of Sweden will be very different from that of an older woman living in an urban area of Spain).
Source: EIGE (2020)
As a gender mainstreaming method, gender-responsive evaluation is applicable to all types of interventions, including those not necessarily aimed at gender equality and gender-specific interventions. This way, gender-responsive evaluation also assesses to what extent an intervention has pursued gender mainstreaming.
The term ‘gender evaluation’ has been used in the literature to refer specifically to the evaluation of gender equality policies[2], which can be both targeted gender equality interventions and gender mainstreaming strategies. The evaluation of gender equality policies is also important to assess the changes and shifts these policies have produced[3]. Similarly, the evaluation of gender mainstreaming strategies allows for judging the extent to which they have been successful in integrating a gender perspective into the work of an institution, a policy area, etc.
Legal and policy context in the EU and Member States
Evaluation is a well-established feature of EU governance and policymaking. It has a legal and political basis in Article 318 of the Treaty on the Functioning of the European Union (TFEU)[6], the EU’s financial regulations[7] and some sector-specific funds. Evaluation plays a significant role in ensuring strong accountability and transparency in EU policymaking. Besides, the ‘evaluate first’ principle adopted by the Commission, by which existing interventions are evaluated before bringing forward new proposals in the same area, aims to closely integrate evaluation findings into EU decision-making and the policy cycle.
The ‘Better regulation’ agenda and the EU evaluation policy
The EU evaluation policy is framed by the ‘Better regulation’ agenda, introduced by the Commission communication on better regulation for better results in 2015[8]. The ‘Better regulation’ agenda sets out the Commission’s approach to designing and evaluating EU policies transparently based on evidence and the views of citizens and stakeholders. The purpose of the ‘Better regulation’ agenda is to simplify EU laws and make them more targeted and easier to comply with[9]. The commitment to the ‘Better regulation’ agenda is also sealed in the interinstitutional agreement on better law-making[10]. It was signed in 2016 by the European Parliament, the Council of the European Union and the European Commission. In this interinstitutional agreement, the three EU institutions recognise their joint responsibility in delivering high-quality legislation for the EU[11]. The most recent communication on better regulation[12], adopted in 2021, commits to improving the use of evaluation findings and the reporting of unintended consequences. The sustainable development goals (SDGs) also feature prominently in the updated ‘Better regulation’ agenda. Under the agenda, the developed legislation and policies must ensure that the EU recovers from the COVID-19 pandemic and shape a more resilient and equal world for the next generation. To ensure this, the Commission plans to identify the relevant SDGs in each policy proposal and examine how they support their achievement. For this purpose, evaluations and impact assessments should establish links to the SDGs as part of their analysis[13].
Legal and policy basis for considering gender equality in evaluations in the EU
The EU treaties enshrine equality between women and men as a core value of the EU and a fundamental right and oblige the EU to combat gender inequalities in all its activities[14]. These provisions form the legal basis for the role of the EU and Member States in promoting gender equality and implementing gender mainstreaming. The EU gender equality strategy, adopted in 2020, sets out the key gender equality policy objectives and measures for the EU to advance towards a gender-equal society[15]. It reinvigorates the EU’s commitment to the dual approach to gender equality that combines targeted measures to achieve gender equality with strengthened gender mainstreaming. Specifically, the Commission commits to enhancing gender mainstreaming by systematically including a gender perspective at all stages of policymaking, including evaluations, and in all EU policy areas and adopting intersectionality as a cross-cutting principle.
The Beijing Platform for Action provides a comprehensive agenda to advance gender equality at the international level. All Member States have signed it, and the EU is committed to monitoring its implementation. Two of its strategic objectives, namely under ‘Area H’ on institutional mechanisms for gender equality and gender mainstreaming, are particularly relevant for the promotion of gender-responsive evaluation: H.2 on the integration of a gender perspective in legislation, public policies, programmes and projects, and H.3 on the generation and dissemination of sex-disaggregated data for planning and evaluation.
Provisions on gender-responsive evaluation in specific EU policy areas and programmes
Apart from the general evaluation and gender equality policy frameworks, other regulations and policy documents of various EU programmes and policies also include specific provisions relevant for gender-responsive evaluation. For example, the common provisions regulation for shared management funds[25] covering the 2021–2027 period establishes gender equality as a horizontal principle and mandates gender mainstreaming and the integration of a gender perspective throughout the preparation, monitoring, report and evaluation of programmes[26]. Provisions on gender-responsive evaluation are also found in EU external action – particularly in the EU’s action plan on gender equality and women’s empowerment (GEWE) in external action 2021–2025. One of its strategic objectives (Objective 4) establishes the systematic use of gender-specific or sex-disaggregated indicators in the monitoring and evaluation system of EU-funded action for EU delegations, the Commission and Member States to ensure gender mainstreaming in all action and targeted action. This provision provides a basis to integrate a gender perspective in the evaluation cycle of the EU external action and international cooperation and development projects.
The approach to gender-responsive evaluations in the field of international cooperation and development is informed by the Directorate-General (DG) for International Partnerships, DG Neighbourhood and Enlargement Negotiations and Foreign Policy Instruments’ approach to evaluations with gender as a cross-cutting dimension, realised in 2018[27]. The document is linked to the human rightsbased approach (HRBA), which puts human rights and the rights holders at the centre of EU external action[28]. A toolbox, adopted in 2014 and updated in 2021, on applying the HRBA to international partnerships, created a framework to integrate the HRBA into all EU development tools and activities[29]. HRBA is a complementary approach to gender mainstreaming, and it is used to integrate the gender perspective in all development activities, including evaluation.
In the field of research and innovation (R & I), the EU has established requirements for gender mainstreaming, including specific emphasis on evaluation. For example, having a gender equality plan with arrangements for data collection, monitoring and evaluation is an eligibility criterion for accessing funding under Horizon Europe[31][32]. Moreover, the European research area agenda commits to developing principles for integrating and evaluating the gender dimension in R & I[33].
Key players in gender-responsive evaluation in the EU
Several European Commission DGs have commissioned or are authors of gender-responsive evaluations. DG International Partnerships (formerly known as DG International Cooperation and Development) and DG European Civil Protection and Humanitarian Aid Operation (DG ECHO) play an important role in assessing the EU’s activity in the field of international development and cooperation and the EU’s humanitarian action respectively. Other DGs, including DG Internal Market, Industry, Entrepreneurship and SMEs, DG Employment, Social Affairs and Inclusion, DG Research and Innovation, DG Justice and Consumers, DG Neighbourhood and Enlargement Negotiations, and the European Parliament’s DG Internal Policies, also carry out evaluations integrating a gender perspective. Other bodies, including EIGE, the European Evaluation Society and especially their working group on gender and evaluation also play a key role in promoting gender-responsive evaluations at the EU level. The European Evaluation Society brings together voluntary organisations for professional evaluation that are organised in associations or informal networks. These organisations play a role in promoting the professionalisation of evaluation, including gender-responsive evaluations.
Gender equality in Member States evaluation policy frameworks
Overall, the institutionalisation of evaluation across Member States remains low. Few Member States have national evaluation laws, regulations or policies that guide or require the implementation of evaluation. When they do, they do not require a gender perspective to be integrated into the evaluation process. Very few Member States have harmonised national evaluation policies that would guide and require systematic implementation of evaluation[34]. Instead, the commitment or obligation to evaluate policies is usually established only within concrete sectoral policies and laws. As a result, evaluations are not systematically carried out but scattered across different institutions and policy areas.
Against this backdrop, the institutional context and the existence of both supporting evaluation and gender mainstreaming structures highly influence whether and what regulations and procedures for gender-responsive evaluation are in place.
Member States’ policies on gender equality generally recall, however, the obligation to incorporate a gender equality perspective at all stages of policymaking and in all areas. These provisions constitute the primary reference point for the development and implementation of gender-responsive evaluation. In some cases, national gender equality policies include specific emphasis on evaluation, concerning both the integration of a gender perspective into the evaluation of all types of interventions and the evaluation of gender-specific interventions (see Box 6).
Overall, the efforts to establish norms and standards for gender-responsive evaluations in the Member States are scarce and concentrated only in some policy fields. Across the Member States, regulations and procedures for gender-responsive evaluation can be found especially in the areas of development and international cooperation, gender equality, employment, social affairs, education and R & I. Given the EU requirements for considering gender equality within the EU funds, the provisions for gender-responsive evaluation also tend to be integrated into the national programming documents for the EU funds programmes.
The lack of integrated evaluation systems that are gender-responsive contradicts the EU and Member States’ gender equality legal and policy frameworks. It is also a missed opportunity to advance the principles of good governance and the achievement of gender-equal results.
How does gender-responsive evaluation work?
A gender-responsive evaluation takes the following procedural steps.
- Institutional validation. This will ensure that there are adequate resources (both financial and human) to implement the evaluation in a gender-responsive manner. It will also facilitate access to data and informants and will contribute to a better buy-in of the findings of the evaluation.
- Preparation of the evaluation strategy. A gender-responsive evaluation requires careful preparation to ensure that it is of high quality, credible and useful.
Establish the evaluation team.
The evaluation team must include sufficient gender expertise, including among those managing and steering the evaluation. A gender-balanced evaluation team is desirable and aspects such as cultural and geographic representation should also be considered.
Assess how the intervention tackles gender equality.
Before preparing the evaluation design, it is important to determine the evaluability of gender equality in the intervention at stake. This requires assessing whether gender equality is the primary focus of the intervention or not, whether the intervention takes account of gender-specific considerations, whether sex-disaggregated data was collected, etc. The information gathered in this evaluability exercise will inform the design of the evaluation.
Define the evaluation objectives, scope and evaluation criteria.
This step requires clarifying what kind of learning the evaluation will produce and how the findings are expected to be used. Assessing the evaluation’s contribution (or not) to gender equality must be stressed as a key objective of the evaluation. While there is a standard set of EU evaluation criteria, other criteria could also be considered, such as sustainability, impact, equity[43], empowerment[44], etc. The evaluation criteria must be revised to ensure they adopt a gender perspective. Evaluation criteria such as equity and fairness consider aspects such as whether an intervention is socially just and whether the benefits or effects of an intervention were evenly distributed across different groups, stakeholders, regions, etc. While the criteria of equity and fairness may incorporate gender equality considerations, a standalone criterion on gender equality is recommended to ensure a detailed evaluation of gender equality aspects.
Integrating a gender perspective into evaluations requires mainstreaming gender through the standard evaluation criteria.
- Relevance: assessing the relevance of an intervention from a gender perspective requires verifying that the intervention responds to the gender-specific needs of the target population, and that its objectives are appropriate for addressing gender equality considerations in the intervention’s social, economic and cultural contexts. For instance, it requires examining whether the intervention is based on sound gender analysis[45] that uses sex-disaggregated data, captures gender gaps and reflects the different roles and needs of women and men.
- Effectiveness: evaluating the effectiveness of an intervention from a gender perspective requires assessing the extent to which, why and how the intervention contributed to the achievement of gender equality objectives and/or the reduction of gender inequalities (e.g. changes in perception on gender roles). It also provides evidence of how women and men benefited from these changes. Under this criterion, a gender-responsive evaluation also assesses whether the intervention or aspects of the intervention led to any unwanted (positive or negative) effects on gender equality.
- Efficiency: assessing the efficiency of an intervention from a gender perspective requires analysing if there was an equitable and efficient allocation of resources. This criterion also assesses the level and adequacy of resources used for tackling gender inequalities compared to other aspects tackled by the intervention. It can also include an assessment of the cost of not providing resources for gender equality, in other words, evaluating the enhanced benefits that could have been achieved with a certain amount of investment[46].
- Coherence: assessing the coherence of intervention from a gender perspective requires looking at the extent to which its various components are coordinated and complement each other to sustain advancements in gender equality (internal coherence) and to what extent they interrelate to other interventions to achieve gender equality in society (external coherence).
- EU added value: assessing the EU added value of an intervention from a gender perspective seeks to gather evidence of the changes towards achieving gender equality that would not have been obtained without an EU-promoted or funded intervention[47].
Define gender-sensitive evaluation questions
Evaluation questions break down the evaluation criteria and help further define the objectives of the intervention. In a gender-responsive evaluation, the evaluation questions must allow all the relevant aspects of an intervention to be evaluated with a gender perspective including the processes, activities, outputs, outcomes and impacts. The following are examples of gendersensitive evaluation questions that can be used to mainstream gender into the EU evaluation criteria[48].
Design the methodological approach and tools
Gender-responsive evaluations apply mixed methods (quantitative and qualitative) for data collection and analysis as it increases the reliability and validity of the evaluation findings. They also adopt participatory approaches that ensure the involvement and consultation of stakeholders.
Conducting and steering the evaluation. During the data collection and analysis stage, a gender-responsive evaluation employs a flexible and culturally sensitive approach that understands and acknowledges the constraints of informants. It also pays utmost attention to ethical considerations such as confidentiality, fair representation and avoidance of harm. Evaluators must also acknowledge their privileged position of power in the process of evaluation. Thus, all their judgements must be based on the evidence gathered and the triangulation of information from various data collection methods and sources. Ultimately, this step of a gender-responsive evaluation seeks to gather solid evidence to assess and report on the gender impacts of an intervention, the expected and unexpected changes related to gender equality it has prompted and the extent to which it has adopted a gender perspective.
Report writing, dissemination of evaluation findings and follow-up actions. Before the final report is produced, it is recommended that preliminary findings are discussed with participants in the evaluation, including stakeholders, in order to ensure that the findings are relevant and meaningful for all. The evaluation report should use sex-disaggregated data to account for the findings across all the evaluation criteria. The report should also use gender-sensitive language. The findings of the evaluation should be used to develop gender-responsive lessons learnt and recommendations to feed into the revision or design of future interventions. The final results of the evaluation should be disseminated among all participants and stakeholders.
Why is gender-responsive evaluation important?
Evaluations provide reliable evidence for decision-making, learning and accountability. For gender-responsive evaluations, the learning produced during the evaluation process seeks to promote social change and advance gender equality in a sustainable way[57]. It can serve as a driver for better and more transparent evidence-based policymaking, a more efficient allocation of resources and greater accountability to gender equality.
Evaluations should inform the decision-making process and the development and revision of policies. Gender-responsive evaluations generate knowledge and evidence about what works and what does not to advance gender equality, why and for whom.
Policies and policymaking, including evaluations, are not gender neutral. On the contrary, how gender is framed or overlooked in policies has consequences. Thus, adopting a gender-responsive approach to evaluation generates reliable evidence about how an intervention has responded to the different needs of women and men and girls and boys, in all their diversity[58].
Gender-responsive evaluations also unravel the (unintended) gendered impacts of the intervention and the achievements, limitations and difficulties encountered during the design and implementation process. These findings constitute valuable evidence against future gender-blind policymaking. They can serve to prompt policymakers to take action and address gender inequalities, consider a gender perspective and deploy resources more efficiently to respond to the needs of all beneficiaries, even in those interventions where the gender dimension is not apparent.
The accountability function of gender-responsive evaluations is extremely relevant as it demonstrates results, transparency and commitment to stakeholders by providing information to participants and stakeholders about the results and unintended effects of the intervention on gender equality[59].
The involvement of the relevant stakeholders (citizens, civil society, researchers, business, etc.) in evaluations through participatory and empowering approaches echoes feminist principles about knowledge and value being culturally, socially and temporally contingent, and the importance of listening to those affected[60]. Through the involvement of different stakeholders, including final beneficiaries, a gender-responsive evaluation contributes to enhancing their knowledge and capacities to promote gender equality and challenge inequalities[61]. It also re-engages citizens with the policymaking process and contributes to social and institutional change[62].
A gender-responsive evaluation can detect where, when and how the gender perspective was lost or weakened across the design and implementation phases of intervention and propose ways to reintegrate it[63]. Frequently, a gender perspective is adopted during the design of an intervention and the planning stage, but that perspective is lost or watered down during the implementation and during its monitoring. The evaluation stage is a privileged stage to promote learning for gender equality because it is the stage where the policy process is reviewed and recommendations for improvement are suggested[64]. The emphasis on learning underlines a key feature of evaluation that is consistent with the need to derive lessons learnt for future policymaking about what works and not to achieve gender equality.
Gender-responsive evaluations promote social change by using the learning produced during the evaluation process to promote gender equality in a sustainable way[65].
Examples of gender-responsive evaluations
Examples at the EU level
Gender-responsive evaluations are more prominent in policy fields that include specific legal requirements to integrate a gender perspective into policymaking, such as external action and development cooperation and R & I. Other examples of evaluations that adopt a gender perspective to some extent were also found in the fields of social and human rights[66], education[67], rights of the child[68] and thematic evaluations in the area of gender equality.
Gender equality is usually considered only under some evaluation criteria, particularly under effectiveness, effectiveness and efficiency, or effectiveness and impact. For instance, the evaluation of the EU humanitarian efforts in Yemen assessed the gender responsiveness of this EU action in the criteria of relevance, coherence and effectiveness (see Box 3). This is done by introducing some gender-sensitive evaluation questions.
At other times, gender equality considerations are introduced under a stand-alone criterion on equity. This is the case of the interim evaluation on the implementation of the ‘Rights, equality and citizenship programme 2014–2020’. Under the criterion of equity, gender equality aspects were considered together with rights of the child and rights of people with disabilities (see Box 4). However, a gender perspective was not adopted in the rest of the criteria.
The strategic ‘Evaluation of EU support to gender equality and women’s empowerment in partner countries (2010–2015)’, commissioned by DG International Partnerships, specifically adopts a gender analytical framework based on the ‘5Cs’ as key factors that contribute to GEWE results. The analytical framework assumed that if the combination of the 5Cs is in place, the EU will contribute to a high level of GEWE results at country level.
In turn, the most recent evaluation of the EU action in this area reconstructs the intervention logic of the EU external action related to GEWE. The evaluation acknowledges, however, the non-linearity of the intervention logic in an effort to recognise that changes related to gender equality do not always follow the pattern of a linear result change (see Box 5).
It should be noted that, contrary to what could be assumed, EU evaluations in the area of gender equality do not necessarily adopt a gender-responsive design[69]. As highlighted by the United Nations Entity for Gender Equality and the Empowerment of Women (UN Women)[70], a gender-responsive evaluation has two essential components: what the evaluation examines and how it is undertaken. This second component, which emphasises aspects such as inclusiveness, participation of all stakeholders and accountability, should be particularly considered when conducting evaluations with a focus on gender equality
Examples at the Member State level
In Member States, gender-responsive evaluations are mostly carried out as evaluations of policies on gender equality and gender mainstreaming. For example, such evaluations can be found in Spain[71], France[72], Luxembourg[73] and the Netherlands[74]. Some Member States also integrate a gender perspective into programme evaluations that dedicate a part of funding to gender interventions, such as in the field of social policy or technology and higher education (see example in Box 6).
Member States usually do not pursue a standard approach to gender-responsive evaluations. Specifically, the evaluations may follow stages of the policy in question (conception, implementation and the results and effects)[76] or rely on the evaluation questions to structure the evaluation report (see Define gender-sensitive evaluation questions).
When evaluation follows a standard approach at country level, it tends to rely on the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD DAC) evaluation criteria of relevance, effectiveness, efficiency, sustainability and impact[78] (see Box 8).
Examples of gender-responsive evaluations at the Member State level usually employ interviews and focus groups for gathering data and participatory methodologies that focus on the direct beneficiaries of the interventions (see Box 9). For another example of an evaluation with a gender perspective that adopts a participatory approach, see Box 10.
Other examples
United Nations
At the international level, the United Nations Evaluation Group (UNEG), an interagency network that brings together the evaluation units of the UN system, has developed standards and practical tools[82] for UN staff to integrate gender equality and human rights into evaluations.
In 2013, UN Women adopted an evaluation policy[83]. This policy establishes a framework for ensuring the independent evaluation of UN Women’s achieved results in the pursuit of gender equality and its role in leading and promoting gender-responsive evaluations across the UN system.
UN Women has also developed tools[85] to support the institutionalisation of gender-responsive evaluations and has a dedicated database[86] that hosts and rates all its evaluations to support accountability and learning.
Independent evaluators and civil society
EvalGender+[87], which is part of the global partnership EvalPartners, is an international network that promotes gender-transformative evaluations to accelerate commitments to achieve the SDGs. They develop knowledge products to advocate for gender-responsive evaluations, provide funding and disseminate knowledge and best practices. They contribute to the Gender & Evaluation Community of Practice[88], which is an open global platform to anyone working on gender-responsive evaluations. Voluntary organisations for professional evaluation play a key role in promoting gender equality in evaluations since the evaluation of public policies is frequently conducted by external evaluators and consultancies.
In civil society sectors, evaluations with a gender perspective have been used as a tool for decision-making, advocacy and to demand accountability from political actors (see Box 11). Other times, civil society organisations cooperate closely with executive powers to conduct an evaluation (see Box 12)