Monitoring and evaluation are important parts of the process of change. As you know by now, a gender equality plan (GEP) will typically address several issues at once, leading to a complex set of measures. Nonetheless, effective monitoring and evaluation instruments are often lacking, which undermines the transformative potential of the planned measures. If objectives are not indexed on relevant progress, success or outreach indicators, it is difficult to assess whether the organisation is actually being transformed. This might also reduce the commitment of stakeholders to meeting those objectives. Having an appropriate monitoring and evaluation plan in place, however, can support the effective implementation of measures, ensure accountability, and enhance your knowledge and understanding of ongoing changes. This way, you also know whether your GEP requires adjusting.
Besides these logical arguments for thinking about monitoring and evaluation from the very beginning, it is also a GEP requirement for participating in Horizon Europe. You need to be aware that, in order to be eligible for Horizon Europe, ‘it is mandatory that organisations collect and publish disaggregated data on the sex and/or gender of personnel (and students, where relevant) and carry out annual reporting based on indicators’ (see Horizon Europe Guidance on Gender Equality Plans, pp. 23–27). While this step comes only after planning and implementing your GEP, as laid out in the step-by-step guide (because that is when you start monitoring the effects of your measures), you need to know that the monitoring and evaluation strategy needs to be set out beforehand.
Ideally, you considered which areas you wanted to focus on in step 2 when analysing and assessing the status quo in your organisation. In step 3, you then set out specific, measurable, attainable, realistic and time-related (SMART) targets and measures addressing these areas. In order to develop a monitoring and evaluation strategy, use the status quo assessment as a starting point. The results of this assessment will establish the baseline, which will allow you to monitor and evaluate your progress.
In order to develop an effective monitoring and evaluation strategy, you need to differentiate between monitoring targets and evaluation targets. In order to understand the difference, consider the following definitions, used by the gender equality monitoring tool (pp. 3–8) of the EU project ‘Taking a reflexive approach to gender equality for institutional transformation’ (TARGET).
The tool defines monitoring as a continuous process, in which data is systematically collected in order to provide management and key stakeholders with regular updates on the progress and achievement of objectives and the use of allocated funds.
Evaluation, on the other hand, relates to a systematic and objective assessment of an ongoing or completed project, programme or policy based on the monitoring data, providing lessons learnt for the planning of future measures.
In other words, ‘Monitoring ensures that the right thing is done, while evaluation ensures that the right outcomes are achieved’ (TARGET gender equality monitoring tool, p. 3), which means that the two go hand in hand. But as you can see from the definitions above, their specific targets differ: the monitoring targets focus on the specific outputs and related processes (implementation level), while the evaluation targets relate directly to the targets set out in your GEP (i.e. to the impact or outcome you wish to achieve) (strategic level). This is also why the respective time frames differ: while an evaluation is a more in-depth analysis, usually conducted at the end of your GEP or funding cycle (of course, interim evaluations are possible), your monitoring targets will be assessed at much shorter intervals, to inform you about the progress. Remember that Horizon Europe requires annual monitoring reports.
The factors that need to be considered in order to identify monitoring targets also generally apply to the evaluation level. However, keep in mind that achieving the desired outputs (as shown by the monitoring) does not necessarily lead to an achievement of the desired outcome or impact (as shown by the evaluation). This might be the case, for instance, if the measured output to increase the number of teachers who participated in gender training to a certain percentage was reached, but the training was not effective enough to ensure the desired outcome of increased gender competence among teachers.
The monitoring and evaluation indicators that you will identify are concrete variables that you can measure in order to assess if a monitoring or evaluation target was reached. For more information on how the different dimensions relate to each other, consult the gender equality monitoring tool directly.
Now that you know about the basics of monitoring and evaluation (and the difference between the two), you can think about a concrete strategy. Note that it is always helpful to involve people who have experience with monitoring and evaluation in this process. You can also check how it was performed in other organisations, by looking at the examples in the (GEAR) action toolbox.
When coming up with a strategy, use the logic model you identified in step 3 (embedded in a theory of change): the impact pathway of the measures implemented will help you identify what you want to monitor and evaluate. As for the details of your strategy, you need to differentiate between monitoring and evaluation. Consider the following steps in order to come up with a monitoring strategy.
Identify concrete output indicators. For this purpose, take a look at your GEP and your status quo assessment and identify output indicators for each of your measures. A list of potential indicators is provided below. Consider that the collection of the relevant data needs to be feasible with the resources available to you.
Select appropriate data collection instruments. Know that, in general, these will be the same instruments used in step 2 for your status quo assessment. Broad data will be available from the funding process, other (internal) data might be available on a regular basis from the human resources department, still other data you might have to collect yourself. For the data you need to collect yourself, you could conduct an annual survey of staff to monitor change, for instance. The gender equality audit and monitoring (GEAM) tool provides ready-to-use surveys (implemented via the LimeSurvey) for this purpose, including a specific part for funding bodies.
Come up with a time frame. Your monitoring should take place annually, including annual monitoring reports published on the organisation website to meet the Horizon Europe requirements.
Plan regular monitoring sessions. Involve the core and/or extended team responsible for your GEP. These meetings can be crucial for reflecting on the progress by looking at the monitoring data and exchanging experiences. This way, you will be able to react to potential issues and steer your measures in the desired direction.
As for creating your evaluation strategy, the general process will be similar, but some things need to be considered in more depth.
Think about the context. When planning the evaluation of your GEP, you need to consider the context of your organisation. Relevant context factors were discussed in step 1. As the evaluation is more extensive than your annual monitoring, you need to consider, in particular, the time and (human and financial) resources available to you. These will also depend on the type and size of your organisation.
Identify additional impact indicators. While you will consider output indicators in your evaluation, it will also focus specifically on the impact of your implemented measures. Make sure to include both quantitative and qualitative indicators, as some measures cannot be properly assessed by looking only at quantitative figures (e.g. the participation of women in senior positions, the number of cases of sexual harassment reported to / dealt with by the relevant body).
Use additional (qualitative) data collection instruments. In your final evaluation, you may want to add additional data collection instruments, such as individual interviews, focus groups, participatory workshops, document analyses or participant observation. These qualitative techniques will allow you to gain a deeper understanding of the impact of your measures.
Take your monitoring results into account. You will carry out your evaluation at the end of your GEP cycle. In your final data analysis, also include the results of your monitoring process.
Keep in mind that monitoring and evaluation should not be a huge burden for you, but that some time needs to be invested to ensure the success of your measures. When drawing upon external expertise to carry out your monitoring and evaluation processes, we recommend bringing together these external evaluators and the people in charge of implementing change within the organisation to ensure that instruments are adapted to your goals and constraints.
For additional resources on how to plan your monitoring and evaluation process, take a look at the tab ‘Tools and resources’.
A list of quantitative indicators (as suggested by the Horizon Europe Guidance on Gender Equality Plans) was provided in step 2. The same indicators should be considered for continuously monitoring progress in your organisation. Of course, the list is not exhaustive and indicators should be selected to fit your specific targets and objectives. Furthermore, the indicators in the list below may have to be adjusted to the context of your planned measures:
staff numbers by gender at all levels, by disciplines, by function (including administrative/support staff) and by contractual relation to the organisation;
average numbers of years needed for women and for men to make career advancements (per grade);
wage gaps by gender and job;,
numbers of women and men in academic and administrative decision-making positions (e.g. boards, committees, juries);
numbers of women and men candidates applying for distinct job positions;
numbers of women and men having left the organisation in the preceding years, specifying the numbers of years spent in the organisation;
number of staff by gender applying for / taking parental leave, for how long they took leave and how many returned after taking the leave;
numbers of absence days taken by women and by men differentiated, by absence motive (sick leave, care leave etc.);
numbers of training hours/credits attended/received by women and men;
numbers of women and men students at all levels and for all disciplines, and academic and employment outcomes;
shares of women and men among employed researchers;
shares of women and men among applicants to research positions, among people recruited and success rate, including by scientific field, academic position and contract status;
shares of women and men in recruitment or promotion boards and as heads of recruitment or promotion boards, and shares of women and men in decision-making bodies, including by scientific field.
Based on the targets set out in your GEP, specific indicators need to be developed to establish a baseline and monitor progress. Such indicators help to build accountability for the success or failure of implemented measures. The ‘Evaluation framework for promoting gender equality in research and innovation’ (EFFORTI) toolbox can support you in identifying quantitative (and qualitative) indicators to measure the output, outcome and impact of your measures.
You may also want to consider breaking down the data even more and looking into additional dimensions besides gender: looking at intersectionality can include individual or group features, such as a migrant or minority background, disabilities, low socioeconomic status or at risk of poverty, sexual orientation, and so on. Pay special attention to data protection issues if you plan on breaking down the data into small groups. Consider also any national regulations on collecting personal data.
Look at the tab ‘Tools and resources’ for examples of quantitative indicators identified by other organisations and resources provided by structural change projects.
Qualitative indicators are especially relevant to see whether your desired outcomes were reached. However, qualitative indicators can also give additional information on your progress and help you understand the dynamics of change (or lack thereof). Qualitative indicators may look at dimensions such as the following.
Mainstreaming of gender knowledge. This can be measured, for instance, by the relevance given to knowledge creation on gender equality within the organisation, the institutionalisation of gender equality (in the form of dedicated programmes or departments), the dissemination of gender equality knowledge across disciplines, and so on.
Awareness among different categories of staff and external stakeholders (reviewers, board/panel members, applicants). This can be measured by the attention given to gender equality by different categories of stakeholders through communication initiatives, codes of conduct and activities centred on gender-related aspects.
The uptake of gender equality objectives set out in your GEP. This can be monitored by observing the participation in and acceptance of your implemented measures and the (human and financial) resources allocated to support these measures.
The actual transformation towards greater gender sensitivity. This should focus on the effects on both formal and informal practices due to the implemented measures. It may, for instance, be shown by increased attention being given to women’s ideas and perspectives in decision-making mechanisms that are dominated by men. Consider different levels of hierarchy.
The diffusion of a gender equality culture. This can be measured in terms of changing working conditions, but also verbal and non-verbal interactions and decision mechanisms (seating arrangements in panels). It could be reflected in changes regarding the management of work–life balance, awareness of sexual harassment and other aspects of gender-based violence, non-sexist communication, and so on.
For more inspiration and sample questionnaires, see the GENDER-NET report on qualitative indicators in research institutions. Know that qualitative indicators have a huge learning potential. They support self-reflexivity and may provide useful indications for a continuous enhancement of the implemented measures. They may also provide evidence that change happens and that gender equality and awareness are not out of sight. Techniques to collect qualitative data include individual interviews, focus groups, participatory workshops, document analyses and participant observation.
To help with the implementation of your monitoring and evaluation instruments, EU-funded structural change projects have developed a number of useful templates and ready-to-use resources. Switch to the tab ‘Tools and resources’ to view the full list. The ‘Gender equality in engineering through communication and commitment’ (GEECCO) data monitoring tool, for instance, provides an Excel template that includes indicators for a variety of targets, detailed definitions, formulas to calculate shares and other useful options. You may also want to take a look at the GEAR action toolbox for best-practice examples of monitoring and evaluation in other organisations.
Once you have collected and analysed the data, you can check if there have been any (significant) changes since your initial status quo assessment (baseline). You should also assess whether the monitoring and evaluation targets have been met. Discuss the results with your team and draw conclusions on what they mean for your GEP. During your monitoring, you might want to check whether you need to adjust some of your targets or the way that measures are implemented. During the final evaluation, you need to ask yourself what you can learn for the next GEP cycle.
Finally, you need to communicate your results.
Provide annual monitoring reports as well as a final evaluation report. These reports should be published on the organisation website. The number of evaluation periods and reports depends on your individual strategy.
Regularly update leadership/management about the results. This will be done through your annual monitoring and final reports. However, you might also want to involve them in meetings or update them more regularly, depending on your organisational structure.
Inform other stakeholders within your organisation. It is not just the leadership that is interested in the progress of change in your organisation. Make sure to communicate the results to all relevant stakeholders. You might also want to keep them engaged by organising a meeting to present and discuss the results of your analysis (e.g. after the final evaluation of your GEP). Note that the monitoring and evaluation process is also an effective way to keep stakeholders (including leadership/management) engaged and to ensure their ongoing support for your measures. It also paves the way for the future by helping you design even more resolute measures for your next GEP.
Consider external stakeholders. Your communication can also target policy stakeholders at regional or national level, professional associations or other institutional partners of your organisation.
When communicating the impact of your measures, know that there might be other positive side effects (or added value) of your implemented measures: the entire process may lead to a strengthened sense of community; more transparent recruitment, appraisal and evaluation procedures; stronger pluri-disciplinarity in research; and improved working conditions in general. All of this may be uncovered by your monitoring and evaluation process. In particular, your final evaluation will show the positive dynamics brought about by gender mainstreaming strategies and their inherent opportunities. Making these positive side effects visible can help strengthen your position and build the foundation for the next GEP cycle.
In order to view videos and webinars or further tools and resources on the topics discussed in step 5, switch between the respective tabs. Otherwise, click below to continue to the next step and read about how to ensure the sustainability of your measures. You can also go back to the previous step.
The EU-funded project SPEAR prepared video presentations to help practitioners understand the steps involved in implementing a GEP. The videos are based on the steps provided in this GEAR step-by-step guide. Watch the videos on steps 5 and 6 to get a better understanding of how the process works and what to consider in these steps. Note that there are also tasks for you to perform at the end of some of the videos, to check your understanding of the topics. GEAR tool – steps 5 and 6
There is a useful tutorial on how to use the ‘Promoting gender balance and inclusion in research, innovation and training’ (PLOTINA) monitoring tool.
The ACT GEAM tool also provides useful training videos.
Guidelines for creating a monitoring and evaluation strategy
For data collection and monitoring, the GEAM tool contains a specific section for funding bodies, covering external funding activities and internal processes.
To assess structural change, the EU-funded project ‘Transforming organisational culture for gender equality in research and innovation’ (GENOVATE) developed comprehensive guidelines for evaluating gender equality action plans. These guidelines include inputs from the evaluation literature and guide you through the steps of the evaluation process.
Another very comprehensive guide was written in the course of the EU-funded project GenderTime, entitled A model for building a gender equality index for academic institutions. This 2016 guide addresses the problem of measuring gender equality in academia. It starts by defining the problems and arguing for the importance of appropriate monitoring and evaluation, then provides detailed definitions and, finally, introduces different approaches. It also describes how to build a system of indicators in great detail.
In the context of the ‘Gender equality in information science and technology’ (EQUAL-IST) project, a report based on the experiences of monitoring and evaluating GEPs in seven research-performing organisations was published. The report presents the assessment methodology and indicators used in the monitoring process and provides a monitoring template plan.
The wheel toolkit designed by the EU-funded ‘Systemic action for gender equality’ (SAGE) project (2017) aims to assist organisations in implementing and sustaining change in cycles. At each cycle, the organisation should reassess itself and carry out changes towards gender equality. It includes a comprehensive three-phase approach to GEPs and institutional changes. Check out the project report entitled Mapping tools for the evaluation of gender equality plans.
If you want to see more examples of how data was collected in other organisations, see Structural Transformation to Achieve Gender Equality in Science – Guidelines (pp. 29–34) produced by the EU-funded project ‘Structural transformation to achieve gender equality in science’ (STAGES). In this project, strategies for structural change were launched in a number of research organisations, with the guidelines being written based on the experiences in these organisations.
The European Institute for Gender Equality (EIGE) website also provides a section on gender monitoring, including guidance on how to build up a set of indicators, as well as a gender statistics database.
Examples of useful monitoring and evaluation indicators
The gender equality monitoring tool of the EU-funded project TARGET provides multiple examples of how to define indicators for different target areas. It builds on a logic model, showing the pathway from the input and set activities to the different outputs, outcomes and general impact. For each dimension in this model, example targets and indicators are provided.
The EU-funded project PLOTINA created a monitoring tool based on 10 core indicators and 40 specific indicators, which can be selected based on the focus of your GEP. Check out the full list of indicators.
Baltic Gender developed an updated version of its handbook of gender indicators, which describes gender-sensitive indicators and provides information on the rationale, the data needed, the computation method, and initial ideas for data analysis and critical issues.
The EU-funded project ‘Female empowerment in science and technology academia’ (FESTA) provided a thorough guide on quantitative indicators and methodology in its FESTA toolkit.
Science Europe developed a Practical guide to improving gender equality in research organisations; the second part of the document (p. 26) deals with ‘How to monitor gender equality’.
The ‘Gender equality network in the European research area’ (GENERA) planning–action–monitoring (PAM) tool can help you find measures, indicators and targets for GEPs in the field of physics. You can choose to click through the online tool or download the entire PAM tool as a PDF document.
The following databases and tools can help you get an overview of how your country is performing compared with other European countries regarding selected gender equality indicators: You can consult She Figures 2021, EIGE’s gender statistics database or GENDERACTION’s data dashboards.
Ready-to-use monitoring and evaluation tools
The EU-funded project GEECCO developed various evaluation and monitoring materials, including an Excel template and a PowerPoint tutorial, for collecting and analysing sex-disaggregated data in research-performing organisations. It covers three core areas: (1) decision-making processes and bodies, (2) recruitment and career development of women researchers and staff and (3) the sex/gender dimension in research and teaching content.
The PLOTINA monitoring tool is an online tool that can help you measure and visualise your progress over different periods. Watch the tutorial video in the tab ‘Videos and webinars’ to get started.
The EU-funded project ‘Gender diversity impact – improving research and innovation through gender diversity’ (GEDII) designed a gender diversity index based on the share of women in different positions, by age and other factors. A self-assessment tool was also developed; you can enter the relevant numbers for your organisation directly on the website and receive your gender diversity score automatically. This score can be calculated repeatedly to monitor change within an organisation.
Report templates to communicate your results
The EU-funded structural change project ‘Institutional transformation for effecting gender equality in research’ (INTEGER) prepared a number of reporting templates, providing suggestions on the structure and relevant content to be covered. Take a look at the context report template, the process report template, the impact report template and the final report template for inspiration and guidance.
