australia-awards-global-monitoring-and-evaluation-framework-2022

Australia Awards Global Monitoring and Evaluation Framework February 2022

Contents Introduction ................................................................................................................. 3 Executive Summary ............................................................................................................................. 4 Document Summary ............................................................................................................................ 5 Purpose ................................................................................................................................................ 6 Guiding Principles ............................................................................................................................... 6 Components ......................................................................................................................................... 6 Section A: Global Program Logic ....................................................................................................... 7 Section B: Monitoring and evaluation across the Australia Awards cycle ..................................... 8 Section C: Global Assessment, Learning and Reporting ............................................................... 12 2 Australia Awards Global Monitoring and Evaluation Framework

Executive Summary Key Audience The primary audience for the MEF are users and with roles and responsibilities for monitoring, evaluation and reporting across the Australia Awards. The MEF seeks to provide Posts, Managing Contractors and the DFAT Global Education and Scholarships Section with clear and detailed information that supports them to meet global data collection and reporting requirements. Purpose The MEF seeks to support the capture and analysis of a range of information that enables Australia to tell the Australia Awards performance story, communicate its impact, and support ongoing adaptation and decision making for improvement. Targets Six performance targets will form the basis for systemically tracking, assessing, learning and reporting on the Australia Awards at a global level. These are related to: 1. Promotion and Selection 2. Inclusion 3. Influencing Development 4. Alumni Network 5. Bilateral Collaboration 6. Women’s Leadership Country and regional programs are required to report against a linked set of Core Indicators on an annual basis. This seeks to ensure consistency of data collection across the global program and enable DFAT to tell the global performance story of the Australia Awards. Importance The Australia Awards has delivered significant impact over several decades. It is critical that we continue to engage with and communicate across whole-of- government, to the public and to partner countries, providing key messaging on why the Australia Awards is important, what it has achieved and ensure accountability for the use of program resources and the achievement of outcomes. Use The MEF will inform on the development of a range of products. This includes a wealth of rich data gathered and used at a country and regional level. This information will also be used by the DFAT Global Education and Scholarships Section to communicate its value and impact to a wider audience, produce an annual performance investment report, and feed into decision making across the Australia Awards. 4 Australia Awards Global Monitoring and Evaluation Framework

Document Summary 5 Australia Awards Global Monitoring and Evaluation Framework

Section A: Global Program Logic The Global Strategic Framework sets out objectives and two Long-term Program Outcomes. These are shown in Figure 1 below along with three Intermediate Outcomes, which are building blocks to the achievement of Long-term Outcomes. Every country or regional program will implement Australia Awards in a way that is tailored to the specific context and its relevant priorities. This may mean that individual outcomes are accorded greater or lesser priority, and receive greater or lesser allocation of resources within country or regional programs. Furthermore, some modalities may be more or less closely associated with particular outcomes. Figure 1: Australia Awards Program Logic The full Australia Awards Program Logic can be found in Guidance Note 1: Program Logic , which can be accessed here . 7 Australia Awards Global Monitoring and Evaluation Framework

Section B: Monitoring and evaluation across the Australia Awards cycle A range of groups and organisations collect, hold and use information about the Australia Awards to support day to day decision making, accountability and communication to a range of stakeholders. This includes:  Country and regional programs that manage Australia Awards – mainly Posts, but in some cases also DFAT Desks in Canberra  Managing Contractors – most Australia Awards country or regional programs are supported by a contractor  DFAT Global Education and Scholarships Section  Academic Institutions in Australia  Academic Institutions in the Pacific region  The survey contractor engaged by DFAT to undertake regular Arrival and On Award surveys of scholarship awardees across all country and regional programs  The Australia Awards Global Tracer Facility  Other specialist contractors engaged to support M&E  Australian Host Organisations  Alumni Associations The section provides information about:  How monitoring and evaluation will be undertaken at key points in the cycle of scholarships and short courses (and the program logic);  Who is responsible for M&E tasks;  How the evidence generated by M&E will be shared and used at each point in the cycle/logic, and by whom;  How information will flow throughout the Australia Awards cycle. 8 Australia Awards Global Monitoring and Evaluation Framework

Australia Awards Monitoring and Evaluation tools This MEF relies on a set of M&E tools – activities and sources of information – to provide the evidence necessary for management and reporting. A number of these are already established and providing the necessary data. Others require further systematisation, are in development, or must be developed in order to fulfil the requirements of effective monitoring, evaluation and management of Australia Awards at the global level. Within this set of M&E tools, several should be understood as the minimum requirements for all programs implementing Australia Awards; others are optional and should be included when relevant and possible for specific investments. Table 1: Australia Awards data collection tools Tool Required/Optional Status Awardee information (including application, academic progress and pastoral care data), stored in OASIS Required by all programs Established On Award Surveys Required by all programs Established Alumni information Required by all programs Stored on country program level databases Managing Contractor M&E Required by all outsourced programs Established Tracer Studies Required by all programs Global Tracer Facility established in 2016 Short Course Awards M&E Required by all programs 1 Media Monitoring Required by all programs Requires systematisation across Posts Post-managed Program M&E Required by all programs Requires development Other Evaluation studies Optional A more detailed description of the above listed tools is provided is provided in Guidance Note 2: Monitoring and evaluation across the Australia Awards cycle , which can be accessed here . 1 Short course M&E must be in place to support reporting against mandatory Core Indicators. 10 Australia Awards Global Monitoring and Evaluation Framework

Information Flows Information is drawn from a number of sources throughout the cycle of Australia Awards planning, implementation, monitoring and evaluation, and is stored in several different ways. Figure 2 below provides an overview of the main sources and repositories of this information, to assist programs to understand where to draw from in seeking to ensure good quality monitoring and evaluation for their implementation of Australia Awards. Further detail on: Capture and use of evidence generated by M&E in relation to the program logic; Responsibilities and Timeframes for Monitoring and Evaluation Activities; and a list of M&E Language and Terminology is outlined in Guidance Note 2: Monitoring and evaluation across the Australia Awards cycle which can be accessed here . Figure 2: Australia Awards information sources and databases 11 Australia Awards Global Monitoring and Evaluation Framework

The Global Education and Scholarships Section will work with Posts to stay abreast of the range of studies and M&E products, and explore ways to communicate findings and results to a broader range of stakeholders in Australia and globally. Figure 3: Global assessment, learning and reporting Our performance: Assessing, analysing, learning and improving Australia Awards is a long-standing initiative with an overarching strategic framework that guides country and regional programming. Other DFAT Official Development Assistance (ODA) funded initiatives typically have a set of End of Program Outcomes that are expected to be achieved at the end of a time- bound investment. Australia Awards are an ongoing investment, whereby the overarching investment logic and long-term outcomes have and are likely to remain relatively unchanged over time. The design, implementation and performance of Australia Awards does not however remain fixed. This global assessment, learning and reporting framework has been designed to ensure the Australia Awards continues to be agile and adaptive. Rather than simply reporting on the achievement of outcomes and activities each year, this framework seeks to build a cumulative picture that supports an assessment of Australia Awards performance over time. Measuring and reporting on outcomes and performance Country and regional programs obtain, analyse and report on performance and the achievement of Australia Awards outcomes in accordance with their own M&E frameworks and systems. The global MEF will draw on this information to examine and communicate results in addition to data managed centrally by the Global Education and Scholarships Section. 13 Australia Awards Global Monitoring and Evaluation Framework

To provide a basis for systemically tracking, assessing, learning and reporting on the Australian Awards at a global level , six performance targets are introduced. These progress markers are framed as targets to emphasise Australia Awards strive for continuous improvement and adaptation, noting that the Program Logic and associated outcomes are expected to remain relatively unchanged over time. The targets are aligned to the global strategy framework, the program logic and encompass elements of Gender Disability and Social Inclusion (GEDSI). With regards to the program logic, Long-term Outcome 1 is included in Target 3 2 , and Long-Term Outcome 2 in included in Target 5 3 . Intermediate Outcome 1 sits within target 5 4 . Intermediate Outcome 2 sits within Target 3 5. Intermediate Outcome 3 sits within Target 5 6. GEDSI is reflected in Targets 2 and 6. Enabling outcomes have not been integrated into the framework 7 . In addition to supporting data collection, analysis and reporting on outcomes, the targets seek to support assessment of how well the Global Education and Scholarships Section, country programs and other stakeholders are working to facilitate outcomes. Collectively, they provide a framework that enables DFAT to tell the global performance story. Six Performance Targets: 1. Promotion and Selection: Country programs attract and select increasing proportions of diverse, high-calibre, eligible applicants, placing them in Awards that align with DFAT priority areas; 7. Inclusion: Country programs effectively promote the Australia Awards to vulnerable and minority groups, have demonstrated capacity and processes in place to support inclusion, and strive to strengthen systems and practice if standards are not met; 8. Influencing Development: Australia Awards is an effective mechanism which contributes to an increasing number of alumni better equipped to influence development outcomes in their home countries across a diversity of sectors; 9. Alumni Network: Country / regional programs build a bigger, stronger network of active alumni by creating meaningful opportunities for quality engagement that contributes to alumni advancement in their home country; 10. Bilateral Collaboration: Australia Awards and alumni engagement increasingly contribute to enduring relationships and mutual cooperation between Australia and the partner country; 11. Women’s Leadership: The Australia Awards contributes to a growing cohort of women leaders who are increasingly able to participate, influence and lead across a diversity of development sectors. The performance targets guide Annual Investment Performance Reporting led by the Global Education and Scholarships Section, and assessment and learning across the global Australia Awards. Additional detail and definitions related to performance targets are provided Guidance Note 3: Core Global Indicators. 2 Long term outcome 1 ‘Alumni use their skills, knowledge and networks to contribute to sustainable development’ is directly captured by indicator 3.3 ‘Number and percentage of alumni able to provide valid examples of contributions to country objectives, using the skills, knowledge or networks gained from their award’. 3 Long term outcome 2 ‘Alumni contribute to cooperation between Australia and partner countries’ is directly captured by indicator 5.2 ‘Number and percentage of alumni able to provide valid examples of participation in mutual collaborations between their country and Australia, after their completion of the Australia Award’. 4 Intermediate Outcome 1 ‘Alumni view Australia, Australians, and Australian expertise positively’ is not captured by indicators that country programs are required to report against on an annual basis as this data is captured centrally. 5 Indicator 3.1 ‘Number and percentage of awardees who report satisfaction with their award’ seeks to capture data on the role of the Award in building skills, knowledge and networks. Indicator 3.2 seeks to capture data on whether alumni are in roles that are relevant to the study. 6 Intermediate Outcome 3 ‘Alumni have relevant and useful networks and relationships’ is captured by indicator ‘1 Number and percentage of alumni who have drawn on bilateral links, established through the Australia Awards’. 7 The targets seek to build on data that has historically been captured by Country Programs and seek to capture priority aspects, rather than measure all components related to outcomes and performance. 14 Australia Awards Global Monitoring and Evaluation Framework

Global Assessment and Learning In addition to synthesising information and reporting on outcomes and performance annually as described above, in-depth analysis and learning will also be facilitated. This will enable DFAT to undertake critical enquiry, reflection and a deeper-dive on critical elements than afforded through annual reporting. DFAT administers the Australia Awards through a devolved architecture, encompassing the Global Education and Scholarships Section, country and regional programs, and Posts. Country and regional Australia Award programs have their own M&E frameworks and capture a wealth of data to track progress and impact. This MEF seeks to facilitate cross-country and regional learning, as well as learning between the Canberra-based Global Education and Scholarships Section and Country and regional programs (inclusive of Posts and associated Managing Contractors) 8 . To support this process, a Monitoring and Evaluation Community of Practice (COP) will be convened by the Global Education and Scholarships Section. The purpose of the COP is to reduce silos and enable DFAT Canberra, Posts, Managing Contractors and the Australian Council for Education Research (ACER), which manages the Global Tracer Facility, to share and discuss their work, explore a range of in-depth issues determined by members, and inform Global Education and Scholarships Section decision making. The Monitoring and Evaluation COP will:  be co-designed and led by members , who will determine areas, issues and topics to be discussed and examined;  enable DFAT and Managing Contractor staff to further refine core indicators discuss and navigate M&E approaches and methodologies associated with implementation and reporting;  enable members to share and discuss insights and learnings from impact studies, evaluations and other learning processes;  support members to discuss what is and is not working across the Australia Awards , and explore a range of in-depth issues and solutions;  provide a two-way feedback mechanism between the Global Education and Scholarships Section and country and regional programs that prompts changes and improvements to the global architecture (systems, policies and guidelines) to ensure flexible and responsive delivery, particularly in the COVID-19 context. The Global Education and Scholarships Section may commission other more in-depth assessments of the Australia Awards such as independent reviews and evaluations, meta-evaluations and analysis of country and regional data, and other research pieces. These will be strongly informed by the COP. To support joint learning and assessment at the global level, a set of guiding questions are presented below. This tool is intended to provide a light-touch guiding framework linked to the Performance Targets. The majority of questions seek to prompt reflection on the role of the Global Education and Scholarships Section. 8 Country programs engage partner country representatives and alumni themselves in country-led M&E processes. In some instances, the COP may seek the inclusion of in-country stakeholders in CoP sessions and global M&E and reflection processes. 16 Australia Awards Global Monitoring and Evaluation Framework

Table 2: Australia Awards Performance: guiding questions for learning and assessment Global Performance Targets Guiding Questions Promotion and Selection 1. Country programs attract and select increasing proportions of diverse, high-calibre, eligible applicants, placing them in Awards that align with DFAT priority areas  Do Australia Awards enable sufficiently tailored and flexible delivery options to meet Awardee study preferences and remain competitive against other international awards programs? 9  Are global strategic framework investment priorities, requirements and delivery options enabling sufficient diversity of awardees?  Are Australia Awards eligibility requirements enabling selection of the right candidates 10 most able to influence change? Inclusion 12.Country programs effectively promote the Australia Awards to vulnerable and minority groups, have demonstrated capacity and processes in place to support inclusion, and strive to strengthen systems and practice if standards are not met.  Are Country Programs increasing efforts 11 to enhance GEDSI, including identifying and addressing barriers to inclusion and which approaches are successfully fostering equal inclusion and benefit?  To what extent have investments actively involved people with disabilities and/or disabled people’s organisations in planning, implementing, monitoring and evaluating?  Are there unintended consequences 12 for minority groups on return and how can Australia Awards minimise harm?  Are global strategies 13 to encourage equity of access 14 sufficient and are country programs adequately supported to implement strategies and dedicate sufficient resources and inputs towards inclusion? 9 The global strategic framework seeks to achieve flexibility through diversification of delivery options including remote, online and hybrid delivery options, a pivot towards long- and short-term awards in-country and provision of broader opportunities beyond face-to-face and in-country completion to achieve people to people links, exposure to Australia, and soft power benefits. This is expected to contribute to the competitiveness and inclusivity of Australia Awards. 10 Australia Awards assumptions on ‘the right candidates’ are outlined in Principle 2: Merit-based selection of the global strategic framework. 11 As per Core Indicator 2.2, efforts are defined at: development or enhancement of GESDI Policies, strategies, guidelines or procedures; research or analytical work; staff capacity development that has resulted in strengthened practice; provision of support to individuals. 12 For example, people with disability make experience a high level of accessibility on Award in Australia, and return to their home country where access is limited and they are unable to apply their knowledge and skills and become despondent. 13 Global strategies are outlined in Principle 3: Equity of access of the global strategic framework. 14 Equity of access includes across the stages of promotion, application, on-award and post-award. 17 Australia Awards Global Monitoring and Evaluation Framework

Bilateral Collaboration 15.Australia Awards and alumni engagement increasingly contribute to enduring relationships and mutual cooperation between Australia and the partner country  What success factors underpin positive awardee experiences, perceptions and connections with Australia, and how can Australia Awards nurture these on-award?  Which strategies and modalities are most effective in facilitating links that endure over time, and what learnings does this hold for the Australia Awards Linkages Framework?  What lessons do Country Program impact studies and evaluations hold for how the Australia Awards can garner alumni participation in mutual collaborations 17 ? Women’s Leadership 16.The Australia Awards contributes to a growing cohort of women leaders who are increasingly able to participate, influence and lead across a diversity of development sectors.  Are Australia Awards increasingly targeting development sectors where women’s participation is most needed 18 and how can gender-targeted placements best be supported?  In what areas are country programs achieving and not achieving progress in effectively implementing strategies to promote gender equality and women’s empowerment’ 19 .  What lessons can be learned for how Australia Awards can best enhance women’s leadership, career advancement, and ability to contribute to outcomes at rates equal to male alumni 20 ? 17 Examples of levels of participation include: professional - joint projects or partnerships, either development or business / private sector related; research – including engagement in a joint research project with an Australian university or research institute; and community initiatives – including bilateral collaborations supported by community grants or initiatives such as with Australian groups (such as rotary, schools or universities or ANGOs). 18 The Global Strategic Framework supports the targeting of development sectors where women’s tertiary skills and increased participation in policy development, decision-making and technical implementation aspects of development are most needed in each country. 19 Progress in this area is measured through Indicator 6.2 which is derived from IMR reporting. 20 It is acknowledged that different enabling environments will afford higher or lower levels of women’s empowerment, and that the Australia Awards may be a relatively limited or significant factor in women’s advancement. However, 19 Australia Awards Global Monitoring and Evaluation Framework

For more information and key resources Website Australia Awards Scholarships | Australian Government Department of Foreign Affairs and Trade ( dfat.gov.au ) Enquiries enquiries.canberra australiaawards.org @AustraliaAwards @australiaawards @AustraliaAwards 20 Australia Awards Global Monitoring and Evaluation Framework