Building an Evaluative Mindset at Hallam #1: Our Approach
by Liz Austen
STEER (Directorate of Student Engagement, Evaluation and Research), as the name suggests, has always had an evaluative function. This has included support for the delivery of the current Access Agreement (2019-2020), and the newly approved Access and Participation Plan 20-25 (APP). The Office for Students states that “Access and participation plans set out how higher education providers will improve equality of opportunity for underrepresented groups to access, succeed in and progress from higher education.”
The 2020-2025 APP includes a section on the ‘evaluation strategy’ of the institution. This was written based on our self-assessment of the scope and scale of evaluation practices. Hallam has some areas of good practice in place across the institution and our overall assessment was positive. However, there is a need to build capacity and capability and address the hidden expectations surrounding evaluation design and implementation. STEER aim to support staff with areas such as data confidence, evaluation methods and objective setting so that interventions and initiatives can be designed for optimum reach and impact.
Expectations of Evaluation
The Office for Students (OfS), as the regulator of higher education and approver of the APP, have provided a comprehensive overview of their expectations of evaluation practice. OfS require robust evaluation so providers can prioritise and understand what works to eliminate inequalities. They have cemented their commitment to three (non-hierarchical) Standards of Evidence:
“the narrative of evaluation (knowing what you are doing and why); empirical enquiry (evaluation to measure changes generated by different activities and practices); and consideration of causal claims (what impact can you identify as a direct result of your activities?).”
They have also outlined an Evaluation Framework which covers 5 key areas of an evaluation cycle. Each of these areas will be discussed in 5 forthcoming blog posts:
- Strategic Context
- Programmes/Initiatives Design
- Evaluation Design
- Evaluation Implementation
- Learning
Hallam’s commitment, outlined within the APP, states that:
“We will embed an evaluation mindset in our approach; it will be a core competency of staff and will drive objective setting, programme design and our teaching and learning… We will work to embed an evaluative mindset, so evaluation is visible, resourced and valued in CPD, staff recruitment and workload management.”
During the 19/20 academic year STEER will prioritise the identification of evaluative requirements, specifically:
- Continuing to support staff working on strategic measure programmes with iterative and systematic evidence-informed programme design and implementation.
- Developing training and development sessions for staff to develop capacity and capability concerning methodology and examples of ‘what works’.
- Designing and implementing a new process of ethical approval to allow institutional service evaluations to be shared externally and inform future iterations of work.
- Creating and piloting an evaluation repository to communicate evaluative outputs and raise awareness of evaluative practices.
What do we evaluate?
Building an evaluative mindset is not just linked to the strategic APP. Developing evaluation capacity at Hallam would also:
- Support leaders and managers who are asked to analyse strategically relevant data sets and provide evidence of the impact of activity. For example: differential student outcomes, retention and progression targets.
- Improve the content and scope of internal quality enhancement processes leading to improved learning and teaching / student satisfaction. For example: Module Review, Departmental Review, and Periodic Review.
- Improve the content and scope of external quality assurance processes leading to improved national recognition of teaching excellence. For example: institutional and subject TEF narratives.
- Support core teams asked to collate and analyse strategically relevant data sets and provide evidence of the impact. For example: Access and Participation Plans via TASO / OfS, REF Impact Case Studies
References
Office for Students – Evaluation and Effective Practice https://www.officeforstudents.org.uk/advice-and-guidance/promoting-equal-opportunities/evaluation-and-effective-practice/