In February 2022, Alan Donnelly and Liz Austen in STEER began to work on an evaluation of the Postgraduate Certificate (PGCert) and Master’s (MA) Student Engagement in Higher Education at the University of Winchester. The programme of study, which was launched in 2018, is described as “a professional development opportunity to critically examine and influence research, policy and practice” in student engagement (Moxey et al., 2022). The student cohort predominantly comprises of practitioners who are working in roles within higher education.
The programme team commissioned an evaluation to find out what impact the PGCert / MA has had on the practitioners studying on the course and within their organisations of employment, and to explore how the design and processes of the course, including its blended delivery, have contributed to the outcomes being achieved. This blog post presents: 1) the experiences of the practitioners on the course who co-designed and participated in the evaluation; 2) the reflections of the programme team about what they learnt from the project; 3) the perspectives of the evaluation team from STEER.
The evaluative approach
The evaluation of the course was structured using a Theory of Change approach, which is defined as “a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context” (Centre for Theory of Change). The aim of the evaluation was to understand ‘how much’, and ‘how and why’, change had occurred on the course since its inception. An Advisory Group of practitioners, who were either studying on the course or had graduated, were recruited to: co-create the Theory of Change and determine their own definitions of ‘success’; design the methods of data gathering; and reflect on emerging findings. Evidence of impact was collated from practitioners studying on the course, graduates and staff on the course via peer-led interviews, online reflective activities and other sources of evidence.
The perspectives of practitioners: “Working in partnership to create the scope of the project”
“What I found most interesting about being part of the project was how much the participants actually guided the work undertaken by the team. We were trusted in being the experts of being a student and moved beyond being an advisory group to working in partnership to create the scope of the project as well as participate. I doubt I’ll be part of an experience like it again so I’m very grateful to have been a part of it.” Contribution from Aimee Cuthbert
“I was positively surprised at how the [Advisory] Group leaders [STEER] managed to move the participants past the point of providing feedback on the course, to working collaboratively in identifying a methodology to evaluate the program. I was involved in the peer-led interviews and once again I had a positive experience. It was great to have the chance to hear what other students thought about the course, especially being able to reflect critically on some issues.” Contribution from Gloria Laurini
“All members of the [Advisory] Group brought equally valuable knowledge and experience, and we were able to work together constructively, feeding off each other’s ideas to inform proposals for the evaluation process. The report that has been produced offers considerable food for thought for the student engagement community and the wider higher education sector and it was a pleasure to be part of the process.” Contribution from Kevin McStravock
The perspectives of the programme team: “The response drew together our research area of student engagement/co-design to offer a considered study fit for the current HE evaluation context”
“For the programme team, the evaluation of the PGCert/MA Student Engagement by the STEER team was valuable for a number of reasons. Students had repeatedly shared anecdotally with us the impact the course had on their professional development, such as going on to get promotions, or feeling more confident when discussing student engagement related issues with senior managers. Having watched students grow in confidence, go on to present their work at national conferences, and writing up their assessments for journals and blogs, we were keen capture and evidence the outcomes for our future practice. So for us, it was exciting to have these benefits (among others) affirmed and documented through an evaluation which was conducted by evaluators who were external to the programme.
The evaluation took place during the time in the programme where second year students were finishing the module entitled ‘Researching and Evaluating Student Engagement in Higher Education’, and embarking on their own research dissertations. Therefore, the evaluation was well-timed as an experiential learning opportunity for the students as the focus of this module was all about teaching research skills suited to evaluating student engagement practice. As the module lecturers, we were able to draw on the evaluation both as an example of participatory research, as well as discussing positionality and methods of reducing potential for bias. The Theory of Change approach being used in practice to shape the research design based on the ethos of student/staff partnership has provided a learning opportunity for the programme team, so we can draw on this in future teaching content for the course.”
Contributions from Maria Moxey (Teaching Fellow in Student Engagement in Higher Education at the University of Winchester) and Tom Lowe (Senior Lecturer in Higher Education at University of Portsmouth and former programme leader of the PGCert/MA Student Engagement)
The perspectives of the evaluators: “A learning experience for all groups involved”
“The experience of working on this evaluation was educational, enrichening, and eye-opening. The Advisory Group provided unique insight into the course that shaped the focus of the evaluation, its outcomes and the methodology that was adopted. Collaborating with these practitioners from the outset made us appreciate the variety of motivations and expectations that they had for studying on the course, which highlighted the importance of accounting for these individual differences within the evaluation design.
The decision of the Advisory Group to use peer-led interviews as part of the evidence collection was an innovative idea that we had not anticipated. This approach was chosen as it was in alignment with the collaborative ethos of the course and the exchange of stories between participants in the interviews became a learning experience for all groups involved. The evaluation demonstrated the potential of using Theory of Change to structure the planning and design of evaluation of a taught course in higher education. At the end of the project, members of the Advisory Group reported developments in their own understanding about approaches to evaluation which they have applied within their own assessments”. Contributions from Alan Donnelly and Liz Austen (STEER)
An executive summary of the project, methodology and findings is available here: https://shura.shu.ac.uk/30905/1/Donnelly_2022_Process_impact_evaluation.pdf Please contact Alan Donnelly (firstname.lastname@example.org) for further information about this project.