Enabling centralisation and closing the loop at Birkbeck, University of London

From devolved to centralised module evaluation

Birkbeck, University of London, is one of the UK’s most distinctive providers of higher education. The majority of Birkbeck students study part-time, alongside full-time students, with high numbers of mature students. However, Birkbeck remains unique in that all its provision (full and part-time) takes place in the evening. Billed as “London’s evening course specialists”, Birkbeck celebrates its 200^th anniversary in 2023.

Seeking to transform its approach to module evaluation feedback, the institution first undertook an Explorance Blue pilot project across five departments in the summer term of the 2018-19 academic year before implementing full roll-out at the start of 2019-20.“ Prior to using Blue we did not having anything in place that was centralised,” explained Academic Services Manager Garmon ap Garth. “It was a completely devolved process, so all 19 departments would have their own systems in place, usually paper but a couple online using different systems. The practice varied considerably between departments – some with 4-5 questions for their surveys, others with 20-30 – so there was no consistency.”

The “biggest concern” over not having a centralised system, Garmon said, was there was no way to capture the knowledge being gained. “It was not clear what students were saying and it was also unclear what staff were doing with the feedback they received, or whether students would hear back again on what they fed back on,” he revealed. “The whole thing was not put together in any coherent way, and there was no longitudinal analysis, so these are the things we wanted to change by implementing Blue.”

‘Closing the loop’ to encourage future participation

As part of his MBA in Higher Education Management, Garmon conducted research on module evaluation practices within the Russell Group of universities. One of the findings of his 2019 study was that only six of the 23 institutions had a policy in place to ‘close the loop’ and this further informed his thinking on what he wanted to achieve at Birkbeck. “Closing the loop was one of the key outcomes we wanted to get when we went out to tender for a system and a key demand for the project with Blue,” he said. “We wanted to ensure consistency in practice and consistency in reporting, and to meet the requirements of providing data to internal and external bodies, including Office for Students (OfS) requirements to better understand the student experience, and explain and eliminate differences between different demographics.”

Garmon continued: “Survey fatigue is a common worry in the sector and shown to impact response rate, but closing the loop is accepted as the most effective way of encouraging participation. If the students know that their feedback is being considered and acted upon they are much more likely to provide feedback in future as well. Clearly there is a drive amongst all universities in the UK to improve National Student Survey (NSS) results and student voice is an aspect of that. Being able to provide feedback should therefore have a benefit for NSS.”

Birkbeck has followed the example of Loughborough University after being impressed with Dr Sarah Williamson’s presentation on closing the loop at Bluenotes Global. “When our evaluation survey (20 core questions with up to five departmental questions, up to three module-specific questions and optional individual tutor evaluation) closes, the results are sent to the module convenor,” Garmon said. “A new project is created in Blue which creates a task to complete three questions for closing the loop. Reports are sent to students, including the closing the loop answers (with actions to be taken), within two weeks of the report closing. We have made it clear to staff that this is not the only way to close the loop, including messaging in Moodle and using the report in the staff-student liaison group, but that it is a key way.”

Immediate impacts and future success measures

Whilst Birkbeck has only completed one year of Blue module evaluation, and the change is “still in very early stages”, Garmon was able to highlight some immediate impacts on NSS student voice results which were released in July 2020: the ‘I have had the right opportunity to provide feedback on my course’ and ‘Staff value students’ views and opinions about the course’ statements increasing in agreement by 2% each. He said: “It is not a conclusive improvement, but at least it is in the right direction. However, one of the most important things is our ability to get the reports out to students without delay otherwise they will be less likely to complete the evaluation in future years.”

Looking ahead, Birkbeck plans to seek to automatically feed the Blue responses into annual reviews and ensure 100% completion rate from module convenors; receive feedback from students on the processes – to what extent do they actually read the student reports and are they useful?; and explore other ways to make the reports available e.g. to help with module selection. “We know the importance of tracking student experience as early as possible in line with the OfS plan and others,” Garmon explained. “So we are seeking more understanding of the granular level experience and have more opportunity to address any issues. Also for students to have a better experience, and to tell us that they are having a better experience, which is then acknowledged formally in the NSS.”

He added: “It has been really good working with Explorance. The processes have been very straightforward and there have been no technical issues whatsoever. The team are responsive to issues, they respond quickly to questions, and give us what we need. We have also benefited from them setting up regular meetings with other universities to give us opportunities to find out things we did not know. I would definitely recommend them to other universities. They deliver on exactly what they say they will do.”

Further innovation through Bluepulse

A new development at Birkbeck for 2020-21 is to trial Bluepulse with the Students’ Union. Through the initial pilot Bluepulse, student class reps are being given access to the system, to gather feedback from students throughout the institution, and aiming to enable rapid responses from the student body and therefore being able to react quickly and effectively. By giving student class reps an additional communications tool to ask questions, results are being fed back to staff through the staff-student liaison group or other relationships they have with staff. “There are challenges with how staff and students engage with formative evaluation tools effectively so finding a system that we can use consistently across all staff and students is something we are keen on. We are excited about utilising Bluepulse,” Garmon said.