Institution:
Liverpool John Moores University
Location:
Liverpool, England
Challenge:
The university needed a solution that can handle their complex module evaluations and easily generate reports that can be customised and shared at all levels.
"The beauty of Blue is that we can very easily set surveys up to run at any time. There is a lot of power in the system."
Key benefits:
- Handles complex large scale evaluations for modules that are offered under more than one programme
- Offers question personalisation to programme leaders
- Tracks live response status across all modules during evaluation periods
- Breaks down results by student demographic information (age, gender, origin, programme, etc.)
At Liverpool John Moores University (LJMU), module evaluation is a key survey for quality assurance and enhancement. However, their evaluation process had a complexity that made it challenging for the institution to measure all aspects of the teaching and learning environment effectively.
“We survey every level and every module taken by students,” says Professor Clare Milsom, Director of Teaching and Learning Academy. “We have large modules which may have a number of different programmes and students take up to five modules a year which are all evaluated simultaneously.”
For a long time leaders of multi-programme modules wanted to be able to see how students from different programmes responded to the module, as combined scores and comments made it difficult to address programme specific feedback.
Furthermore, modules with a non-standard delivery patterns (e.g. those that run over summer or with multiple starting times) were difficult to capture. “We also do not evaluate individual members of staff. We evaluate the delivery – the teaching and learning.”
With these and other criteria in mind, the university decided to pilot three solutions for their module evaluation process, amongst which Blue stood out prominently.
A module evaluation system that fits
For over 10 years, LJMU was using a standardised online survey that was being outsourced to an external company. In 2014, with the help of the Pro-Vice Chancellor (Education) they decided to pilot Blue module evaluations.
“We met the Explorance team many years ago and were following the development of Blue because we were interested in the instrument”, says Dr. Elena Zaitseva, Academic Research and Development Officer. “We wanted to evaluate this pilot thoroughly and looked at student engagement, response rates, as well as how staff and module leaders engage with the instrument and with the results.”
During the pilot, the impact of Blue on the university’s module evaluation process was impressive.
“Response rates improved in semester one where we surveyed a limited number of modules. We saw a big jump from about 30% to over 50%,” she says.
Blue’s flexible and easy-to-use interface also resulted in greater engagement in the process by members of staff. “Reports were immediately available and members of staff were much more willing to share Blue reports with their students because of the quality and professional look.”
“We can split the data into parts and give module leaders evidence about what their own students think about that particular module.”
When asked if they would recommend Blue for institution-wide use, the majority of staff who were involved in the pilot said yes.
Digging deep into response rate data
LJMU moved from piloting Blue to administering it institution-wide with approximately 80,000 evaluations across 2,000 modules. Powerful and flexible, Blue’s response rate dashboard delivered visibility into module evaluations that the university did not have before.
“Now staff can go into the portal and monitor their own module response rates in real time,” says Dr. Graham Sherwood, Faculty Registrar.
“We can see the time of day that students reply and what we are finding is that although we have portal access, most responses come as a direct response to a reminder email.”
Blue is integrated with the university’s IT infrastructure to synchronise reporting and provide access to relevant levels of information.
“There is a lot more information coming in about how students reply. We can see satisfaction by programme, by school, by gender,” he says. “We can split the data into parts and give module leaders evidence about what specific groups of students think about that particular module. Once reports are published they are in a repository in Blackboard where staff can access them at any time.”
Providing staff with data they can use
Something else that members of staff had asked about for the past few years is the ability to add their own questions to the evaluations. With Blue, module leaders were given the option to choose up to two additional questions from a list of 16, or write their own.
“We were pleased that the majority of staff used question personalisation, but we were enormously pleased that over 83% of staff at the School of Natural Sciences and Psychology used QP,” says Dr. Zaitseva. “This was staff that used it for the first time last year.”
84% of staff also believed that the information value and usability in reports were excellent or good. For the first time the voices of students from different programmes were heard and you can clearly see that for some module leaders it was very helpful information.”
Blue enabled the university to link student demographic data to individual reports and those reports can be customised for programme leaders or senior management. “It’s all done automatically in Blue. You could play with the format of the reports. It’s a very powerful system.”
“We are very aware of how powerful Blue is and are still yet to explore all functions that the platform offers. The training and support exemplary, we were fully assisted through our semester one survey. We are now managing on our own as we deliver 1500 module surveys across a range of programme structures.”
Professor Clare Milsom,
Director of Teaching and Learning Academy
Blue•Course evaluations•Customer story•Higher education•Student insight solutions•
