AI-powered Student Voice Analysis for NSS and Module Evaluation at Liverpool John Moores University

Challenge: Unlocking Timely and Actionable Insights from Qualitative Analysis

Liverpool John Moores University (LJMU) had been using generic text-mining software to analyze open-text comments derived from student surveys for over 10 years. However, the process proved challenging due to an overreliance on the interpretation of a single researcher, a lack of dynamic learning, and the resource-intensive nature of the system. Since the model was not trained on student feedback, extensive manual thematic analysis was required to contextualize the results generated by the software.

“The value of open-text feedback is that it leads to actionable insights and can actively inform local decision-making and broader institutional strategies related to student retention and success,” explained Fenna Boerkamp, Institutional Research and Evaluation Officer in the University’s Student Voice and Evaluation Team. “Whilst we had used text analytics technology for institutional survey analysis for a long time, we were exploring opportunities to introduce a model that was specifically designed to analyze student feedback to better support timely decision-making.”

Solution: Implementing Purpose-built Feedback Analytics

LJMU selected Explorance MLY due to the platform being specifically trained to analyze student feedback, and for the additional features it offers. These include MLY’s alerts (instant access to concerning/problematic comments, or comments indicative of a student being in distress), analysis-sharing functions, and the option to compare demographic groups.

“MLY not only derives topics but also generates recommendations from the comments which is a useful feature”, Fenna said. “It allows us to break down the data by demographics using the filters in the tool, and also create widgets for specific demographic groups, for example, the most frequent topics for only male or female students. Additionally, MLY enables sharing and collaboration on the projects, allowing all key stakeholders to participate in interpreting and finetuning the findings together.

For our institution specifically, the alert function that MLY offers has been really useful as we still, according to our policy, have to redact all names out of negative comments. That means we read all comments in our module evaluation survey, and we try to figure out which comments we may need to remove names or parts, and the triggers can significantly reduce the time needed to complete this process.”

The University applied MLY to two specific projects during the first six months. The first was for analysis of the 2023 National Student Survey (NSS). The test project was successful, as the majority of themes identified by MLY matched LJMU’s rapid manual analysis. Additionally, the recommendations from MLY were valuable in supporting the institution’s action plans, proving its effectiveness for NSS-type analysis.”

Outcome:  A Faster and More Scalable Approach to Processing Qualitative Data

For LJMU, the application of MLY has already delivered valuable insights that have enhanced the institution’s understanding of the student experience.

“Using MLY to analyze qualitative data is ultimately far less time-consuming and resource intensive as MLY does more of the work for you,” Fenna commented. “It is also possible to process very large amounts of data: for example, we have generated initial insights from over 38,000 comments for module evaluation in a few minutes. Hence, scalability is a huge benefit of MLY. It is easier to cover a much larger amount of comments and derive more recommendations from the comments than you would manually be able to, and that provides more actionable insights that inform institutional change.”

Fenna concluded: “Adding MLY to LJMU’s portfolio of analytical tools has enhanced understanding of student needs and improved the student experience by providing faster, real-time analysis without the need for lengthy manual data processing. The constantly learning algorithm and new features have definitely added continuous value in analyzing student feedback at scale.”