Institution:
University of Newcastle
Location:
Newcastle, New South Wales, Australia
Challenge:
The University of Newcastle needed ways to improve the quality of its feedback data to give students a stronger voice in decisions concerning their academic experience.
"The pre-trained models were a key difference for us because other off-the-shelf text analytics tools weren’t trained using Higher Education comments. With MLY, we’re speaking the same language."
Key benefits:
- Quickly identify emerging issues and prioritize comments of concern.
- Improved response rates on student surveys.
- Transitioned to a fully qualitative student feedback approach.
- Efficient, accurate, and repeatable process for qualitative feedback analysis.
The Challenges of Building a Feedback Culture
In 2011, the University of Newcastle launched its online student feedback program, utilizing Explorance Blue for web surveys and online course evaluations. This had a profound impact on data accessibility, leading to a culture of data-driven decision-making processes to shape and track institutional priorities.
Over the past 10 years, the feedback culture has matured, and in recent years, significant gains have been made with the help of Explorance solutions. “We have put a significant amount of effort into increasing the quality of feedback that we collect, and it is changing the entire survey landscape,” said Mrs. Meagan Morrissey, Manager Student and Staff Insights at the University of Newcastle. “This resulted in a higher level of engagement from our staff and students.”
In 2020, the university sought to improve the quality of the data collected and explored innovative avenues to amplify the student voice. It boldly undertook a radical transformation of its student surveys, eliminating all 30 quantitative questions and retaining only a single mandatory qualitative question, at the same time shifting the focus from quantitative scores to qualitative insights.
“We no longer specify to our students what they can provide feedback on,” Meagan explained. “Instead, we offer an open text field where students can provide feedback on the elements of their university experience that matter most to them.”
The resulting change provided The University of Newcastle with a much richer data source and actionable insights for effecting tangible improvements in the learning experience. This initiative was paired with a targeted marketing campaign aimed at cultivating feedback literacy. It involved imparting students training on the art of delivering constructive feedback and framing the entire process as an indispensable skill for future workforce readiness.
The university successfully captured students’ attention by presenting feedback as a meaningful aspect of the student journey, fostering further engagement. Students understood that their feedback was being listened to, valued, and acted upon. As a result, survey response rates doubled, and the more detailed responses uncovered various issues previously undetected.
However, the success of this new program presented the university with a compelling challenge: an abundance of qualitative data lacking an efficient and systematic method of analysis.
In response, the university invested in an in-house solution that enabled them to identify comments that had been previously read and verified and flag any comments containing words from a dictionary-based reference file for inappropriate or concerning content. While this improved the situation, it had significant limitations given the evolving nature of language and slang variations.
The University of Newcastle Feedback Analysis Methods
The university explored various options. At first, they accepted that analyzing qualitative comments was a lengthy process and then considered outsourcing the work. However, neither worked due to strategic and ethical concerns. The university stumbled upon machine learning, AI, and Explorance MLY during its research.
Explorance MLY emerged as an ideal solution, enabling the university to conduct accurate analyses of entire comments rather than relying on a word-for-word approach. This technology proved invaluable in prioritizing comments of concern and uncovering themes, sentiments, and recommendations that might have eluded human analysis. With the platform’s specialized models for student experience, it naturally categorized higher education terminology, providing insights into both in-class and extracurricular experiences.
Meagan explained that “The pre-trained models were a key difference for us because other off-the-shelf text analytics tools weren’t trained using higher education comments. With MLY, we’re speaking the same language.”
The integration of machine learning eliminated bias and naivety and markedly reduced human error in the process. Consequently, MLY has become a key component of the University of Newcastle’s feedback analytics toolbox. It offers the university a straightforward, reproducible, and defensible method for consistently analyzing qualitative feedback over time.
“We were reading comments most of the year, and now MLY is processing that workload in minutes,” Meagan continued. Her team leverages this extra time to take their feedback culture to the next level. This includes developing strategies to prevent harm and continuing to reinforce their feedback culture for both students and staff.
Explorance MLY alerts model is critical to helping the University of Newcastle quickly identify comments of concern. By automatically flagging inappropriate comments and courses with increased issues, university heads and executives have an initial starting point to focus on emerging topics before they escalate out of control.
With a straightforward drag and drop of a CSV file containing filtered answers, MLY effortlessly takes over. “The Recommendations Model is key to what you could do more or less of within the course,” Meagan stated. Whether it involves additional resources, online tutorials, or reducing time allocated to a specific topic, this model provides a comprehensive overview of tangible changes students suggested for the course.
Explorance MLY Amplifies the Student Voice
Building on the seamless integration of Explorance MLY into the university’s feedback analytics toolbox, this AI-powered qualitative analysis tool empowers academic leaders to analyze and categorize student qualitative feedback swiftly and efficiently. MLY excels at processing unstructured student comments from diverse sources, offering detailed insights, student experience-specific categorization, sentiment analysis, alerts on sensitive topics, and actionable recommendations.
MLY is versatile in its feedback-source compatibility, accommodating student comments from various channels, including:
• Course evaluations
• Central and major surveys
• External review sites (e.g., Rate My Courses, EDUopinion)
• Social media (e.g., Facebook, Reddit, X)
This broad spectrum of feedback sources facilitated a more authentic and impactful analysis of the student voice, unveiling trends that would have remained undiscovered without this depth of data.
The University of Newcastle Embraces MLY
Contrary to concerns about artificial intelligence and machine learning replacing human staff, the University of Newcastle has had a different experience. MLY continues to be viewed as a valuable tool that empowers university staff to elevate their programs and operations to new heights.
Expanding beyond student feedback, the University of Newcastle also harnesses MLY’s employee experience model, gaining valuable insights into the staff’s commitment to the university. When comparing staff feedback to student feedback, the university has observed more elaborate employee responses, often fueled by heightened emotion and passion. Recognizing the significance of this, the University of Newcastle increasingly emphasizes employee comments in its ever-evolving feedback landscape.
About the University of Newcastle
With a focus on providing an inclusive and supportive learning environment, the University of Newcastle offers a diverse range of undergraduate, postgraduate, and research programs. It caters to over 37,000 students, including a substantial international cohort from over 115 countries, enriching the university’s cultural fabric and global perspective.
The University is celebrated for its strong commitment to equity and social justice, reflected in its various initiatives to support indigenous students and communities. Its campuses, located in vibrant urban and regional areas, boast state-of-the-art facilities and a research-driven environment, fostering innovation and creativity among students and faculty alike.
The University’s commitment to real-world impact is evident in its approach to education, research, and student feedback. By aligning its strengths with industry needs and societal challenges, it has become a hub for transformative solutions, nurturing the next generation of leaders and change-makers.
Course evaluation•Explorance MLY•Higher education•Student feedback•Student voice•
