James Cook University Takes Course Survey Analysis to a New Level with Blue
James Cook University
Cairns, Townsville, Mackay, Mount Isa, Thursday Island and Brisbane, Queensland, Australia
A university wanted an automated course evaluation system that could separate course-based data from teaching-based data for more sophisticated analysis.
"We made several requests specific not just to our needs but to an Australian context, and Explorance was responsive...We are quite amazed at how quickly Explorance was able to respond to [our] need."
- Doubled response rates
- 99% of students responded that they liked the new system (small sample)
- Running course and instructor surveys seamlessly from
- Blackboard LearnTM LMS across international campuses
- Sophisticated reports and analysis
When the Teaching and Learning Development Unit (TLD) at James Cook University (JCU) started looking for a new course evaluation survey system, they had a long list of requirements.
DETAILED DATA ON COURSE AND INSTRUCTOR EVALUATIONS
One of the world’s leading institutions focused on the tropics, JCU is known for its cutting edge research. Those high standards are apparent in TLD, which gathers student feedback data and conducts extensive reporting analysis. JCU separates teacher assessment survey reports from the course survey reports, in order to get a more precise picture of each.
“Often individual teachers are not in control of the design of the content that they’re delivering in our multi-campus environment. We want to make sure that we’re working with staff around the issues that they can change with their courses and/or teaching,” said Dr. Glenn Harrison, Academic Developer at James Cook University.
“Having two surveys but in the same instrument enables us to separate those student comments that are course-based from those that are teaching-based.”
A former physiology researcher and lecturer with 20-years of teaching experience, Dr. Harrison now works closely with teaching staff to improve their teaching and course delivery while assisting the institution to establish benchmarks and maintain standards.
ADVANCING CHANGE THROUGHOUT THE UNIVERSITY
To manage these instruments, JCU previously ran course evaluations on a home-grown online system and the teacher evaluations on paper.
The separate systems were not ideal. Students would often confuse the instruments, which affected response rates and the quality of the data. Crucially, the paper-based system was also resource-intensive and costly.
To establish better processes, JCU assembled a university-wide reference group. The group comprised representatives from every stakeholder in the course evaluation process, including each Faculty, the Student Association, Corporate Planning and Performance, IT, and the Singapore campus among others. It also included external consultant Dr. Sid Nair, a well-known international consultant in quality and evaluations in higher education.
“The reference group was essential to the success of this project,” said Katrina Green, Project Manager for Teaching and Learning Development at James Cook University and who played an integral role in the implementation and management of Blue.
“They gave important feedback on what the new system and processes should look like, each from their own perspective. They also went back to their areas as well-versed spokespersons for the new initiative.” Under the stewardship of TLD, the group spent about one year developing a comprehensive new policy for teacher and course surveys, which outlined not only guiding principles and procedures but important details such as the question bank.
PUTTING PROVIDERS THROUGH THEIR PACES
With a clearly defined policy to guide them, TLD set out looking for a new system to replace their old technology. The replacement system had several assessment criteria. The new system had to be a unified survey platform but could issue separate reports on teacher and course surveys, it had to work within a multi-campus environment (they have several campuses across the state of Queensland and internationally in Singapore), and it had to meet their advanced analytics requirements.
JCU evaluated several systems, including some developed by Australian universities, Australian software companies, and North American software companies.
They selected Blue by Explorance, an online course evaluation and survey system developed specifically for an academic environment.
BLUE COMES OUT AHEAD IN SEVERAL CRITERIA
“We had several reasons for selecting Blue,” said Green. “It’s a flexible system, and it gave us the control we needed over the process.”
JCU also liked several key features of the product like the Blue Portal Integrator, a semi-independent Web application for integrating with Learning Management Systems (LMS) and school portals.
“It was really important to us that Blue could integrate with our Blackboard LearnTM LMS,” Green said.
PHASED ROLLOUT ACROSS ENTIRE UNIVERSITY IN UNDER ONE YEAR
Rollout of the new Blue course and teacher survey system took place in phases, going from start to finish in 12 months.
JCU first implemented in December 2012, and ran a pilot in January with a small number of courses. They went live with the entire university for course surveys in May 2013. This phase included integration with their Blackboard Learn LMS, allowing students to access course evaluations directly from the university student portal. They went live in October 2013 with the teacher surveys added to the course survey.
The system is fully hosted by Explorance so that the IT team at JCU doesn’t have to perform routine maintenance or other requirements.
“We have three major campuses plus satellite campuses around Australia, and one in Singapore. They are now doing all course and instructor evaluations online. The surveys are running well between us and Singapore via Blackboard — we are very pleased about that success.”
Project Manager for Teaching and Learning Development
DOUBLED RESPONSE RATES AND RAISED STUDENT SATISFACTION
JCU undertook several initiatives to promote course surveys, even before they went live with Blue.
“We developed a communication strategy, which included a new website, targeted communications such as posters, graphics, bookmarks, and slide shows for instructors to show in classrooms,” Green said.
“One of our most effective tactics was the ‘roving’ students, who spoke directly to students and answered questions, particularly around confidentiality. The roving students carried iPads, which students could use to complete their course evaluations on the spot. That worked much better than offering an overall prize.”
Those initiatives help raise response rates by 20%. Once the university went live with Blue, they were able to increase response rates by yet another 20%.
“We like the mobile display capabilities, the personalized emails, and the automated email reminders. We think the Blackboard integration is probably the biggest contributor to the 20% increase,” she said. “We placed a big blue button on the Welcome page of the portal, which takes students directly to their evaluation forms. These are now available from a single place and listed in an intuitive order. They don’t even have to login separately.”
The Teaching and Learning Development Unit asked students how they felt about the new system, and got a near perfect score.
“We asked students in our pilot whether the new system made sense, whether they liked it, and if there was anything we should change. It was a small trial — a few hundred students — out of those, 99 per cent said they liked it,” said Green.
“This kind of data is very valuable when introducing change,” added Harrison. “You have to make a convincing case when approaching staff. Numbers like this make a big impact.”
IMPROVING TEACHER DEVELOPMENT
As the teacher evaluations are not mandatory at JCU, Explorance had to ensure that Blue provided the ability for instructors to opt into an evaluation if they so desired.
“We are quite amazed at how quickly Explorance was able to respond to this need – it was a matter of weeks,” said Harrison. “A staff member can now select, in a completely safe and accurate way, to opt-in for teacher evaluations. It’s been really successful.”
JCU decided to place the course coordinator in charge of activating the teacher evaluations in Blue, rather than allow every instructor access to make this selection. Explorance set up Blue to match their processes.
“Teachers work with the course coordinator to determine whether a teaching evaluation is appropriate for their class,” said Harrison. “The idea is to have the teaching team having a discussion about it. Then they can look at the data together and make improvements together.”
The Evaluations team conducts extensive analysis on the evaluation data, combining results with demographic data for better understanding.
The university has a mandate, for example, to ensure all student populations are represented in the responses. Blue reports show demographic information on the student respondents, including gender, socio-economic differences, indigenous status, and other categories tracked in their student information system.
“Reports to staff and administrators show pie charts illustrating the demographics of the respondents,” said Harrison. “We already know, for example, that there is a female gender bias in the data. We have a 60/40 split of females and males enrolled at the university, but we have a 70/30 split on students who do the survey.
That’s something to keep an eye on.” They can also dig deeper into the data for greater understanding.
“That split varies: on some campuses and in some disciplines it’s exaggerated further or is brought back to neutral,” he said.
To research how they can improve response rates, JCU uses Blue embedded Google Analytics to understand student behaviour with the surveys.
“With Google Analytics, we can look at aspects like what time students are completing the survey, what devices they’re using to complete them, whether they’re coming in from the email link or the Blackboard link. We can make adjustments based on this data to help improve response rates.”
DETAILED REPORTS FOR STAFF AND ADMINISTRATORS
JCU appreciates the robust data they’re getting from Blue, and have created several types of reports for the various stakeholders.
“The quality of quantitative information that’s coming to staff is already very apparent, and the staff is very appreciative,” he said.
For the course survey, the Evaluations team issues a report to staff and administrators that shows the mean score for every question, the statistical distribution for every question, a chart and tables that illustrate aggregated frequency, and pie charts that show respondent demographics information.
“One of our favourite reports is the Individual Course report. This report shows mean scores for every question, as well as comparative mean data, which allows staff to see responses from one class compared to all classes in their discipline, their school, faculty, and the university as a whole. They can also see how responses compare to other classes of similar size,” said Harrison.
“The second section contains histograms with the mean, median, and mode for each question, and frequency, which is important. For those that don’t like means and histograms, there is the third section, which shows a graph and table showing combined overall agreement. The fourth section shows the answers to the open-ended questions.”
Harrison works closely with instructors to ensure these reports are used effectively for reflection and professional development of teaching staff. He is careful to get to the details that matter when using aggregated data.
“Knowing class size is important when making comparisons, which is something we weren’t able to compare data on before we had Blue,” he said. “There’s no doubt that there are differences between very large classes and very small classes, and at JCU we have a lot of very small classes. It’s very comforting for a staff member teaching an undergraduate class of 600 students that they know they aren’t being compared to a postgraduate subject of 10 students.”
REPORTS THAT ENGAGE STUDENTS IN THE PROCESS
The Evaluations team also likes the Student report, a one-page report that shows a histogram displaying the aggregated responses for every question. Once the survey is closed, students can see the percentages on how many of their peers agreed, were neutral, or disagreed with each question.
“The student report allows us to send information back to students in a way we never could before, so it feels less like a one-way avenue for them,” said Green.
“We’re trying to develop a culture here around how data is collected and how we interact with students,” Harrison added. “When you keep students in the loop and show them that you are responding to their feedback, they’ll accept change and keep responding at a good rate.”
For the future, JCU has a number of projects lined up that involve even more analysis, using other tools such as learning analytics and free text analysis. One particularly intriguing project is to create comparative reports and benchmarks using data from other universities in Australia.
“We’ve been speaking with another university here in Australia that is also using Blue, and talking about how we can combine data to show comparative results between the two universities,” Harrison said.
A TRUE PARTNERSHIP WITH THEIR SOFTWARE PROVIDER
JCU had high praise for the customer service they received from the support team at Explorance, which has worked closely with JCU to ensure their needs were met.
“Customer service has been fantastic,” said Green. “They were always prompt in their response to our questions — we also benefited very much from the training they gave to us here.”
With a high quality survey system, excellent customer service, and exceptional reporting output, JCU is ultimately very pleased with their decision to implement Blue.
“I think Explorance understands the Australian market well — the regulatory requirements, and even that the culture is different from North America. Their people appreciate these differences and adjust to meet our needs, not the other way around.”