The Drive for Evidence-based Decision Making at the University of Johannesburg

With a student population of over 50,000, of which more than 3,000 are international students from 80 countries, the University of Johannesburg is the third biggest residential university in South Africa. Since 2018, the University has been on a course evaluation journey, from paper-based to Blue-based evaluation that is semi-automated and fully integrated with Blackboard.

As a Senior Manager in the University’s Centre for Academic Technologies (CAT), set up to provide a richer learning experience for students and fulfil the institution’s vision of 21^st century skilled academia through the promotion of innovative technologies, I have effectively been driving the University’s ‘journey’ on this. Going back to where we started the old system of course evaluation was ad hoc, paper-based and cumbersome. The bubble sheet scanners we used were reaching the end of their life and, with a steady increase in evaluations, operationally we had to do something different.

A change in thinking was also underway. Evaluation reports had been primarily used for academic professional development and promotion purposes, so the University’s management team had no insights from this valuable data source. Additionally, module and lecturer evaluations did not contribute to the University’s overall strategy to improve student success through evidence-based decision making.

A decision was taken that all first-year undergraduate modules and all priority modules had to be evaluated. Therefore, a new approach was needed that was efficient at handling high volumes of evaluations – 4,000 in all – while also providing meaningful data to the University as a whole. After investigating Blue, we were really happy with the scope of the offering and appointed Explorance as our course evaluation partner.

With colleagues in the Centre for Academic Staff Development (CASD) we began work in 2018 with a small group of volunteers in the University’s College of Business and Economics and ultimately piloted the online system over a two-week period. We were particularly pleased with Blue’s integration with Blackboard – the anonymous and easy-to-use interface, enabling strategies for increased responses, improved reporting, automated feedback and data analytics on the feedback received.

The University has developed two categories of evaluation: taught courses – undergraduate and postgraduate (module evaluations and teaching evaluations) – and supervision programmes (different sets of questions), and full implementation is underway. Last year was the first time we ran all undergraduate evaluations online – featuring 1,043 modules and 1,388 academics, with over 40,000 students surveyed, and more than 1,000 reports being released.

The biggest success has undoubtedly been:

• Automated evidence-driven reporting for individual lecturers
• Aggregated reports for the Department, School, Faculty or College
• Aggregated reports with relevance to student successes and staff development
• Aggregated report on all evaluations for the Deputy Vice-Chancellor: Academic, to whom CAT/CASD colleagues and I ultimately report to.

Our course evaluation journey continues, and we have a number of goals that we want to achieve. Closing the loop, and feeding back to both students and academics, is one area for development alongside some more operational targets. However, going back to the rationale that module and lecturer evaluations should form part of our overall strategy to improve student success through evidence-based decision making, we are definitely beginning to do that. While some evaluation is still paper based, with Blue we now have the ability to move everything online at any time, and hopefully that will be the case by the end of this calendar year. Blue is definitely the most complete supporting tool I have seen, from its data handling to ability to capture multiple evaluations.

While some evaluation is still paper based, with Blue we now have the ability to move everything online at any time, and hopefully that will be the case by the end of this calendar year.