Creating a New Measurement Instrument for Student Ratings – Fresno State at Bluenotes GLOBAL 2021

Reading Time: Less than 7min.

Synopsis: With student success and engagement a top priority for many academic leaders, student ratings and feedback are a key pathway to improving any course or institution. In a recent session at Explorance’s Bluenotes GLOBAL VX 2021 conference, Kathleen Dyer, PhD, and Dermot Donnelly-Hermosillo, PhD of California State University, Fresno discussed the criticisms that student ratings have received and how creating a new measurement instrument can effectively respond to these concerns. 

Student ratings and feedback are instrumental in improving course delivery, course assessment, and the overall Student Experience. Empowering the student voice brings value to any Higher Education institution.

However, in recent years, there have been many academic news articles criticizing the value of student ratings.  With many Higher Education publications criticizing student ratings for being biased, invalid, and harmful to instructors, some institutions have even thought about eliminating student ratings completely.

In a recent presentation at Explorance’s Bluenotes GLOBAL VX 2021 conference, California State University, Fresno (Fresno State) discussed the validity of this criticism, where it comes from, and how academic institutions can continue to gather student feedback in an effective manner.

Recognizing Criticisms of Student Ratings

This presentation began by recognizing that those studies being carried-out that focus on student ratings are validated with academic and scientific research. Although the research may back up the criticisms, Fresno State still believed there was value in the student voice.

Recognizing that there can be a problem of bias and lack of validity in student ratings, Fresno State focused on the instrument of measurement instead of student ratings overall.

Many of these instruments implemented to measure student feedback are not constructed by scientists, nor are they tested for reliability. Therefore, the bias and validity issues often associated with student ratings is exacerbated by the measurement instrument, causing student ratings to be less effective and valuable.

Fresno State discovered the following 3 common problems found in existing measure instruments:

• Global ratings that allow bias – Many instruments tend to ask, “How good is this class?” This rating is extremely biased as every class has a different value and rating to each individual student, dependent on a host of different factors.
• Judgement of the invisible – Existing instruments may present a statement such as “My instructor cares…”, a statement that is difficult to validate and hard for students to attach a specific value to.
• Impossible judgements – Measurement instruments also prompt students to make impossible judgements such as, “My instructor is knowledgeable.” For the most part, students are taking these courses for the very first time, therefore do not have the knowledge or experience themselves to make this judgement.

The presentation team stressed that the student voice can provide value to both the Student Experience and overall institution. Student feedback can be used to identify valuable insights and make actionable decisions to improve an institution. Feedback allows the opportunity for students to share their opinions anonymously, which could then be viewed by faculty and academic leaders to see what could be improved upon.

Additionally, the student voice does not only improve the Student Experience, but also assists with professional development for instructors. By gathering student feedback, instructors are given the opportunity to hear what they can improve upon in the classroom, ultimately improving course delivery and the academic experience in general.

Creating a New Instrument

Having come to an understanding that these biases and problems are created by the measuring instrument, and not overall problems inherent in student course evaluations or ratings, Fresno State created a task force set out to create a new instrument.

With a team that included survey construction experts and content experts, the task force’s goal was to eliminate all problems with existing instruments and create a more valuable surveying process, while also making the instrument meaningful to faculty.

The task force created an instrument that demonstrated reliability, validity, and flexibility; and is measured by instructional design, delivery, and assessment. The content of the instrument was expertly peer-reviewed.

Noting the existing problematic strategies regarding student ratings, the task force set up the instrument in line with the following concepts:

• Directly observable behaviors of faculty – Allowing students to directly validate what their instructors said or did by asking “My teacher did…”.
• Student self-report of their own understanding – Instead of asking if the instructor was knowledgeable of the subject, students are asked to self-report on what they were able to understand and learn in the course by stating, “I understood…”.
• Student reports are evidence-based – If students are asked to report on content outside of the typical course setting, they are asked to do so using evidence such as peer-reviewed scientific publications.

In addition, this new instrument was especially meaningful to faculty as it was faculty generated – it was created with representatives from every college on campus at Fresno State. The content within the instrument is also clearer and students know exactly what they are being asked, with instructors understanding where they need to improve upon, based on the scores provided by the student feedback.

Managing the Student Experience

This presentation made the point that although the criticisms that student ratings have received are fair and proven, the issues regarding student ratings are largely due to the measuring instrument.

By creating a new measuring instrument, Fresno State was able to eliminate biases and other problems that were caused by existing instruments. With re-evaluated questions and a new overall survey construction, Fresno State was able to reap the full benefits of student feedback, improving both the experience of its students and faculty.

Student feedback provides value for students, instructors, and institutions. It is important for academic leaders to keep student experience as a top priority – as the landscape continuously becomes more competitive and ever-changing.

Learn how to better manage the student experience from application to graduation