4 Ways to Evaluate Massive Open Online Courses (MOOCs)

In our last post examining MOOCs we identified some reasons why these courses are challenging to evaluate. MOOCs are difficult to assess because there are no established evaluation criteria, low completion rates, varied instructor involvement and accessibility issues. However, that does not mean that MOOCs are impossible to evaluate.

As MOOCs are still evolving, different approaches to assessing their quality and effectiveness continue to adapt. Today’s post will review four methods that can be utilized to evaluate MOOCs. These approaches attempt to currently assess MOOCs prior to the establishment of accepted evaluation criteria. A number of foundations are calling for proposals on how to effectively evaluate MOOCs (i.e. The MOOC Research Initiative and the European MOOCs Stakeholder Summit). It will be interesting to see the proposed approaches and subsequent successful implementations.

1. As an eLearning course:

   Depending on the structure and organization of the MOOC, one may be able to assess it as any other eLearning course. This would only work if the MOOC has set goals, objectives and learning outcomes. The MOOC can then be assessed with one of the established online course evaluation rubrics such as the ‘Quality Online Course Initiative Rubric’ by the Illinois Online Network.

2. By the process perspective:

   Stephen Downes proposes two approaches to evaluating MOOCs, both of which assess these courses as networks. The first method is referred to as the ‘process perspective’ whereby the MOOC is evaluated by the criteria of successful networks (autonomy, diversity, openness, interactivity).

3. Via the outcomes perspective:

   The second method that Downes proposes is the ‘outcomes perspective’. In this approach MOOCs are evaluated as knowing systems; as entities which learn as a whole. MOOC effectiveness is then based on the system’s success and not on individual participant outcomes. Both of Downes’ approaches are very interesting and deserve more thought and exploration.

4. By self-assessment:

   Some MOOCs are experimenting in using self-assessment to evaluate student progress. However, this type of evaluation only examines participant learning and not the quality of the course or the teaching approach. With MOOCs not establishing learning goals and with students creating their own, the evaluation process becomes even more complex. Another factor to consider is that relying solely on self-assessment can lead to inaccurate and unreliable results.

Useful resource: ‘The Fundamentals of Feedback for Online Students’; the Education Database Online Blog.