Following on from my last blog post Design and Data in which I introduced the latest research I have been conducting for my doctorate, in this post I’ll be unpacking how I will be reviewing the learning performance data from our course dashboards on FutureLearn.
For each presentation of an Open Online Course (OOC) on FutureLearn there is an associated performance dashboard. Within the dashboard are the overview statistics giving a snapshot review of the performance of each presentation and a number of datasets for export (comments, enrolments, question response, and step activity). In analysing the data from a combination of these datasets and the overview dashboard in conjunction with the learning design it is possible to determine the success of the learning design in correlation to the learning performance data. For example, to review from the learning performance data whether the learning activities designed are engaging learners to reaching their learning outcomes.
As the OOCs selected have presented on more than one occasion since their production in 2014/15 it is possible to determine whether the engagement or disengagement of the learners in particular steps repeats in subsequent presentations and therefore associated with the learning design, or is isolated to one presentation whereby mitigating factors may be the cause (e.g. anniversary of an associated event).
The dashboards available from FutureLearn represent the learners’ progress through the course and are categorised into the following Course Measures:
- Joiners – Number of learners who have enrolled on the course
- Leavers – Number of learners who have elected to unenrol from the course
- Learners – Number of learners who view at least one step in any week at any time within the course
- Active Learners – Number of learners with user interaction with at least one step in the course (marking steps as complete, submitting content for review, or attempting a quiz or test)
- Returning Learners – Learners who have interacted with at least one step in two or more weeks of the course
- Social Learners – Learners who have commented on at least one step
- Fully Participating Learners – Learners who have interacted with at least 50% of the steps and attempted the tests
The dashboards displayed in these progress reports are taken directly from FutureLearn, it is important to note that FutureLearn calculate their percentages for Active Learners, Returning Learners, Social Learners and Fully Participating Learners from the number of learners who have returned to the course (Learners) and not from the number who have enrolled (Joiners).
When edX reviewed their Harvardx and MITx courses from their first presentation year Ho et al (2015) stated that “Participation initially declines in repeated courses, then stabilises” based on the review of data from 11 of their courses. Ho et al found that participation (defined in the paper as enrolment) declined on average by 43% from first to second presentation. Five courses had a third presentation, whereby enrolment remained unchanged from second to third presentation, thus stabilising. This method of declining engagement will be used when reviewing OU OOCs.
It is important to note that engagement is not tangible and measured primarily through enrolment or course completion (frequently seen as successful engagement), learners may engage with elements of the course, but not reach the requirement of a Fully Participating Learner. Therefore it is important to acknowledge all the course measures identified in the dashboard as part of the review process. To my knowledge (readers please correct me if I am wrong), that this will be the first literature to map across multiple course measurements to ascertain engagement and more importantly disengagement.
In my blog posts next week I’ll be discussing the engagement survey that I created for the purpose of this research and also the hypothesis that I have been forming during this process on how I think learners engage with MOOCs and therefore how and why they disengage.
Until then…I’m doctor in waiting
Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, fall 2012-summer 2013. Ho, AD, Reich, J., Nesterko, S., Seaton, DT, Mullaney, T., Waldo, J., & Chuang, I.(2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).