Year Two: Review of Initial Study

  1. Review of Initial Study

4.1 Aim of Initial Study

The aim of the initial study was to:

  • Trial a combination of methods to be applied to the main study (interviews and surveys)
  • To understand the view point of learners engaging with free courses
  • To select learners to interview from two different platforms seeking to highlight any commonalities that may occur in the selection of the course and then their engagement with it (as it is the same course, but delivered differently)
  • For the selection to highlight any differences in responses and whether they were viewed as either positive or negative to their learning experience
  • To gather responses from interview questions to help set survey questions for beta survey
  • To survey a sample of learners from the same courses on two different platforms seeking to highlight any commonalities that may occur in the selection of the course and then their engagement with it (as it is the same course, but delivered differently)
  • For data from the initial study interviews and beta survey to cascade into methods and question types used in the main study.

4.2 Methodology of Initial Study

For the initial study a combination of quantitative and qualitative methods were piloted to ensure that the methods identified for the main study were suitable and for the research questions to be thoroughly tested.

The initial study comprised of four parts. Firstly, that a small sample of 12 candidates were selected via random sampling for interview (Appendix One) from the same open online course but hosted on two different platforms (one with option for learners social features and one without). The second phases was the analysis of the responses gathered (Appendix Two) that aided the refinement of the survey questions being drafted (Appendix Three). The third phases was the testing of the beta survey on a small population to ensure the design, layout and question content were suitable for use in the main study (Appendix Four). The final phase was the development of the draft survey questions for use in the main study from the analysis of the results to date and the continuation of review of literature (Appendix Five).

There was an adjustment to the sampling of learners for the initial study from the original strategy this was due to a lower response rate than what was initially expected. In hindsight this has aided in the strengthening of the research in preparation for the main study. Originally, the strategy was to select learners from one particular course (Introduction to Ecosystems) that had been hosted on both the FutureLearn and OpenLearn platforms. As response was low, this strategy expanded to also incorporate; Managing My Money, Forensic Psychology: Eye Witness Investigation and Introduction to Cyber Security. From these expansion further data was gathered and upon analysis and reflection is a process that would benefit that of the main study to be addressed later in this report.

  1. Data Findings from the Initial Study

5.1 Analysis and Results of Interviews

All the interviews (Appendix Two) were allocated a number on point of contact; this number remained allocated to them upon point of response and interview. At no point were the numbers reallocated (e.g. 1-6) to allow for further interviews for late replying candidates, the interviewing of substitute candidates if one the finalised six were to cancel, or the additional interviewing of candidates if required. Tables 1 and 2 demonstrate excerpts of the interviews transcribed.

All but one of the participants (OL12) had responded that they had since gone on to enrol on a further open online course or formal study since they enrolled on Moons. All the participants stated that they liked to study courses out of personal interest, with one (FL11) adding the relevancy of study to their profession. Due to the subject of the course selected to contact participants through (Moons) the correlation of the course subject and the relevance to the workplace may have been higher in an alternative course.

All participants stated that they liked to select courses that were intellectually challenging; however all commented that they selected courses that were within subjects that already interested them. Two (FL01 and FL08) commented that they would never pick a course in the ‘fine arts’ as it would be of no interest to them.

One participant (FL08) commented he found the discussions on the FutureLearn course to be ‘chaotic’ and ‘difficult to learn from others’ as the other learners were just commenting on their activity. All of the learners said that they preferred to learn on their own, with two added that they would participate in social activity (FL05 and FL08) however neither stated that it would be as an alternative to loan study. Two participants (Fl04 and OL11) adding that if they join or rejoin a course late due to personal commitments then they didn’t need to comment on the activities as the cohort would have been ahead of them or completed the course.

All but one (FL11) said that they had registered for more than one course at the time, but only one of them (FL11) had a specific time management strategy, the remainder stating that they ‘just make time’ (FL01), and learn ‘when free’ (OL09), and ‘when time allows’ (OL11).  Two (FL04 and FL12) commented that they had completely abandoned a course at least once, yet all stated that they didn’t always study to the course schedule and had intentions of returning to unfinished courses after course closure at a later date to complete them.

All stated that they generally followed the course structure, except two participants (FL12 and OL07) however two stated that they skipped parts that were optional (FL04 and FL05), such as long videos (FL04), if they found it uninteresting (OL01), felt they already knew the topic being covered (FL12 and OL11), or if it was an activity such as an assignment which they felt they wouldn’t benefit from (OL09).

5.2 Analysis and Results of Surveys

From these interviews the beta survey for the initial study was developed (Appendix Three). As stated previously this survey was sent to a random sample of learners from Introduction to Ecosystems, Managing My Money, Forensic Psychology: Eye Witness Investigation and Introduction to Cyber Security (Appendix Four). All of these courses had been hosted on both FutureLearn and OpenLearn, so therefore comparable for analysis.

There were 26 responses in total, which are broken down as follows:

Table One: Initial Study Beta Survey Reponses

Platform Title of Course Responses
FutureLearn Introduction to Ecosystems 6
OpenLearn Introduction to Ecosystems 3
FutureLearn Managing My Money 3
OpenLearn Managing My Money 1
FutureLearn Forensic Psychology: Eye Witness Investigation 3
OpenLearn Forensic Psychology: Eye Witness Investigation 3
FutureLearn Introduction to Cyber Security 7
OpenLearn Introduction to Cyber Security 2

As this data for analysis is too small for individual course comparison for the initial study the data for the courses for each platform were combined. For the main study this practice won’t be used, as the data sample will be much larger it will be possible to compare data between the same individual course hosted on both platforms and the collective data from all courses hosted on both platforms.

Table Two: Initial Study Combination of Survey Responses per Platform

Platform Number of Responses
FutureLearn 19
OpenLearn 9


Though it is not ideal that the data is not more evenly distributed for analysis, it provides a strong lesson to be learned for the development and data collected for the main study.

 5.2.1 Comparison of Demographical Data

For the learners to take part in the survey they all were required to agree to the terms set out in the ethical statement at the start of the survey to proceed. Of the responses from the FutureLearn surveys 54% stated that they were female and 46% stated they were male. In comparison from the OpenLearn responses 63% stated that they were male and 37% female. None preferred not to say.

With regards to age, there were no responses for the age brackets from under 16 to 25-35 from OpenLearners, with the majority (38%) selecting 36-45, then 56-65 and over 65 both at 25% with the remaining 13% selecting 46-55. In comparison 15% of FutureLearners selected 18-24 with the majority selecting 56-65 and over 65 (both 31%) and the remaining selecting 46-55 (15%) and 25-35 (8%).

In response to the location of learners, the majority in both sets of combined responses was the United Kingdom with 77% and 75% for FutureLearn and OpenLearn respectively. For OpenLearners the remaining 25% resided in Europe, for FutureLearn however 15% resided in the United States of America and 8% in Asia. With both platform combined responses English was the learner’s first language (77% FutureLearn, 75% OpenLearn).

When asked what was the learner’s highest educational qualification, for OpenLearn college diploma or certificate (38%) and undergraduate/Bachelors University degree (25%) scored the highest. The remaining responses were evenly split at 13% each for school leaving qualification, postgraduate, and doctorate. For FutureLearn undergraduate and postgraduate degrees scored the highest with 31% each, with 15 % obtaining a doctorate resulting in the FutureLearners being more highly qualified than the OpenLearners. In which 92% of FutureLearners had studied an OOC before in comparison to 63% of OpenLearn.

5.2.2 Analysis of Responses

In the follow up question 67% of FutureLearners had studied 1-4 OOCs in the past two years (with 33% studying 5-9) in comparison to 100% of OpenLearners only studying 1-4 OOCs. The answers in Question Nine demonstrate that a range of courses are selected for study, especially noted by FutureLearners demonstrating the wide interest in the range of subjects available for free learning. However the responses in Question Ten denote that both OpenLearners and FutureLearners prefer to remain loyal to a small number of platforms, or were not aware of other platforms available. Both combined responses demonstrated a preference to study one course at a time (54% FutureLearn, 63% OpenLearn), followed by two courses simultaneously (38% FutureLearn, 25% OpenLearn), and finally four courses collectively (8% FutureLearn, 13% OpenLearn).

When asked what devices the learners preferred to study on, of the responses given FutureLearners preferred not to study on a smartphone or tablet, opting for a desktop or laptop instead. OpenLearners responses were similar, which may be related to the demographic of the learner as noted in Question Three, when upon expansion of learners in the main study may differ. The same in responses is noted in Question Thirteen in regard to registration for formal university study with the majority of both not registered (92% FutureLearn, 88% OpenLearn). However of those enrolled 100% of OpenLearners were registered on part-time distance/online study with an even split of FutureLearners registered in full-time face-to-face and part-time distance/online study.

In Question Fifteen learners were asked as to their motivations for studying OOCs, the majority in both were for personal development (9 selections for FutureLearn, 5 for OpenLearn) and leisure or enjoyment (8 selections for FutureLearn and 4 for OpenLearn), with the majority selecting the courses for the interest in the subject (13 selections for FutureLearn and 8 for OpenLearn) with attraction to the title both ranking second (5 selections for FutureLearn, 3 for OpenLearn).

The survey then asked the average number of weeks spent on studying an OOC, the majority of FutureLearners (54%) selected 5-6 weeks (note, that this is the average duration of a FutureLearn course) with the majority of OpenLearners (38%) selecting 7-8 weeks (note, that OpenLearn courses are available on perpetual cycle and of the courses targeted were all 8 weeks in length). The second majority was held at 3 weeks (FutureLearn 15% and OpenLearn 13%). When asked the length of time within a week dedicated to learning OOCs 50% of OpenLearners selected 3-4 hours in comparison to 23% of FutureLearners for the same time, with 38% of FutureLearners preferring to spend 0-2 hours. In contrast, 0% of OpenLearners selected 0-2 hours with 38% selecting 5-6 hours instead, for which only 23% of FutureLearners selected the same timeframe. When asked if the learners preferred more time to study 62% of FutureLearners said no, with an even split of OpenLearners selecting yes and no (50% each), as in Question Twenty 55% of FutureLearners felt that they the time spent already studying was enough, whilst 50% of OpenLearners selected family commitments and 38% of OpenLearners stating that they felt the understanding of the subject was achieved. Attention will be given in the main study as to the optimum length of time selected as this will have a significant impact on learning design.

Throughout the review of literature in Year One, papers regarding the emphasis of social learning within an OOC were given emphasis. However when asked, both platform learners selected articles, videos, learning activities and quizzes as the most enjoyable parts of the OOCs, with discussions/forums, peer review and peer assessment being selected as the least most enjoyable. Will be beneficial with regards to learning design to collate and analyse these answers again for the main study.

When asked about the pace to which the learners prefer to study at the majority of FutureLearners agreed that they preferred to keep to the pace of the course schedule (7 selections), and to learn at their own pace within the schedule (8 selections) opting for the courses that have end dates to focus their minds (6 selections). OpenLearners demonstrated a fairly even distribution between wishing to study at the pace of the course, but also wishing to learn ahead to finish early and a similar response to the presence or absence of set start and end dates. Such a response may be due to the perpetual cycle of course presentation found on OpenLearn and will benefit from further data collection in the main study.

In response to Question Twenty Four, 75% of OpenLearners and 50% of FutureLearners stated that they watched all videos to the end, with 42% of FutureLearners stating that they do skip videos, but use the transcript to understand what the video covers. This question response would benefit from secondary platform analysis of the step duration of learners when active on video content steps.

The next question asked learners as to the importance of receiving acknowledgments after completing an OOC. The learners were given a range of options to rate, however it is to be noted that a pattern emerged from FutureLearners that these acknowledgements were not important in comparison to the OpenLearners, which may be due to the a range of acknowledgements available from FutureLearn incurring a charge (whereas the alternatives are free on OpenLearn) or to the age and qualification demographic of the learners as denoted earlier in the survey.

In the final two questions, learners were asked what three things keep you interested and where least interesting in the course. For both questions a free text field was set to provide responses. A range of responses were given that have been grouped into themes for use in the main study survey (Appendix Five).

5.2.3 Analysis of Initial Study Survey for Development for Main Study Survey

From the analysis of the initial survey a number of lessons have been learnt and reflected upon for amendments for the main study (Appendix Five).

For Question Nine subject themes have replaced the free text field as a method of categorising the types of OOCs that learners prefer to study. An extra question (Question Ten, Appendix Five) has been added to ascertain the awareness of the learners to the platforms available prior to asking them on which platforms have they accessed OOCs (newly number Question Eleven, Appendix Five).

In Question Twenty the option to select professional commitments was absent in the initial study which has been added for the main study. And finally for questions twenty seven and twenty eight the responses from the initial study have been grouped into themes to aid with the analysis of patterns in responses in the main study when the response numbers increase in scale.

  1. Continuing the Review of Literature

Utilising the feedback from Year One Final Report gave the opportunity for further direction on the continuation of the review of literature with the selection of papers suggested in preparation for the final stages of the initial study leading to the main study.

In counter to the evidence of the initial study, the literature gave strong emphasis to the need for participant engagement (Clouse and Evans (2003), Coppola et al (2002), Marks et al (2005) and Swan 2002), peer influence (Yang et al. 2014) and socially conducive environments (Rosé et al. 2014), with Arbaugh and Benbunan-Fich stating it to be ‘one of the strongest predictors of positive outcomes in online educational environments’ (2007) also giving indication to the need for collaborative activities for positive learning outcomes (Gunawardena and Zittle (1997) and Rovai (2002) in particular the types of learner interaction classified by More (1989) as learner-instructor, learner-learner, and learner-content. It may be possible that the results of the initial study, as it is a small sample, may have produced an anomaly of favour towards the anti-social learner-content which counters this literature, to which further exploration within the main study is required. It may be possible as found with Caspi, Gorksy and Chajut (2003) that the majority of students contributed to a small amount of messages, or that learners had difficulty finding interesting discussion opportunities (Yang et al. 2014) and that in the situation of the small sample may have only highlighted this pattern of activity.

Motivation continues to be a theme within the literature as there are links to be made between motivation and engagement as motivation is multidimensional and multilevel in construct (Boaekaerts, 1997). Though Tai (2008) states that strong motivation is a prerequisite for online learning, it is in the field of formal study, if learners are choosing OOCs for personal developments and leisure learning then the level of motivation may differ to that of an online student studying towards a formal qualification. Whilst a teacher seems to hold a strong presence in face-to-face learning (Roth et al (2007) and Legault et al (2006) and Junco (2012) stating that it was critical to increase the participation of students in the course where the level of interaction or guidance is limited from instructors to reduce the level of dropouts, this doesn’t seem to translate into the findings of the initial study with learners rating alternative features of OOCs before Lead Educators and Facilitators.

Interestingly whilst Rienties et al (2009) found that learners that were highly extrinsically motivated contributed less actively to what Veerman and Veldhuis-Diermanse (2001) consider to be ‘social contributions’ this would aid in the explanation of the ranking of the importance of social engagement within the initial study, it would not aid in the explanation as to the extrinsic acknowledgements that FutureLearners rated as ‘Not Important’.

In researching engagement versus performance Aguiar et al. (2014) noted that the understanding of retention has changed considerably over time, and therefore more complex than initially quantified. This theory is important to the understanding and answering of the research questions as given the heterogeneity of the learners (Lackner et al. (2015), the understanding of retention and completion may vary considerably within the learners community in contrast to that of the academic and the quantifying of performance with (Nicol and Macfarlane-Dick 2006).

Whilst Gibbs and Simpson (2004) have argued that assessment has a positive effect on student’s learning and engagement within traditional teaching environments, this is not a pattern depicted in the results of the initial study. Within the main study a clearer pattern is expected to develop, the requirement for further analysis of this statement would be required through post-survey interviews to ascertain whether it is a strong requirement for learning design of OOCs.

From the results of the main study further research into the links between the studies by Hew (2015) in support of literature on motivation by Reeve (2012) and Skinner et al. (2009) into the categorisation of student engagement as; behavioural, affective and cognitive, will be given.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s