Category Archives: Year Two

Year Two: Methodology of Main Study

Though there are a range of methodologies that can be used in research such as interviews, focus groups, and observations, due to the volume of learners undertaking open courses the most suitable methodology would be an online survey as a survey will be able to collate both quantitative and qualitative large-scale data for analysis. This became evident in the execution of the initial study due to the length of time it took to contact learners, arrange interview times, carry out interviews, and transcribe notes. There are a number of attractions in using a survey; one-shot data gathering, wide population representation, ascertains correlations, accurate data capture, and statistical data processing (Morrison, 1993: 38-40).

The self-completion survey is to be hosted online as the sample population of learners enrol and study open courses online, so the required demographic is suitably targeted. In a change to the original strategy for the main study, learners who have completed start of course surveys for courses that are hosted on both FutureLearn and OpenLearn will be targeted. Then the response data for individual courses can be analysed for comparison, and also collated for collective comparison of open online courses hosted on OpenLearn and FutureLearn. As the platform functionality and features differ this will aid in developing an understanding as to whether the presence or absence of such features impact on engagement and learning design.

The survey includes a combination of nominal data (for comparison with ‘traditional’ MOOC data (Jordan, 2014) to ascertain whether this open course community of informal learners is different), and scaled questions to establish attitudes of participants towards course engagement and learning design. Capturing large scale data through an online survey will aid to determine factual information; preferences, attitudes, behaviour, experiences and beliefs (Weisberg et al, 1996).

The design of the survey has taken into consideration Hoinville and Jowell’s (1978) three prerequisites of survey design; purpose of inquiry, population specification, and resources available. The survey questions strongly address the research questions of engagement, disengagement and learning design. Three populations of learner strategically aligned to the JIFL journey have been identified (address in the Participants and Samples Chosen for Main Study section below), and the survey is to be hosted in the three said locations online within the research timescales. Concern for participants (Sapsford, 1999: 34-40) has been taken into consideration through ensuring anonymity of participants and the fourteen stage process identified by Cohen et al (2009: 209) was followed.

The main study will also include the development of interview questions to be conducted on an individual 1:1 basis or as part of a focus group. For this purpose, and to confirm, clarify, and question any commonalities, trends and anomalies the final survey question allows participants to submit their personal details for further contact.

The evaluation report data from the first ten MOOCs presented by The OU on FutureLearn will also be reviewed in conjunction with the demographic data collected by the survey as historical documentary research to ascertain whether non-MOOC (therefore JIFL prospective) learners are being successfully targeted.

 

  1. Implication of Research and Dissemination Strategy

The implications of this research have strong implications upon professional practice as Senior Producer: MOOCs for The Open University. The findings from this research have drawn the attention and interest of the Director, Open Media who commissions open online courses for The Open University, Lead Educators and academic authors who create the content for open online courses and departments with the University who edit and construct the courses for publish and hosting. Interest has also been received from parties (academic and non-academic) external to the University, as findings may have implications on the development of their open online course strategies also.

At present, the blog ‘Doctor in Waiting’ is being utilised as a research journal (Burgess, 1984) documenting the process of the doctorate, the research, data findings, analysis and review of literature. From this there have been invitations to co-write academic papers and book chapters on the research and the implications on open online course development. This would be in addition to the development of thesis for submission.

Key to the dissemination of research is the development of a strategy to address the ways to disseminate for; awareness, understanding and action. From this aids the understanding of want is to be disseminated, to whom would the dissemination be to (external, internal and connected audiences), the timely manner as to when to disseminate (after main study, after thesis), the channels through which to disseminate (papers, conferences, blog, reports, workshops, book chapters, etc.), who may assist in the dissemination (supervisors, co-authors, publishers, event organisers) and how to develop this dissemination strategy into an action plan.

Year Two: Review of Initial Study

  1. Review of Initial Study

4.1 Aim of Initial Study

The aim of the initial study was to:

  • Trial a combination of methods to be applied to the main study (interviews and surveys)
  • To understand the view point of learners engaging with free courses
  • To select learners to interview from two different platforms seeking to highlight any commonalities that may occur in the selection of the course and then their engagement with it (as it is the same course, but delivered differently)
  • For the selection to highlight any differences in responses and whether they were viewed as either positive or negative to their learning experience
  • To gather responses from interview questions to help set survey questions for beta survey
  • To survey a sample of learners from the same courses on two different platforms seeking to highlight any commonalities that may occur in the selection of the course and then their engagement with it (as it is the same course, but delivered differently)
  • For data from the initial study interviews and beta survey to cascade into methods and question types used in the main study.

4.2 Methodology of Initial Study

For the initial study a combination of quantitative and qualitative methods were piloted to ensure that the methods identified for the main study were suitable and for the research questions to be thoroughly tested.

The initial study comprised of four parts. Firstly, that a small sample of 12 candidates were selected via random sampling for interview (Appendix One) from the same open online course but hosted on two different platforms (one with option for learners social features and one without). The second phases was the analysis of the responses gathered (Appendix Two) that aided the refinement of the survey questions being drafted (Appendix Three). The third phases was the testing of the beta survey on a small population to ensure the design, layout and question content were suitable for use in the main study (Appendix Four). The final phase was the development of the draft survey questions for use in the main study from the analysis of the results to date and the continuation of review of literature (Appendix Five).

There was an adjustment to the sampling of learners for the initial study from the original strategy this was due to a lower response rate than what was initially expected. In hindsight this has aided in the strengthening of the research in preparation for the main study. Originally, the strategy was to select learners from one particular course (Introduction to Ecosystems) that had been hosted on both the FutureLearn and OpenLearn platforms. As response was low, this strategy expanded to also incorporate; Managing My Money, Forensic Psychology: Eye Witness Investigation and Introduction to Cyber Security. From these expansion further data was gathered and upon analysis and reflection is a process that would benefit that of the main study to be addressed later in this report.

  1. Data Findings from the Initial Study

5.1 Analysis and Results of Interviews

All the interviews (Appendix Two) were allocated a number on point of contact; this number remained allocated to them upon point of response and interview. At no point were the numbers reallocated (e.g. 1-6) to allow for further interviews for late replying candidates, the interviewing of substitute candidates if one the finalised six were to cancel, or the additional interviewing of candidates if required. Tables 1 and 2 demonstrate excerpts of the interviews transcribed.

All but one of the participants (OL12) had responded that they had since gone on to enrol on a further open online course or formal study since they enrolled on Moons. All the participants stated that they liked to study courses out of personal interest, with one (FL11) adding the relevancy of study to their profession. Due to the subject of the course selected to contact participants through (Moons) the correlation of the course subject and the relevance to the workplace may have been higher in an alternative course.

All participants stated that they liked to select courses that were intellectually challenging; however all commented that they selected courses that were within subjects that already interested them. Two (FL01 and FL08) commented that they would never pick a course in the ‘fine arts’ as it would be of no interest to them.

One participant (FL08) commented he found the discussions on the FutureLearn course to be ‘chaotic’ and ‘difficult to learn from others’ as the other learners were just commenting on their activity. All of the learners said that they preferred to learn on their own, with two added that they would participate in social activity (FL05 and FL08) however neither stated that it would be as an alternative to loan study. Two participants (Fl04 and OL11) adding that if they join or rejoin a course late due to personal commitments then they didn’t need to comment on the activities as the cohort would have been ahead of them or completed the course.

All but one (FL11) said that they had registered for more than one course at the time, but only one of them (FL11) had a specific time management strategy, the remainder stating that they ‘just make time’ (FL01), and learn ‘when free’ (OL09), and ‘when time allows’ (OL11).  Two (FL04 and FL12) commented that they had completely abandoned a course at least once, yet all stated that they didn’t always study to the course schedule and had intentions of returning to unfinished courses after course closure at a later date to complete them.

All stated that they generally followed the course structure, except two participants (FL12 and OL07) however two stated that they skipped parts that were optional (FL04 and FL05), such as long videos (FL04), if they found it uninteresting (OL01), felt they already knew the topic being covered (FL12 and OL11), or if it was an activity such as an assignment which they felt they wouldn’t benefit from (OL09).

5.2 Analysis and Results of Surveys

From these interviews the beta survey for the initial study was developed (Appendix Three). As stated previously this survey was sent to a random sample of learners from Introduction to Ecosystems, Managing My Money, Forensic Psychology: Eye Witness Investigation and Introduction to Cyber Security (Appendix Four). All of these courses had been hosted on both FutureLearn and OpenLearn, so therefore comparable for analysis.

There were 26 responses in total, which are broken down as follows:

Table One: Initial Study Beta Survey Reponses

Platform Title of Course Responses
FutureLearn Introduction to Ecosystems 6
OpenLearn Introduction to Ecosystems 3
FutureLearn Managing My Money 3
OpenLearn Managing My Money 1
FutureLearn Forensic Psychology: Eye Witness Investigation 3
OpenLearn Forensic Psychology: Eye Witness Investigation 3
FutureLearn Introduction to Cyber Security 7
OpenLearn Introduction to Cyber Security 2

As this data for analysis is too small for individual course comparison for the initial study the data for the courses for each platform were combined. For the main study this practice won’t be used, as the data sample will be much larger it will be possible to compare data between the same individual course hosted on both platforms and the collective data from all courses hosted on both platforms.

Table Two: Initial Study Combination of Survey Responses per Platform

Platform Number of Responses
FutureLearn 19
OpenLearn 9

 

Though it is not ideal that the data is not more evenly distributed for analysis, it provides a strong lesson to be learned for the development and data collected for the main study.

 5.2.1 Comparison of Demographical Data

For the learners to take part in the survey they all were required to agree to the terms set out in the ethical statement at the start of the survey to proceed. Of the responses from the FutureLearn surveys 54% stated that they were female and 46% stated they were male. In comparison from the OpenLearn responses 63% stated that they were male and 37% female. None preferred not to say.

With regards to age, there were no responses for the age brackets from under 16 to 25-35 from OpenLearners, with the majority (38%) selecting 36-45, then 56-65 and over 65 both at 25% with the remaining 13% selecting 46-55. In comparison 15% of FutureLearners selected 18-24 with the majority selecting 56-65 and over 65 (both 31%) and the remaining selecting 46-55 (15%) and 25-35 (8%).

In response to the location of learners, the majority in both sets of combined responses was the United Kingdom with 77% and 75% for FutureLearn and OpenLearn respectively. For OpenLearners the remaining 25% resided in Europe, for FutureLearn however 15% resided in the United States of America and 8% in Asia. With both platform combined responses English was the learner’s first language (77% FutureLearn, 75% OpenLearn).

When asked what was the learner’s highest educational qualification, for OpenLearn college diploma or certificate (38%) and undergraduate/Bachelors University degree (25%) scored the highest. The remaining responses were evenly split at 13% each for school leaving qualification, postgraduate, and doctorate. For FutureLearn undergraduate and postgraduate degrees scored the highest with 31% each, with 15 % obtaining a doctorate resulting in the FutureLearners being more highly qualified than the OpenLearners. In which 92% of FutureLearners had studied an OOC before in comparison to 63% of OpenLearn.

5.2.2 Analysis of Responses

In the follow up question 67% of FutureLearners had studied 1-4 OOCs in the past two years (with 33% studying 5-9) in comparison to 100% of OpenLearners only studying 1-4 OOCs. The answers in Question Nine demonstrate that a range of courses are selected for study, especially noted by FutureLearners demonstrating the wide interest in the range of subjects available for free learning. However the responses in Question Ten denote that both OpenLearners and FutureLearners prefer to remain loyal to a small number of platforms, or were not aware of other platforms available. Both combined responses demonstrated a preference to study one course at a time (54% FutureLearn, 63% OpenLearn), followed by two courses simultaneously (38% FutureLearn, 25% OpenLearn), and finally four courses collectively (8% FutureLearn, 13% OpenLearn).

When asked what devices the learners preferred to study on, of the responses given FutureLearners preferred not to study on a smartphone or tablet, opting for a desktop or laptop instead. OpenLearners responses were similar, which may be related to the demographic of the learner as noted in Question Three, when upon expansion of learners in the main study may differ. The same in responses is noted in Question Thirteen in regard to registration for formal university study with the majority of both not registered (92% FutureLearn, 88% OpenLearn). However of those enrolled 100% of OpenLearners were registered on part-time distance/online study with an even split of FutureLearners registered in full-time face-to-face and part-time distance/online study.

In Question Fifteen learners were asked as to their motivations for studying OOCs, the majority in both were for personal development (9 selections for FutureLearn, 5 for OpenLearn) and leisure or enjoyment (8 selections for FutureLearn and 4 for OpenLearn), with the majority selecting the courses for the interest in the subject (13 selections for FutureLearn and 8 for OpenLearn) with attraction to the title both ranking second (5 selections for FutureLearn, 3 for OpenLearn).

The survey then asked the average number of weeks spent on studying an OOC, the majority of FutureLearners (54%) selected 5-6 weeks (note, that this is the average duration of a FutureLearn course) with the majority of OpenLearners (38%) selecting 7-8 weeks (note, that OpenLearn courses are available on perpetual cycle and of the courses targeted were all 8 weeks in length). The second majority was held at 3 weeks (FutureLearn 15% and OpenLearn 13%). When asked the length of time within a week dedicated to learning OOCs 50% of OpenLearners selected 3-4 hours in comparison to 23% of FutureLearners for the same time, with 38% of FutureLearners preferring to spend 0-2 hours. In contrast, 0% of OpenLearners selected 0-2 hours with 38% selecting 5-6 hours instead, for which only 23% of FutureLearners selected the same timeframe. When asked if the learners preferred more time to study 62% of FutureLearners said no, with an even split of OpenLearners selecting yes and no (50% each), as in Question Twenty 55% of FutureLearners felt that they the time spent already studying was enough, whilst 50% of OpenLearners selected family commitments and 38% of OpenLearners stating that they felt the understanding of the subject was achieved. Attention will be given in the main study as to the optimum length of time selected as this will have a significant impact on learning design.

Throughout the review of literature in Year One, papers regarding the emphasis of social learning within an OOC were given emphasis. However when asked, both platform learners selected articles, videos, learning activities and quizzes as the most enjoyable parts of the OOCs, with discussions/forums, peer review and peer assessment being selected as the least most enjoyable. Will be beneficial with regards to learning design to collate and analyse these answers again for the main study.

When asked about the pace to which the learners prefer to study at the majority of FutureLearners agreed that they preferred to keep to the pace of the course schedule (7 selections), and to learn at their own pace within the schedule (8 selections) opting for the courses that have end dates to focus their minds (6 selections). OpenLearners demonstrated a fairly even distribution between wishing to study at the pace of the course, but also wishing to learn ahead to finish early and a similar response to the presence or absence of set start and end dates. Such a response may be due to the perpetual cycle of course presentation found on OpenLearn and will benefit from further data collection in the main study.

In response to Question Twenty Four, 75% of OpenLearners and 50% of FutureLearners stated that they watched all videos to the end, with 42% of FutureLearners stating that they do skip videos, but use the transcript to understand what the video covers. This question response would benefit from secondary platform analysis of the step duration of learners when active on video content steps.

The next question asked learners as to the importance of receiving acknowledgments after completing an OOC. The learners were given a range of options to rate, however it is to be noted that a pattern emerged from FutureLearners that these acknowledgements were not important in comparison to the OpenLearners, which may be due to the a range of acknowledgements available from FutureLearn incurring a charge (whereas the alternatives are free on OpenLearn) or to the age and qualification demographic of the learners as denoted earlier in the survey.

In the final two questions, learners were asked what three things keep you interested and where least interesting in the course. For both questions a free text field was set to provide responses. A range of responses were given that have been grouped into themes for use in the main study survey (Appendix Five).

5.2.3 Analysis of Initial Study Survey for Development for Main Study Survey

From the analysis of the initial survey a number of lessons have been learnt and reflected upon for amendments for the main study (Appendix Five).

For Question Nine subject themes have replaced the free text field as a method of categorising the types of OOCs that learners prefer to study. An extra question (Question Ten, Appendix Five) has been added to ascertain the awareness of the learners to the platforms available prior to asking them on which platforms have they accessed OOCs (newly number Question Eleven, Appendix Five).

In Question Twenty the option to select professional commitments was absent in the initial study which has been added for the main study. And finally for questions twenty seven and twenty eight the responses from the initial study have been grouped into themes to aid with the analysis of patterns in responses in the main study when the response numbers increase in scale.

  1. Continuing the Review of Literature

Utilising the feedback from Year One Final Report gave the opportunity for further direction on the continuation of the review of literature with the selection of papers suggested in preparation for the final stages of the initial study leading to the main study.

In counter to the evidence of the initial study, the literature gave strong emphasis to the need for participant engagement (Clouse and Evans (2003), Coppola et al (2002), Marks et al (2005) and Swan 2002), peer influence (Yang et al. 2014) and socially conducive environments (Rosé et al. 2014), with Arbaugh and Benbunan-Fich stating it to be ‘one of the strongest predictors of positive outcomes in online educational environments’ (2007) also giving indication to the need for collaborative activities for positive learning outcomes (Gunawardena and Zittle (1997) and Rovai (2002) in particular the types of learner interaction classified by More (1989) as learner-instructor, learner-learner, and learner-content. It may be possible that the results of the initial study, as it is a small sample, may have produced an anomaly of favour towards the anti-social learner-content which counters this literature, to which further exploration within the main study is required. It may be possible as found with Caspi, Gorksy and Chajut (2003) that the majority of students contributed to a small amount of messages, or that learners had difficulty finding interesting discussion opportunities (Yang et al. 2014) and that in the situation of the small sample may have only highlighted this pattern of activity.

Motivation continues to be a theme within the literature as there are links to be made between motivation and engagement as motivation is multidimensional and multilevel in construct (Boaekaerts, 1997). Though Tai (2008) states that strong motivation is a prerequisite for online learning, it is in the field of formal study, if learners are choosing OOCs for personal developments and leisure learning then the level of motivation may differ to that of an online student studying towards a formal qualification. Whilst a teacher seems to hold a strong presence in face-to-face learning (Roth et al (2007) and Legault et al (2006) and Junco (2012) stating that it was critical to increase the participation of students in the course where the level of interaction or guidance is limited from instructors to reduce the level of dropouts, this doesn’t seem to translate into the findings of the initial study with learners rating alternative features of OOCs before Lead Educators and Facilitators.

Interestingly whilst Rienties et al (2009) found that learners that were highly extrinsically motivated contributed less actively to what Veerman and Veldhuis-Diermanse (2001) consider to be ‘social contributions’ this would aid in the explanation of the ranking of the importance of social engagement within the initial study, it would not aid in the explanation as to the extrinsic acknowledgements that FutureLearners rated as ‘Not Important’.

In researching engagement versus performance Aguiar et al. (2014) noted that the understanding of retention has changed considerably over time, and therefore more complex than initially quantified. This theory is important to the understanding and answering of the research questions as given the heterogeneity of the learners (Lackner et al. (2015), the understanding of retention and completion may vary considerably within the learners community in contrast to that of the academic and the quantifying of performance with (Nicol and Macfarlane-Dick 2006).

Whilst Gibbs and Simpson (2004) have argued that assessment has a positive effect on student’s learning and engagement within traditional teaching environments, this is not a pattern depicted in the results of the initial study. Within the main study a clearer pattern is expected to develop, the requirement for further analysis of this statement would be required through post-survey interviews to ascertain whether it is a strong requirement for learning design of OOCs.

From the results of the main study further research into the links between the studies by Hew (2015) in support of literature on motivation by Reeve (2012) and Skinner et al. (2009) into the categorisation of student engagement as; behavioural, affective and cognitive, will be given.

Progressing Forward into the Second Year

After the excitement (and the jumping around) of the confirmation of successfully completing my first year of my doctorate subsided, I went back to the grindstone to draft my next progress report on my studies since submitting my final year one report. Due to the length I’ve split this across several posts…

  1. Introduction

This research is a continuation into the investigation as to the attraction of open online courses and what elements of learning design engages learners through to course completion. The purpose of this research is to identify what elements of open online courses that learners engage and disengage with, and how these research findings can influence the learning design of open online courses (OOCs).

In the second academic year of the doctorate the focus will be on the review of the initial study, the development and action of the main study, and finally the analysis of the results leading to the dissemination of research findings and development toward the construction of the thesis for submission.            

  1. Feedback from Year One Final Report and Action Plan

The feedback from the Year One Final Report was insightful, and an opportunity to receive constructive criticism on my research which is crucial to ensure that what is research and then reported upon stands against academic rigour.

The feedback gave the opportunity to develop an action plan not only for the second year of the doctorate in the development of the progress reports and research but also to the methods used in writing the content that will be presented in these reports and ultimately the thesis.

Through follow on discussions with Dr Ferguson themed on the feedback given within the Year One Final Report, further clarity has been given towards the elements of the main study, the platforms and courses being brought into focus, and a stronger linking to the literature and research questions, which will be explored further in the relevant sections of this report.

Key features of this feedback have been highlighted and then condensed to produce a guideline for the forthcoming year. This should aid in the methodology used in the execution of the main study and the analysis of the results received from it and the development of writing style in reporting the findings in preparation for the thesis.

Further literature was recommended for reading, which has been undertaken in preparation for this report and for the foundations of the main study.

  1. Reviewing the Title of the Thesis and Research Questions

3.1 Reviewing the Title of the Thesis

As part of the preparation for this Progress Report and utilising the feedback from the Year One Final Report and subsequent discussions with Dr Ferguson as part of the activities within Doctorate Day School, is the review of the title of the thesis and the research questions within.

As my progress reports developed throughout Year One it became clear that the initial title I had submitted as part of the proposal ‘Effective Processes and Structures in Online Learning through a Social Paradigm’ was no longer suitable.

Throughout the development of the progress reports within year one discussions with Professor Whitelock and Dr Ferguson were held as to the relevancy of not only the title of the thesis but also the research questions within.

From these discussions and continual review of the literature the title of the thesis was formed to reflect the gaps with the research currently published. The literature reviewed to date demonstrated that although research has been undertaken with regards to open courses in the form of MOOCs (Massive Open Online Courses) and engagement with formal courses, little research had been undertaken with respect to the engagement levels of learners that are a stronger fit to the JIFL (Journeys from Informal to Formal Learning) journey and therefore are of more interest to universities that the present MOOC demographic. Though MOOCs are receiving much interest at present, the wider concept of open online courses will be addressed in the title, research questions and subsequent research, not just concentrating on MOOCs as learners engage in a range of open online courses.

From this the thesis title was formed to be:

“Engagement of Informal Learners Undertaking Open Online Courses and the Impact of Design”

What this review and feedback to date has demonstrated is that there is a strong requirement for research into the engagement of informal learners that are suitable for a JIFL journey into either undergraduate or postgraduate formal education, as it is the strategy of many universities to convert informal learners through open courses into formal students. Even if an informal learner has no desire to become a formal student it is still within the educational provider’s interests to ensure that the learner feels that they are capable to undertake and complete informal learning.

3.2 Reviewing the Research Questions

One of the themes emerging from the literature reviewed to date and from the feedback given is that there has been an expression of academic interest in the retention and completion figures of a range of MOOCs and OOCs, however very little literature has been dedicated to the engagement of the learner with the content, how they were initially attracted to the course (much emphasis is placed on ‘free’ rather than the content, the Lead Educator, the university facilitating the course, how it is delivered, how it can be studied etc.).

Understanding the attraction to engage and then the elements that maintain engagement to completion need addressing as there is a distinct gap in the literature regarding this, and would be of benefit to academics and learning design teams in the creation of open online courses. Hence the research questions for the main study and thesis are:

  1. Why do people engage, and remain engaged in free open online courses?
  2. What elements of the design of the free open online courses increase or maintain learner engagement?

Addressing these questions should guide the understanding of these issues. It is important to note that this title and research questions have a wider application beyond that of this doctorate as its findings and recommendations may be able to translate through to formal offering to aid student engagement to qualification completion.