© ISTOCK
Researchers at the Centre of Sociology at HSE’s Institute of Education conducted an experiment to evaluate the effectiveness of massive open online courses (MOOCs) at three Russian universities. According to their findings, students’ results are independent of course format, that is, whether the course is online or offline.
Over the past five years, MOOCs have traversed a difficult path, from initial hype to recent crisis. Dubbed ‘the phenomenon of the year’ in 2012, massive open online courses were then announced ‘dead’ in 2017 by one of the directors of the influential online learning platform, Udacity. Today, developers of massive open online courses and the various platforms that promote them are in search of new business models and development strategies. One possible approach is to integrate MOOCs into traditional education.
The idea to use massive open online courses in traditional educational programmes is extremely controversial. Can they be considered an alternative to conventional courses? There are advantages and disadvantages to such an approach. ‘Technophiles’ promote MOOCs as a way to make quality content from the best teachers of leading universities accessible to students everywhere and to give students the opportunity to tailor their educational trajectories to suit their needs, while reducing their lecture load. However, those against the integration of MOOCs argue that students studying online cannot master the material as effectively, and that it is difficult to synchronise MOOC schedules with the curriculum. It is also difficult to establish how credits for such courses can be awarded.
Studies examining the effectiveness of MOOCs, which have been conducted all over the world over the past three to four years, indicate that, in general, the use of massive open online courses as an additional tool in teaching has a positive impact on educational outcomes. However, at the same time, students demonstrate a low level of interest in this format, and a low level of satisfaction. One of the problematic areas of previous studies concerning massive open online courses is the inability to establish a causal relationship between the format of learning (completely online, mixed or traditional) and student outcomes. To establish such a relationship, it is necessary to use experimental or quasi-experimental methodology. Several studies have used this methodology to analyze the effectiveness of online formats, but not specifically of MOOCs. Rather, these studies have only examined online components of traditional courses.
In 2017, the Ural Federal University and HSE were commissioned by the Ministry of Education and Science of the Russian Federation to carry out the project, ‘Research on new ways of organising the educational process using open online courses.’ As part of this project, a group of researchers from the Centre of Sociology at HSE’s Institute of Education conducted the first experimental study devoted to the study of MOOCs as an effective alternative to traditional courses.
Second-year students studying mechanical engineering and construction at three regional universities took two mandatory courses offered in different formats. Before the courses began, they were randomly assigned to one of three groups. The first group (101 students) studied according to the traditional format, the second group watched online lectures and also attended face-to-face lectures (mixed format - 100 students), and the third group (124 students) completed the entire course online, interacting with teachers on the online platform.
All participants in the experiment completed a pre-test before the start of the classes, as well as several psychological tests, in particular, to determine their level of procrastination, educational motivation and self-regulation. The groups did not differ significantly in terms of the results of these tests, nor in composition in terms of gender, age, USE results (Unified State Exam) or previous experience of online courses. After completing the course, all students took the same exam, which was treated as a post-test for the experiment.
Three universities took part in the experiment. All three are in the third quartile according to size of student population (10,000 - 25,000). Students at two of the three universities completed the course ‘Engineering mechanics’ and students at the third university completed the course ‘Technology of structural materials.’ Online courses were developed by teachers at Ural Federal University according to the National Learning Standards (Russian: ФГОС) in the selected areas of education. The model of the study implied that teachers of the MOOCs have a better educational background, are more widely published and teach for longer at university than teachers of face-to-face courses.
Three variables were measured in order to compare the three groups: the extent to which the students mastered the material (based on post-test results), the level of satisfaction with the course format, the level of involvement in mastering the material (how much time was spent). Three conclusions can be drawn from the results. Firstly, the course format (online, mixed or traditional) does not affect students’ results, and there is no difference in terms of course format in students' subjective assessment of the knowledge they acquired. Secondly, students who completed the online courses were less satisfied with the course and tend to prefer a traditional or mixed format. Thirdly, the researchers note that some of the MOOCs may be more difficult to pass if only offered online, and that online courses differ in terms of how much time is required to master the material.
Researchers note some limitations of the research related to the complexity of applying these research results to other types of higher education institutions and non-engineering specialisations. Furthermore, it is difficult to monitor students' communication with each other and the influence of a particular teacher or university. But according to leading research fellow at HSE’s Institute of Education, Igor Chirikov, the proposed model for testing the quality of online courses is reproducible and can be used by other universities. In particular, the experiment will make it possible to understand the compatibility of specific MOOCs with courses at a particular university and to assess potential negative impact on students’ educational outcomes. Researchers are open to sharing their methodological experience with interested colleagues.