Using Learning Analytic Tools to Enhance Quality of Hands-on-Activities in Online Technology Courses

In the last few years, there has been a proliferation of online Information Technology (IT) and management related programs in many Thai universities. Big data analytic has turned into a new science in refining businesses strategies, but there is a lack of strategy when it comes to analyzing the data captured in various Learning Management Systems (LMSs) used for hosting the course contents. This case study provides an insight into how to use of Moodle “Reports” as a Learning Analytic (LA) tool for analyzing the use of Hands-on-Activities (HOAs), lab activities and assignments provided in technology related courses. This small study is also used to propose a new framework for using LA to improve quality assurance (QA) in such programs.


Introduction
Big data analytic has become a prominent part of new business model. It has been used for crafting marketing strategies and tactics. With increasing use of social media, mobile, cloud and ecommerce the amount of online data being generated is enormous. The internet or the cloud has become a new target for data analytic but the data about students enrolled in online courses offered by colleges and universities in Thailand and elsewhere in Asia is not fully harvested. Online technology related programs are lagging behind in using the Learning Analytic (LA) tools available in various Learning Management Systems (LMSs). As a result, the quality of courses and learning outcomes in technology related courses continues to suffer. In this small case study, students are provided a series of Hands-on-Activities (HoAs) and asked to perform important tasks on networked devices. This exploration is aimed at integrating HoAs in online courses and how they can be used to improve students' experiences. This is also a small effort in providing new insight into using Moodle "Reports" as LA tool for enhancing quality of online technology programs and courses. The captured data in the logs is used to track usage of HoAs and other tasks to learn about the strengths and weaknesses of an online technology courses. Figure-1 given below summarizes the 3-step Measure-Learn-Take Action (MLT) process for using LA to create rigorous online HOAs or laboratory exercises.

Literature Review
Before the advent of 21st century big data analytic was known by several other names-the most prominent was called data mining. Special programs were written in Java and other languages and many algorithms were created for the purpose of data mining, especially for business purposes. Since the beginning of this new century online learning is being used as alternate to traditional face-to-face (F2F) learning. This mode requires a new framework in which reports, logs, data and analytic tools provided with an LMS are being extensively used for a better understanding of learning processes and outcomes in a Virtual Learning Environment (VLE). It is a soft technology that involves the orchestration of the human process of engaging with the analytics as part of the larger teaching and learning process [1]. In the last decade various LMSs have become the main drivers of online learning. As early as 2008 Wexler et al. pointed various advantages and disadvantages of using LMS for teaching technology courses. Many experts now believe that online ecosystems should be able to break the technological constraints of existing LMSs and achieve an effective improvement in learning processes [2]. Owlia and Aspinwall presented a conceptual framework that covers six criteria to measures quality in online courses. They are tangibles, competence, attitude, content, delivery and reliability. It has also been found that participative, peer-based approaches affect teaching and learning and development of dynamic curricula in many ways [3]. Angela van Barneveld et al in 2012 developed a conceptual model of LA to demonstrate not only the connections between business and academia but also within higher education. The field of LA is relatively new; however, in its short duration it has produced numerous terms and conceptualizations [4]. Current LA related proposals pivot around five specific types of actions that are listed below: 1) A new framework architecture that supports learning through service-based ecosystems 2) Learning analytics for educational decision making 3) Adaptive knowledge systems 4) Gamification of learning processes 5) Semantic portfolios to collect evidence of learning In the last 5 years many new technologies have come to play a prominent role in online learning. Siemens and many other researchers have concluded that LA is in its early stages of research and implementation in higher education. Several organizations (Society for Learning Analytics Research and the International Educational Data Mining Society) have been formed to foster a research community around the role of data analytics in education. Many powerful forensic tools, technologies, such as, sniffers, reports, and logs are extensively used in analyzing various hacking and cyber-attacks. LA is now considered as a new research field that will lead to the development of new tools and technologies and new models and their use in enhancing QA in educational settings [5] [6]. The challenges facing LA as a field are being gradually resolved through new and more efficient data capture techniques within various systems, including LMSs which are commonly used for hosting eLearning course content.

Methodology
For the purposes of this study Moodle, an open source LMS, is used. Moodle is a widely used LMS and provides a variety of tools and modules to manage and monitor the various teaching and learning activities in a VLE. The latest version of Moodle 3.2.3 is freely available for download. Moodle is built by the Moodle project which is led and coordinated by Moodle HQ, an Australian company of 30 developers which is financially supported by a network of over 60 Moodle Partner service companies worldwide.

Objectives
There are three main objectives of this study. The first objective is to highlight the significance of LA in the era of big data. Second objective is to show the usefulness of a tool called "Reports" as a LA tool provided in the Moodle. The third objective is show LA can be used to assess the quality of various HoAs, assignments and other activities in an online technology courses.

Research Questions (RQs)
This brief study has been designed to find answers to the following two important questions: • RQ1-Can "Reports" provided in the LMS be treated as a LA tool to dissect the use of HoAs in a technology course? • RQ2-Can LA be used in analyzing and synthesizing the results to enhance the quality of online technology courses?

Sample
It is quite disheartening to note that enrollment in many online programs offered in English language in Thai universities remains very low. Majority of students still prefer F2F classroom based teaching and learning and commute to various colleges and university campuses on daily basis. This study is based on a small purposive sample consisting of 9 students enrolled an online M.S. technology program. Although the number of students is limited but the main focus here is to show how to harvest the data provided in 5 logs to understand the content and quality of an online technology course. This study was conducted in 2017.

Learning Management System (LMS)
Rapid expansion of internet and 3G/4G wireless technologies has given rise to various types of LMSs and social networks for online learning [7] [10] [11]. Moodle is one of the open source LMS widely used by schools, colleges and universities around the world to conduct the online academic and training programs. The interface or the arrangement of contents in a VLE may vary from LMS to LMS. For example, Blackboard, Sakai or Desire2Learn (D2L) will have their own interface as well management tools, modules and dashboards. Most online courses consist of Learning Objects (LOs) or modular resources, usually digital and web-based, that can be used and re-used to support learning activities. One of the main advantages of online courses is that provides the opportunity to manage various LOs which are compliant with international standards, such as, SCORM-2004, AICC, IMS-LTI and Tincan. The same LOs can be adapted to teach different courses. Using SCORM compliant LOs can help in fine-tuning online course materials for specific purposes or learning outcomes. A typical display of LOs or learning resources in a VLE is shown in the Fig. 2.

Reports
For managing student's activities in a VLE the Moodle provides a series of tools. It is shown in Fig.3. One of the most important tools for gauging engagement in an online course is "Reports" which consists of 5 distinct components-Logs, Live logs, Activity report, Course participation and Statistics. For this study, the data captured in the following 5 logs listed in Fig.3 was analyzed. The five logs provided in the Moodle are-Logs, Live logs, Activity report, Course participation and Statistics.

Analysis and Discussion
This section of the paper contains results and discussion based on the data collected through the Moodle Reports. For the purposes of discussion Reports are being treated as an LA tool. Although the sample size is small, the main purpose of this study is to elucidate the significance of Reports, which provides 5 different data sets in the form of 5 distinct logs.

Uniqueness of Technology Courses
The primary goal of any IT/ICT course is to help students to learn how new technologies and innovations are disrupting existing business models and are enabling organizational transformation. Such courses also require that students develop critical thinking skills through hands on activities (HOAs) or virtual lab work or simulations.
Hence it is important that the HOA's title is clear, precise, and meaningful and is able to catch the students' attention. It is also very important the objectives or the main purpose of each HOA in an explained in a clear way. Each HOA has to be focused on a single and very specific task. While creating the HOAs, the instructors need to focus on the "logic" behind every HOAs. This logic can be deductive or inductive. Deductive logic is often used to transmit strong and clear concepts. It usually includes clear steps. On the other hand, the inductive logic is based on the description of different and specific situations. Hence it allows instructors as well as the students to go back to a specific concept and repeat the HOAs. A cutting-edge curriculum in Science-Technology-Engineering-Math (STEM) is aimed at deepening problem-solving skills. It can help train students to successfully use information systems and new technologies for business operations. The STEM based programs, especially the ones related with Information Technology (IT) should also meet regional standards to help graduates to work anywhere in the Asean Economic Community (AEC).

Table 1. List of 6-HoAs
To achieve this goal the online technology courses should prepare students to work as systems analysts, IT /ICT managers, and technology officers both within and outside the IT/ICT industry. For the purposes of this study the following 6-HOAs were setup for the sample course.

Reports as a Diagnostic Tool
In medical sciences the "Reports" based on the lab tests are used for diagnosing an ailment or a body condition. Similarly, the "Reports" provided in a LMS can also be used as an effective diagnostic tool. It can also be used as a forensic tool to dissect various activities in a VLE. They provide a clear insight into how various learning objects (LOs) or resources provided in a technology course are being used by the students. As an example, the Table-1 provides the list of 6 HOAs. Fig.4 derived from this table shows the scores on these HOAs for 9 students (S1-S9) enrolled in the class. The figure indicates that the average scores on all 6 HOAs are in the same range. The scores emphasize that all students enrolled in the course successfully completed the 6 hands on activities.
Similarly, the students' scores on 6 corresponding assignments (A1-A6) related to 6-HOAs are depicted in the Fig.5. The captured data clearly indicates that there is no marked discrepancy in terms of individual scores on 6 assignments. The six assignments were carefully prepared to help the students to master the theoretical concepts. The positive correlation between assignment and HOAs scores further strengthen the argument that providing reading assignments related to HOAs can improve the quality of experiences in an online technology course. HOAs. It is very clear from the graph that only on HOA-5 and HOA-6 there was slight variation in the scores of 9 students (S1-S9). In other words, the pattern of scores on all 6 HOAs does not show any marked differences.  In a nutshell, the efforts put in by instructor in providing HOAs were not wasted. Fig.7 maps the scores of 6 HOAs and 6 Assignments side by side. Only HOA-5 and Assignment-5 shows marked differences in terms of scores. All other lines in the graphs demonstrate a positive correlation between assignments and the HOAs.

Report as a Quality Assurance (QA) Tool
New information visualization techniques including a variety of colorful displays and dashboards are being provided for instructors and students, so that they no longer need to "drive blind" during the tenure of an online course. As shown in Fig.8 the new and emerging field of LA can help in developing logical pathways for enhancing quality assurance (QA) in online courses.
The field of LA is relatively new. However, within the last 3 years it has evolved to becoming an integral part of teaching-learning processes. In an effort to add clarity to this landscape many types of LA models and frameworks have been proposed [ "Reports" is used to propose a simple model to illustrate (Fig. 9) the use of logs in improving quality of online technology courses. Along with many other trends this small study provides a step forward in exploring the use of LA in higher education. It is quite clear that just like the big data and the business analytic, the LA will also become a powerful tool for bringing continuous improvement of HOAs, assignments and other tasks in online technology courses [9] [10]. As mentioned above, Quality Assurance (QA) remains a critical issue in eLearning programs. On a meta-level, LA provides exciting opportunities to analyze and explore captured data from the system reports and logs to improve the overall learning which is often bogged down by subjective opinions and impossible-to-verify learning styles, theories and outcomes and surveys [11]. The QA process in technology courses incorporates several core QA functions: learning resources, HOAs, Assignments, Learning Analytic and Usage Evaluation. The QA triangle shown in Fig.8 effectively illustrates the synergy among these functions.
Each core function actually represents a constellation of activities in a VLE. The triangle indicates that rather than a unique sequence of steps that initiate QA activities, all core functions need to take place in a balanced manner for a QA strategy to be effective. The greatest impact on quality is achieved only when all functions are implemented in a coordinated fashion. Hence the justification for using LA is based upon issues relevant to the measurement of quality. Of course, one of the big problems around use of LA is the lack of clarity about what exactly should be measured to get a deeper understanding of quality of learning. A simple model as shown in Fig, 9 can help resolve key QA issues about HOAs in an online technology course. As a precursor to QA, it is important that instructors learn more about the significance of data sets captured in the logs of various LMSs.

Conclusions
In an online course the typical measurements related to engagement include time spent on number of logins, number of accessed LOs, and number of activities or tasks finished, but are these measures really getting to the heart of the matter? The remedies provided in an online course on the basis of LA alone cannot help in improving the quality of online courses. Hence QA should be treated as a continuous process which allows eliminating the waste and introducing new and improved contents and tasks that are hardened over time to become solid indicators of quality of learning experience. For effective use of LA, a simple framework proposed in Figure-9 can alleviate some of the issues related to QA. In the case of Moodle, the reports tool can be used to help the instructors understand whether or not students are engaging with the HOAs and other tasks. In coming years analytics will dominate all businesses. Big data and analytics will rule all information assimilation, storage, processing, distribution, utilization, and commercialization. While knowing what students are learning in an online course is important, knowing how they are doing on hands-on tasks is also extremely important. Technology related online courses can be made more meaningful with rich HOAs and virtual labs.