Program Evaluation on Learning Materials, Tutors, Assessment Matters and Overall Program Structure from ODL Students' Perspective

Open Distance Learning (ODL) is an approach in education practices in Open University Malaysia (OUM). One of its programs that are offered through ODL is the Bachelor of Psychology (BPSY) and it is one of the programs with the highest student intake in OUM. Since the program has been commencing for more than 10 years, it is the time to evaluate whether the program needs any revisions or improvements. Thus, the aim of the study is to determine the level of satisfaction of students on the Psychology program offered in OUM. A questionnaire was developed for this purpose which includes 4 areas of students’ satisfaction, namely the learning materials, student perceptions on their tutors, assessment matters, as well as the overall program structure. The questionnaires were distributed to more than 1000 active students through online and face-to-face tutorials. A total of 439 students participated in the study. The results indicated that students are most satisfied with the overall program structure and least satisfied with the assessment matters with the mean score of 3.98 and 3.59 respectively. The study certainly benefits the university in taking action to revise and improve the program. Recommendation for future research is included at the end of the paper.


Introduction
The way we communicate, operate, produce and live is changing because of the implementation of information and communication technology (Frye & Hemmer 2013). The emersion of new user interfaces and web 2.0 technologies changes our way of living, including the educational system. In Open University Malaysia (OUM), we introduced the open and distance learning (ODL) where it provides opportunities for all citizen of Malaysia to study online.
ODL is more than e learning. In the ODL approach, teachers and students are distanced from each other, and they communicate mostly using technology. Students do not have to be physically present to learn. This is definitely different from the conservative or traditional approach. In ODL, content is not delivered, but produced by students. Students are encouraged to develop and construct knowledge as opposed to given or spoon-fed by teachers (Elías, Baracic, Tomasegovic, & Mrvac, 2010).
OUM offers such opportunity to its students. Although some learning materials are given to students in the form of modules and learning kits, students are still encouraged to seek more than that. They are needed to dig more information from the Internet or printed resources, to explore further on the relevant research and publication to their courses (Frye & Hemmer 2013). OUM students are independent students, and they have more freedom to learn.

Problem of Statement
One of the programs offered in OUM is the Bachelor of Psychology (BPSY). It was introduced and offered to students since 2008. There are almost 800 students who have graduated from the program. As far as the academic staffs involved and the OUM management is concern, the program is working well and the students are learning, thus, it seems that it is unnecessary to evaluate or assess the program. However, according to the University of Central Florida (2005), the above statement is a misconception about program evaluation. Although the program is running well, there is still a need for improvement in the quality of the education program as well as in improving students' learning (University of Central Florida, 2008). There is always room for improvement even if it is felt that the program is good enough. Thus, since the program has been offered for more than 10 years, it is necessary to evaluate the program as to be aware of its effectiveness to students and to obtain constructive feedback about the program from the students.

Objectives of the Study
There are four objectives in this study, as follows: 1. To determine students' level of satisfaction on their learning materials; 2. To assess how students perceived their tutors; 3. To examine students' feedback on the assessment matters; 4. To determine students' perception on the overall program structure.

Research Questions
Specifically, this study shall seek to answer the following research questions: 1. What is the level of satisfaction among students on the teaching materials? 2. What is the students' perception on their tutors? 3. What is the students' feedback on the assessment matters? 4. What is the students' perception on the overall BPSY program structure?

Operational Terms
In this research project, the terms evaluation is referring to the educational program evaluation. It is the "systematic collection and analysis of information related to the design, implementation, and outcomes of a program, for the purpose of monitoring and improving the quality and effectiveness of the program." (ACGME, 2016) In this context, program evaluation is about understanding the program through a routine, systematic, deliberate gathering of information to uncover and/or identify what contributes to the 'success' of the program and what actions need to be taken in order to address the findings of the evaluation process (Frye & Hemmer, 2013).

Significant of the Study
This study is crucial and very significant to OUM especially CESS to take the appropriate measures to enhance the program. The data obtained can also be used as a useful source to identify the programs' strengths and weaknesses. As such, BPSY program could be remained as one of OUM programs with the highest intake and relevant in the market. Besides, the finding will help OUM to revise the program accordingly and eventually comply with the Malaysian Qualification Accreditation (MQA) standard. Finally, this feedback will be used to revamp, update and refine the program to make it more relevant.

Literature Studies
Distance learning is not a new term. It has been introduced over 100 years ago in Europe where students learn from audiotapes and lessons that were sent through mail (Mahmood, Mahmood, & Malik, 2012). The contemporary definition of distance learning was given by Greenberg (1998 as cited in Mahmood, Mahmood & Malik, 2016) as "a planned teaching/learning experience that uses a wide spectrum of technologies to reach students at a distance and is designed to encourage learner interaction and certification of learning" (p.36).
In a study done by Mahmood and Malik (2016) to compare between distance learning and traditional method, it was found that only in one area, ODL is better than the traditional method, which is on the appropriateness of assignment/project for the course. The rest of the items, namely feedback to students, preparedness of instructions, materials of the course, student interaction as well as consultation with students, tend to favor the traditional method.
However, that study focused on full time students whereas OUM offers part time program to students, which undoubtedly will result in different phenomena. Thus, it would be beneficial to assess the program in OUM.
There are a number of components to look into when a program needs to be assessed and improved. According to Carnegie Mellon University (2016), these few components are divided into three major components; program outcomes, direct measures and indirect measures.
In program outcomes, knowledge skills and abilities identify and students be able to demonstrate their knowledge skills and abilities upon completion of the program.
Direct measures of the assessment methods used by the Faculty to assess their students' achievement of the program outcomes. This measurement includes exam karma case studies, or representation, written assignment and portfolio.
In indirect measures, on the other hand include the student's perception on whether or not they have or achieving a learning outcome (Carnegie Mellon University, 2016).
Apart from that, the quality of learning materials is also important in ODL. According to Woo (2011), modules in OUM comprise a number of good HTML characteristics. Those characteristics are 1) html module should allow nonlinear learning; 2) encourage exploratory learning; 3) respect different learning styles; 4) state learning outcomes; 5) communicate high expectation; and 6) promote interaction and interactivity. She also added that although she found some researchers who are doubtful about interaction in ODL, while others believe that ODL programs can be more interactive than traditional one with proper interaction design (Woo, 2011).
Instructors or tutors is another expect that this study is looking into. According Kerwin and Mayes (1999, as cited in Banda & Kaphesi, 2017), students were highly satisfied with instructor and courses in ODL and direct interaction with instructors who played no role in determining students' satisfaction with their course. This implies that in overall, students were contended with their instructors despite lack of direct interaction with them.
However, in more recent studies, it was found that with the fact that in ODL students and instructors geographically dispersed, ODL instructors/practitioners have to deal with numerous challenges. Those challenges such as the ability to offer prompt and effective feedback, regular contact, and should be able to fully utilize the latest trend in technology as a tool for distance education (Cant, Wiid, & Machado, 2011). They also found that there are four top characteristics of ODL practitioners or instructor, namely having 1) good subject knowledge; 2) fair and ethical; 3) good writing skills; and 4) ability to guide students.
Assessment is another aspect that this study is looking into as it is an integral part of the learning process. A research suggested that in ODL system, content-based testing has shifted to performance-based assessment where assessment is linked with learning and skill development of the students because they are required to search from various sources to attend to assignment questions and quizzes (Chaudhary & Dey, 2013).
Nevertheless, it was found that there are some challenges in assessment in ODL. According to Ngara, Ngwarai and Mhute (2012) ODL assessment are lacking of diversity while the students in ODL programs are highly diverse in terms of age race and socioeconomic status. It is also challenging to address special needs students as they are uneasily recognized through this distanced learning. Plagiarism is another issue in ODL as it is difficult to be certain of the originality of the students' work when that the tutors or instructors are not in presence for close monitoring (Ngara, Ngwarai and Mhute, 2012). This is however incongruent with Nyika (2016) who suggested that electronic assessment can help to counter plagiarism using plagiarism check software. Furthermore, through e-assessment, there are a few other advantages such as low cost, instant feedback to students, flexibility with respect to time and location, reliability, as machines are more reliable than human marking especially in term of MCQ's and the reduce of paper usage (Carroll, 2002, as cited in Nyika, 2016.
Having mentioned on these researches done, it is clear that there is no research found to explore the effectiveness of psychology course in OUM. Thus, this study will determine students' feedbacks on the Bachelor of Psychology program offered as an ODL program to students.

Methodology
This research project employed questionnaire survey method. The survey instrument comprised of several components such as biodata of the respondent, learning materials, assessment, tutor and general feedback on the program. The questionnaire was distributed to the students via Google form and face to face interaction during the class.

Population and Sampling
The population and the sampling involved in this study is BPSY students who are still active students showed in the admission and record unit system. They must be senior students with at least a year of study in OUM regardless of gender, age, ethnic, religious and region. Convenience sampling technique was used in the study.

Data Collection
The data was collected through a developed questionnaire by the researchers. A pilot study was done prior to this study to determine its reliability. Results from the pilot test are indicated in Table 1 below

Data Analysis
The collected data was analysed descriptively using SPSS software. The mean and frequency score were gathered from the data.  In terms of ethnic group, majority of participants are Malay with a total of 211 participants (48.1%), Chinese 70 participants (15.9%), Indian 76 participants (17.3%) and other races constitute 82 participants (18.7%).

Participants' Demographic Profile
On the age aspect, there are 4 major age categories which are 21-30 years old with 163 participants (37.1%); 31 to 40 years with 198 participants (45.1%); 41 to 50 years with 57 participants (13%) and 21 participants (4.8%). were above 51 years above. The data showed that the majority of participants are in the range of 21 to 30 years.
For the year of study, it was found that 205 (46.7%) are students who have been in the program for 1 to 2 years; 171 (39%) are students for 3 to 4 years; 47 participants (10.7%) are student for 5 to 6 years and finally, 16 participants (3.6%) have been with the program for 7 to 8 years. The long period of study is due to the fact that most of the students are working adults and study on part time basis. Table 2 presents the results obtained from the participants' responses to each item in the scale. The students' level of satisfaction on their learning materials scale of the participants was measured ranging from 1.00-2.33 as low level of satisfaction to 2.34-3.66 as medium and 3.67-5.00 as high level of satisfaction about their learning materials. From the result, the participants showed medium and high level of satisfaction on the learning materials. It was found that students were hoping that other learning materials should also be included in the online platform. The activities and self-checks in the modules are sufficient to help learning.
Other learning materials should also be included in myINSPIRE.
The learning materials are applicable to the current situation.
3.818 High *Indication: mean 1.00-2.33 Low, mean 2.34-3.66 Medium, mean 3.67-5.00 High Table 4 shows the average value of each question posed to the participants and it was found that item 6 scored the highest.  Table 5 shows that each of the 11 questions under feedback on assessment methods is at a high level of satisfaction. The table stated that students preferred multiple choice questions (MCQ) type the most and least preferred on the item that stated the final exam question is easy.  Table 6 showed the results of students' satisfaction on the overall program structure. It was found that the item that stated I enjoy the psychology core courses offered in this program' score the highest.  Table 7 showed the overall students' level of satisfaction and it was found that students are most satisfied with the overall program structure of the program and least satisfied with the assessment matters.

Discussions and Recommendations
The aim of the study is to determine ODL students' level of satisfaction on BPSY program's learning materials, tutors, assessment matters, and overall program structure. Results showed that majority of the items for each construct score at the high level of satisfaction.
Results showed that ODL students were most satisfied with the overall program structure. This is congruent with Rajadurai, Alias, Jaaffar and Hanafi (2018) who found that the learners were satisfied with their usage of learning materials, academic facilitation and the services provided by the universities. This is supported by Elías, Baracic, Tomasegovic, and Mrvac that e-learning provides direct student's involvement in planning and development of the class. Banda (2017) also found that students found the ODL program rewarding in terms of the content of the program. However, students were found to be least satisfied with the assessment matters. This is congruent with Ngara, Ngwarai and Mhute (2012) who mentioned that ODL assessment are lacking of diversity while the students in ODL programs are highly diverse in terms of age race and socioeconomic status.
For future research, it is recommended to look into other aspects in ODL such as students' time management, the online learning platform and the availability of internet connection as to obtain more robust findings of feedback of any ODL program.