Comparison between the Senior and Junior Academics' Perceptions on Criteria in Measuring Teaching Effectiveness

Teaching in Malaysian universities has undergone a few transformations. In the effort of fully embracing outcome-based education, other initiatives had also been injected to further enhance teaching and learning. Hence planning teaching has become more complex and may be more challenging to both junior and senior academics. The focus of the study was to compare senior and junior academics’ perceptions on what they viewed as important criteria in measuring teaching effectiveness. The instrument used in the survey was developed based on the individual standards under each of the seven criteria of the Australian University Teaching Criteria and Standards (AUTSAC) Framework. Sixty-eight standards were identified as relevant to the Malaysian context. Multi-stage sampling procedure was utilized to identify samples among the lecturers of the Malaysian public universities. In general, the younger academics considered five of the seven criteria as important in the measurement of teaching effectiveness, whereas the senior academics only considered four as important. The findings were not surprising as the junior academics were in the early stages of their career and were more focused on developing their expertise. It is hoped that universities will recognize these criteria and standards and include them in their assessment for teaching effectiveness for yearly appraisal as well as promotion and teaching awards.


Introduction
The criteria used in determining academic performance in teaching are mainly decided upon by the faculty or university management team. The pressure that one must perform as an effective and superb teacher may come from both the university management and the students. Teaching in Malaysian universities has undergone several transformations. With the decision to adopt outcome-based education as a nation's effort in 2009 and the challenges that needed to be unfolded in realizing the initiatives of the Malaysian Education Blueprint for Higher Education (MEB:HE) 2015 -2025 [1], planning teaching has become more complex. It involves the preparation of materials, activities, tasks and planning of alternative assessments which may be new and challenging to some lecturers. Lecturers also need to adopt blended leaning strategies and focus not only on developing students' in the cognitive, affective and psychomotor domains but also on developing their soft skills. Lecturers must also report on the extent their courses are contributing to the achievement of the programme outcomes.
Report on students' achievement in terms of the cumulative grade point average need also reflect their acquisition of skills, deemed as important for employability, such as critical thinking and problem solving skills, team work skill, and leadership skill. Thus, effective teaching in the Malaysian higher education context include the success of the lecturer in grooming the students holistically. This implies that criteria in determining effectiveness of teaching must also be improved, in response to the changing role of the lecturer.

18
Comparison between the Senior and Junior Academics' Perceptions on Criteria in Measuring Teaching Effectiveness Most Malaysian universities rely on surveys to determine teaching effectiveness. This is also the main source for assessing teaching, as asserted by Berk [2] that survey of 40,000 department chairs in the United States indicated that 97% used "student evaluations" to assess teaching performance. Earlier, Seldin [3] had also highlighted that student ratings have been used as primary measure of teaching effectiveness for the past 30 yrs. The widespread use of student surveys is useful but insufficient to provide evidence of the effectiveness of teaching. Survey rating may be easily administered and analyzed but it needs to be complimented with other sources of data to avoid bias and to provide a more holistic assessment. Other sources of data that can be explored in evaluating teaching effectiveness include interviews, review of course materials, observation and teaching portfolio.
Numerous researches on development and identification of measures for teaching effectiveness had been conducted across many systems of higher education. In the October 2010 report [4] on perceptions on effective teaching and learning, survey respondents identified effective teaching and learning practices to enhance learning must include providing timely feedbacks to learners in improving learning, connecting learning outcomes with appropriate assessments facilitated learning, learners feeling encouraged to approach teachers who demonstrated interests in student learning, holding frequent teacher-student meetings to discuss student queries and concerns in guiding learning, organising lectures in a coherent format facilitated learning, and recognising the importance of regularly reviewing teaching to enhance learning. In Flinders University's context, effective teaching practice refers to the strategies and techniques that create optimal learning opportunities that motivate and inspire learners to achieve the desired learning outcomes.
Catano and Harvey [5] did a study on students' perceptions of teaching effectiveness to ensure the Evaluation of Teaching Competencies Scale (ETCS) that they developed and tested reflected students' perspectives. Using the critical incidents job analysis, the students identified nine teaching effectiveness competencies: communication, availability, creativity, individual consideration, social awareness, feedback, professionalism, conscientiousness and problem-solving.
To spearhead teaching in an institution, there is a need to develop teaching leaders. Those who develop teaching and learning strategies, provide training on teaching, and/or lead curriculum development may be regarded as leaders in teaching. Thus, leadership in teaching may also be a criterion to be considered in measuring teaching effectiveness. Catano and Harvey [5] cited Harvey, Royal and Stout [6] who stressed that very good teachers were transformational leaders with defined characteristics such as sacrificing themselves for the advancement of their students, engaging students in and out of the classroom, developing them intellectually and internalising the values associated with the discovery of new knowledge. Catano and Harvey [5] found most of the competencies identified by students for the ETCS were components of transformational leadership.
A more extensive characteristics is provided by the National Research Council [7], a synthesis based on the committee's identification of best practices from an examination of scholarly literature, which are (i) knowledge and enthusiasm for subject matter, (ii) skill, experience, and creativity with a range of appropriate pedagogies and technologies, (iii) understanding of and skill in using appropriate testing practices, (iv) professional interactions with students within and beyond the classroom, and (v) involvement with and contributions to one's profession in enhancing teaching and learning In 2015, Innovative Research Universities of Australia (IRU) and the Malaysian Research Universities Network (MRUN) collaborated on a project entitled "Criteria and Standards for Effective Teaching in Higher Education". This project focused on improving the professionalization of teaching through establishing and implementing criteria and standards for teaching effectiveness in universities. Among the objectives were to identify a common teaching criteria and standards used in Malaysia and Australian universities; to propose a common framework to be tailored for each university; and to identify and implement one or more applications of the framework for each participating university. This project would expand and clarify the evidence used to assess and determine teaching effectiveness. This is of utmost importance because it is used in yearly assessment of lecturers, evaluation for promotion and teaching awards. It may also be used as a guide in the setting of the key performance indicator (KPI) for teaching, and above all to ensure that teaching in universities is not side-lined due to the overemphasis on research.
In this study, the emphasis is on the quality of teaching and this needs to be aligned to criteria in academic appointment and promotion. Previously, most universities relied on student evaluation surveys (SETs) to determine teaching effectiveness and its related alignment. To overcome the weaknesses of SETs, an objective regulatory and accrediting framework with criteria and standards of achievement is needed that could measure teaching effectiveness. This is even more pertinent as in the learning environment where the students and teaching staff are becoming more diverse.
In identifying a framework as a base for the study, the researchers had compared the Australian University Teaching Criteria and Standards (AUTCS) [8] [9] (Figure 1) and the UK Professional Standards Framework (UKPSF) [10] (see Figure 2). Both frameworks had almost similar aims which include professional development for those in the teaching and learning field, fostering various creative and innovative teaching approaches and strategies, quality teaching, learning and assessment practices, and research and management activities. It was decided that this study adopt the AUTCS framework because of its flexibility. It was customized to suit each institution's values, criteria and context. Another reason for using the AUTCS framework was its ensured quality. It was designed by a project team in collaboration with the Australian academic community where the key intended outcome was to assist universities in reviewing their policy and practice around teaching quality and recognition. Besides that, the criteria selection and its indicative standards of achievement were based on extensive literature reviews on New Zealand, Australian and United States teaching criteria. In the formulation of the AUTCS framework, the developers had also referred to the UK Higher Education Authority (HEA) Professional Standards Framework [9]. The development project for AUTCS was conducted over a period of one year. There was an extension project which was a continuation of the original. Twenty-one universities were involved where the project approached the development and implementation of the teaching criteria in several ways reflecting the diversity of the institutions. The validity and reliability of AUTCS framework was assured for it underwent several drafts and were trialed in five Western Australian universities and several other selected Australian universities. Feedback and comments were received from university academics across Australia and New Zealand. The summary of the AUTCS is shown in Figure 3. Since its inception, it has received wide international recognition and it is still growing.
Finster and Milanowski [11] argued that in addition to establishing elements (referring to criteria and standards) for the teacher evaluation system, teacher perceptions of the quality of these elements are just as critical for making the performance evaluation system work. They suggested a potential theory of action, referred to as the Performance Evaluation System Theory of Action that links the performance evaluation system to improved instruction and student learning. It highlighted that teachers must generally agree that the standards are illustrative of "good" teaching practices and it is critical that teachers perceive the multiple measures as fair, valid, and reliable. Hence, it will lead to improved instruction and consequently, improved student achievement.
In defining a suitable measure for teaching effectiveness that meets the expectations and culture of the academics and the universities, it is essential to determine the perception of academics on what they viewed as important standards to be used as criteria. Since junior academics face different challenges and expectations, thus, it is important to view the perspectives of both junior and senior academics to identify criteria and standards that they perceived as important in the measurement of their teaching effectiveness. The academics need to agree on the criteria and the fairness of the measures, as highlighted by Finster and Milanowski [11]. Specifically, this paper sets to compare the senior and junior academics' perceptions on relevant indicators for measuring effective teaching.

Research Design and Procedures
The study adopted the survey research design. The initial draft was developed by the Malaysian team and then a question by question analysis was conducted to ensure the survey captured the required information and can be administered in both printed and online format. The Australian team further refined the draft. The survey comprised of a demographic section containing four questions seeking information including administrative position, academic level and duration of employment in higher education. The bulk of the survey comprised items on the individual standards under each of the seven criteria in the AUTSAC Framework. Based on the framework, 68 standards were identified as relevant to the Malaysian context. In the survey, items correspond to each of the 68 standards and respondents were asked to rate how important they felt the standard was for their institution as a measure of teaching effectiveness. On these questions, they were asked to respond on a four-point scale from Not Important at All (1), Not so Important (2), Important (3), and Very Important/Essential (4). The 5-point Likert scale was changed to 4-point to avoid the tendency to choose neutral responses. The results were analysed descriptively and using the t-test. Listed below (Table 1) are the seven criteria and the number of standards for each criterion that were listed in the questionnaire.
A pilot test was conducted on 30 academic staffs in one of the universities. Cronbach's Alpha test was conducted to determine internal consistency of the questionnaire, which was found to be .95. The questionnaire was then sent to two language experts. Table 2 shows the Cronbach Alpha reliability index for each criteria in this questionnaire, which indicates that it has good reliability. The survey in Malaysia was administered in printed form. Enumerators were appointed to help in distributing and collecting the questionnaire.

20
Comparison between the Senior and Junior Academics' Perceptions on Criteria in Measuring Teaching Effectiveness

Population and Sampling
The population of the study were 26,802 lecturers from the Malaysian public universities [11]. The 20 public universities in Malaysia are categorized as research, focus, technical, and comprehensive university. Research universities focus on the field of research, focus and technical universities on specific areas related to their establishment, comprehensive universities on diverse range of courses and fields of study. In this study, three research universities, three technical universities, one comprehensive university, and one focus university participated. The multi-stage sampling procedure was adopted. In the first stage, the universities were selected at random based on the types of universities. Consequently, the number of samples for each university was determined by proportionate sampling which was determined based on their total number of academics according to their ranks. Selection of samples according to academic ranks was done by random sampling. In this study, associate professors and professors were categorized as senior academics while the lecturers and senior lecturers were categorized as junior academics.
The minimum sample size was determined based on Cochran's formula [13]. According to the 2015 statistics of Malaysian Higher Education [12], there were 6,025 senior academics and 20,767 junior academics. Thus, based on the formula, only 361 respondents were needed to represent the senior academics and 384 for the junior academics. To reach that target, 1010 questionnaires were distributed, taking into account the possibility of low response rate. A total of 874 (87.53%) were received, however only 787 completed questionnaires were included in the analysis. Of those analyzed, 266 were senior academics and 521 were junior academics. Table 3 shows the number of universities selected and the number of respondents who completed the questionnaires.

Results
For the descriptive analysis, the mean score for each standard was calculated and the resulted range of the mean score was from 2.86 to 3.47 (Table 4). It was found the standard considered most important by the respondents was for Criteria 2, Use of innovative teaching approaches (mean = 3.47). The standard, leadership in planning and/or development of curriculum from Criteria 1 got the lowest mean score which was 2.86. The criteria considered most important for teaching effectiveness came from four of the seven criteria: Criterion 4 -Developing Effective Environments, Student Support and Guidance; Criterion 3 -Assessment and Giving Feedback to Students, Criterion 2 -Teaching and Supporting Student Learning, and Criterion 7 -Professional and Personal Effectiveness ( Figure 4). The 68 standards were ranked according to the mean scores for importance based on feedbacks from the respondents. In determining the degree of importance, the top 25 percent of the standards (i.e. 17 standards out of the 68 standards) were identified from the list. The standards were then traced back to the criteria they belonged to. Out of the seven criteria in the survey, in general the younger academics considered five of the seven criteria as important in the measurement of teaching effectiveness, whereas the senior academics only considered four as important (Table 5). Design and planning of learning activities were considered important by the senior academics but not by the junior ones.
Unlike their junior counterparts, the senior academics did not seem to give importance to assessment and giving feedbacks to students, the integration of scholarships of teaching and learning (SoTL), classroom research and professional activities with teaching and in support of learning. The biggest gap between criteria considered important by junior and senior academics is for Criteria 1, followed by Criteria 6 ( Figure 5).

Perceptions of Academics on Standards Considered Important as Measures of Teaching Effectiveness
From Table 6

Perception of Senior and Junior Academics on Standards that are Important as Indicators of Effective Teaching
In order to understand the differences in the perceptions of the junior and senior academics, the standards that came in as top 25% are highlighted in the table on standards considered important as measures of teaching effectiveness (See Table 7). Five standards that made to the top 25% considered as important by senior academics came from Criteria 1 (mean = 3.41 to 3.59). However, this was not true for the junior academics, who did not prioritise any of the standards within Criteria 1. The use of technology and effort to integrate on-line learning were regarded as low priority by the senior academics. However, the standard of leadership related to curriculum review and development was not given due importance by both the senior and junior academics.
Both senior and junior academics did prioritise on Criteria 2, namely innovative and collaborative teaching approaches, innovation in teaching practices and supporting student, effective supervision of student research activity and knowledge and use a range of teaching activities. However only the junior academics considered effective supervision of student internship/practice and leadership in teaching practices and supporting student as important. The Malaysian academics did not view peer review as an important standard in determining teaching effectiveness. Although scores in student evaluation survey were widely used in most Malaysian universities as indicators for teaching performance, both the senior and junior academics set low priority for this standard.
Assessment was a criterion that was given priority only by the junior academics. However, Innovation in assessment tasks and giving constructive and timely feedbacks were the two standards that were not considered.
As for Criterion 4 which included effective environments, both the senior and junior academics prioritised the standards on student support and guidance, on being initiative and innovative in supporting students learning, and providing support and guidance of student with appropriate academic needs. However, the junior academics also prioritised on initiative in creating engaging learning environment whereas the senior academics focused more on being innovative in creating engaging learning environment. However, the standards on being available for consultation, creating supportive learning environment to support student diversity, and demonstrating effective practice in developing collaborative learning were not considered as priorities.
For standards in Criteria 5, the junior academics only prioritised on authorship/co-authorship of publication/s in a nationally or internationally respected journal relevant to teaching and learning. None of the other eight standards in this Criterion were given similar priorities. On the other hand, Criteria 5 was not a priority for the senior academics.
None of the standards in Criteria 6 were prioritised by any of the academics. For Criteria 7, the junior academics considered responding positively to opportunities and new teaching approaches as a priority. However, the senior academics placed more importance on two other standards. These were approaching teaching with enthusiasm, passion and confidence, and being proactive in supporting peers to develop personal qualities. None of them prioritised the standard on responding to the needs of the teaching and learning transformation plan or to demonstrate resilience, perseverance, self-reflection, commitment, mentorship, supporting junior colleagues and peers. An independent t-test were conducted to determine if the differences between the perception of the senior and junior academics were significantly different. It was found that there was no significant difference in the overall perception of the junior academics and the senior academics on the criteria and standards with regard to teaching effectiveness. However, in testing further, significant differences was established for Criteria 1 -Design and Planning of Learning Activities, (t (785) = 8.67, p = .000) as can be seen by mean of 30.22 for senior academis and 25.21 for junior academics. Significant differences were also established for Criteria 5 -Integration of Scholarship of Teaching and Learning (SoTL), Research and Professional Activities with Teaching and Support of Learning, (t (785) = -3.09, p = .002) and Criteria 6 -Evaluation of Practice and Continuing Professional Development, (t (785) = -4.31, p = .000).

28
Comparison between the Senior and Junior Academics' Perceptions on Criteria in Measuring Teaching Effectiveness

Discussion
In comparing junior and senior academics, a common trend on the standards considered as priorities within Criterion 1 -Design and Planning of Learning Activities was observed, however the perception scores of the junior academics were much lower and did not make it to the top 25% of the 68 standards. Standards on leadership in designing and review of curriculum and leadership in planning and/or development of curriculum were at the bottom of the list of priorities for the junior academics. This was not surprising as they were still in the early stages of their career and more focused on developing their expertise.
For Criterion 2 -Teaching and Supporting Student Learning, the senior and junior academics had different views on the standards they considered as priorities. One exception was the standard C2S5 -Use of Innovative Teaching Approaches which was considered as the most important priority by both the senior and junior academics. The standard C2S9 -Innovation in Teaching Practices and Supporting Students was also considered very important by the senior academics, although second in priority. They also placed importance on Engaging and Stimulating Student Participation as third in priority (C2S1). On the other hand, the fourth and fifth priorities were the Standards C2S6 -Effective Supervision of Student Research Activity and C2S3 -Knowledge and Use Range of Teaching Activities. The standard C2S4 -Use of Collaborative Teaching Approaches was the bottom priority for the senior academics and Leadership in Teaching Practices and Supporting Students (C2S10) for the junior academics.
In terms of Criterion 3 -Assessment and Giving Feedback to Students, both groups agreed the standard C3S7 -Assessment Constructively Aligned to Learning Outcomes was most important in teaching effectiveness. They also agreed that the standard C3S6 -Innovation in Assessment Tasks was least important. However, they had different priorities for other standards, when one standard was important for one group but less important for the other group.
For Criterion 4 -Developing Effective Environment, Student Support and Guidance, both the senior and junior academics had almost similar perceptions except that it was different in terms of priority. One consensus that the two groups had was, the Standard C4S8 -Support and Guidance of Student with Appropriate Academic Needs which was the first priority for the senior academics but a second priority for the junior academics. On the other hand, the first priority of the junior academics was C4S4 -Innovative in Supporting Students Learning but a third priority for the senior academics. Three Standards that were considered not important in terms of priority were C4S1 -Availability for Consultation (email, telephone etc.), C4S2 -Creating Supportive Learning Environment to Support Student Diversity and C4S7 -Demonstrate Effective Practice in Developing Collaborative Learning.
For the Criterion 5 -Integration of Scholarship of Teaching and Learning (SoTL), Research and Professional Activities with Teaching and Support of Learning, the senior and junior academics did not show any agreement in terms of the level of priority for any of the standards. However, in terms of the importance of the standards, the senior academics considered the standard C5S2 -Share Teaching and Learning Research within Your Discipline as the most important, while the young academics considered it as fourth in importance. However, the junior academics did consider the standard that was most important in terms of priority was C5S3 -Authorship/Co-Authorship of Publications and a Nationally or Internationally Respected Journal Relevant to Teaching and Learning. However, both standards did include publications, though C5S3 did not specifically refer to authoring publications.
As for Criteria 6 -Evaluation of Practice and Continuing Professional Development, both the senior and junior academics did not perceive any of the standards as important priorities. With regard to Criteria 7-Professional and Personal Effectiveness, the senior academics perceived the standard C7S8 -Approaching Teaching with Enthusiasm, Passion and Confidence as the most important priority and C7S18 -Proactive in Supporting Peers to Develop Personal Qualities as their second priority. Based on teaching experiences, these perceptions were expected. They would be highly knowledgeable and very passionate about their work and at the same time approachable and with their valuable teaching experience, they would be ever willing to help and groom others especially their students in particular as well as their peers. On the other hand, the junior academics perceived the standard C7S1 -Responding Positively to Opportunities and New Teaching Approaches as their first priority in importance. The junior academicians found this standard important. Perhaps they were enthusiastic about teaching, resourceful and ever willing to learn.
The study established significant differences in the perceptions of the senior and junior academics on teaching effectiveness. Perhaps this was expected of the junior academics for they perceived that building supportive relationship with students was their immediate concern. They were focusing more on research and professional activities. The senior academics were more concerned with effective work such as designing and planning learning activities. They perceived there should be more student participation and by using student-centred approach it would result in effective student learning.
Although the senior and junior academicians differed in their perception on the priority of importance of the standards, they however perceived almost all the standards in the seven criteria as important measures for teaching effectiveness. In comparing the perceptions of the academics on the criteria and standards, they did not greatly differ except that the junior academics did not consider Criteria 1 and the standards subsumed under it as important in determining teaching effectiveness.

Conclusions
Most literature reviews and researches had found that student surveys in measuring teaching effectiveness were insufficient hence, the need to establish and implement criteria and standards to further improve measures for teaching effectiveness, especially in higher education. Apart from the over dependence on the results of the student surveys to illustrate teaching effectiveness of academics, most surveys are constructed according to common criteria and standards to measure all academics, may they be senior or junior academics. Based on the Performance Evaluation System Theory of Action [11], the academics would be able to use this feedback to guide and improve their practice. Since, the challenges and expectations are different for the junior and seniors, their agreement on the criteria of measurement must be taken into consideration to ensure fairness, validity and reliabilty of the measures [11. In this study, seven criteria and 68 standards were ranked according to the mean scores for importance. In general, the younger academics considered five of the seven criteria as important in the measurement of teaching effectiveness, whereas the senior academics only considered four as important. Finster and Milanowski [11] argued that teacher performance evaluation system needs to be treated as coherent systems and how teachers perceive it needs to be linked with the components of the evaluation system. In this context, the evaluation criteria used for junior and senior academics need to be differentiated because the juniors are still learning on how best to teach and how teaching supports their career aspirations. The evaluation criteria and standards for teaching need to be differentiated for the purpose of yearly assessment or for promotion.
These criteria and standards were deemed reliable and valid in measuring teaching effectiveness for they were based on the senior and junior perceptions themselves on the importance of the criteria and standards, although there were differences in perceptions between the senior and junior academics with regard to teaching effectiveness. The findings showed the criteria and standards were perceived important by both groups of academicians, and hence it is hoped that universities will recognise these criteria and standards in measuring teaching effectiveness and seek to adopt them as their own. Some of the more profound standards identified in this study can be extracted and simplified to be used as survey items for student evaluation of lecturers' teaching. The standards are also relevant in assessing promotion and teaching awards, which require compilation in the form of a portfolio.
In Malaysia as in many other countries, evaluation for promotion and teaching awards is often done using teaching portfolio. Diamond [13] also affirmed that the primary use of portfolio is for promotion and tenure. A portfolio that provides a comprehensive documentation of the academic's accomplishment, practices, reflections, evidences of effective practices, innovative teaching and assessments require considerable amount of time for preparation. Thus, more studies should be done to develop an exhaustive criteria and standards and simpler form of documentations in providing evidences of one's good practices in teaching especially in higher education.