The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills

Computational thinking skill is a new skill needed in twenty-first century learning. However, teaching practitioners have been struggling to find effective techniques to be used in the teaching and learning process that can help students to develop such a skill. Several studies have proposed a number of models to guide the teaching and learning process, but most lacked a focus on the improvement of computational thinking skill among students. As such, this study was carried out to identify and validate the elements and sub-elements of an assessment instrument prior to developing an effective teaching and learning model for enhancing students’ computational thinking skill with the use of the Delphi technique, which was carried out through three rounds of implementation involving a panel of experts made up of five (5) academicians and two (2) industrial practitioners. Essentially, the proposed elements and sub-elements were derived from those based on the combination of the problem-solving techniques and metacognitive skills. After the completion of the three rounds of such technique, six (6) elements and 25 sub-elements of the assessment instrument were validated to be highly important, entailing them to be embedded in the proposed teaching and learning model for enhancing students’ computational thinking skills. The use of Delphi technique in this study successfully identified and validated six (6) elements and 25 sub-elements of the teaching and learning model. This study contributes to the development of a model that can be used by teaching practitioners as a guide in their teaching and learning process for enhancing students’ computational thinking.


Introduction
Essentially, computational thinking skill is a skill that helps an individual to systematically and logically understand the occurrence of an event. According to [1], computational thinking is defined as a way of thinking that can be expressed algorithmically that enables humans in solving problems with the use of specific devices. On the other hand, [2] asserts that problem-solving is a concept of computational thinking, which, according to [1], can be enhanced through learning activities that train specific thinking to enable students use computers more efficiently. However, this skill encompasses not only computing skill but also basic skill that every individual need to possess to help produce digital makers. Admittedly, the literature is replete with many definitions of computational thinking constituting a list of 66 elements, with five topping the list, namely abstraction, 8202 The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills algorithm, decomposition, evaluation, and generalization [3]. Taking cognizance of its importance in the twenty-first century, many nations have started to embed this skill in their school curricula encompassing all levels of education, from elementary to tertiary level. As such, teaching practitioners must use suitable learning and teaching techniques and strategies to help develop such a skill among their students. In this regard, appropriate learning models can be used to guide teachers and lecturers to identify relevant techniques for their classes.
Even though many studies of computational thinking skills have been carried out, most lacked a focus on the development of appropriate models for effective learning and teaching that could help improve students' computational thinking skills. As such, there is an urgent need to develop effective learning and teaching models by taking into account appropriate techniques that can help students develop and improve their computational thinking skills. In the field of computer science, such an important skill is highly relevant to computer programming, which has been demonstrated in several studies of the teaching of computer programming [4][5][6], the findings of which led to the development of several popular learning and teaching models and techniques to help improve students' computational thinking skills. In this study, the researchers analyzed five leading models to determine their effectiveness in improving students' computational thinking skills, which are closely related to computer programming and problem-solving [7][8][9][10][11]. The analysis helped reveal the essential elements and sub-elements of the five models, which formed the basis of this investigation. Also, several researchers [12][13][14][15][16] have used the metacognitive skill technique in the teaching and learning of computer programming. However, existing teaching and learning models have not used such a technique in teaching the subject matter. The analysis showed a combination of appropriate elements and sub-elements of the five models, problem-solving techniques, and metacognitive skill techniques would be effective in improving students' computational thinking skills.
As such, a test was entailed to verify the validity of elements and sub-elements that had been selected for the development of a learning and teaching model to guide the development of strong computational thinking skills among students. In this respect, the researchers used the Delphi technique to validate the elements and sub-elements of a proposed model. As revealed in many qualitative studies, this technique has been proven to be highly reliable to help practitioners to make informed decisions and reach important consensus in various fields, including education [17]. In principle, this technique operates by searching an agreement or a consensus with the use of assessment instruments or questionnaires that are administered for several rounds to gather data from a panel of experts. In this regard, [18] argues that the Delphi technique is used to filter and identify relevant factors and to help experts reach an agreement in a particular study. As such, this study used this technique to validate the selected elements and sub-elements of the proposed teaching and learning model.
Premised in this context, the objectives of this study were as follows: (a) To identify and validate appropriate elements and sub-elements of a teaching and learning model that can help improve computational thinking skills with the use of the Delphi technique and (b) To explore the contribution of each element and sub-element to the development of computational thinking skills. The outcomes of the validation process helped lay the foundation for the formulation of the proposed teaching and learning model that would help students to improve their computational thinking skills.

Materials and Methods
The Delphi technique was used in this study used to address the research objective given its high effectiveness in gathering and analyzing experts' opinions of the field of this study. A survey was carried out for three (3) rounds to ensure the experts would finally reach a strong consensus or agreement of the elements and sub-elements of the proposed teaching and learning model.

Participants
An expert is an individual who has vast experience and strong knowledge of a particular topic. In this study, the selection of experts was based on their expertise and experience in the field of computational thinking. As such, seven (7) experts with 11 years of working experience or more in computer programming and computational thinking-related fields were recruited. Such a number of experts was in line with the recommendations made by [19] and [20], who assert that there will be no significant differences in the consensus reached by a panel consisting of three, five, nine, and 11 experts. Purposeful sampling technique was used to select the experts based on their experiences and involvements in teaching computer programming courses and as master trainers in the field of computational thinking, which resulted in the selection of five (5) academicians and two (2) industrial practitioners.

Assessment Instrument
An assessment instrument was used in this study, the design of which was first carried out by surveying several sources of information regarding elements and sub-elements that were deemed relevant to the development of the proposed learning and teaching model. Essentially, the contents of such sources, such as journal articles, technical reports, Master's theses, doctoral theses, and the websites of certain organizations, were screened to identify elements relevant to the proposed model.
Five (5) participants with vast experiences in computational thinking and computer programming were selected for a pilot test to validate the research instrument, the results of which helped to improve its structure and contents. Also, the reliability of the assessment instrument was further enhanced by incorporating their feedback in the instrument accordingly. Each item was rated along 5-point Likert-type scales to measure respondents' levels of agreement as summarized in Table 1. The results of the pilot test were also used to guide the development of the assessment instrument which would be used for the implementation of the Delphi technique.

Implementation Procedure of the Delphi Technique
In this study, the Delphi technique was carried out for three (3) rounds as detailed in the following sub-sections: In the first round, the assessment instrument, named Round1 Assessment Instrument (R1-AI), consisting of seven (7) elements and 17 sub-elements were sent to the experts through official postal mails and emails to elicit their opinions. Their constructive feedback helped to improve the reliability of the instrument by enabling the researchers to re-examine the instrument and make necessary changes deemed important. The revised instrument would then be subjected to the second round of the Delphi technique.

Round 2
The aim of the second round of this technique is to elicit a consensus of experts' opinions. For this purpose, the researchers distributed the revised assessment instrument, named Round2 Assessment Instrument (R2-AI), to the same experts to seek their comments or feedback. The acceptance and rejection of elements and sub-elements was based on a cut-off point of 3.5 of the assessment continuum, ranging from '1' (Extremely not imperative) to '5' (Extremely imperative). Thus, elements and sub-elements attaining a rating of 3.5 and above would be accepted. Conversely, those with ratings lower than 3.5 would be rejected [21]. The second round of this technique was deemed complete after all elements and sub-elements had been either accepted or rejected. The R2-AI instrument was checked and refined prior to the implementation of the third round of this technique.

Round 3
The revised assessment instrument from the second round was renamed Round3 Assessment Instrument (R3-AI). The instrument was distributed to the same experts to elicit their final agreement on the validity of the elements and sub-elements of the revised version. In the third round, a consensus of experts' opinions of the validity of the revised elements and sub-elements of the R3-AI instrument was obtained. Figure 1 shows the implementation of the Delphi technique carried out in this study. 8204 The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills

Findings and Discussion
The panel of experts recruited in this study was made up of five (5) lecturers who were involved in the teaching of computer programming and were also computational thinking master trainers and two (2) industrial practitioners who were also computational thinking master trainers. All of the experts were vastly experienced whose working experiences ranged from 15 to 25 years. The findings of the first and second rounds of the Delphi technique showed that there were several elements and sub-elements that had to be refined and discarded and new elements and sub-elements had also to be introduced. The results of the third round of this technique helped to establish the validity and reliability of elements and sub-elements. Table 2 summarizes the demographic backgrounds of the experts.

Findings of the First and Second Round of the Delphi Technique
Initially, prior to the first two rounds of the Delphi technique, the assessment instrument consisted of seven (7) elements, namely Understanding and Defining Problems (E1), Planning Problem-Solving (E2), Designing Problem-Solving (E3), Coding (E4), Testing (E5), Documenting (E6), and Metacognitive Skills (E7), all of which were made up of 17 sub-elements. After the implementation of these rounds, several changes were made, such as maintaining, refining, adding, and exchanging some of the sub-elements and elements, which are summarized in Table 3. The details of these changes Several changes were made after the implementation of the first two rounds of the Delphi technique. For example, the wording of E2.1 was refined to make it more intelligible, while E2.2 was a new sub-element transferred from the first element, E1. The last two sub-elements, namely E2.3 and E2.4, were retained. Another new sub-element was introduced, namely E2.5 (Adapting or using the original codes to solve a new problem), which was originally a sub-element of the sixth element, E6. (c) Initially, the third element, E3, constituted two (2) sub-elements, namely 'Writing appropriate algorithms based on the analysis and planning carried out earlier' (E3.1) and 'Examining the written algorithms to ensure accurate output' (E3.2). After the first two rounds of the Delphi technique, no changes were made to these two sub-elements. However, a new sub-element was added to the third element, E3. As such, the third element, E3, now comprises three sub-elements.  Table 3 summarizes the elements and sub-elements of the teaching and learning model for the improvement of computational thinking skills after the first and second round of the Delphi technique.
In addition to refining, adding, and discarding several elements of the assessment instrument, some experts also suggested that the definition of metacognition had to be made more detail at the beginning of the instrument to 8206 The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills foster a better understanding of such a concept. All the comments and suggestions made by the experts in the first and second round of the Delphi technique were taken into account by making the necessary changes to the assessment instrument, which would be subjected to the third or final round of the same technique.  Table 4 shows the revised elements and sub-elements of the teaching and learning model for the improvement of computational thinking skill after the third round of the Delphi technique. As shown, there were six (6) Table 4 also shows the mean scores and standard deviations of the elements and sub-elements of the proposed model.

Findings of the Third Round of the Delphi Technique
The followings are the detailed description of the embedded elements and sub-elements of the teaching and learning model for the improvement of computational thinking skill. (a) As shown, the mean score of the second element, E2, was 5, which was extremely high, signifying that it is an extremely important element that is needed to improve students' computational thinking skills. Specifically, the first sub-element, E2.1 (Using existing knowledge to plan several problem-solving methods to solve problems), and the fifth sub-element, E2.5 (Adapting or using the original codes to solve new problems) are highly important because they are consistent with the concept of generalization of computational thinking, which is a concept of solving a new problem based on previously-solved problems. Likewise, the fourth sub-element, E2.4, is also highly important as it divides a problem into several manageable sub-problems and functions, which closely relates to the concept of decomposition of computational thinking that breaks down a complex problem into smaller problems that are easily discerned and managed [22]. Thus, the implementation of the planning activities of the second element (E2) of the proposed model can help develop the two sub-components of computational thinking skill, namely generalization and decomposition. (b) Similarly, the findings suggest that the third element, E3 (Designing problem-solving), is highly important too, entailing it to be embedded in the teaching and learning model for the improvement of computational thinking skill. Essentially, this element is closely related to the algorithmic element of computational thinking where an algorithm refers to a series of structured commands to solve a problem or to perform logical reasoning [23]. Specifically, the second sub-element, E3.2 (Examining the written algorithms to ensure accurate output), and the third sub-element, E3.3 (Checking the developed algorithms to ensure efficient problem-solving), are in line with the concept of assessment of computational thinking, which is to ensure the solution of an algorithm, system or process is precise and suitable for its intended use [22]. In particular, the activities carried out during the designing process to generate computer programs relate to algorithmic thinking of computational thinking. Thus, as students perform designing activities to produce computer programs, they are also developing their computational thinking skills. This finding is consistent with those of other studies, clearly emphasizing the importance of this element in improving the algorithmic thinking and assessment of students' computational thinking [23].
As such, this element should be embedded in the proposed teaching and learning model, given the high rating of its sub-elements, with all attaining a mean score of 5. (c) The findings showed that the fourth element, E4, was also deemed highly important and valid by the experts, with four of its sub-elements attaining a mean score of 5, while the reaming sub-element achieving a mean score of 4.86. Such findings are hardly surprising given that one of its sub-element is closely related to the concept of decomposition, algorithmic thinking, and assessment of computational thinking. In this regard, coding activities entail strong logical thinking and high creativity among students to generate precise, quality computer output. Thus, the implementation of the fourth element, E4, in the teaching and learning process can help improve students' computational thinking in terms of decomposition, algorithmic thinking, and assessment. In light if this revelation, element E4 together with its sub-elements should be embedded in the proposed teaching and learning model to ensure its effectiveness in developing students' computational thinking skills. (d) The findings also indicated that sixth element, E6 (consisting of seven sub-elements), was highly rated by the experts, with five of its sub-elements attaining a mean score of 5. Again, this specific finding is not surprising as all the sub-elements of E6, namely E6.1, E6.2, E6.3, E6.4, and E6.5 (which involve planning, monitoring, selecting, checking, and assessing activities) are closely related to the specific elements of computational thinking, namely decomposition, algorithm, generalization, and evaluation. Thus, such an element is crucial to the development of students' computational thinking skills. Likewise, the same sub-elements have been studied by other researchers that focused on computer programming [12][13][14][15][16], indicating its significance in problem-solving techniques. On the other hand, the remaining sub-elements, namely E6.6 and E6.7, attained mean scores of 4.86 and 4.71, respectively, underscoring 8208 The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills their importance in the development of students' socialization skills. Overall, the above findings suggest that element E6 is one of the important elements that needs to be embedded in the proposed teaching and learning model. (e) As shown in Table 4, the first element, E1 (consisting of two sub-elements), was also deemed reliable and valid by the experts, as evidenced by the high ratings of its sub-elements, namely E1.1 (Defining a problem by reading and understanding it) and E1.2 (Analyzing the problem by identifying the input, process, and output), with the former and latter attaining mean scores of 4.7 and 5.0, respectively. Arguably, such a finding is tenable as this element is closely related to one of the important aspects of computational thinking, namely abstraction, an ability that helps the abstraction of data that are only relevant to a problem and eliminates those deemed irrelevant. Surely, the inclusion of this element in the proposed teaching and learning model can help students develop such an important skill or ability. Such an inclusion can be justified by the evidence provided by the findings of previous studies, in which several researchers used this element in the teaching of computer-related courses which helped students to develop strong abstraction ability or skill [12,[23][24][25]. Table 4 summarizes the elements and sub-elements of the teaching and learning model for the improvement of computational thinking skills after the final round of the Delphi technique. Analyzing the problems by identifying the input (the given data), process (the appropriate problem-solving method), and output (the intended outcome) [8][9]11].  (f) The same findings also indicated that the fifth element, E5 (consisting of three sub-elements), received a strong consensus regarding its validity from the experts, with its first two sub-elements, E5.1 (Executing the program repetitively by entering different sets of inputs to generate a precise output) and E5.2 (Examining the output in terms of readability), attaining a mean score of 5 each. By contrast, with a mean score of 4, the last sub-element of this element, E5.3 (Examining the output in terms of efficiency), was deemed moderately important. These findings suggest that all the sub-elements, especially the first two, are important to help improve students' computational thinking skills, entailing their inclusion in the proposed teaching and learning model. However, the experts cautioned that the third sub-element E5.3, must not be overly emphasized for novice students, as they were quite new to the concept of computational thinking. As such, only basic aspects of computational thinking should be exposed to such students.
Overall, these findings highlight the importance of this testing element, E5, which is closely related to two important components of computational thinking, namely evaluation and generalization. To lend credence to such revelations, similar findings had been reported in previous studies [26][27][28][29], which showed the element of testing was important to help improve students' computational thinking skills.
As a whole, the findings discussed in this section helped the researchers to address the research objectives, which were to validate the elements and sub-elements of a teaching and learning model for the improvement of computational thinking skill with the use of the Delphi technique and to examine the relations between the proposed model' elements and sub-elements and those of computational thinking skill. Table 5 highlights the mapping of the elements and sub-elements of the proposed model and computational thinking skill. 8210 The Use of Delphi Technique in Validating a Teaching and Learning Model for Enhancing Students' Computational Thinking Skills The summary of the findings of the Delphi technique used to validate the elements and sub-elements of the proposed teaching and learning model are as follows: (a) Prior to implementing the Delphi technique, the assessment instrument consisted of seven (7) elements and 17s sub-elements. Subsequently, after applying such a technique, several changes were made to the proposed model, which by then had six (6) elements and 25 sub-elements. (b) The most important elements were E2 and E3, followed by E4, E6, E1, and E5. (c) The above elements and sub-elements, which had been vigorously validated by a panel of experts, should be embedded in the proposed teaching and learning model, which could be used to guide the development of learning activities to improve students' computational thinking skills.

Conclusions
Every student should acquire strong computational thinking skill, which is a new but important skill needed in twenty-first century learning. Thus, it needs to be cultivated as early as possible, but teaching practitioners have been struggling to find effective techniques to be used in the teaching and learning process that can help students to develop such a skill. Against this backdrop, this study was carried out to develop and validate the elements and sub-elements of a proposed teaching and learning model for such a purpose. Specifically, such elements and sub-elements were conceptualized from the combination of elements of problem-solving and metacognition, which are two popular techniques used in the teaching and learning of computer programming. After a rigorous validation process through the Delphi technique, six (6) elements and 25 sub-elements were found to be highly valid. In conclusion, the findings of this research can be used to guide the development of an effective teaching and learning model with which teaching practitioners can use to select the most appropriate technique for their teaching practices. This study contributes to the development of the teaching and learning model for enhancing students' computational thinking skills as illustrated in Figure 2.