Crowdsourcing as Content Collaboration for STEM Edutainment

In this project, crowdsourcing science, technology, engineering and mathematics (STEM) content was used to promote enjoyable, challenge-based English language gamification to young ESL learners and instructors altogether. The aim is to optimize the participants’ engagement in learning STEM disciplines using the target language. However, relevant literature in this area shows inadequate empirical evidence on engagement when these two activities combined are implemented in real-life settings. This research therefore, was conducted to explore the participants’ engagement in learning STEM contents, outside their classroom setting, using crowdsourcing and gamification activities. Through needs analysis of 48 participants (school children aged 13 to 14 years) who were assisted by 20 university undergraduates who were involved in the ‘Young Scientist Camp’ project, it was discovered that the participants had minimal exposure to the use of technology as well as the language in learning STEM disciplines. The participants were assigned to crowd-source for online STEM contents because the target is to maximize the use of English language. The contents were used to form several questions which they then took turns to test their peers from other groups using Kahoot! The qualitative data of the study were gathered via video recorded interaction, observations, and focus group interviews. Tesch’s thematic analysis, Kappa coefficient for inter-rater reliability and an observation checklist using Kapp’s motivational characteristics of game elements were used to determine crowdsourcing characteristics and elements of engagement from the activities. Interestingly, the pupils showed evidence of engagement when faced by unexpected problems in their groups. They also expressed strong engagement when the autonomy was given to them in deciding the contents and flow of the activities, which was evidence in the process of completing the tasks. This implicates emerging recommendations in engaging young learners via crowdsourcing techniques and gamification to foster interest in learning the language for STEM.


Introduction
Personalized learning technologies in education 4.0 with disruptive technology are constantly changing the landscape of learning to an independence, collaborative and networked. By customizing learners' learning needs, it is now a challenge to develop a 'one-size-fit-all' content and create efficient personalization materials and strategies [1]. By engaging millennial learners in the learning process, the role of teachers is no longer the 'sage of the stage'. This means learners are now the partial producer of knowledge through a series of meaningful engagement during the process [2]. For that, employing educational gamification can be a stimulus for a more motivating and engaging classroom activity [3]. In fact, gamification has become a popular tactic to encourage learners' specific behaviors, and increase motivation and engagement. Huang and Soman [4] assert that learning contents eventually become exciting and interactive when an element of gamification is introduced.
Apparently, this has caused the line between content creators and content consumers becoming unclear due to large crowd empowerment of users to collaborate, organize and share knowledge known as 'crowdsourcing'. Instructors and learners have been crowdsourcing content on almost any topic or domain including the creation of textbooks and other class materials [5,6]. In fact, there are various crowdsourcing platforms that support research activities such as EpiCollect.net. Citsci.org etc. However, Law et al. [7] highlight the issue of uncertainties in crowdsourcing processes in terms of activities that they choose to engage, how it influences their navigation, and the strategies to implement the task. There are also uncertainties on the amount of information needed for decision-making in designing and executing a research project. In fact, despite the increasing trend and capacity of crowdsourcing as an open innovation practice [8] for gathering data and enabling collaboration between researchers, "crowdsourcing has not made its way into mainstream research methodologies" (p. 1544) [7]. Indeed, this brings about scarcity of empirical evidence of learners' engagement when crowdsourcing and gamification altogether are employed in real life settings. In fact, despite the widespread adaptation of gamification and its proven effects, there have been a minimal number of works that focused exclusively on Higher Education (HE). A systematic review of literature on 562 potentially relevant articles published between 2000 and 2016 by Ortiz eRojas [3] identified only 30 studies were accepted using the inclusion criteria of 'Higher education' and 'STEM related courses'.
On the other hand, there have been heightened efforts given to Science, Technology, Engineering, and Mathematics (STEM) disciplines due to its role as foundational to economic growth. Lee, Min, and Mamerow [9] find students' choice of a STEM career is particularly influenced by their teachers' expectations. This has led to the emphasis on teachers' pedagogical content knowledge (PCK) which has become greater with the need to develop PCK for interdisciplinary STEM learning [10]. Hence, Gilliam, Jagoda, Fabiyi, Lyman, Wilson, Hill, and Bouris [11] encourage the incorporation of technology-supported collaborative learning to create interest in STEM.
Correspondingly, it raises questions on the impact of gamification when combined with crowdsourcing in learning STEM disciplines in a real-life setting. Could gamified crowdsourcing activities stimulate participants' engagement in learning STEM disciplines? Hence, this paper aims to explore evidence of engagement by observing participants' interaction during crowd-sourcing and gamification activities when revising STEM contents using the target language.

Crowdsourcing in Learning
Crowdsourcing is defined as "a new web-based business model that harnesses the creative solutions of a distributed network of individuals through what amounts to an open call for proposals" (p. 8) [12]. Several interactive and participatory webs using crowdsourcing have attracted people because it is fun and exciting, portrays altruism and genuine willingness to help others, and is as one significant motivator [13]. However, though these webs or automatic solutions have been designed to serve the purposes, the tasks remain difficult for machines as it requires humans' level of abstraction and creativity [14]. This gives the idea to crowdsource learning content in encouraging learners' construction of their own learning paradigms. Literature has shown learners can be prompted for information that can be used for improving learning conditions for other learners [15]. Estellés-Arolas and González-Ladrón-De-Guevara [16] operationalize crowdsourcing in accordance to three criteria i.e. Crowd, Initiator or Crowdsourcer, and Process. The three criteria are further explained in the following sub-criteria (Table 1) which are applied in this study to identify evidence of crowdsourcing that took place during the activities. What they have to do 3.
What they get in return 4. Who it is 5.
What they get in return for the work of the crowd 6.
The type of process it is 7.
The type of call used 8.
The medium used

Gamified Crowdsourcing in Educational Context
As a theoretical ground for the study, the educational qualities of gamified crowdsourcing applications are based on constructionism, which implies learning by constructing one's own knowledge. Cocciolo [17] asserts that this constructive pedagogy is closely related to Lave and Wenger (1990) situated learning theory. Central to the theory of constructionism is the social nature of learning which is the best condition for learning where the learner is actively engaged in creating something that is meaningful to themselves or to others around them [18]. Therefore, this theory gives engagement a core position in the equation of learning. The United Kingdom's Department for Culture, Media, and Sport [19] has defined engagement levels, in a more structured model i.e. Attending (to pay conscious attention to content); Participating (to interact and thereby contribute to creation of content); Deciding (to make decisions about resources for content creation); and Producing (to create content which has a public impact). The model and other complemented concepts such as enjoyment, curiosity, and meaning-making serve as a theoretical framework for this exploration.
The use of games in class have been proven to provide enjoyable learning and promote attentiveness to learners [20]. Games play a significant role in changing a traditional teacher-centered classroom to learner-centered classroom [21]. Gamification has been proven to improve learners' motivation in formal and informal conditions [22][23][24]. Myhre [25] asserts gamification as a design approach which is utilized to encourage and build users' motivation and give them a better user experience.
Gamification in this study is defined as "using gamebased mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and solve problems" (p. 10) [26]. Utmostly, Huang and Soman [4] recommend a five-step process to develop effective gamification (Figure 1), which is adapted in this study in finding out evidence of learners' engagement.
Notably, the objective for integrating gamification in second language learning (L2) is to provide a more attractive and effective learning STEM experience. The gamification offers L2 learners to acquire L2 skills by solving STEM related tasks or challenges, and an opportunity to interact among them as it is implied in a social game [27]. When people perceive any form of social presence, they tend to respond in a natural way to feelings like happiness, empathy, frustration etc. Kapp [26] hence, lists 12 motivational interrelationship characteristics of game elements which are 1) 1. Abstractions of Concepts and Reality; 2) Goals; 3) Rules; 4) Conflict, Competition, or Cooperation; 5) Time; 6) Reward Structures; 7) Feedback; 8) Levels; 9) Storytelling; 10) Curve of Interest; 11) Aesthetics; and 12) Replay or Do Over.
Kahoot! has one of the ways to enrich educational assessment in the classroom and large-scale testing setting. The application offers the widely applied standard multiple-choice question type, which generally includes a prompt followed by a small set of responses from which learners are expected to select the best choice [28]. The task is readily scorable by a variety of electronic means and offers innovative features like the technological enhancements of sound, graphics, animation, video or other new media like response options or both. Given this definition, there are many ways Kahoot! can be innovative when revising content knowledge like STEM, and in which crowdsourcing can contribute to the innovations. This study then, has taken a small step in applying small-team, apprenticeship-like interactions that are currently common in research settings.

Research Design
This case study consists of a process that analyzes practices and maps them to accepted definitions to prove the empirical terms existence. It also has a participatory process that attempts to introduce a positive change through practical interventions i.e. crowdsourcing and gamification. Since this descriptive qualitative study has the natural alignment between teaching and learning and research processes, the researchers, instructors, and pupils were involved as participants and contributed to the research design to gather rich, reliable data.

Participants
The participants for 'Young Scientist Camp' were 48 secondary school pupils (P), aged 13 to 14 years old from two districts, in Pahang, Malaysia, and 20 facilitators (F) of undergraduate students. The project was organized to provide the pupils with positive learning experiences, and to increase their interest in STEM disciplines.

Procedures and Analyses
Using Huang and Soman's [4] five-step process to develop effective gamification (Figure 1), the evidence of the participants' engagement was collected at the fourth step (Identifying resources), and fifth step (Applying gamification elements). Prior to that, the first three steps were crucial in preparing and determining the participants' readiness for the activities. A quick survey was conducted as needs analysis before the lesson started, followed by observations of the crowd-sourcing initiatives and the gamification conducted by the pupils and instructors.
Video recording was used to capture the evidence of engagement among the participants during crowdsourcing and gamification activities. A game-based learning application, Kahoot! was used to test the crowdsourced contents using the language. Observations of the participants' attitudes were recorded to document their actions and reactions towards the activities. At the end of the activities, reflective sessions were carried out via three focus group interviews with five pupils and an instructor in each. Kapp's [26] twelve (12) motivational characteristics of game elements were used as a checklist in analyzing the participants' engagement during Kahoot!. The procedures are illustrated in Figure 2.
The video-based data were transcribed, coded and categorized using the eight steps of Tesch's [29] thematic analysis. The analysis was adopted to specifically aid the coding process and in categorizing themes. Cohen Kappa inter-rater reliability of coding was used to measure the agreement between two identified raters (experts). Using SPSS software, the Kappa value of 153 identified themes and sub-themes was identified i.e. κ value =.644 (Table 2) which was interpreted as 'Substantial Agreement' (0.61-0.80) according to Viera & Garrett [30] Kappa interpretation.

Step 1 to 3 (Understanding Audience and Context): Process for Effective Gamification
Before the crowdsourcing and gamification activities were carried out, the researchers did a quick survey on the participants' abilities in constructing simple questions in English language, and their prior knowledge about Kahoot! (step 1). This was to ensure the smoothness of the planned crowdsourcing and gamification activities in achieving the learning objectives of the camp (step 2). An application, Poll Everywhere, which can be accessed at www.polleverywhere.com was used to find out about these required knowledge and abilities from the participants. It was identified that the majority of the pupils needed some input in constructing English questions, and possessed no knowledge about Kahoot! Hence, some lessons about forming the questions and explanations about Kahoot! were carried out so that all participants would have an equal amount of knowledge and abilities to participate in the next activities. This also helped the researchers to structure or monitor the participants' engagement (step 3).

Step 4 (Identifying Resources): Evidence of Crowdsourcing
The pupils were assigned into groups of four to five members with two facilitators to guide them. They had access to the Internet to crowdsource relevant STEM contents. Prior to this camp, the pupils had participated in a series of STEM content workshops with similar aims i.e. to foster interest in STEM and motivate these young learners to further pursue STEM fields in their future studies. For this crowdsourcing activity, the pupils were required to form five questions in relation to STEM contents which were used to test other groups. Video-based data during the activity were analyzed using Estellés-Arolas and González-Ladrón-De-Guevara's [16] eight characteristics to determine the participants' engagement during crowdsourcing of STEM contents. The themes and sub themes of the data were coded and categorized according to the eight characteristics. Table 3 has some samples of participants' interaction, and Figure  3 is a snapshot from the video recorded data during the crowdsourcing activity. The results fulfilled all Estellés-Arolas and González-Ladrón-De-Guevara's [16] eight (8) characteristics of engagement during crowdsourcing activities. Both the pupils and facilitators, took turns to be the 'crowd' and 'initiator/ crowdsourcer, to complete the assigned tasks. Interestingly, the participants even interacted using English language spontaneously because they gathered more STEM contents in the language when crowdsourcing for it online. This is in contrast to learning the contents in a formal classroom context, which mostly depends on the language the textbooks typically developed in the first language. The pupils were put in a situation that necessitated them to use the language because most of the contents discovered online were in English language. This seemingly impulsive use of English was evidence even when they were interacting with each other in their groups. Apparently, crowdsourcing STEM contents made them realize the importance of English language. In later discussions, they admitted knowing certain specific terms in STEM disciplines was futile without knowing how to use it correctly. In other words, the pupils were excited to discover STEM contents from the online sources during the crowdsourcing activity, which has indirectly made them to be actively interacting about the contents using the target language. P14 -Let's construct the question together c).
P23 -Good! looks like a good question to ask 2.
Who it is b).
What they get in return for the work of the crowd a). F4 -where's the group leader (pupil) that we've appointed yesterday? Please lead the task b).
P22 -without the questions we can't proceed to the next task 3. About the process: a). The type of process it is b).
The type of call used c).
The medium used a).

P2 -I think the internet can help us b).
F31 -Everyone is encouraged to answer c).
P47 -Let's use the Microsoft first, then we transfer

Step 5 (Applying Gamification Elements): Evidence of Learners' Engagement
This final step gathered the evidence of the pupils' engagement. Firstly, the video-based data of the pupils' participation during the gamification activity were analyzed using the checklist by Kapp's [26]. Later, the four engagement levels model of CASE [19] were used to analyze the qualitative data collected from the focus group interviews during the reflective sessions.

Observation Checklist based on the Video-based Data
The evidence of motivating engagement during Kahoot! activity was video recorded, transcribed and categorized, using Kapp's [26] twelve (12) (Table 4). No evidence from the participants because input about STEM was provided prior to the program. Minimum control on the language use -fluency, rather than accuracy.

Goal √
The goal of Kahoot! were explained at the onset. The goals of the game gave the participants a purpose to be involved and focused; the facilitators then had a way to measure their participation. Clear goals were important to boost the participants' morale hence, the quality of their engagement with the assigned task. The researchers needed to be careful because it was demoralizing if the goal was too difficult to achieve, whereas the game could end earlier than the expected plan if the goals were too simple.

Rule √
Kahoot! itself is a game that has a set of defined rules, designed to limit player actions and keep the game manageable. It was identified to have two types of rules i.e. Operational Rules: the rules describing how the game was played (presented in form of explicit instructions along with implicit guidance); Instructional Rules: the rules the game developer wants the player to learn and internalize after the game is played. The rules gauged the participants to engage, educate, and form a mindset where there must be one correct answer for each question.

√
Kapp's definition of conflict is 'a challenge provided by a meaningful opponent'. There were competitive elements in Kahoot! hence, competition between groups and cooperation among group members were high. There was an explicit manifestation of cooperation within the game when the participants rushed to help other members in answering.

5.
Time √ Since Kahoot! is a form of assessment, time is highly prioritized and used as a motivator for player activity and action.

Reward Structures √
Plausibly, rewards were the ones that drove the learners' engagement and cooperation. The fact that the scores were shown immediately after each response, the result of top three most active and/or accurate players created mixed feelings among the participants. The competitive elements, however, must be toned-down to avoid stress and lack of concentration.

Feedback √
Throughout the game the participant received immediate results of their responses to the questions. The feedback after completing a task was in the form of cheering phrases, or of more motivating words, urging participants to not become demoralized when making mistakes, and more importantly was to keep the participants in the game, focused and on track. Though only six characteristics of motivating engagement were identified during the gamification activity, it is important to note that Kahoot! has all the characteristics to offer [28]. Unfortunately, time has caused limits for the other six criteria to be clearly proven in this project. The characteristics like Levels, Storytelling, Curve of Interest, Aesthetics, and Replay or Do Over need a longer duration to be clearly ascertained based on the participants' action and reaction. For instance, the frequencies of similar questions were suggesting certain patterns of interest among the participants. However, it could be clouded by the input they had just received before the camp, and the knowledge was still fresh in their minds. Still, six characteristics have proven the motivating engagement the pupils experienced from the activity.

Thematic Analysis of the Focus Group Discussion
During the reflective sessions, focus group interviews with five pupils and instructors were conducted. The questions were used to address the participants' engagement during the gamified crowdsourcing activities. It served to gather the overall information about the participants' engagement during the both activities. The qualitative data from the videos were analyzed using the four engagement levels model of CASE [19] as the themes. The analyses were agreed by the raters as evidence for the participants' engagement (see Table 2). The participants recalled what they had learned about STEM contents and the language forms and structure in constructing the test questions for Kahoot!. Resultantly, the data showed evidence of the participants' engagement from the beginning (attending) to the end phase (producing) throughout the process of learning STEM contents. Table 5 has a summary of the interactions from the interviews that took place with one of the groups. Table 5. Evidences of engagement from feedbacks of focus group interview (after the game)

Levels
The Recorded Observations 1.
Attending (to pay conscious attention to content) They were researching for English STEM terms using the Internet as well as asking the facilitators to confirm their findings.

2.
Participating (to interact and thereby contribute to creation of content) When asked about their contributions, the participants were excited to show their contributions which were the questions they formed in the groups before the questions were screened to be included in the games.

3.
Deciding (to make decisions about resources for content creation) They found it hard to choose as everyone was excited to test their peers' (from the other groups) knowledge of STEM that they had recalled. Some said they had to draw lots as to which questions should be selected, and others just let the facilitators choose.

4.
Producing (to create content which has a public impact) During the game, they admitted feeling proud to see their questions were used to the others and enjoyed listening to comments after the answers were revealed. The participants were anxious with their questions because they were unsure of the translated terms.
In overall, after analyzing the qualitative data, there are three aspects of learning in crowdsourcing and gamification that can be generally recommended for learners' engagement:

 Group Work Interaction
The nature of crowds and games involve more than one entity. Getting the learners to work in groups to participate in the activities urged them to interact. Hence, group interactions should be part of the teaching and learning process.

 Learners' Autonomy
Content and activities should be fluid and organic with minimal scaffolding from the instructors by letting learners to decide on its flow. Giving them the 'say' in learning increases the sense of importance and responsibilities in achieving the intended outcome of lessons.

 Process vs Outcome
Engagement through active learning has occurred during crowdsourcing and gamification activities. However, the externally driven behavior changes during these two activities are short-lived. As such, instructors should encourage learners to work towards achieving the outcome, but evaluation begins when the process takes place.

Conclusions
In this study, the majority of the pupils started with zero knowledge of Kahoot! but the later observations proved otherwise. They crowd-sourced their STEM knowledge and independently set up Kahoot! to test their peers. They experienced difficulties in translating certain STEM terms to English and in constructing questions. The observations also notified some technical complications which in turn, interestingly increased the pupils and instructors' engagement and participation in the activities. At the end, they reflected their experiences revising STEM contents in English via gamification as real and motivating despite its brevity and the difficulties. This study therefore, recommends learners' crowdsourcing especially when conducting content revisions.
In essence, crowd-sourcing allows the structure of lessons to be fluid and organic because users generated and designed their own content. To add, using crowdsourcing and gamification can drive learners' passion and create opportunities for learners to transfer knowledge to other learners. Crowdsourcing is not just about sourcing out tasks to the crowd; people's participation and co-creation of content could further tie a bound between them and stimulate long term engagement. Learners should be encouraged to participate and convince that their involvement is important hence, making them feel willing to contribute. Gamification on the other hand, can successfully make the overarching goal of crowdsourcing understandable and motivating for learners.