Divergent Activities in History Teaching: History Teachers Perceptions and an Initiative for Evaluation as Preparing Rating Scale

Assessment is one of the basic elements of teaching. However, there are not enough studies on the evaluation of divergent questions, particularly, in history education. Therefore, after reviewing the past and current literature on the topic, this study describes the design and implementation of a rating scale which can be used in the evaluation of divergent questions. The study based on action research examines history teachers’ views on divergent questions and how they evaluate the answers to such questions. The study was conducted with four history teachers who are working in different schools attached to the Ministry of National Education (MoNE) and 68 high school students in Turkey. Interview and document analysis were used as data collection tools. The study consists of two parts. In the first part of the study teachers' thoughts about divergent questions were determined; and the second part involves the generation of a rating scale that can be used in assessing the answers given to divergent questions. The result reveals that the rating scale provides the sufficient reliability and validity value, and can be used by history teachers in terms of divergent questions.


Introduction
The most basic reason for learning history is the feeling of curiosity about the past. This feeling means that people have the desire of learning about how they and communities came to current days in chronological process [1,2,3]. Besides satisfying the curiosity, history is one of the branches of social sciences, which liberates the mind with its contribution to reasoning. Also it has a moral content in terms of individual's self-identification and control [4]. Wineburg [5] points out that history is a wide area of study which contributes to depth-thinking. According to him thinking in history or historical thinking refer to a process of using critical and higher level thinking skills. Collingwood [6] underlined that history is very important in the context of past imagination which contributes questioning and thinking.
The meaning attached to the history and teaching history varies with time. The developments in social sciences after 19 th century and WWII have influenced the importance and aims of history teaching [7]. After this era the aims of history teaching addressed under two main headings like helping students to create an identity sensation and contribute to the development of historical consciousness [8]. Apart this general classification there are many other studies which deal with the purposes and benefits of teaching history [9,10,11,12].
History lessons existed for transferring culture and constructing national identity in the past. However, after WWII, history lessons were thought of as a lesson in which, thinking skills could be developed. Accordingly, students who replace themselves as historians in history lessons will be able to develop historian skills and enrich historical thinking [13]. So that the principles of historians habit of minds [14] and guides posts for historical thinking [13] have come to forefront. This understanding of teaching history can be seen post-2005 history curricula in Turkey. Identity transfer and strengthening citizenship highlighted in the sections of 'General Objectives of National Education' and 'General Objectives of History Education'. But, in the section of 'Special Aims of History Teaching' it is seen that transferring information and the skills put the foreground [15]. The skills categorised as 'basic skills' and 'historical thinking skills' and active learning supported in the context of improving learning history. These changes in curricula allowed teaching past knowledge but, it made teaching history as a tool which improves student's skills such as solving problems, creativity, critical thinking, emphatic thinking and democratic thinking. The new curricula gave emphasize research studies rather than reciting; focused on facts and casualty rather than singular events. This new approach gave more importance to the skills and understanding and led to improving students' knowledge and skills through activities. With this understanding activitybased examples have been prepared and made available to all history teachers and history teacher candidates [16,17].
Although progress has been made with activity-based approach at the level of curricula and teaching materials, history teachers could not keep up with these changes enough yet. As Diriöz [18] stated that history teachers seem to have limited historical thinking and reasoning activities in their classes.
One of the key elements that has improved the students' high-level thinking skills are open-ended questions. Open-ended questions which commonly used in education contribute to the development of skills such as problem solving, creative thinking, reasoning, generalization, and decision making [19,20]. On the other hand, open-ended questions have their own limitations such as ensuring the reliability of the scorer and establishing objective evaluation criteria [21]. Evaluation of divergent and creative thinking questions also faced with these limitations.
This study focuses on how history teachers assessed the answers to divergent thinking questions. The emphasis was placed on criteria that used to evaluate responses and established by history teachers in the scope of application. Beside this the study underlined the importance of using divergent questions and the difficulties in assessing their answers within history classes.

Divergent Thinking
Beyond the process of transferring information from the source to the learner, education is an inquiry about how the information is created and formed, in the period of choosing the right information. Interrogating sources, making comparisons, empathy and historical imagination are the basic elements that can be used in the process of creating knowledge in social sciences disciplines. In addition to these, it is possible to consolidate the structured information with discussion, questions and animation activities.
Divergent thinking is associated with critical and creative thinking has some sub-dimensions such as lateral thinking, reverse thinking, empathic thinking, and hypothetical thinking. Divergent thinking is to initiate extraordinary thoughts by expanding the boundaries of knowledge. The selections of different options, the creation of new and original ideas are the parts of the divergent thinking process [22]. Reaching unique answers based on available knowledge is the most important feature of divergent thinking. These answers are evaluated as new, original or creative [23].
Divergent thinking is also related to the lateral thinking expressed by Bono (1992). Lateral thinking is defined as the arrangement of information in an unusual way with an original point of view. And it involves approaching the issues with different perspectives by combining creativity, art, humour and intelligence. Lateral thinking is a form of thinking based on hypotheses created in the context of provocative thinking, speculation, and thinking in reverse. Any content that starts with second conditional expressions such as "if it were, if so, if I were in his place, if you were in his place etc." is a part of lateral thinking [24]. These elements are also associated with empathy and hypothetical thinking that take part in divergent thinking.
There is a contradiction with divergent thinking as Woodcock [25] notes. Creating or speculating about events which did not happen in the past is against what history should be about. History deals with things that really happened in the past, not the edited, created or speculated past. At this point there is a right questioning as "why should a person think something that is unrealistic or did not exist?" Although the history does not welcome the creativity or speculation they can be used in terms of teaching.
When divergent questions are used in the context of the causality they become a powerful tool in history teaching. With the use of divergent questions, events that are not actually happened are speculated and new holistic fictions are created. Thus, students are provided with their own explanation and reflection on how and why events in the past happened. Thinking hypothetically allows students to see differences and conflicts, and provides better understanding of opposite situations. It also facilitates learning by keeping historical dilemmas alive and contributes to the learning of causality [25,26].
People are considered to have developed divergent and hypothetical thinking skills around the age of 12 which corresponds Piaget's formal operational state. In this period of abstract and hypothetical thinking, reasoning and assumptions can be made and all possibilities can be considered systematically [27]. From the perspective of the educational process in Turkey, the divergent thinking skills of students arise from the beginning of the second stage of primary school. For the development of divergent thinking, students should be encouraged to build evidence-based hypotheses. Students should be provided with examples from the world of facts and fictions and be asked to defend their opinions as if they were correct [28].
An activity that can be used in the context of developing divergent thinking skills is to ask to the students questions. The divergent questions that lead to divergent thinking allow students to reason through analogies and comparisons across borders [29]. Divergent questions reinforce understanding of historical events and contribute to historical thinking. When we look at examples of divergent questions that can be used in history teaching in the literature, it is seen that there are various examples of questions under the headings of empathic thinking, lateral thinking, and thinking in reverse or hypothetical thinking [10,26].
If history science is considered to be an inquiry method, the relation of history teaching and divergent thinking becomes clear. Studying history provides opportunities to teach higher level thinking skills such as recognizing perspectives, making generalizations, discovering biases and 1270 Divergent Activities in History Teaching: History Teachers Perceptions and an Initiative for Evaluation as Preparing Rating Scale thinking creatively [5]. As Fordham [30] points out that one of the most important goals of history lessons taught in schools is to feed thinking. Using divergent activities in history classes contribute to depth-thinking and improve students understanding. And, it is one of the most influential aspects of the history of science to be able to think about what might happen if events happened differently. By using divergent thinking activities and divergent questions in history lessons students will be able to see the potential effects of possible differences [26]. This is reflected in the history teaching curricula in Turkey and the thinking skills are emphasized in teaching programmes [15].

Evaluation of Divergent Questions
There are not enough researches and sufficient studies on how students could be improved through divergent activities and their assessment in Turkey. The absence of common acceptance of the definition of creativity and divergent thinking is the main reason for this deficiency. Tests, evaluated by different experts and analytic rubrics are the main evaluation methods of divergent questions.
The first is the tests that measure creativity like Guilford and Torrance 's tests. But according to the Piffer [31] these tests are not functional for measuring and evaluating creativity and divergent activities. Creativity measuring tests are advanced tests and cannot be applied in a classroom environment. Fluency, objectivity, originality and enrichment of ideas are the main dimensions of these tests.
Fluency of ideas: This dimension concerns the production of ideas, solutions and options within a certain period of time. It is related to the ability to produce a large number of ideas, to use knowledge when needed, and to produce different solutions.
Flexibility of ideas: This dimension is related to the ability to look at events differently and to solve problems with different approaches. The basic questions to be used to determine this dimension are: How many different opinions were produced, and how many type responses were created?
Originality of ideas: It is about the ideas being extraordinary, unique and unorthodox. It indicates that the given answer is new and has not been expressed by others (or expressed by very few people).

Enrichment of ideas:
It is the dimension of giving details, making samples, expanding thought and gathering thought [32,33].
One of the most used techniques in the evaluation of open-ended questions is the evaluation of different experts and the creation of an average score. Increasing the number of experts increases the reliability of measurement and strengthens the results. However, different experts' evaluations create limitations and weaknesses in terms of time and labour. The subjectivity of experts is the second limitation to be faced in such an approach.
Analytic rubrics are one of the techniques that can be used to evaluate open-ended questions. However, the use of such rubrics has also limitations. An analytical rubric was developed to measure creativity in creative writing by Mozaffari [34]. Image, characterization, voice and story are the basic criteria of this analytical rubric. With this scale, attempts have been made to measure features such as the ability to imagine, to create a systematic way of thinking, and to tell stories. The content evaluated in the study was considered in four categories as, very good, good, moderate and weak. In addition, the standards for each category are given in a table.
In the international literature, it appears that these traditional assessment tools have been replaced by multiple and dynamic assessment methods. In this context, approaches such as subjective rating, self-evaluation and peer evaluation are seen as the foregrounds in measuring and evaluating creativity [25,34,35]. However, such evaluation approaches also have their own difficulties and limitations.
The history teaching programs in Turkey [15] also emphasized the dynamic evaluation tools. It is stated that students' work can be evaluated through anecdotal records, checklists and rating scales in the context of performance-based (authentic) evaluation. However, programs have not been given enough information on how to build these tools. Although these techniques that are not sufficient for evaluation of creativity and divergent thinking; they can be regarded as guides in the context of the criteria to be set for divergent thinking and creativity measurement.

Purpose and Method
The aim of the study is to determine the knowledge and opinions of history teachers about the use of divergent questions in history lessons and to create a rating scale to be used in evaluating the answers given when divergent questions are used. The problematic of the study is determined as "What are the things that history teachers pay attention in evaluating divergent questions." Sub-problems of the research are: 1-What is the level of knowledge of history teachers about divergent thinking and divergent questions? 2-What do history teachers think about using divergent questions in their lessons? 3-What are the things that history teachers pay attention to when evaluating the answers to the divergent questions?
The study is a qualitative research to examine an event in its natural environment. The subjects investigated in the study are what is the current situation regarding the use of divergent questions in history lessons is; and what are the problems that teachers experience in evaluation such questions' answers are? After these determinations, a working group was formed with the participating teachers. It has been tried to determine a rating scale that can be used in the evaluation of divergent questions in the direction of negotiations with this group. The researcher has been in constant interaction with the teachers during the study. Thus, opinions about what could be done to divergent activities were shared between the teacher and the researcher, and solutions to the problems encountered during the process were found. In this context, the research is a qualitative study in which the method of action research is used.
Action research is a method that used to make improvements in education and training. This method, mostly used by teachers, provides opportunities to improve the education. The action research method contributes to a better understanding of educational process and enables realistic solutions to problems. The method enables teachers to participate in scientific research by observing their own practice and thus provide an opportunity to fill the gap between theory and practice in the field of education [36,37,38]. Problems in action research are dealt step by step and the results are not generalized. The active participants became both subject and objects in the action research. So action research is a disciplined process of inquiry which is conducted by and for those taking the action. And the active participants affect the research process as they are affected by the results of the research [39,40].
Participants of the study were selected through convenience sampling which is a part of purposeful sampling. Convenience sampling refers to individuals who are currently available for the study and close to hand. The researcher using such a sample cannot make scientific generalizations about the total population [39,40,41]. This method saves time, money and gives practicality and speed to research. For this reason, it is one of the most used sampling methods in qualitative researches [37]. Four history teachers and their volunteer students are the participants of the research. The teachers are working in secondary schools affiliated to MoNE in Trabzon and Rize. Information on participant teachers is shown in Table 1.
In the first phase of the research interviews were held with teachers on the use of divergent questions in history teaching. Then these teachers were encouraged to do divergent activities like using a hypothetical thinking question in their classes, and then they were asked to evaluate the written answers. In this process, firstly the teachers were interviewed separately then the researcher and four teachers came together and review the written answers in order to establish the criteria for the evaluation of divergent questions.

Data Collection and Analysis
Initially, four history teachers were interviewed separately to identify their opinions about development of thinking skills and to use divergent activities in history classes. The interview form which includes five basic questions beside the personal data was prepared for the research. Drilling questions were used when it is needed to obtain the desired in-depth information. The interviews are held in the teachers' own schools and every interview lasted in 40-60 minutes. Each interview was recorded by researcher and these records were used during the writing process.
After the interviews, it was decided that the teachers would ask the student's a hypothetical questions in the context of divergent activities and take the answers in written. The hypothetical question that is decided to be asked to students is as "What would have happened if the Ottoman State had not participated in World War I?" Teachers were asked to take the given answers in the written form to evaluate and rate these answers. In addition, teachers were asked to set their own scoring criteria in the evaluation of divergent questions.
After this phase, the research group came together to discuss the criteria established in the context of the evaluations. In this group interview, the criteria created by the teachers for divergent activities were compared with the evaluation elements in the literature. As a result of these group discussions, a common rating scale used in the evaluation of divergent questions has been established.
Data diversity in the study was provided by taking evaluation criteria from different teachers. Examination of documents on students' written responses and opinions from different teachers about the evaluation of divergent questions contributed to the validity and reliability of the study by increasing data diversity.

History Teachers' Thoughts on Thinking and Divergent Questions
In the first phase of the study, the history teachers' opinions on teaching thinking and using divergent questions in history classes were determined. For this purpose, five basic questions were asked to history teachers who participated in the study. This part includes the questions which asked participant teachers and their answers.

What is Thinking and What is it's the Place in
Education?
The interviewed teachers have generally defined thinking as an exercise of mind. Expressions that stand out in the definitions made by the teachers are: analysing the events in mind, curiosity, abstract interpretation of concrete affairs, decision making process, designing process, to look at events from a distance or from above, internal perceptions about the external world, the process of perceiving and interpreting the events that surround us, mind gymnastics. Although participatory teachers have a view that thinking is a human characteristic, they do not have a clear knowledge about its context and functioning.
Participating teachers have considered thinking as a mental act from one side and as a talent from the other side. According to the teachers, as a mental act thinking is the process of questioning and reaching the conclusion. On the other hand it has also been seen as a talent as a design and editing field. The teacher C, who defines thinking as talent, emphasizes design and editing, looking at events from different angles, and stated that "thinking is a feature that can be improved rather than being taught." But, other teachers who participated in the study stated that thinking can be taught. According to them, thinking can be taught by making inferences from the realities. According to these teachers, stories, anecdotes, biographies, empathic and provocative questions, comparisons, and case studies contributes to teaching of thinking.

What is the Place of History Lessons in Thinking
Education?
Teacher A stated that, history lessons have unique opportunities for thinking in the context of change and continuity. According to him, focusing on similarities and differences in history contributes to students' thinking. He used the following expressions: "What did people eat in the past, what they eat now? How was it, how is it now? Why did this happen? Etc... You can operate these simple questions with every subject. Of course the curricula restrict us, but history still puts many questions and events ahead to think about. Our job is to make students aware of these questions and events, but students do not think so much." Similarly, Teacher B noted that history lessons have a rich content for enhancement thought. He stated that history lessons are an essay in which the mankind experience passes to the future generations and used the following expressions: "Every event that has been analyzed in depth contributes to the development of thought. Lots of questions arise inevitably when learning history, and some of these questions are as why did he do it, why did he so, if I were, and etc... Seeking answers to these questions leads people to think." Teacher C, emphasized that asking questions in lessons will improve thinking. He stated that "Focusing on 'why' and 'how' questions prompted students to think." He also emphasized the importance of the level of students and stated that it would be beneficial to have open-ended questions in examinations. According to him, history has the potential to contribute to thinking, but this can change according to the situation of teachers and students.
Teacher D stated that interpreting historical events and examine the attitudes of persons or states against these events would contribute to thinking. "We can contribute to students' thinking if we go through examples like Fatih's dream of the world state and Atatürk's Republican ideals during the war period." The teacher D who used these expressions stated that historical events and historical personalities inevitably lead people thinking. According to her, popular history topics are more interesting and they can be used for students' thinking. She stated that: "Unfortunately, the historical events in the series are influencing the thinking of the students more than the history lessons. So, the series sometimes can be more effective than the teachers in history teaching. Students can better understand the role of women during the ottoman period via popular series; or it can be even more noticeable when Magnificent Suleyman's children are watched in series or films." She also stated that classroom teaching should be organized in the context of thinking education.

What Activities do You Organize in Your Lessons to
Contribute to Students Thinking and Develop Their Thinking Skills?
Teacher A stated that he uses the techniques such as discussion, brain storming and six-hat thinking together. The teacher added that these techniques can be productive when the students are interested but students usually do not show interest and the desired result does not obtained from these techniques. Learners who read more can improve their language and commenting skills and this can raise the quality of discussion. From this reality, teacher A stated that he recommended books to students. He also pointed out the case studies, historical narratives, anecdotes, and films which are creating opportunities in the context of thinking. According to him such extensions also enrich the class discussions and contribute to students' thinking.
Teacher B stated that emphasizing empathy over case studies is useful for students' thinking. Teacher B used the following expression regarding empathy: "From the beginning of Turkish history in Central Asia to Armenian issue reviewing and analyzing events contribute students' thinking." He explained his argument with the example of Timur and Yıldırım's correspondences before the Ankara War. He emphasized that empathizing Timur and Yıldırım will develop students' multiple thinking skills. Teacher B also emphasized that questions are also important for thinking, and expressed the need for high-level thinking questions in the context of Bloom's taxonomy.
Teacher C has stated that he encourages students to think through comparisons, analogies, concepts and images. He also added that he tried to use different expressions instead of expressions drawn directly from the textbook because the implications of the new concepts and visuals have had a positive effect on thinking through beneficial discussions. Teacher C also drew attention to the importance of the questions and stated that students should be asked the right questions in order to recognize similarities and differences.
Teacher D, who expresses that popular history and out-ofschool teaching have opportunities to improve thinking in history classes, stated that she uses discussions and questions as much as possible. She also stated that "students will be attracted by asking questions and discussions will lead them to think deeply, so such activities should be used more often." She added that she uses such activities as much as the interest of students continues.

What is Divergent Thinking? Do You Use Divergent
Activities in Your Classes?
Another question asked to interviewed teachers is whether they include divergent activities in their lessons. As a precursor of this problem teachers were asked about "what is divergent thinking?" Teachers have stated that they do not know enough knowledge about divergent thinking. Two of the teachers have defined divergent thinking as interpreting and original thinking. Definition of teacher A is as "Divergent thinking should be related to interpretation and high-level thinking." Likewise teacher D also defines it as "a unique thought which is not stereotype." The researcher gave brief information about divergent thinking and divergent activities to the teachers in the context of the literature. After the description, it has been determined that without knowing it the teachers have included divergent activities in their lessons. Teacher A stated that they were handling some events with some techniques such as brain storming and six hat thinking which could be evaluated as divergent activities. He exemplified this with the establishment of printing house in Ottoman Empire. Related to the issue he expressed that they discussed the development of printing house from different groups' viewpoints such as calligraphers, Muslim scholars and, Christian or Jewish minorities. In a similar vein, he stated that the Tanzimat and Islahat fermans and Mahmut II's reforms are some other topics that can be used in this context.
Teacher B stated that as an example to be considered as divergent activity he asked to his students to do interviews with historical figures like journalists. Teacher C indicated that due to the intensity of the program they did not have the opportunity to do activities as they want but, they made some comparisons on historical events which can be regarded as divergent activities. In the context of these activities, students make comparisons about the establishment and demolition of states and they use how and why questions instead of who and what. Teacher D stated that they use discussion to reveal unique and original thoughts. She or sometimes the students use questions to initiate these discussions by using popular historical elements such as articles from newspapers or scenes from historical series. For example: "a student asks about the outfit of a historical personality in the series; then we discuss it as if they were like that or how could they be dressed and what were the clothes made from in the past..."

How do You Evaluate Divergent Activities if You Do?
The last question that asked to interviewed teachers is about how they assess divergent activities. The teachers, who use divergent activities in the context of various techniques, do not find themselves adequately to evaluate these activities. The answer given by teacher A is as follow: "In the brain storm method you do not give importance to the truth. More different ideas and answers are the best, and I do not evaluate these activities. At the end of the activity I just repeat the basic ideas that emerged during the activity." Teacher B stated that he reads students writings and if he encounters good performances he use some marks for rewarding the student. If the writing is a rewarding assignment, in some cases, it shared to the classroom, which is also a kind of prize. Teacher C and D stated that they do not evaluate the discussions or comments which held in the context of divergent activities.
As a result of the interviews, it was observed that history teachers providing students some activities for improving their thinking skills such as multiperspectivity and creativity but these attempts are not done with full awareness. The teachers are not evaluating the divergent thinking activities which performed in their classes and this means that there is no expectation from such activities. But Assessment is necessary in order to determine the achievements of the students and to see if they have achieved the intended success. For this reason every activity should be evaluated by formative or summative assessment techniques.

Generating Evaluation Criteria and a Rating Scale
After the interviews with teachers in the context of thinking education and divergent activities, it was decided that the students should be asked a hypothetical question and the answers should be taken in writing in the context of divergent activities. Teachers stated that a problem related to World War I should be used because it is a subject in 10th and 11th grade curriculum. Based on the teachers' views it was decided to use the question determined by Tokdemir and Erol [42], which students can answer, and which was determined as "What would have happened if the Ottoman 1274 Divergent Activities in History Teaching: History Teachers Perceptions and an Initiative for Evaluation as Preparing Rating Scale Empire had never entered World War I?" Teachers were asked to evaluate the answers given to this question and establish evaluation criteria when doing the assessment. The research is a voluntary work and was conducted in the spring semester of 2016-2017 academic year with the participation of 68 students and four teachers from four different schools. It was observed that the teachers who participated in the study marked the answers of the students between 45 and 90. Teacher A has given grades in the range of 45-75 points and become the teacher who used the lowest grades. The grade given by teacher B is the highest grade in the range of 70-90 points. Teacher C scored students' answers with 50-80 grades while teacher D scored them between 50-85 points.
After the implementation, the teachers who participated in the research and the researcher dealt with the evaluation criteria which established during the divergent activity as a group interview. In this group interview, it was asked to the teachers to explain the criteria they used when evaluating students' answers. The criteria created by the teachers and the dimensions they are associated with are shown below in the Table 2.
When the table was examined, it can be seen that the teachers have established similar criteria but use different scoring. First inference from the table is the analogy of the process of evaluation of divergent thinking to the process of creativity and discussion. When we look at the criteria that teachers make, it is seen that the created criteria have features that can be used in evaluating creativity and also in discussions. Criteria such as establishing original thoughts, advocating argumentative arguments and counter arguments, and using the concepts correctly are both necessary for creativity, divergent thinking and discussions. This reveals a reciprocal link between divergent thinking and creativity and discussion. The second derivation is that the teachers considered the skills of using language as a criterion.
In the group interview, the teachers explained that the evaluation option by different experts can be applied to the schools where more than one history teacher works. Teachers also have stated that although this is theoretically possible, they do not go to practice. According to the teachers, even though open-ended questions are used in the joint examinations in the schools, there is no practice in the way of evaluating the same student's answer by more than one teacher. They also emphasized that self-assessment and peer evaluation techniques will be limited in the context of evaluating divergent questions. Teachers A and B stated that such practices could be applicable to the schools with more qualified students, such as Social Science High Schools but teacher A, who is working such a school, did not fully agree with this opinion, and stated that his students are more likely to be exam-oriented and their tendency to evaluate each other objectively is weak.
Teachers are more likely to apply classical assessment methods such as tests and open ended questions which improve students' high level thinking skills. They do not use the innovative or authentic evaluation methods. But they believe that, evaluation techniques such as rating scales and checklists could be used for evaluating divergent questions.  15 15 In the group interview, teachers stated that it would be more functional to make individual scoring with the joint rating scale, rather than using collective scoring in the evaluation of divergent questions. Participant teachers have identified the criteria that can be used in evaluating divergent questions, taking into account both the information contained in the literature and the criteria they have established in the context of the practice in the research. Six of these criteria were related to the quality of the content, while the last two were related to the use of language and the grammatical structure. These criteria and their scores are shown in Table  3.
According to the teachers participating in the study, these evaluation criteria will be applicable as well as minimizing the subjectivity. Moreover, these criteria can be used despite the intensity of the program and the crowd of the classes. The rating scale that participant teachers use to establish these criteria is shown in Table 4. In the generated rating scale, students' responses were categorized as 'excellent ',' good' and 'poor ', and minimum criteria were set for each category. Participant teachers have applied and tested applicability of this rating scale on the student responses which they have previously assessed. The results mostly were found in categories of 'good' and 'poor' as expected and a limited number of answers were evaluated in 'excellent' categories.  Using historical events and concepts The generated text contains maximum historical events and actors. The generated fiction has a versatile content that include conjunctions such as 'because', if it were', 'whereas', 'hence' etc. The text includes historical concepts and actors in a possible context that is not actually real.
The generated text contains at least a historical actor and an event which is related to the actor and used in a limited way. The text is developed in a consistent and reasonably manner in There is a limited number of historical concepts in the text.
The historical event and the actor that involved in the generated text are irrelevant, or actors are never included. Text does not serve for any purpose and does not constitute integrity between subject, title, and result.
Creating and support arguments The generated text contains maximum number of arguments and these arguments form a logical whole in the context of their relation to each other. This generated whole is rational and acceptable. The arguments are supported in line with the conditions of the considered historical period The generated text contains at least a reasoned argument. This argument is appropriate to the circumstances of the events of considered historical period There is no supported argument and consistent thought in the generated text.

Using creative ideas and expressions
The generated text includes maximum of different ideas. These ideas are presented with original, surprising and provocative expressions (Reasonable and relevant expressions should be established; otherwise, they will be excluded even if they are original.) There is a unique idea and approach that will influence the reader. This approach is appropriate in the context of subject matter and logic.
There are no creative expressions or the expressions used are absurd and irrelevant in the context of the subject and time.

Conclusion of ideas
Thoughts are concluded within a reasonable framework. Judgments and decision-making expressions take place in the conclusion section. The result is compatible with the title and content.
Thoughts are concluded within a reasonable framework. The result is compatible with the title and content.
The result is not compatible with the title and content.

Associate with current events and making generalizations
The generated fiction is associated with the current era in the context of change and continuity. Generalization and examples are used for this association.
At least one example and a generalization are used for the association of generated fiction and current era.
There is no association with generated fiction and current era.

Creating futuristic implications and projections
Some results are extracted from the generated fiction as it were real and some predictions about the future are created that based on these results.
Some results are extracted from the fiction as it were real but there is no prediction about the future.
There are no futuristic projections.
Writing skills (spelling, punctuation, and grammar) There are no errors in terms of grammar and writing rules. Academic writing skills reflected in the The generated text is sufficient in terms of grammar and writing rules.
There may be minor errors in exceptional cases There are grammatical and spelling mistakes.
Creating clear and understandable sentences and paragraphs A fluent expression which includes short and clear phrases is used. Paragraphs have integrity and there is a strong context in transitions between paragraphs. The narrative is enriched with metaphors and analogies.
A fluent expression which includes short and clear phrases is used. Paragraphs and the fiction have integrity.
The narration is ambiguous or manipulated with unnecessary and exaggerated expressions.

1276
Divergent Activities in History Teaching: History Teachers Perceptions and an Initiative for Evaluation as Preparing Rating Scale When the evaluation scale of the teacher is examined it can be seen that it is in a usable structure. Teachers have included the basic criteria of divergent thinking in their criteria. The first criterion emphasizes the link with history lesson and historiography, and the second, third, fifth and sixth criteria relate to divergent thinking but they also have strong associations with creativity, discussion and high-level thinking skills. While the fourth, seventh and eighth criteria seem to be related scientifically in general terms of academic writing skills.

Conclusions
Although history teaching curricula emphasize high-level thinking skills in the context of historical thinking skills, the development of these skills depends on the practice of the teachers and is not at the desired level. History teachers are acting with the influence of the educational system which is test and result oriented in Turkey. The intensity of the program also limits the ability to develop thinking skills.
One of the problems faced by the teachers in the limited studies on the development of high level thinking skills in history teaching is that it is not known how such studies can be evaluated. In this study, participant history teachers were tried to determine their views on divergent activities, teachers were asked to evaluate written responses to divergent questions after the implementation, and they were asked to form evaluation criteria when doing this evaluation. It has been seen that the evaluation criteria used by the participating teachers such as putting out new ideas, using creative ideas, associate with current event, making original comments etc. can be used in the evaluation of divergent activities and can be associated with the tests that measure creative thinking. However, it was seen that some of the criteria could not be used in evaluating creative and divergent activities. For example the criterion as 'using the actors and the condition of the period correctly' cannot be used as an evaluation criterion because it prevents the construction of manipulated reality.
With the research it has been determined that participant teachers have limited theoretical knowledge about divergent activities and their evaluation. During the research the information in the literature on how to evaluate divergent questions has been reviewed and an analytic rubric was prepared for the evaluation of written materials related to divergent activities considering the criteria used by the participating teachers. The prepared rubric has been test on students' written answers and founded functional. It was also found that the prepared rubric was compatible with the studies in the literature on the evaluation of open-ended questions such as İlhan [19], Doğan [21], Piffer [31], and Mozaffari [34].
Divergent activities require a new and original perspective other than the usual, but they serve to reach the desired point in the context of the results. Bono [21] likened this to 'trying to drop fruit by shaking a tree rather than sitting under a fruit tree and waiting for a fruit to fall'. It is possible to increase mental activity, to reach different solutions, and eventually to feed creative and critical thinking skills with a provocative thought or a question. The divergent activities to be formed by manipulating a past event in history teaching will contribute to the students to make correct generalizations about the historical events and the present, to make judgments and to reach the correct results. In this context, using more divergent activities in history classes and training the teachers about evaluation of divergent activities emerges as the recommendations of this study.