Evaluation of Nonverbal Elements in Mathematics Textbooks

This article comments on mathematics textbooks for lower secondary schools. Authors do not focus on texts in the books but on the nonverbal elements instead. A possible system of categories which enables mapping and classifying of such elements is suggested in this article. As a result of that, it is possible to evaluate and compare the textbooks which the authors have analysed. A pilot research of putting this evaluation system in use is also described in the article. The research ascertains whether the system of categories is useable to obtain sets of nonverbal elements suitable for further analysis.


Introduction
Recently there have been a number of publications on evaluation of textual content in textbooks [1][2][3][4][5].There are also researches that focus on nonverbal elements, specifically in geography and civics textbooks [6][7][8].Other researches deal with nonverbal elements from the multicultural and gender point of view e.g.Moraová [9] which we are also going to focus on in one of our following papers.
As a follow-up to the researches mentioned, we present an article whose aim is to deal with evaluation of mathematics textbooks from the point of view of nonverbal elements.With respect to the small scope of our pilot research we have focused on the topic of planar and spatial geometry as it is defined in the framework education program: mathematics and its application.The specific chapter we have been focusing on is a chapter on quadrilaterals.Further in this article we will present a suggestion of how nonverbal elements can be categorized and how we can compare mathematics textbooks using this categorization.This would enable us to evaluate the quality of a textbook with respect to a criterion other than textual content.
Nonverbal elements are introduced by many definitions, which do not vary from each other too much.A number of researches belonging to a research area called research on pictures [10][11][12][13] studied the interaction between a nonverbal element and the reader.We can distinguish four main research areas which differ in the way they look at nonverbal elements and the process of their interpretation.Janko [7] defines subareas focused on (1) the process of interpretation of an image; (2) the factors which effect remembering of images and creating mental representations of imaging elements; (3) research on nonverbal elements from the educational point of view and (4) description of aesthetic reactions and emotions in connection with nonverbal elements.
If we focus on publications defining qualities of nonverbal elements from the perspective of the way they convey learning contents we can find a number of different approaches.For example Goldsmith [14], who evaluates nonverbal elements from the perspective of literacy, distinguishes three categories of nonverbal elements' qualities: syntactical, semantic and pragmatic.Another research realized by Levin [15] focuses on the qualities of nonverbal elements from the perspective of cognitive strategies when dealing with these elements.The quality of a nonverbal element relates mostly to its ability to transform learning content into an understandable form.Levin also states that when it comes to cognitive processes we need to keep in mind that every pupil perceives a nonverbal element in a slightly different way based on their own cognitive characteristics.The nature of an educational goal, according to Levin, also effects the correct function of a nonverbal element while it's cognitive processing.
Ballstaedt [16], Peeck [17] or Evans [18] pay attention to topology of nonverbal elements with respect to specific qualities and determinant attributes.All these experts distinguish more or less similar qualities: type of nonverbal element, level of abstractness, relation of the element to a text and label of the element.Also Schallert or Levin and Mayer [19,20] were concerned with a quality called relation of the element to a text.Apart from these, Peeck [17] lists other qualities of nonverbal elements, e.g.There are researches that deal with different aspects of nonverbal elements in textbooks.Approaches based on quantification of all types are usually used for such surveys, without a deeper analysis, i.e. an analysis of qualitative characteristics of nonverbal elements.Our long-term goal is to contribute to the problem of nonverbal elements as structural components of textbooks and to try to describe their function as didactic instruments.
Pressley and Miller [21], among others, state what methods should be used for nonverbal elements' evaluation.They claim that it is necessary to make use of various methods and that such research needs to be carried out continuously because of constant innovations in textbook publishing.They also mention drawbacks of some research methods that are often used.One of the inadequacies is a shallow approach resulting in shallow outcomes of the research.It is hence necessary to analyse nonverbal elements more deeply with respect to their effect on the learning process.Isolated evaluation of elements regardless their possible interconnections and relations in a given textbook is seen as another inadequacy.
Particular methodological practices used for classification and evaluation of nonverbal elements based on its qualities like type of nonverbal element, level of abstractness, etc. (see above) are mostly of a quantitative character.Methodological practices used for their categorization are quantitative content analyses of inspected textbooks.Results of the researches are usually given as relative or absolute frequencies of the elements in given textbooks.It can be the quantitative character of the research methods that causes the shallowness of results which then do not cover for example an influence of nonverbal elements on educational situations when pupils or teachers work with a textbook [21].Nevertheless, it is very difficult to design a research that would focus on educational situations.For that reason there are also alternative methods focusing on pupils' and teachers' work with nonverbal elements.We can mention for example observation, interview, verbalization methods or projective techniques [22][23][24].
In this article we handle nonverbal elements mostly as visual representations.These allow us to illustrate difficult mathematical phenomena in case we want to highlight only important features without disturbing a complicated structure.In the world of teaching, nonverbal elements play an important part in pupils' creation of a concept of the given mathematical phenomenon [7].For that reason, choosing a suitable set of visual representations, hence also textbooks is crucial.
Nonverbal elements understood as visual representations can be static as well as dynamic images.Dynamic images cover animations or interactive geometrical shapes created with a computer, e.g. using the GeoGebra software.This article focuses only on static image which means all pictures in textbooks.

Research aims
A general aim of our research is to take part in clarification of nonverbal elements' role as structural components of mathematics textbooks.Our findings will be used as a basis for an analysis of nonverbal elements as didactical instruments.More specifically, our research aims are of a methodological and contextual character.
The methodological aim of our research is to design a system of categories that would allow us to classify types and qualities of nonverbal elements in mathematics textbooks and also to confirm reliability of this system.
The contextual aim is to evaluate the potential of nonverbal elements in selected Czech mathematics textbooks to support acquiring of new learning content and its understanding.This aim is to be accomplished via the following partial aims: 1) To carry out a comparative analysis of contemporary mathematics textbooks using the system of categories for nonverbal elements' types and qualities evaluation (survey 1).
2) To realize interviews with pupils focusing on their evaluation and cognitive interaction with nonverbal elements in textbooks (survey 2).This article describes a pilot research during our survey 1, i.e. suggesting and altering a system of categories to evaluate types and selected qualities of nonverbal elements in mathematics textbooks and also confirming reliability of the system.
With respect to our previous findings [25], which showed that pupils' conceptions and mental representations of geometrical shapes with all necessary features are very insufficient, we have intentionally chosen a wide range of textbooks for this pilot research including a fifty-years-old one, whose frequency of nonverbal elements is very low.
In our research on nonverbal elements [26] which focused on knowledge transfer we have discovered that mental representations are laden with drawbacks like formalism.In relation to this, another question comes to mind.How should we categorize nonverbal elements in mathematics textbooks and consequently, which categories are determining for forming mental representations?An aspect taken into account was the fact that pupils read without understanding and their perception of nonverbal elements is very superficial despite the fact that today's textbooks are full of these elements.Another aim of ours when testing older teaching materials was to observe such categories which are not present in contemporary books.

Materials and Methods
The research problem and consequent research aims were transformed into research questions.We distinguish questions from the preparative phase of our research and questions from the phase of realization.In the preparative phase we set a research question: What criteria do experts Evaluation of Nonverbal Elements in Mathematics Textbooks consider as crucial while evaluating nonverbal elements in mathematics textbooks?For the phase of realization we formulated two research questions: 1) What types of nonverbal element do occur in mathematics textbooks?
2) What qualities do these elements show?
In the preparative phase we carried out a literature search.We identified which qualities of nonverbal elements effect pupils understanding of mathematics.We also created a list of characteristics of nonverbal elements in mathematics textbooks.As a result we obtained a first version of system of categories.In this phase our main interest was to find out how pupils perceive some of the characterized types of nonverbal elements.In a survey [26] we showed that pupils perceive some of the types incorrectly.The results of this survey led us to a conclusion that working with nonverbal elements often causes problems to children and their mental representations are burdened with many deficiencies including formalism [26].

System of Categories
The system of categories we suggest in this article follows the system of categories designed for classification of nonverbal elements in geography textbooks, which was introduced by Janko [7].However, we are aware of the fact that in mathematics we classify the nonverbal elements in a rather different way.In geography it is important to show for example photos that highlight key features or interesting objects that characterize a concept.On the other hand, when we want to highlight important features of a square, photography is not a very suitable nonverbal element.Before we start with the evaluation, we need to divide all nonverbal elements in mathematics textbooks into a set of basic types, regardless how employable the element might be in a lesson.
The first two groups of nonverbal elements are dynamic and static nonverbal elements.As we have already mentioned, further on in this article we are going to focus on the static ones only.These we have divided into six basic types: geometrical shape (U), graph (G), tabular type (T), pictorial type (O), combined type (K) and other types (S).Each of these types is further subdivided into more specific categories.To each category there is a description to explain the features of corresponding nonverbal elements.In the legend there is an example to each category.This division is shown in a table, see Table 1.

Geometrical Shape (U)
This type of nonverbal elements is divided into three categories: precisely constructed shape, drawing and photography.It covers images where we can clearly identify geometrical shapes.
Precise shapes with highlighted features are labelled U1.It can be for example a precise shape of a right angled triangle with marked right angle, see Figure 1(a).Shapes without any important features marked (right angle, sizes of sides, etc.) are labelled as a drawing U2.This category covers also shapes which are not precise or drawn by hand but still result in a recognizable geometrical shape, see Figure 1

Graph (G)
This type of nonverbal elements covers all kinds of graphs and diagrams, category G1, see Figure 2

Tabular Type (T)
The tabular type of nonverbal elements includes all types of filling gaps of various shapes and styles, which follows a given rule (category T1).A prototype example could be a magic square or a chart with empty boxes to fill, see Figure 3

Pictorial Type (O)
The difference between this category of nonverbal elements and the category geometrical shape is that the pictures do not present any geometrical or mathematical element, which would accompany the text.An example is a picture of a car when students are to calculate its speed.Individual categories then classify only the technique used to create the picture (drawing or painting, photography).

Combined Type (K)
A combined nonverbal element is composed of more than one image.The K1 category covers pictures that can be separated but there is a connection between them.An example is figure with instructions for origami, see Figure 4 (a), or a schematic overview of quadrilaterals.The K2 category covers images that combine two or more types of nonverbal elements.For example photography of a butterfly (category O2) with marked axis of symmetry (category U1).K2 also covers dimensional sketches or photography, see

Other Types (S)
This type of nonverbal elements is divided into three categories.The S1 category includes highlighted mathematical symbols or marks, see Figure 5

First Pilot Research
In our first pilot research we used these categories to classify nonverbal elements in four textbooks.Each time we focused on chapters on quadrilaterals.The aim of our research was only to map the nonverbal elements in these chapters, using our system of categories.In the research we have been testing whether our system of categories includes all types of nonverbal elements and whether all these categories are defined unambiguously.For that purpose we have asked three mathematics teachers to classify nonverbal elements in the four textbooks.Another aim of our pilot research was to compare the textbooks as far as the relative frequency of the nonverbal elements is concerned.Using our system of categories we have classified these textbooks: A. Molnár [28] B. Novotný [29] C. Reichel [30] D. Cihlář [31] The reason for choosing these textbooks in particular was to discover all the possible types of nonverbal elements.Textbook A was published in 1999, i.e. before the publication of the Framework educational program.Textbook B was published more than 50 years ago and its concept is very different from contemporary textbooks.Book C is an Austrian textbook and textbook D was published in 2013 and hence follows the Framework educational program.There are various types present various frequencies present in each of these books.

Second Pilot Research
The results we gained in our first pilot research led us to a number of changes resulting in a second version of system of categories (see Table 2).In our second pilot research we evaluated a different set of textbooks and we focused on a different chapter called circle and disc.Book A in our second pilot research was published in 1997 and is intended for selective schools.Textbook B is from the USA and textbook C is intended for lower secondary schools.The reason for these changes was to observe whether it will affect the frequencies and types of nonverbal elements of our system of categories.The evaluators were four mathematics teachers who did not take part in the previous research and hence were not affected by it.All these changes were made to make it easier to test the reliability of the system of categories.The three following textbooks were tested in our second pilot research: A) Herman [32] B) Quast [33] C) Müllerová [34] There were two reasons to alternate the system of categories as well.The first reason was to make our system of categories more precise and better arranged.In our previous research, the "Geometrical shape" category was the most frequent one.For that reason we have divided it into three new types.On the other hand, this time we have not divided those categories which did not occur or occurred only to a small extent in the previous research.The second reason was a content evaluation of the potential of nonverbal elements in selected textbooks to support acquiring new educational content and help to create mental representations and understanding.
The aim of our second pilot research was to confirm the reliability of the system of categories so that we could create its final version.Using this final version we can carry out a high-quality analysis of mathematics textbooks.
The reliability of the system of categories was tested using Cohen's kappa coefficient [35] that allows us to measure level of consensus between two evaluators.The value of this coefficient considered to be satisfying is 0.75 and more (e.g.Fleiss, 1981).Some authors also accept values between 0.75 and 0.4 (e.g.Landis & Koch, 1977).The level of consensus for values of Kappa less than 0.4 is considered small or minor (e.g.Sim & Wright, 2005).

Results and Discussion
The results we gained in our first pilot research [25] can be seen in Figure 6, which shows the relative frequencies of single categories of nonverbal elements presented in textbooks A, B, C, D. They can be summed up into three conclusions: 1) The textbooks differ from each other a lot as far as the "Geometrical shape" type of nonverbal elements is concerned.2) Some of the categories of our system cannot be easily distinguished from each other.
3) The results handed by our evaluators were very diverse.The following study carried out in June 2015 is based on the results of our previous research.Our second pilot research was testing the new system of categories shown in Table 2. Our main focus was to confirm reliability of the new system.
The results of our second pilot research are shown in Table 3, Table 4 and Table 5.There is a single table for each of the textbooks A, B and C. The data presented there were collected using a content analysis method and coding.The results were obtained using descriptive statistics and nominal data analysis.Specifically, in columns called respondent (from 1 to 4) we display relative frequencies, which were calculated to determine the distribution of nonverbal elements in given categories.The last column displays Cohen's kappa which characterises intercoder reliability of a given type of nonverbal elements.It is obvious from the results that for each type of nonverbal elements the evaluators reached different level of consensus when using the system of categories.
Suggested system of categories designed to evaluate nonverbal elements and their educational potential was based on national and foreign researches described in the introduction of this article.The system was later enhanced with our own categories to reflect innovations in nonverbal components of mathematics textbooks.It turned out that already existing classifications cannot be simply adopted without any changes.Some adjustments were necessary so that it allowed us to evaluate nonverbal elements' qualities.At the same time, it was our goal to create such system of categories that would be comfortable to use and that would not expect evaluators' deduction.

Conclusions
Our aim was to show that in mathematics education it is important to develop pupils' visual literacy.Visual literacy can be seen as a condition for working with nonverbal elements but also as one of the aims of education.We evaluate didactical elements from the point of view of their illustrative qualities.We see nonverbal elements as a key part of nontext component of textbooks.The importance of nonverbal elements is obvious in situations when it is difficult to represent the learning content verbally.Nonverbal elements make it possible to present learning contents in a way that is closer to pupils and hence also more understandable and acceptable.
Given the extension of our research sample as well as the requirements related to reliability confirmation, we had three evaluators carrying out the pilot coding.These evaluators were instructed how to use the system of categories so that they were able to use it for nonverbal elements classification in analysed textbooks.After a successful confirmation of reliability the system of categories was used in the phase of realization.In this phase there were four evaluators using the system of categories for coding.It occurred that the evaluation of various types of nonverbal elements is a rather demanding activity.We discovered that our evaluators were capable of focused coding for only about 60 minutes.After this period of time the evaluators became tired and less focused which might have caused lower precision of coding.
After an analysis of empirical data obtained during reliability confirmation of the system of categories we came to a conclusion that the low level of consensus between the evaluators could be caused by a large number of categories of nonverbal elements.Based on these facts we have decided to design a modified system of categories.In our final research these changes will be done: The number of categories will be adjusted for the sectional system of categories for evaluation of the nonverbal elements and text connection.Together with alternations regarding the connection between nonverbal elements and a text there should also be other changes to the system to increase the intercoder reliability.The aim of these steps is to make the system more user-friendly and well-arranged.In general, we want our final version of system of categories to be a reliable research tool that can be used for mathematics textbooks' evaluation and development.
In an article Günzel [25] we came to a conclusion that the system of categories needed to be specified because of some overlapping categories.It also needed to be adjusted to make it possible to confirm its reliability.Another aim resulting from our previous outcomes was to focus on a larger number of chapters and textbooks which are to be evaluated using the system of categories.
The results of our second pilot research carried out with three different textbooks and four different mathematics teachers can be again summed up into three conclusions: 1) The most frequent category of nonverbal elements in all textbooks is "precise geometrical shape".The three types (inquiry based, explanation, constructional) further divide this category according to the function of the element in a textbook.Each of the types does occur in every textbook with "precise geometrical shape -explanation" being the most frequent one in all cases.
2) If we compare the results from all four evaluators it is obvious that they still differ a lot in all three cases (textbooks A, B and C).It shows that the system of categories is still not reliable enough, although the number of evaluators as well as textbooks was too small to enable a deeper statistical survey.3) Some of the new categories are still difficult to be distinguished from one another which is made obvious in case of the three types of the first category "precise geometrical shape" and their frequencies in textbooks according to our evaluators.Another ambiguous set of types are those from the category "photography/drawing" -mathematical and nonmathematical type.
The results of our research point us to yet another specification of our system of categories that would become reliable and whose types of nonverbal elements would be disjoint.We intend to use Cohen's kappa [35,36] to confirm the reliability of the system of categories.This test enables to measure the match of the evaluators using a match coefficient.Our research will also focus on more points of view of the nonverbal elements categories, e.g. a function of the nonverbal element in the textbook, its connection to a text, level of abstractness, its fidelity and the meaning of its label [7].Our long-term goal is to carry out a research on nonverbal elements in context with understanding.A number of specialists points out that it is not fully relevant to draw conclusions about nonverbal elements' educational potential based on their quantification only.Pupils apply various cognitive strategies when dealing with nonverbal elements and make use of different qualities when interpreting the meaning of the elements.Furthermore, we would like to find out how pupils evaluate the forms of nonverbal elements and how they understand phenomena represented by nonverbal elements.
(b).The last category, photography U3, covers only such pictures, which contain one or more geometrical shapes, see Figure 1(c) [27].

Figure 1 .
Figure 1.Examples of precise shape (a), drawing (b) and photography (c) (a).Number lines, vertical as well as horizontal, and various scales are labelled G2.An example of a G2 category is Figure2(b) that displays a scale on a thermometer in a chapter on integers.

Figure 2 .
Figure 2. Examples of graph (a) and vertical number line (b) (a).Default tables with data, which are already filled, are classified as T2, see Figure 3 (b).

Figure 3 .
Figure 3. .Examples of filling gaps (a) and default table with data (b)

Figure 4 .
Figure 4. Examples of separable pictures with a connection between them (a) and a combine of more types of nonverbal elements (b) (a).Category S2 covers maps, see Figure5(b).At last, the S3 category includes any other image, which does not relate to mathematics, e.g. a logo of a publisher, see Figure5(c).

Figure 5 .
Figure 5. Examples of highlighted mathematical symbols (a), map (b) and logo (c)

Figure 6 .
Figure 6.Relative frequency of the first three categories of nonverbal elements

Table 1 .
System of categories of nonverbal elements -first version

Table 2 .
System of categories of nonverbal elements -second version

Table 3 .
Relative frequency of single categories in textbook A

Table 4 .
Relative frequency of single categories in textbook B

Table 5 .
Relative frequency of single categories in textbook C