Dependency on Verbal Versus Graphic Representations among Second Language Learners When Reading Illustrated Expository Materials

The use of graphic representations is ubiquitous and essential in science-related domains. Research into learners’ reading of materials that incorporate verbal and graphic representations (i.e. multi-representational materials) however suggests that graphics receive very little attention compared to text. Such a tendency is not ideal as comprehension of multi-representational materials requires learners to attend to the graphics and integrate information contained in these representations with those in the text. This field of research, however, has focused primarily on native language learners. Since the number of people learning in their non-native language is growing rapidly, it is crucial to understand how they engage and learn with these materials. The present paper is part of a larger study that focuses on the multi-representational reading of 42 Malay ESL undergraduates. Their eye fixations during the reading of a science passage and an accompanying diagram were collected and analyzed. To our knowledge, no such study has been undertaken in the Malaysian context. Hence, the findings will be significant contributions to this field. The results indicate that similar to the reading of multi-representational texts by native speakers in other contexts, the multi-representational reading by the Malay ESL undergraduates was also heavily text-based.


Introduction
Reading materials used in science-related domains contain an abundance of graphic representations. Graphs, tables, diagrams, photos etc. are used to accompany verbal representations for a myriad of purposes, be it affective or for more pedagogical functions. In the science domains, visual representations help learners understand new and abstract concepts, solve problems and develop a deeper understanding of complex scientific domains [1][2][3]. Given this wide range of purposes, it is not surprising that graphic representations used in science-related textbooks have multiplied in numbers and variety in recent decades [2,4].
Given the above trend and the pedagogical values of graphic representations, it would seem logical to assume that learners would utilize them accordingly in their reading of scientific texts. However, research studies have revealed that learners tend to spend more time reading the verbal representations rather than graphic representations when both are provided [5,6]. Predominant attention on verbal representations (i.e. text in this study context) is not harmful unless the readers focus so much attention on the text that they ignore the graphic representations which are included to aid comprehension [7,8]. However, all these studies have been on learning of scientific materials by native speakers and not much has been done in this field on English as a second language (ESL) learners. The present paper will attempt to rectify this issue by presenting preliminary findings from a larger study that examines the reading of an illustrated scientific material, which consists of both verbal and graphic representations, by a group of ESL learners.
It would be interesting to find out whether these learners manifest the same tendency as native speakers who are reading in their native language. Given that they are reading the materials in a language that they are less proficient in, it is plausible that they may utilize different strategies. This paper intends to investigate this by examining the learners' reading behavior. More specifically, eye fixation data would be collected and analyzed to examine the attention distribution of a group of Malay ESL undergraduates between text and graphic.

Learning from Multi-Representational Materials
Graphics can play a number of functions when used alongside text. According to Levin,Anglin and Carney's [9] functional taxonomy, graphics can play five different functions in relation to text content, i.e. decorative, representational, organizational, interpretational or transformational. A graphic can merely decorate a page when it has no or very little relationship to the text. As such, this type of graphics has little pedagogical value. Representational, organizational and interpretational graphics in contrast are pedagogically relevant. A representational graphic may correspond one-to-one or partially with a text to concretize the text content. An organizational graphic, on the other hand, can organize a complex text into a more coherent structure, whereas an interpretational graphic serves to make difficult text more comprehensible to learners. These three types of graphic representations are very useful in presenting scientific materials.
The language commonly used in scientific materials, according to Fang [10] has distinctive lexicon, syntax and semantics and may contain technical vocabulary, lengthy nouns and complex sentences. Considering this difficulty, it comes as no surprise that learners encounter many difficulties in comprehending and learning them. The graphic representations as described earlier can help overcome these difficulties if used appropriately. In addition to making scientific texts more concrete, coherent and comprehensible, they also serve other pedagogical roles. Reference [1] further suggests that graphic representations can help in the construction of a deeper understanding of a particularly complex domain. In chemistry, for instance, competence in interpreting and manipulating graphic representations in addition to text is a particularly valued skill [11].
Since graphics and texts are often used in conjunction in the learning of scientific concepts; learners need to study both forms of representations adequately in order to fully exploit their benefits. Indeed, the proponents of multiple representations believe that integration of verbal and graphic information can help learners construct a coherent mental model of the subject matter [1,12,13]. A number of studies have also found that text-graphic integration is positively associated with performances on different types of learning outcomes [12,14]. Reference [12] for instance identified that in addition to integrating verbal and graphic representations longer and more frequently, inspecting graphics for a longer time upon initial encounter (i.e. first-pass fixation duration) with them was ideal for learning.
It is further discovered that insufficient attention on graphic representations impede effective learning of science-related domains. Neglecting graphic representations may lead to difficulties in understanding new concepts, consequently jeopardizing learners' chance of constructing a deeper understanding of their complex domain [1][2][3].
Despite these findings, many research studies have found that graphics do not receive adequate amount of attention during learning. Reference [6] for instance reported that their participants spent only 65.7 seconds on an animation accompanying text. The latter in contrast was fixated for 112.4 seconds. In their subsequent experiment, the text was attended to for 131.2 seconds compared to 41.9 seconds on the animation. In this subsequent experiment, the text was attended to three times more than the animation (131.20 versus 41.90 seconds). In an earlier seminal study, Hannus and Hyönä [7] observed that their participants only spent six per cent of their total study time on multi-representational stimuli. These studies were all conducted on first language learners. It is indeed a worrying trend and this paper is interested to find out whether this trend extends to second language learners.
The reasons for this phenomenon are also worth looking into. Multimedia researchers have forwarded several reasons why graphic representations are often neglected. Firstly, the processing of visual information in graphic representations is normally easier and faster than the processing of verbal information in verbal representations [15,8]. This can easily make the readers think that they have fully understood what a particular graphic is trying to convey, leading them to spend less time on it. Secondly, the graphic and the verbal representations may contain redundant information. When this is the case, learners would tend to focus on only one representation [16]. It has been argued that the learners would focus on the more familiar representation [17].

The Study
Several assumptions with regard to reading behavior can be made when readers are not native to the language being used. One assumption would be that similar to the native readers, the non-native readers would also exhibit a heavily text-based reading approach. It could be possible that they would pay more attention to the text rather than the picture because they are having difficulties with the text, which is written in their second language (L2). As pointed out by Scheiter and Eitel [18], higher fixation values, be it in terms of frequency or duration, are normally associated with comprehension difficulties. In addition, late unbalanced bilinguals such as the ESL learners in this paper, who acquired their L2 during late childhood tend to read in their L2 less fluently and hence may struggle to understand the text [19]. They have also been observed to make more frequent and longer fixations when reading in their L2 than in their first language (L1). Another point to support this assumption can be deduced from the body of research examining how a second language is learned with the support of multiple representations. In this literature, exposing language learners to graphics in addition to the verbal L2 input does not deter them from paying more attention to the latter [20,21]. This is also evident in the case of videos with subtitles [20,21] and passages with glosses (i.e. verbal and/or visual annotations for key vocabulary items) [22]. In fact, the tendency for text-based reading is also evident in this literature. This tendency is also displayed when readers are reading narratives in their L2 [23,24].
Alternatively, comprehension difficulties could also lead to more and longer fixations on the graphic. To illustrate, upon encountering unfamiliar words in the text, a reader could turn her attention to the graphic in search for clues to their meanings. In this way, the graphic acts as an alternative source of information to the unfamiliar vocabulary items [1]. Mayer and colleagues also proposed similar ideas suggesting that graphic representations via videos could assist comprehension of narration in the L2. They proposed that the images in the video can help learners form connections between the verbal and graphic representations of difficult words thereby resolving vocabulary issues more quickly. If the graphic was indeed used an alternative source of information, then the non-native readers can be expected to be less heavily text dependent as there might be a number of long fixations on the graphic.
Given these arguments, proficiency in the L2 appears rather crucial in determining the L2 learners' reading behavior. This variable will therefore be examined further in this paper. Since several hypotheses can be formulated with regard to the L2 learners' reading behavior, an exploratory study was deemed appropriate to fulfill the objective of understanding how ESL learners read an illustrated expository material in English.

Materials and Methods
As previously mentioned, this paper is part of a larger study and will report its preliminary findings.

Participants
The study was conducted on 42 first-year Malay undergraduates majoring in social sciences. They were all native speakers of the Malay language and reported English as their second language. The sample consisted of learners with different levels of English proficiency. English proficiency was determined from the participants' scores in the Malaysian University English Test (MUET), a national English language proficiency test for university admissions. Although MUET has six bands, the ESL learners in this study were those who scored bands 3 and 4 (average proficiency), and 1 and 2 (low proficiency) in the test. In total, there were 15 average (AP) and 27 low proficient (LP) learners of English. All participants had normal or corrected-to-normal visions and received an inconvenience allowance for their participation.

Material
The stimulus used in this study was a neuroscience passage taken from the Science American magazine [25]. It described the pathway of human's fear response system and contained 160 words. Its readability according to the Flesch reading ease was 40.6, and its Flesch-Kincaid grade level, 12.5. On the whole, these values indicate that the text was rather difficult. In addition to the text, a diagram depicting the pathway and relevant parts of the brain involved was included. Based on Lohse, Biolsi, Walker, and Rueter's [26] classification of graphic representations, this diagram can be classified as a process diagram, as it visualizes the processes and relationships among objects in the graphic representation. This diagram also serves an organizational role in relation to the text as it helped organized the complex textual content into a more coherent, structural framework [9].

Procedure
Data collection took place in an eye tracking laboratory and all participants were tested individually. Before obtaining their informed consent, all participants were briefed about the study and its purpose. They then completed a background questionnaire before being exposed to a trial stimulus. A trial stimulus was presented to familiarize the participants with the research procedures involved. Similar to the actual stimulus, the trial stimulus also consisted of a science-related passage with an accompanying diagram. Following this trial, the actual stimulus was presented. The participants were asked to read it at their own pace. After reading, they were asked to answer five yes/ no comprehension questions. Comprehension performance however was not examined in this paper. All participants took about 20 to 30 minutes to complete all procedures.

Apparatus and Eye Tracking Measures
The EyeNTNU-120 eye tracker with a sampling rate of 120 Hz was used in this study. For the purpose of eye tracking analysis, the stimulus was divided into six areas of interest (AOIs); five text AOIs and one graphic AOI. Since this paper focused on the distribution of attention across text and graphic, the multiple text AOIs were grouped together for analysis as one text area. The total surface area occupied by the graphic area was less than half of the area occupied by the text. This called for the use of eye tracking measures expressed as proportional values because the text area, which is larger than the graphic, may be attended to longer simply on the basis of its size. The proportion of total fixation duration (TFD) was used to measure participants' distribution of attention across text and graphic areas. TFD totals up the time for all fixations on a particular AOI. TFD on text area and graphic area were the main focus of this paper therefore proportions of TFD on each area were gathered and analyzed.

Data Analysis
Prior to analysis, the eye tracking data were first screened to ensure that low precision or invalid data were not included in the analysis. As a result of this screening, 14 participants were removed from the sample, leaving only 28 for a subsequent analysis. In total, there were 20 LP and 8 AP participants.

Results
The means and standard deviations of the proportions of TFD on text and graphic areas are presented in Table 1. As can be seen, the text area received much longer fixations than the graphic area. A paired sample t-test carried out indicated that the difference was 0.88, t(27) = 28.30, p < .001, d = 10.99. The above values suggest that the Malay undergraduates paid very little attention to the diagram that organized the complex content of the text. 94 per cent of their reading time was dedicated to the text area alone. A closer look at the descriptive statistics indicate that 11 of the participants (39 per cent) did not even inspect the graphic at all and those who did, only spent an average of 991 milliseconds (i.e. less than one second) inspecting this area. This is in stark contrast with the statistics on the text area which received an average fixation lasting 8,436 milliseconds.
We were also interested in whether or not the participants' level of English proficiency would be related to their reading behavior. We therefore compared both groups' attention on text and graphic areas. The means and standard deviations of the proportions of TFD on text and graphic areas are presented in Table 2. A Mann-Whitney test carried out indicated that there were no significant differences in the way that both groups attended to either the text or the graphic area. For proportion of TFD on the text area, U = 104, z = 1.26, p = .24, r = 0.24. For proportion of TFD on the graphic area, U = 56, z = -1.26, p = .24, r = -0.24. Both the LP and AP participants, despite their varying degree of English proficiency, spent similar amounts of time on the text and graphic areas of the stimulus.

Discussion
This paper has presented preliminary findings of a study that explores how Malay learners read an illustrated expository material in their L2. The learners' distribution of attention between text and graphic areas was particularly examined.
The eye fixation data revealed that similar to reading an illustrated expository material in a native tongue, the Malay ESL learners' reading of a text in L2 was also heavily text-based. They spent 94 per cent of their reading time on the text area alone, leaving only 6 per cent on the graphic. This suggests that the participants did not pay sufficient amount of attention on the diagram accompanying the heavily attended text. This heavily text-based reading might be less strategic for these learners especially when considering that the diagram functioned to organize the complex neuroscience text into more comprehensible pieces of information. The learners might therefore be unable to reap benefits from the diagram. This paper also examined the relationship between reading behavior and English proficiency considering that low versus more proficient English learners might display longer fixations due to their comprehension difficulties [18]. Comparison of the LP and AP participants' attention on text and graphic areas, however, revealed that both groups spent similar amounts of time fixating on either area. This non-significant finding could be attributed to the low readability level of the stimulus. Given that the participants were at most average users of English (as in the case of the AP participants), a text that is more appropriate for the American Grade 12 learners (as in the case of the present stimulus) could be too difficult for both proficiency groups. Hence, rather than turning to the graphic representation as an alternative source of information upon encountering comprehension difficulties as previously assumed, both the LP and AP participants might have instead become more fixated on the text. Furthermore, two other factors could make the text more difficult and complex than it already was, hence, contributing to the heavily text-based reading. The first factor was the readers were not native to the language being used, and the second factor was the readers did not major in any science-related disciplines. The latter made them less familiar with the vocabulary, styles and structures of 458 Dependency on Verbal Versus Graphic Representations among Second Language Learners When Reading Illustrated Expository Materials scientific writing which is distinctive from the ordinary day-to-day writing and has been identified as the main source of difficulty, especially for language learners [10].
The organization of the diagram may also be unfamiliar to them. Their pre-dominant attention on the text could therefore be a result of comprehension difficulties. Although these participants were not necessarily the intended audience of these materials, the findings still painted a worrying picture on the reality of multi-representational reading, not only in an L2 but in any language in general. It appears that a paradox is evident as what is currently practiced by learners (i.e. the heavy reliance on verbal representations) is in contrast with what is ideal and more strategic for learning (i.e. adequate attention on graphic representations).
Proponents of multi-representational learning have proposed a few strategies that could help learners pay more attention to graphic representations such as the signaling principle which cues learners' attention to the graphics by using, for instance, deictic expressions and color-coding [13]. Though these strategies are useful, it cannot be denied that teacher intervention is equally important. It would be ideal for learners if their teachers are more proactive in emphasizing the importance of valuing graphics in the classroom and supporting them in doing so.

Conclusions
This paper has demonstrated that multi-representational reading in an L2 is similarly heavily text-based like reading multi-representational materials in the native language. It could be the case that the participants in this study had issues when trying to make sense of the text content, leading them to fixate almost exclusively on the text. Complementing learners' eye tracking data with their verbal reports could potentially shed more light into this matter.
Despite some contributions to the multi-representational learning literature, this paper however has some limitations. Firstly, the paper did not take into consideration the role of individual differences such as L1 proficiency and spatial ability in explaining the reading behavior of the participants. Secondly, it did not incorporate learning outcome performance to see whether this text-based reading was indeed less strategic when learning from this particular set of materials. Finally, as indicated above, the text readability could be rather low for the participants. Consequently, future work could experiment with simpler texts or a more appropriate group of learners. Additionally, different types and functions of graphic can be manipulated to see whether this heavily text-based multi-representational reading would persist.