Examining Students' Gains in Natural History Museum Science Content Knowledge with Access to Signed Vocabulary

With findings from a study conducted at the Museum of Science, Boston showed how family and school visitors used signing dictionaries in situations that involve physical interaction with phenomena. Research described in this article extended this study to an examination of how school visitors who are deaf or hard of hearing use the dictionaries in a setting, in which interactions with the phenomena are primarily observational. This additional research was conducted by TERC at the Harvard Museum of Natural History (HMNH). Its purpose was to examine the degree to which use of a Signing Science Pictionary and a Signing Science Dictionary results in gains in knowledge of the science content incorporated into exhibits focusing on Arthropods. The study involved 20 grade 1-6 students from a school for the deaf. They were divided into a treatment and a control group. The mixed methods design incorporated a protocol that involved a pre-test, a treatment group that used the dictionaries and a control group did not, and a post-test. Results from the preand post-test scores show that both groups, on average, demonstrated some gain in content knowledge. They also show that gains were similar for the treatment group and control group. Students’ perceptions that they learned something support these findings. This first-of-its kind study confirmed how challenging it is to measure how the dictionaries enhance content learning beyond vocabulary learning. It also showed that additional research is needed to demonstrate impacts of the signing dictionaries on informal learning for this population.


The Problem
Permanent hearing loss at birth afflicts about 24,000 infants annually in the USA. In addition, approximately 2 to 3 infants per 1,000 will have a hearing loss in at least one ear that affects communication, cognition, and educational development (National Institutes of Health, 2010). As these children mature, their literacy levels lag behind those of their hearing peers. This results in significant literacy limitations. In fact, the English vocabulary of the average 15-year-old deaf child is about the size of that of the average 9-year-old hearing child and will not improve significantly (Karchmer & Mitchell, 2006;Qi & Mitchell, 2012). As a consequence, deaf and hard of hearing children often miss out on opportunities to learn science or mathematics content from visits to informal learning environments such as natural history museums (Lang & Steely, 2003; Moon, Todd, Morton, & Ivey, 2012).

Study Overview
Handheld devices appear to have promise for use by museum visitors. They offer independence and enable visitors to experience exhibits at their own pace (Reich, 2009). The research that is discussed in this paper examines the impact of use of such devices by students who are deaf or hard of hearing to increase knowledge of the exhibit content they encounter at a natural history museum. The study was conducted by TERC, an educational institution, in partnership with the HMNH. Its objective was to collect data to answer the research question: Does having access to signed vocabulary result in gains in content knowledge for deaf and hard of hearing students visiting a natural history museum?

The Signing Dictionaries
Each signing dictionary is a complete assistive tool that includes approximately 700 standards-based science terms. The SSP (funded in part by grants from the Shapiro Family Foundation and the U. S. Department of Education, Grant #H327A080040) is intended for use by children ages 5-8. The SSD (funded in part by grants from NEC Foundation of America, the National Science Foundation [NSF] Grant #HRD-0533057, and the U.S. Department of Education Grant #H327A060026) is intended for use by ages 9-12.
Both the SSP and SSD incorporate signing avatars and support the Universal Design for Learning (UDL) framework. The framework emphasizes three principles of pedagogy: multiple means of representing information, multiple means for the expression of knowledge, and multiple means of engagement in learning (Rose & Meyer, 2006). These principles are implemented in the instructional design of the dictionaries as follows: Static images, text, human narration, and signing give users ways of acquiring knowledge (Principle 1). Allowing students to select American Sign Language (ASL) or Signed English (SE) translations, illustrations, or voiced text to help them explain their thinking provides the learner with alternatives for demonstrating what they know (Principle 2). Offering opportunities to work in ways that make sense and are interesting, such as choosing from a group of avatars of different ages, ethnicities, and genders, or changing the signing speed and size of the text, taps into users' interests, offers challenges, and increases motivation (Principle 3). Figure  1 shows pages from the SSP that incorporates these UDL features.

Participant Selection
To recruit participants for the study, TERC researchers invited an urban school for the deaf and hard of hearing that was located in close proximity to the HMNH to be part of the study. This resulted in identification of a group of 20 student participants. Characteristics of this group are presented in Table 1. In support of TERC's policies for working with human subjects, identifiers have been removed to ensure confidentiality of the participants.

Research Procedure
A mixed-methods design that integrated qualitative and quantitative methods was used to accomplish the study objective of examining gains in the use of mobile versions of the SSP and SSD by school visitors who are deaf or hard of hearing results in increased knowledge of science content that is incorporated into the HMNH exhibits Implementing it was intended to enable collection of data to answer the research question: Does having access to signed vocabulary result in gains in content knowledge for deaf and hard of hearing students visiting a natural history museum? The research procedure was designed for the study and variations that were made to it on the day of testing due to a bomb scare that occurred during implementation follows.
Before the visit, teachers used a Participant Data Form to provide demographic information for each student. TERC researchers, in consultation with the students' teachers, identified the Arthropod exhibit as the exhibit to use for data collection and designed a pre-test. Teachers administered the pre-test prior to the visit. Teachers paired students according to grade level, hearing status, hearing loss level, language of communication and signing skill. Taking the characteristics of the pairs into consideration, the teacher then assigned them to group A or B to create two groups that were matched as closely as possible.
Upon arrival at the HMNH, a coin toss determined which of the two groups (A or B) would be the treatment group. The other group became the control group. The treatment group was directed to the Arthropod exhibit that was selected for data collection while the control group and their teachers explored other exhibits. The researchers then introduced the treatment group and their teacher chaperones to use of the SSP and SSD and demonstrated the interactive dictionary features. Next, they gave each pair of students a mobile device that had the dictionaries loaded on to it and a Quest card. The Quest card was developed for the study. It included questions that were intended to help students search the exhibit to identify arthropods and find out about their characteristics, including where they live, and what they eat. The card also had a list of key terms related to the exhibit that could be found in the dictionaries. Findings from prior studies indicated that visitors needed to know upfront what words to look up to get them started using the dictionaries (Vesel 2015). Most of them were not sufficiently knowledgeable about the content to do this independently. Researchers instructed students to explore the Arthropod exhibit using the questions on the Quest card and dictionaries.
Following completion of their exploration, they were to be directed to a room where they would take the post-test.
After the treatment group had completed their exploration of the Arthropod exhibit, the control group was to be assembled and directed to the Arthropod exhibit. They were to explore the exhibit without access to the dictionaries. They were, however, to use a Quest card that did not include a list of key terms. After completing their exploration of the exhibit, they were to be directed to a room that was different from that used by the treatment group where they would take the post-test. Before leaving the museum, teachers were to be given an envelope of follow-up surveys. Each participated teacher and student were to complete a survey that teachers returned to TERC.
Due to a bomb threat that occurred, testing was interrupted. Since the treatment group had completed their exploration of the Arthropod exhibit, they were able to complete the post-test the next day in class. A second visit was subsequently arranged to give the control group the opportunity to explore the Arthropod exhibit and complete the post-test the next day in class.

Content Knowledge
The pre-and post-tests included content questions that were identical. They were designed to assess each student's knowledge of the science content in the Arthropod exhibit prior to visiting it and after having visited it. Table 2 shows the test questions included in the matched pre-and post-tests and their point values. Table  3 shows the mean pre-and post-test scores for the treatment and control group for each question and improvement as the difference between the scores. For Questions 1-3, data are represented as the pre-and post-test mean number of points out of the number of points possible for that question. Question 4 was scored using a 0-3 scale where identifying no arthropods resulted in a score of 0; identifying 1 or 2 resulted in a score of 1; and identifying 3 or more resulted in a score of 3.

Quest Experiences
The post-test included a section that was designed to provide the researchers with information of students' experiences using the questions and list of terms on the Quest card. This section included four questions. Each question had several responses. Students were to select the response that captured their experience. Table 4 summarizes the treatment and control group's responses.

Discussion
Twenty students in grades 1-6 who were deaf or hard of hearing and used ASL for communication participated in the study. They were randomly assigned to a treatment or control group. Each group was comprised of 10 students and visited the Arthropod exhibit. The treatment group used iPods with the SSP and SSD installed and a Quest card with a list of terms that could be found in the dictionaries and questions to help them find out about the exhibit science content. The control group did not use the iPods with the dictionaries. Therefore, they used a Quest card with questions but without a list of terms.
The purpose of the study was to examine if use of mobile versions of the (SSP) and SSD by school visitors who are deaf or hard of hearing results in gains in knowledge of the science content that is incorporated into the exhibits. As the pre-and post-test scores presented above show, the treatment group and control group, on average, demonstrated minimal gain in content knowledge. They also show that these gains were similar for both groups. Additionally, students' perception that they learned something about arthropods support these findings. Specifically, results show that at the end of the visit, students in both groups understood the exhibit content presented less than about 33% of the time. One reason for this might be that many terms students looked up were not included in the dictionaries. Those they did find were fingerspelled. This was problematic as their English language skills were limited and they could not put the letters together to form words. It is also possible that the students did not look up the words they needed to know to answer the assessment questions. Another reason for their limited understanding may be that although having access to signed terms and definitions which may lead to some increase in science content knowledge, it does not necessarily result in a robust understanding. Therefore, the treatment effect that occurred cannot necessarily be attributed to the signing dictionaries. Additionally, even though students could sign their answers to the questions, we cannot be certain that they did so or that their responses actually conveyed their understanding. Lastly, it is important to emphasize that our protocol included giving the post-test at the museum. Because of the bomb threat, students completed it several days later at school where implementation conditions could have varied and the lapse in time between exhibit use and testing could have had an effect on the results.

Conclusions
Modifications made to the protocol due to unavoidable circumstances compromised fidelity of implementation. For this reason, it would be important to conduct further testing at the HMNH that involves implementing the established protocol without modification, incorporating additional exhibits, and including additional participants from other schools.