A Validation Study of the Culturally Responsive Teaching Survey

Amidst the ethnic and linguistic diversity in adult English language classes, there is heightened importance to using culturally responsive teaching practices. However, there are limited quantitative examinations of this approach in adult learning environments. The purpose of this investigation was to describe patterns of culturally responsive teaching practices of adult ESOL teachers and to establish the psychometric properties of the Culturally Responsive Teaching Survey (CRTS), a newly-developed self-assessment survey. Based on Ginsberg and Wlodkowski’s Motivational Framework for Culturally Responsive Teaching [1], this 17-item online survey establishes patterns of teaching praxis appropriate for adult English language classrooms. Findings revealed a trend of regular to frequent use of the majority of the culturally responsive teaching practices indicated in the CRTS. In addition, analyses demonstrated that the CRTS is a reliable, uni-dimensional scale which yielded positive correlations with multi-cultural knowledge and teaching skills. Thus, the CRTS provides a useful tool for examining the praxis of culturally responsive teaching in adult, second-language classrooms. These findings will lead to improved understanding of how adult educators incorporate culturally responsive teaching practices in ethnically and linguistically diverse learning environments, in addition to supporting the use of this instrument in future research studies.


Introduction
Given the ethnic and linguistic diversity of adult English for Speakers of Other Languages (ESOL) students, the incorporation of culture and student cultural identities into the learning environment has heightened importance [2][3][4].While highly diverse in age, country of origin, and native language [5], adult English language learners all experience the acculturation process, simultaneously maintaining ties to their native cultural group and developing relationships within their adopted cultural group [6].Culturally responsive teaching mediates the challenges of this process by placing students' cultural norms and practices at the center of the learning process [7].In essence, culturally responsive educators are proficient at discerning subtle and overt differences and developing culturally-sensitive and appropriate learning environments [8,9].However, while interest in culturally responsive teaching is high, there are limited quantitative examinations of this approach in adult learning environments.In general, research has been conducted through qualitative methods or assessments of teachers' cultural sensitivity, racial bias or cultural competence [10], not teaching practices.Therefore, there is limited quantitative data describing how adult ESOL teachers incorporate their student's cultural norms and values into their curriculum and instruction.
The purpose of this investigation was twofold: to describe patterns of culturally responsive teaching practices of adult ESOL teachers and to establish the psychometric properties of the Culturally Responsive Teaching Survey (CRTS), a newly-developed self-assessment survey.Specifically, I examined: (a) the self-reported frequency of use of 17 teaching practices, (b) the factorial structure of the CRTS, and (c) the construct validity of the CRTS in studies conducted with English language teachers of adults.These findings will lead to improved understanding of how adult educators incorporate culturally responsive teaching practices in ethnically and linguistically diverse learning environments, in addition to supporting the use of this instrument by researchers and practitioners.

Theoretical Framework
Culturally responsive teaching is an umbrella term which encompasses a variety of approaches, such as culturally relevant, culturally sensitive, culturally congruent, and culturally contextualized pedagogies [7].This equity-based approach places students' cultures at the core of the learning process and utilizes the "cultural knowledge, prior experiences, frames of reference, and performance styles of ethnically diverse students" [7 p. 29].By creating classroom norms reflective of the students' native cultures, and not those of mainstream culture, the culturally responsive educator mitigates the challenges of overcoming "cultural mismatches" between the home and school [11,12].Culturally responsive teaching is distinguished by its emphasis on validating, facilitating, liberating, and empowering minority students by "cultivating their cultural integrity, individual abilities, and academic success" [7 p. 44] and is based on the four pillars of "teacher attitude and expectations, cultural communication in the classroom, culturally diverse context in the curriculum, and culturally congruent instructional strategies" [7, p. 44].
While much of culturally responsive teaching theory refers to K-12 settings, Ginsberg and Wlodkowski developed the Motivational Framework for Culturally Responsive Teaching [1] for application in adult learning environments.They posited that culturally responsive teaching increases the intrinsic motivation of students of non-dominant cultural groups.This framework describes the norms and practices of an environment in which "inquiry, respect, and the opportunity for full participation by diverse adults is the norm" [14, p. 161] and is based on the integrated use of four elements: establishing inclusion, developing attitude, enhancing meaning, and engendering competence [1,14].Each element, or criteria, has corresponding norms and practices that adult educators can use in creating or evaluating their praxis.
Teaching practices that create an environment of respect and connectedness and that use cooperation and equitable treatment of all learners reflect the element of establishing inclusion.The element, developing attitude, includes norms and practices that help students develop a positive attitude toward the learning process by building on students' personal experiences and knowledge and by allowing learners to make choices throughout the learning process.The third element, enhancing meaning, includes norms and practices that encourage students to engage in deep reflection and critical inquiry, such as role-plays and simulations.The final element, engendering competence, are practices that show the learner evidence of his or her learning and proficiency and the use of assessments that are contextualized in the learners' experiences [1,14].
Ginsberg and Wlodkowski's framework served as the operational framework of the CRTS due to its original design as "a tool for continual reflection" [1, p. 39] to help educators examine their teaching in an effort to improve the cultural responsiveness of their practices.In addition, although this framework was designed for the higher education classroom, it could be adapted to include the unique patterns of teaching practice relevant to adult language learning environments.

Survey Development
Development of a survey instrument can be accomplished through various methods; however, the process entails the following steps: clear identification of the construct to be measured, item generation, expert review and refinement of item pool, a pilot or development study, and item evaluation and reduction [13].A list of culturally responsive teaching characteristics compiled during a literature review served as the theoretical foundation of the item development process.A literature review using the key terms "culturally responsive", "culturally relevant", and "culturally congruent" teaching yielded findings from predominantly qualitative examinations of culturally responsive teachers in a variety of teaching environments.All characteristics were compiled into a master list and categorized by the four elements of the Motivational Framework for Culturally Responsive Teaching [1].The list was then consolidated by eliminating redundant characteristics, resulting in a master list of 23 characteristics.These represented general beliefs and experiences and were used to develop items of specific teaching practices appropriate for an adult English language learning environment.This was achieved by a panel of ESOL practitioners describing teaching practices that manifested each characteristic.The draft item pool contained 27 teaching practices.
To assess the content validity of the item pool, the expertise of two panels was utilized.The first validation panel consisted of individuals with extensive teaching experience in the field of adult ESOL and Teaching English to Speakers of Other Languages (TESOL) who evaluated the items for clarity and relevance to second language teaching theory [15].The second validation panel consisted of experts in culturally responsive teaching who evaluated the items for relevance to the theories of adult learning and culturally responsive pedagogy.Ranked on a five-point scale, items with means of 3 or below were deleted, while 2 items were reworded or combined.At the conclusion of this stage, the draft survey included 17 culturally responsive teaching practices.The final phase of survey development consisted of both cognitive interviewing and a pilot study with a representative sample of 100 adult education ESOL teachers.To assess the reliability of the pilot survey, Cronbach's Alpha Coefficient was calculated and deemed acceptable at .752.Based on these data, all 17 items were retained for the final version of the CRTS.

Study 1 4.1. Sampling and Procedure
The CRTS was first used in a study of adult education ESOL teachers in Florida in late 2012 [16,17].The purpose of this study was twofold: to develop and validate the CRTS and to describe the self-reported use of 17 specific culturally responsive teaching practices of adult ESOL teachers throughout the state of Florida.The target population consisted of teachers in non-credit, adult education ESOL programs.The sampling frame was compiled using publicly-accessible faculty directories from 15 state or community colleges, 2 universities, 8 school districts, and Bay Area Regional TESOL (BART).From the 430 person sampling frame, there were 134 completed surveys resulting in a 31.2%response rate.The majority of respondents were females (78.38%) from community or state colleges (92%).Four follow-up reminder emails were sent over a one-month period.Responses were somewhat evenly distributed with 46, 49, and 39 responses in consecutive weeks, and no further responses following the fourth reminder email.To examine patterns of response by completion date, ANOVA and Tukey tests were conducted.Significant differences were found using a predetermined Type I error rate of .05 in three items, but there was no discernible pattern to these differences.Therefore, the dataset was analyzed without sub-grouping of response times.

Instrumentation
The CRTS includes 17 teaching practices about which participants report their frequency of use on a 5-point frequency scale with levels of: never, rarely, sometimes, usually, and always.Participants respond to items such as, "I ask students to compare their culture with American culture" and "I include lessons about the acculturation process" by choosing how often they utilize that practice.See Appendix A for the complete item pool.

Reliability
When examining latent traits such as culturally relevant teaching, the reporting of the reliability of sample scores establishes a level of consistency of these unobservable characteristics [18].The Cronbach Alpha Coefficient value of these means was calculated at .781, revealing appropriate levels of internal consistency for data analysis.

Frequency of Use
Various descriptive analyses were conducted to describe the respondents' frequency of use.The first was an examination and representation of the frequency distributions of each item, which were analyzed for central tendency, dispersion, and shape.A cumulative frequency chart was developed in order to examine overall patterns within the sample.Additionally, variance, standard deviation, skewness and kurtosis of both the overall and item scores were compiled and examined.
Item means ranged from 2.51 to 4.26 with thirteen of the seventeen items falling in the moderate to high range of 3.02 to 4.26 corresponding to the frequency levels of sometimes, usually, or always.The remaining four item means ranged from 2.5 to 3.0 corresponding to the frequency level between rarely and sometimes.
The most frequently used practice was "provide rubrics and progress reports to students" (M = 4.26; SD = .98),followed closely by "elicit students' experiences in pre-reading and pre-listening activities" (M = 4.24; SD = .748).The items, "ask students to compare their culture with American culture" and "make an effort to get to know students' families and background", were also noted as being frequently used with means of 4.16 and 4.10, respectively.
Examination of respondents' item responses reveals further details about the two most frequently used teaching practices.While three respondents indicated that they never "provide rubrics and progress reports to students", roughly 80% of all surveyed teachers indicated that they did so on a highly regular basis.A larger percentage of the sample (86.6%), however, indicated that they usually or always "elicit students' experiences in pre-reading and pre-listening activities".Thus, while both item means were high, more teachers use students' experiences than providing rubrics or progress reports.A full list of the mean scores and standard deviations, can be found in Table 1.
Among the least frequently used practices were "include lessons about anti-immigrant discrimination or bias" (M = 2.51; SD = 1.017), and "students work independently, selecting their own learning activities" (M = 2.76; SD = .860),followed by "ask for student input when planning lessons and activities" (M =2.91; SD = .921)and "use student surveys to learn about students' classroom preferences" (M =2.94; SD = 1.102).When these item responses were examined, there was greater dispersion of the least frequently used culturally responsive teaching practices than the most frequently used ones.A greater number of teachers indicated the moderate use of these four practices with the frequency level sometimes, in addition to both the levels of rarely and usually.For example, even though 18.7% of the sample indicated they never included lessons about anti-immigrant discrimination or bias, 15 teachers indicated that they usually or always did so comprising 11.2 % of the sample.Slightly more than one-third of the sample indicated that they never or rarely have students work independently or select their own learning activities.When the teachers who sometimes engage in this practice are added, an overwhelming majority of the total sample (86%) did not support the use of this culturally responsive teaching practice.A complete percentage distribution of the item responses by scale value of the four least used teaching practices can be found in Table 2.

Exploratory Factor Analysis
In this study, an exploratory factor analysis (EFA) was conducted to ascertain if certain items functioned as a group, or factor, of the construct of culturally responsive teaching practices [13].The appropriateness of the data for an EFA was examined based on examining the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett's test of sphericity.The KMO was .767,which is considered acceptable for EFA [19,20].Bartlett's test of sphericity produced a statistically significant value, (χ2(136) = 473.19,p < .05);therefore, the data were deemed acceptable for EFA.
The EFA produced a five-factor solution using a varimax rotation and was used for the final solution.The factor pattern coefficients revealed a majority of the 17 items with factor loadings of .55 or greater and thus, deemed significant indicators of their respective factors [21,22].Using this cutoff point to compare the factor structure of the items in this study to the four-element theoretical framework provided by the Motivational Framework of Culturally Responsive Teaching [1], limited similarities were found.The factors on the CRTS accounted for 56% of the total variance with eigenvalues higher than 1.0 for each of the factors.Therefore, the eigenvalues met the criteria for Kaiser's rule [23], but did not meet the criteria for the total variance (greater than 60%) that is considered acceptable for research in the social sciences [22,24,25].A complete list of the factor pattern coefficients is presented in Table 3.
In summary, the findings demonstrated a limited relationship to the operational framework.Therefore, while additional studies with expanded sampling frames may yield a different internal structure, these findings suggest a uni-dimensional construct to the CRTS, not the multiple element of the theoretical foundation, the Motivational Framework of Culturally Responsive Teaching [1].

Study 2 5.1. Sampling and Procedure
The CRTS was also used in a nationwide study of adult education ESOL teachers from 2014 to 2016.The purpose of this study was also to describe the culturally responsive teaching practices of the sample, while also examining the construct validity of the CRTS through the administration of the Multicultural Teaching Competencies Scale (MTCS) [26].The sampling and survey administration followed the same procedures as Study 1.The survey was administered to three groups, with the results of ANOVA and Turkey tests revealing no discernible pattern to differences in date of response.Therefore, the dataset was analyzed without sub-grouping.There were 219 respondents, the majority of whom were female (88.9%), with males representing 10.1% of the sample, leaving 1% of undisclosed gender.The majority of respondents were from community colleges (26.9%) and colleges or universities (58.5%), while individuals from school districts comprised only 1.6% of the respondents.

Instrumentation
In addition to the CRTS, participants completed the MTCS.The MTCS is a self-report questionnaire that assesses skills, behaviors, and knowledge of multicultural teaching practices and theory [26].
The MTCS is composed of 16 items divided into two sub-scales: Multicultural Teaching Skills and Multicultural Knowledge.Each item is assessed on a six-point Likert-type scale ranging from 1 (Strongly Disagree) to 6 (Strongly Agree).

Examples of the Multicultural Teaching Skills sub-scale are I plan many activities to celebrate diverse cultural practices in my classroom and I often include examples of the experiences and perspectives of racial and ethnic groups during my classroom lessons.
Examples of the Multicultural Knowledge sub-scale are I understand the various communication styles among different racial and ethnic minority students in my classroom and I am knowledgeable about the particular teaching strategies that affirm the racial and ethnic identities of all students.For this study, the item I establish strong, supportive relationships with racial and ethnic minority parents was not included, due to its lack of relevance to the adult learning environment.

Reliability
Similar to Study 1, the Cronbach Alpha Coefficient of the CRTS demonstrated a high level of internal reliability at .834.Additionally, both Cronbach Alpha Coefficients of the MCTS were acceptable at .850 (Teaching sub-scale) and .840(Knowledge sub-scale) [16].Therefore, scores from both instruments were considered appropriate for further analysis.

Frequency of Use
The data analysis utilized in Study 1 was replicated in this study, yielding frequency distributions of each item and a cumulative frequency chart to examine overall patterns of this sample.The high patterns of use in Study 1 were also reported in Study 2. In overall item responses, item means ranged from 2.84 to 4.32 with fourteen items falling in the moderate range of 3 and above corresponding to the frequency category of sometimes, usually and always.The remaining three item means ranged from 2.84 to 2.97, corresponding to the frequency level between rarely and sometimes.
The most frequently used practices in this study included those found in Study 1 (noted above) and the additional items "examine class materials for appropriate images and themes" (M = 4.32; SD = .796),followed by "use mixed-language and mixed-cultural pairings in group work" (M = 4.28; SD = .88).While these item means had been high in Study 1, they now ranked as the top two most frequently used items.Further similarities were found in the least frequently used items with the same two items "include lessons about anti-immigrant discrimination or bias" (M = 2.84; SD = 1.03) and "students work independently, selecting their own learning activities" (M = 2.86; SD = .96)receiving the lowest reported frequency.A full list of the mean scores and standard deviations can be found in Table 1.
Examination of item responses once again yields further details about the patterns of use of the most and least frequently used practices.The item most respondents (49%) described as always using was "provide rubrics and progress reports to students", followed by "examine class materials for appropriate images and themes" and "use mixed-language and mixed-cultural pairings in group work", 43.1% and 42.8% respectively.In contrast, the item that received the highest number of never responses was "encourage students to use their native language with their children".The overall pattern of responses demonstrates great variability, with 18.4% of the respondents stating they never do so, 16.1% stating they rarely do so, 19.3% stating they sometimes do so, and 17% and 29.2% stating they usually and always use that practice.

Convergent Validity
Convergent validity describes the relationship between assessments [27].If there is a positive correlation, the instruments can be seen to measure related constructs.The results of the Pearson-product moment correlations between the CRTS and the two sub-scales of the MCTS demonstrated predominantly significant and positive relationships.There were positive correlations between the scores of the CRTS and both the Teaching Skills (r = .587;p < .01)and Knowledge sub-scales (r = .506;p < .01) of the MCTS.A correlation coefficient matrix of these scores can be seen in Table 4.These significant correlations in the expected positive direction provide support for the convergent validity of the CRTS, by indicating that those who use or desire to use more culturally responsive teaching practices are also more skillful at and knowledgeable about teaching students of diverse backgrounds.These significant correlations in the expected positive direction provide support for the convergent validity of the CRTS, by indicating that those who use or desire to use more culturally responsive teaching practices are also more skillful at teaching students of diverse backgrounds.

Discussion
The purpose of this investigation was to describe the culturally responsive teaching practices of adult ESOL teachers and to establish the psychometric properties of the Culturally Responsive Teaching Survey.These findings will lead to improved understanding of how adult educators incorporate culturally responsive teaching practices in ethnically and linguistically diverse learning environments, in addition to supporting the use of this instrument by researchers and practitioners.

Patterns of Use
These studies revealed a trend of regular to frequent use of the majority of the culturally responsive teaching practices indicated in the CRTS.The findings revealed that the sampled adult ESOL teachers respond to the ethnically and linguistically heterogeneous learning environment by reaching out and incorporating students' learning styles and ways of knowing into their teaching instead of establishing classrooms which represent only mainstream American culture.There are unique characteristics of this learning environment that may contribute to this heightened level of culturally responsiveness.All adult English language learners enter the classroom with values and norms of their native cultural group.As a result, the need to position students' cultures at the forefront of the learning process and utilize their values and experiences may be more compelling and obvious to ESOL teachers.
However, there were some culturally responsive teaching practices that were used less frequently by adult English language educators.These teaching practices involved asking for student input about the learning process, practices which are grounded in learner autonomy and self-directed learning.They are also reflective of the element of developing attitude in the Motivational Framework of Culturally Responsive Teaching [1] and are practices posited to increase the learners' positive regard for the content.Another area of limited use was the inclusion of lessons dedicated to discrimination.Promoting critical inquiry and addressing real-world issues are tenets of culturally responsive teaching.However, findings from both studies revealed that adult education ESOL teachers did not regularly include lessons about anti-immigrant discrimination or bias.It is necessary to develop a better understanding of the rationale behind the less frequent use of these practices to determine how they relate to the overall construct of culturally responsive pedagogy.

Factor Structure
Using the four elements of Ginsberg and Wlodkowski's model [1], this researcher hypothesized a multi-dimensional construct of the CRTS.However, findings from the exploratory factor analysis conducted in Study I support a uni-dimensional structure.An explanation may be found in the interconnected and holistic nature of the model's design.The Motivational Framework of Culturally Responsive Teaching was designed to assist practitioners in instructional planning with self-reflection at its foundation.The four elements are described as interconnected parts of a "holistic and systemic" [1, p. 34] model of classroom practice.All elements work in partnership to create an environment conducive to learning for students of diverse backgrounds.
Another explanation may be linked to the survey items' lack of sensitivity to nuances involved in culturally responsive teaching.The survey includes 17 items which were developed and validated through online questionnaires with multiple choice questions.The lack of open-ended questions may have resulted in an overly restrictive validation process and could be improved upon by conducting live focus groups.Using these, and additional qualitative methods, would allow for probing of participants in order to gain a better understanding of this discrepancy between culturally responsive teaching theory and its practice in adult ESOL classrooms.

Convergent Validity
Convergent validity describes the relationship between assessments [27].If there is a positive correlation, the instruments can be seen to measure related constructs.In Study 2, scores from the CRTS and the MCTS sub-scales were analyzed for correlations through Pearson-product moment correlations.The hypothesis of a positive relationship between the two areas examined by the MCTS, multicultural knowledge and multicultural teaching, and the CRTS guided the examination of data.
The MCTS approaches multicultural teaching as a complex and ongoing activity.The theoretical framework of the MCTS is a multi-dimensional construct of: continual critical reflection, motivation to increase awareness of diversity, and the connection between educator beliefs and praxis.It has been used in a variety of educational settings with acceptable levels of internal consistency and validity.Therefore, the positive correlations found in Study 2 demonstrate the usefulness of the CRTS to adult educators interested in exploring the cultural responsiveness of their teaching.
The results of these studies provide support for the reliability and validity of the CRTS.Findings suggest that the CRTS is a reliable uni-dimensional measure, whose scores demonstrate convergent validity through positive correlation with multicultural teaching knowledge and skills.The CRTS provides a useful tool for researchers to expand understanding of adult ESOL teachers' strategies to incorporate students' cultural identities into the classroom in the presence of ethnic, racial, and linguistic diversity.Further research will yield additional information about general patterns of behavior and should include examinations of differences based on teacher demographics.

Implications for Future Studies
There are areas to consider as the CRTS is used in additional research studies.Based on this investigation, it is a reliable and valid self-assessment survey of culturally responsive teaching practices appropriate for adult English language classrooms.However, additional refinement and expansion of the current item pool would add to its usefulness in describing teacher praxis.The survey includes 17 items which were developed and validated through online questionnaires.The use of online questionnaires in the item pool development stage could be improved upon by conducting live focus groups with adult ESOL teachers.For example, during the validation stage, two items related to the use of native language were deemed not relevant to the adult ESOL classroom.Additionally, two items of the same nature were combined to create one item.This resulted in the inclusion of only two survey items related to the use of the students' native language.However, a tenet of culturally responsive teaching theory is the importance of incorporating and facilitating the development of students' native languages in order to promote academic success.During a focus group, the researcher could probe participants in order to gain a better understanding of this discrepancy between culturally responsive teaching theory and its practice in adult ESOL classrooms.Additionally, participants could describe the rationale behind the frequency of usage.This information would greatly add to the understanding of the implementation of a culturally responsive teaching approach.I ask students to compare their culture with American culture.

4
I make an effort to get to know my students' families and backgrounds.12 I spend time outside of class learning about the cultures and languages of my students.

13
I include lessons about anti-immigrant discrimination or bias.
14 I supplement the curriculum with lessons about international events.
15 I ask for student input when planning lessons and activities.

16
I encourage students to use cross-cultural comparisons when analyzing material.17 I provide rubrics and progress reports to students.

5 I
learn words in my students' native languages.6Iuse mixed-language and mixed-cultural pairings in group work.7 I use peer tutors or student-led discussions.8 I use surveys to find out about my students' classroom preferences.9 I elicit students' experiences in pre-reading and pre-listening activities.10 I encourage students to speak their native languages with their children.11 I have students work independently, selecting their own learning activities.

Table 1 .
Descriptive Statistics of Responses to How Frequently Teachers Used Culturally Responsive Teaching Practices

Table 2 .
Percentage Distribution of Responses to Least Frequently Used Culturally Responsive Teaching Practices by Scale Values

Table 3 .
Factor Pattern Coefficients Based on a Principle Components Analysis for Items Related to Frequency of Use Sub-Scale in Study 1

Table 4 .
Correlations between CRTS and MCTS Sub-Scales **Significant at the p < 0.01 level.