Surveying hard-to-reach research population groups: A mixed methods approach to questionnaire development and evaluation
*Gordon B Willis, National Cancer Institute
Keywords: Questionnaire design, cross-cultural, comparability, multicultural, multilingual
An often overlooked aspect of the concept of "Hard to Reach" is the development of questionnaires that function effectively across culturally diverse population groups. Challenges include: (a) asking survey questions that adequately capture respondents’ experiences; (b) using terminology that is well-understood; and (c) achieving comparability across survey translations. Although these issues are not unique to H2R populations, it is especially important to address them when surveying these populations, to ensure measurement comparability across groups. We applied a mixed-methods design to pretest questionnaire items administered in two field tests of the Discrimination Module within the California Health Interview Survey (N = 12,144). The first test focused on contrasts between six racial/ethnic groups (African-Americans/Blacks, American Indian/Alaska Natives, Asian Americans, Latinos, Multiracial respondents and Non-Latino Whites). The second test focused on six language groups (English, Spanish, Korean, Vietnamese, Cantonese, and Mandarin). We applied both qualitative and quantitative methods: cognitive interviewing, behavior coding (with multi-level statistical modeling), and psychometric testing (including assessment of Differential Item Functioning).
We will present examples from our mixed methods approach that demonstrate how threats to item validity across groups were identified and resolved. Results indicated that: (a) many problems crossed cultural boundaries (e.g., the terms ‘courtesy’ and ‘respect’ were generally perceived as redundant); (b) response categories such as ‘always’ and ‘sometimes’ tended not to align across languages, requiring re-translation; and (c) questions presented problems when they made assumptions that did not apply to certain groups (e.g., questions about lifetime discrimination created problems for immigrants who had lived within multiple socio-cultural contexts). We conclude by encouraging the use of a mixed-method approach for developing instruments intended for hard-to-reach populations, and for ensuring measurement comparability of items across a range of cultural groups and languages.