So this week the American Evaluation Association (AEA) asked me to write a post for their Disabilities and Other Vulnerable Populations TIG (DOVP) Week:

I know this isn't usually what I post about but hey... so here is me on "Diversity, Evaluation, and Better Data".

I’m Pat Campbell, president of Campbell-Kibler Associates, Inc. Under NSF funding, Eric Jolly, president of the Science Museum of Minnesota and I, with the help of a lot of friends, have been generating research-based tips, such as those below, to improve the accuracy of data collection, the quality of the analysis and the appropriateness of the data collected over diverse populations.

Hot Tips:

Ask for demographic information ONLY at the end of measures. There may be exceptions in cases for people with disabilities who will need accommodations in order to complete the measures.

Have participants define their own race/ethnicity and disability status rather than having the identification done by data collectors or project/program staff. If a standard set of categories for race/ethnicity and/or disability is used, also, in an open-ended question, ask participants to indicate their own race/ethnicity and disability status.

Have members of the target population review affective and psychosocial measures for clarity. Ask them what concepts they think are being measured. If what is being measured is obvious and there are sex, race, or disability stereotypes associated with the concepts, consider using a less obvious measure if an equally valid measure is available.

Be aware that there can be heterogeneity within subgroups. For example, while people who are visually impaired, hearing impaired, and learning disabled are all classified as having disabilities, the differences among them are very large and it might be appropriate to disaggregate by different categories of disability.

When race/ethnicity, gender, or disability status is used as an independent variable, specify the reason for its use and include the reason in documentation of the results.

Lessons Learned:

All populations are diverse: The diversity may be in terms of race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. It may be visible or invisible. Most likely in every group there is a multiplicity of diversities. High quality evaluations need to pay attention to the diversity of all populations being served.

Each individual is diverse. As individuals, we have many demographic characteristics including our race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. Rather than focusing on only one demographic category, high quality evaluations need to determine which categories are integral to the evaluation and focus on them.

Rad Resources:

Universal Design for Evaluation Checklist, 4th Edition. The title, says it all. Jennifer Sullivan-Sulewski, & June Gothberg have developed a short planning tool that helps evaluators include people of all ages and all abilities in evaluations.

As soon as it goes live, we hope our website, Beyond Rigor will be another rad resource. Let me know (Campbell@campbell-kibler.com) if you would like to be notified when that happens.