The Challenges of Cross-Culturally Fair Assessments

“Does not address the identified developmental areas directly and clearly.” I am scribbling this on my observation sheet during an Assessment Center (AC). Thus, showing that the candidate will not get the full score in this competency. After all, we are looking for managers who say what they think directly, without a lot of "talking around", and who also address critical points clearly. Or are we? “Hold on”, I am thinking to myself. “Aren’t I thinking way too ‘German’ right now? What does ‘direct’ mean?” The role-player, playing the role of the “employee” of the candidate, obviously understood the message very clearly.

The next exercise in this AC is a competency-based interview. My notes read “Does not make own contribution clear, uses a lot of ‘we’ instead of ‘I’”. Am I being unfair again? I come from an individualistic culture, in Germany, where from an early age we are trained to talk about our individual contributions and successes. However, most countries are collectivistic cultures, where it is often rather uncommon to highlight one's own successes. In this type of culture, people see themselves as part of a group and thus tend to communicate, also in assessment scenarios, accordingly. “So what am I supposed to do? What is the desirable behavior and style we are looking for?” I am thinking, confused.

These two examples highlight some of the challenges observers face in assessments where people from different cultures are being assessed. In fact, there is no aspect of an AC or DC (Development Center) that is not affected by culture. Thus, there is a huge potential for bias, miscommunication, and misunderstanding due to cultural diversity.

The challenge to building cross-culturally fair and inclusive assessments

• Leadership and feedback styles
  Leadership styles differ among cultures. What makes up a good leader in China is very different from how a good leader is perceived in the UK, and chances are high that the difference in leadership styles will be shown in ACs or DCs as well. While in Western countries employees giving feedback to their managers is welcome (E.g., in the context of 360-degree feedback), it is unacceptable in most Asian countries. I have seen many customers fail with the intention of implementing 360-degree feedback in Chinese organizations. Indeed, most cultures do not give feedback as bluntly and directly as I do as a German. In some cultures, communication even only works one way: Top-down. Thus, a role play that measures the interaction of a participant with their (role player) manager that aims at assessing them openly disagreeing with their superior will not be a fair exercise as it puts some cultures at a disadvantage. 
• Group exercises
  While allowing the assessment of multiple participants at the same time is highly efficient, this practice struggles with low reliability and a problematic methodology. As the fellow participants are not trained assessors and, in contrast to e.g. 1:1 role plays, do not follow a predefined script to trigger the (un)desired behavior, the results of group exercises cannot really be compared and depend strongly on how strong or weak the other participants are. If you then add the challenge of different cultures — with some people (e.g. US Americans) learning from when they are young to “sell” themselves and their ideas, and other people, especially from collectivistic cultures, being taught to not stand out of the crowd too much and to follow the group consensus — the predictive validity is negligible. 
• Personality models
  Personality models that are highly accepted in the Western world are less useful in other cultures. The famous Big 5 for example is influenced by individualistic cultures. As the Big 5 influences many personality models used in assessment centers, they almost automatically disadvantage certain cultural groups. Research has shown that in Western cultures, traits, opinions, targets, and other very personal characteristics form the basis for a coherent and stable identity. In Asian culture, however, this continuity may be formed by a person’s role, e.g., being an employee or a manager. Thus, research suggests that the connection between personality traits and behavior can be less in Asian culture than in Western culture, and thus the validity of these questionnaires, too.

Hence, for organizations, one of the most fundamental questions to ask, and answer, is how much they expect their (future) employees and managers to adapt to the organizational culture (usually shaped by its headquarters) or how much they want to embrace cultural differences and allow people to stick to their known behavior. None of the options is necessarily better; both have strong implications for the organizational culture and assessments when selecting or developing staff and leaders.

What makes up a good leader in China is very different from how a good leader is perceived in the UK, and chances are high that the difference in leadership styles will be shown in ACs or DCs as well.

 

Moving toward interculturally inclusive assessments

One of the most cross-culturally applicable assessment solutions is online questionnaires that allow one to make more objective and less biased people decisions. A prerequisite for this is that they have been validated internationally. For its most popular assessment tool, the Occupational Personality Questionnaire (OPQ), SHL offers local comparison groups. It allows you to compare candidates with respondents of the same cultural background and therefore reduces intercultural biases. (Even though research shows that its forced choice technique significantly reduces socially desirable responses, it still cannot fully prevent the fact that in some cultures, e.g., China, questionnaires are generally not popular and respondents tend to give socially desirable answers, thus skewing the results).

Another possibility to reduce cultural bias is to have observers from the cultures of the participants or with experience living in these cultures. It can also help to interpret the behavioral anchors “more loosely”  than in “one-culture” assessments in a way that all cultures have the same chance of showing the desired behaviors.

In my opinion, the topic of cross-culturally inclusive assessments is more relevant than ever before. But, as so often, the first important step is to have awareness of these complexities. Furthermore, it is important to see interculturally fair assessments not only as a challenge but also as something precious that is worth investing time and energy in.

In the above-mentioned AC, I openly shared my thoughts with the other observers. We then defined clear minimum standards of behavior that we wanted to see, e.g. regarding the leadership and feedback style. For other competencies, we decided to interpret the predefined behavioral indicators a bit more loosely so that they reflect intercultural differences.

SHL can help your organization to create more cross-culturally inclusive assessments. Contact us so we can help you with your selection and development of a diverse workforce!