Not Your Grandad’s Cognitive Ability Assessment
Increasing candidate attraction and reach by cracking the code on cognitive ability testing across device types.
As we shared in an earlier blog on this topic, more candidates are choosing to take selection assessments on mobile devices each year. For many, especially those in lower income communities, a mobile device is their only way to access the internet. Naturally, enabling content for mobile devices is a great way to appeal to candidates and reach individuals who might otherwise have difficulty accessing a computer.
However, up until now, simply enabling tests of cognitive ability developed for PCs on mobile devices has been largely unsuccessful. Candidates completing traditional tests of cognitive ability on mobile devices achieve lower scores than they do when completing the same test on a PC. SHL researchers knew that a very different approach to tackle this thorny issue would be necessary to ensure that candidates could demonstrate their full potential regardless of which device – laptop, tablet, or mobile – they used to complete a test.
The result was a brand-new portfolio of ground-breaking cognitive ability assessments – SHL Verify Interactive.
Mobile-First Responsive Web Design
Design for content that will be viewed on multiple devices is most successful when it begins with optimizing the content for the smallest devices. To maintain a consistent experience across devices, the content should respond to larger devices while maintaining the same experience as on the mobile device. Some of the design principles SHL researchers followed to achieve this include:
- Use minimal text
- Utilize graphics to convey information wherever possible
- Eliminate any need for horizontal scrolling– all content must fit within the width of the screen
- Minimize vertical scrolling to every extent possible
- Where vertical scrolling must exist on smaller screens, ensure it is also required on larger screens
We also eliminated the need for multiple-choice response options, which can take up a lot of room on screen, by building in user mechanics described here that allow candidates to input responses in the question itself.
Advanced Scoring Methodologies
In a standard multiple-choice question, a candidate only has two scoring options: right or wrong. We don’t know much about the steps the candidate took to arrive at the answer they chose, so the person that was very close to the right answer and the person who had no idea both get the question wrong.
The innovative question formats used in SHL Verify Interactive have multiple data points, and we can give candidates partial credit for the pieces of the question they respond to correctly if they don’t get the whole question correct. This means we can capture up to ten times as much information about a candidate’s ability level within a single question. So, while the typical question takes a bit longer than a traditional multiple-choice question, we learn so much more about the candidate with each question that fewer questions are needed overall.
All Verify tests released after 2012 have been computer adaptive. We continue that tradition with SHL Verify Interactive in this novel application of a partial credit model for which we built an entirely new computer-adaptive algorithm. The theory behind our algorithms are well researched in the peer-reviewed literature, but to our knowledge, this is the first time this body of work has been applied to an adaptive selection assessment.
Candidate performance is not affected by device choice.
Measurement Equivalence
In order to demonstrate empirically that a candidate’s performance on SHL Verify Interactive would not be affected by device choice, we conducted a study in which participants completed SHL Verify Interactive on a mobile device and then on a PC a few days later (half of the sample completed the tests in the opposite device order). We then did a statistical comparison of the scores from both administrations and found negligible and statistically non-significant differences between scores. The scores were also very highly correlated. Both pieces of evidence indicate that candidate performance is not affected by device choice. This evidence was supplemented by further analyses. providing overwhelming evidence of measurement equivalence across devices¹.
The outcome of our work was SHL Verify Interactive – a ground-breaking portfolio of assessments that attracts candidates and provides them with the opportunity to showcase their skills to potential employers. While SHL is proud of the efforts that went into crafting the engaging look and feel of these tests, we are just as excited about what candidates can’t see. That is the strong psychometric theory that ensures that candidates’ critical reasoning skills are measured as precisely and fairly as possible – regardless of device type. Not your Grandad’s cognitive ability assessment indeed!
Learn more about SHL Verify Interactive and how SHL can help you win by providing deep people insights to predict and drive performance.
¹ These results will be published in a forthcoming peer-reviewed journal article titled “Developing Device-Equivalent and Effective Measures of Complex Thinking with an Information Processing Framework and Mobile First Design Principles” in the journal Personnel Assessment and Decisions.
