coding skills UN031

How SHL Handles Uncompilable Code in Coding Skill Assessments

Uncompilable codes in coding skill assessments—is that bad? Learn how SHL’s technology helps you to detect a program’s quality, despite those uncompilable codes.

If you have ever taken a coding test for a job on a machine, you probably frown if you could not make your code compile. Your program might be almost right, but due to some silly bug, unidentified in a small-time frame, you will get a ZERO score. According to SHL’s Tech Talent report, on average, around 25% of candidates in a hiring pool have strong coding skills, but often get sorted out due to the limited capability of traditional coding skill assessment platforms—a missed opportunity.



But fret not! SHL’s AI team has created a technology that can detect how good the program’s algorithm is, even if it does not compile. Our coding skill assessment platform can understand uncompilable programs using Artificial Intelligence and provides semantic feedback to improve.

 
Uncompilable code is not equivalent to a zero score in the coding skill assessment

How do we do it? First, we can fix some of the codes using Artificial Intelligence. By looking at patterns in good compilable codes, our algorithms minimally modify existing programs to make them compilable. And by using this approach, we can compile about 40% of uncompilable codes. Once compilable, our patented machine learning-based algorithm can generate a grade that mimics human raters.

Fancy as it may seem, we had a harder problem to solve.

What about the codes that do not compile? The remaining 60% of the codes go through smart static analysis and derive features, and signatures of the logic of the program automatically. With these features and a customized form of our machine learning algorithm, we provide grades as accurate as you could think!

Where does it make a difference? Let’s look at some real data. On a set of programs attempted for a job in a large e-commerce player in the USA, we find that 46% of codes were not compiling but were not blank.

Our AI-based algorithm found that 6% of these codes, for 596 candidates, had nearly correct logic. Another 29% of candidates, with a little bit of guidance, would have reached the right logic. All these candidates deserved a shot with the company!

By looking at patterns in good compilable codes, our algorithms minimally modify existing programs to make them compilable. And by using this approach, we can compile about 40% of uncompilable codes.

This system was also used by a large IT company in India for the large-scale hiring of entry-level software engineers. The company hires thousands of software engineers every year and struggles with filling all open positions. By way of grading non-compilable codes, they have been able to improve hiring throughput by 30-40%.

More recently for a technology giant in China, the system helped in selecting around 26% more candidates for the interviews. These candidates had written a logically meaningful code, but the code did not compile. Out of these candidates, around 19% of candidates were finally hired (detailed analysis in Table 1). This is a big win as many worthy candidates who would have been missed out by traditional program grading systems were hired.

 

 What is more? Our AI algorithm provides feedback to all candidates whose code does not compile. To some, we can tell how to fix their programs and make them compilable. To all, we can give them feedback on their algorithmic approach, tips to reach the correct logic, and provide feedback on the stylistic and maintainability issues in their code.

Disappointed with coding skill assessment platforms which give everyone a poor score and no feedback… We have corrected this for all times to come!

Stay tuned for more to come as SHL’s AI team continues to work on advancing the state-of-the-art coding skill assessment and AI

 

If you want to see SHL’s coding skill assessment in action, please book a demo with one of our product specialists. Also, check out our case studies to learn how we helped our clients!

headshot unnam abhishek

Author

Abhishek Unnam

Abhishek is a Senior Researcher within the SHL’s AI Team. He has been influential in translating the latest research in Computer Vision, Speech, NLP, and programming languages into SHL’s product suite.

Contributors
headshot varun aggarwal

Co-Author

Varun Aggarwal

Varun Aggarwal is a researcher, entrepreneur, and author. In 2008, he co-founded Aspiring Minds and now heads Research. His work has led to the world’s first ML-based assessment of coding skills and the world’s first automated motor skills assessment. Varun has published more than 30 research papers and his work has been covered in The Economist, MIT Technology Review, Wall Street Journal, HT Mint, Economic Times and IEEE Spectrum. He started ML India (www.ml-india.org) to build India’s AI ecosystem and organized the world’s first data science camp for school children. Varun was awarded the HUMIES award in 2006 for developing algorithms that mimic human intelligence. He enjoys writing poems and stories.

Make Transformation Possible for Your People

Experience the greatest business outcomes possible with SHL’s products, insights, experiences, and services.

Find Out How

Explore SHL’s Wide Range of Solutions

With our platform of pre-configured talent acquisition and talent management solutions, maximize the potential of your company’s greatest asset—your people.

See Our Solutions