The guide for conducting online code screening and recruiting the best software engineer candidates

Last Updated: January 2021

Due to the COVID-19 pandemic, most companies moved hiring processes from in-person to remote. On top of that, mass layoffs across industries mean that your job postings receive even more candidates than ever before. One of the companies we work with recently posted a full-stack engineering job that could work anywhere in the United States, and they got 600 applications in 90 days. How is any technical recruiter supposed to possibly review that many résumés?

The learn to code movement has introduced new career opportunities to many people, which is fantastic. But it’s also made technical recruiting more challenging, as the overwhelming majority of applicants don’t have the required experience for the jobs they’re applying to. That’s where online code screening comes in.

→ How a tech startup instantly finds the best engineering candidates from a pile of résumés

Code screening is how you filter out unqualified candidates from your recruiting pipeline in an unbiased way. In many scenarios, that will remove as much as 75%-90% of candidates, leaving you with a much more manageable list of résumés to review and consider. This saves recruiters substantial time and of course eliminates the possibility of moving forward with someone completely unqualified and wasting precious engineering time to interview them. Amongst the thousands of recruiters that we work with, we’ve estimated that an effective code screening process saves businesses about $4,500 per hire when valuing time saved.

What is a code screening assessment?

Code screening assessments will vary based on the roles you’re hiring for, tools you use, and number and quality of applicants you receive. That being said, virtually all code screening assessments consist of the following:

  • The ability for candidates to take the assessment on their own time
  • The ability to ensure that candidates don’t cheat or identify if they are cheating by copying answers or partial answers that they find online
  • The ability to administer some combination of coding challenges, multiple choice questions, and open ended questions
  • The ability to automatically grade all or part of the assessment
  • The ability to compare candidate reports and set qualifying criteria for candidates to move to the next stage of the interview process

There are countless tools for creating and managing code assessments. Virtually any feature you can think of is available in the market. You can find a complete list of tools on G2 and reviews of the most popular ones on Medium.

G2 Grid® for Technical Skills Screening Software

What are the factors to consider when selecting a code screening platform?

Generally speaking, here are the factors that matter when selecting a platform:

  • High-quality candidate experience. The code screening assessment is likely the candidate’s first real interaction with your organization, so it’s important that the experience reflects positively on your employer brand. Many platforms unfortunately focus exclusively on the recruiter experience and overlook the candidate experience. They don’t offer adequate employer branding capabilities or a seamless candidate experience. For candidates, especially your top ones, that likely have other job opportunities, it’s important to test code screening platforms thoroughly as if you were the candidate before moving forward.
  • Clear, straightforward, and affordable pricing. You don’t want a platform that is cost-prohibitive and severely limits how many candidates you can screen because that defeats the point of creating an equitable unbiased process where all candidates get a fair shot. Look for platforms that offer unlimited candidates or at least have soft limits with reasonable overage charges.
  • Robust, modern challenge library. You need to ensure that the platform you select has challenges that match the skills and experience that you’re hiring for. Many will list on their websites which languages and skills that they have assessments for. Make sure to find one that offers real-world challenges for modern languages and frameworks like React, and not just algorithmic challenges for legacy languages. In terms of challenge libraries, you can always create a custom assessment, but it’s generally better to use one of the available ones which has been vetted for clarity and accessibility.
  • Easy-to-use interface for recruiters and non-technical stakeholders. Even if you’re an engineer, you probably won’t always have the time to configure and manage every screening process. Pick a platform that even a recruiter can use to select a template based on a job description and run a screening process without tons of technical implementation required.

Other important features like cheating detection and API integrations depend on your business, workflows, and candidate pool. Further, you might want to consider platforms that offer an end-to-end recruiting solution beyond screening, including sourcing, online interviewing, take-home projects, and even facilitating offer letters and negotiation. Once you’ve selected a platform, it’s important to properly configure assessments, otherwise you risk not actually qualifying candidates properly.

What are the factors to consider when creating each code screening assessment?

No matter which code screening platform you’re using, there’s a few important tips to keep in mind when creating assessments.

  • You are filtering out bad candidates, not identifying which candidates to give offers to. When hiring software engineers, you are by definition looking for very specialized and unique skills to solve complex problems in a collaborative environment. A code screening assessment can’t possibly replace a deep technical interview yet alone determine culture fit. The goal is merely to filter out candidates who are clearly unqualified, not to run the most comprehensive assessment possible. You’d be surprised how many candidates will likely fail even just a basic assessment.
  • Make sure the assessment and qualification criteria accurately match the job requirements. While any good software engineer can learn a new language or skill on the job, it’s important for a code screening assessment to filter out candidates who have too big a bridge to gap. Make sure the challenges and questions have relevance to the actual job, but aren’t so restrictive that only someone with overly specific experiences can qualify. Also, make sure to set a reasonable qualification score and monitor it. Too many candidates failing could mean that your assessment is too challenging, but it can also mean that you are sourcing a large number of bad candidates.
  • Your best candidates will ghost you if your assessment is too exhausting. Remember that a good software engineer has many career options. If you use a platform with a terrible candidate experience that requires them to turn their video camera on, prevents them from using normal functions in their browser, and answer endless algorithmic riddles, don’t be surprised when most candidates, especially the best ones, simply don’t complete your screening process. Or worse, they’ll shame your company on social media (example and example).
  • Set clear expectations before and after the assessment. As the employer, you have full context about the interview process. But for candidates, the interview process can vary greatly from company to company, causing a lot of stress. It’s important for the candidate to receive clear information about how long the assessment should take, what skills they’ll be tested for, and what the next steps are after completing the assessment.
  • Look out for and avoid a few common mistakes. Inviting all candidates at the same time is almost always a mistake because it doesn’t give you a chance to catch and fix a problem early! By inviting just a handful of candidates at first, you can catch any errors in the initiation process, assessment instructions, qualifying criteria, etc. The most common mistake is that assessments are simply too long or confusing, and most candidates simply refuse to complete them. Companies like Google and Facebook can get away with grueling assessments because they offer top compensation and prestige, but everyone else should be more reasonable.

→ Improve your candidate experience to eliminate ghosting and hire software engineers faster

Now you know what a code screening assessment is, when to use them, and how to think about creating assessments. You’re well on your way to a more efficient recruiting process.

What are the steps for recruiters to create a code screening assessment?

If you’re an engineering manager, you have the necessary expertise to set up an appropriate assessment on any platform. With today’s selection of user-friendly code screening platforms, so can recruiters if they carefully follow a few steps. Below is an example of the process if you were using Coderbyte.

→ Sign up for our next expert recruiting webinar

  • Open up the job description and take note of the seniority, languages, and frameworks that the engineering team is looking for in the "requirements" section. Don’t worry about the "bonus" or "nice to haves" – those are the sorts of details you can get from résumés after the screening process. Instead, focus on types of skills, like ‘front-end’, ‘data science’, ‘DevOps’, or ‘database.’ Specific languages, like ‘Javascript’ or ‘SQL’ Experience like ‘2+ years’ or ‘senior’
  • Once you’ve identified the key parts of the job description, go ahead and create a new code screening assessment. Most platforms will have an option for role-based templates and for skill-based templates. It’s a matter of preference for which one you’d rather select: think of role-based like top-down, while skill-based are bottom up. In a role-based template, you select the relevant job title and then pick the relevant experience level and languages. In a skill-based template, you select the relevant skills and then pick the relevant experience level. In either case, you’ll always be able to fully modify and customize the assessment afterward. This is just a shortcut for getting started.
  • After going through the assessment template creation process, you’ll be brought to the edit and customization experience. In Coderbyte, each coding challenge will have a suggested amount of time that they take. As a general rule, you want to have at least 1 type of question for each critical skill in the job description. So if it’s a junior role where only one type of skill or experience is required, rely mostly on easy coding challenges and maybe 1 or 2 multiple choice questions. For more senior roles that require more diverse skills, use just 1 hard challenge and then ask a variety of multiple choice and open-ended questions.
  • After you’ve configured your challenges and questions, it’s time to configure the assessment settings. On Coderbyte you’ll have a plethora of options which you can read about in our help guides.
  • Now that your assessment is created, ask a colleague in engineering to take it as though they’re a candidate to give you feedback and make sure everything works. It’s really important to test the end-to-end process before inviting actual candidates.
  • Invite about 10-20 candidates using the email invites rather than public links. Email invites generate one-time links that a candidate can use, whereas public links are too easy for candidates to share and get help. You should only use public links for mass hiring activities like career fairs or hackathons.
  • If you’ve run a batch of candidates through the assessment and the results look good, go ahead and invite everyone. You can even integrate your ATS with many code screening platforms to automatically send the assessment to applicants.
  • When candidates pass the assessment, share their report and résumé with engineering managers to see who they’d like to move forward to the next stage of the hiring process.

At first, this might all seem like an exhausting process but you’ll get the hang of it quickly. The code screening process has improved dramatically in the last few years. Not long ago, most companies just asked candidates a series of riddles and hired based on vague, irrelevant, and biased criteria. Today’s hiring process is far more equitable, sustainable, and scalable thanks to code screening platforms.

What are examples of code screening assessments for different engineering roles?

Let’s analyze a job description and demonstrate how anyone can create a great code screening assessment based on it. Here is the requirements section from one of our customer’s “Full-Stack Engineer” job description.

REQUIREMENTS

  • 2+ years of professional experience working in a fast-paced environment.
  • Significant experience building and shipping MVC and/or state-driven UIs, preferably with Vue.js or React, and have a productive relationship with CSS
  • Professional experience with some database technology such as SQL or MongoDB.
  • Desire to make a huge impact at a growing startup in NYC serving some of the largest and most influential brands in the world.
  • You can work well both independently and with a team, and have strong communication skills.
  • You enjoy venturing outside your comfort zone and experimenting with new technologies and architectures.
  • Nice if you have experience in: AWS (ECS, RDS Aurora, Lambda, API Gateway, MSK, S3, Elastic Transcoder, Others), Streaming Data Pipelines, Python (Flask, SpaCy), SVG and Canvas, WebRTC, Web Sockets, Heroku, Electron, CI/CD

Here are the relevant parts we can use to craft an assessment:

  • Less than 3 years of experience means that this is a junior position
  • Vue.js and React are front-end frameworks so that needs to be a component of the assessment
  • SQL and MongoDB are databases so that needs to be a component of the assessment
  • The company wants candidates that are self-starters and interested in learning new technologies

As stated earlier, leave out any ‘bonus’ experiences as that will be present on the resume and are best assessed during the interview process. Screening is for removing unqualified candidates, not identifying the very best candidate.

In a case like this, we’d recommend one easy challenge using Vue.js or React, one easy challenge using SQL or MondoDB, and an open-ended question about which new technologies the candidate has explored on their own and why.