Making the Engineering Phone Interview Less Awful

Dhaivat Pandya

Feb 17, 2022 • 7 min read

As we're scaling up our software engineering team at Verse, we've been improving our process of identifying great engineers. Our company is predicated on building modern, robust software on top of legacy healthcare systems. We need a strong engineering team to do this well.

All engineering interview processes are terrible in some ways, but we've tried hard to make ours less awful.

We've tried a lot of things. Some have failed in surprising ways and others have worked in surprising ways. I'll talk about how we came up with our fairly unique phone interviews through a set of failed attempts.

What we look for

Our process is meant to find us engineers that are a great fit for our team. A great hire for us has the following characteristics from a technical standpoint:

At least a few years of experience writing code, ideally in a production setting.
An ability to build new services, features, etc. quickly.
An ability to debug things quickly. They should be fast at finding out the root cause of issues, big and small.
Enough experience with systems and tools to operate independently on a day to day basis. Get stuck infrequently.
Good judgement around what constitutes readable & maintainable code and what doesn't.
Great communication skills within technical discussions.

A few things that we explicitly don't care about:

Knowledge of textbook algorithms and data structures.
Knowledge of specific frontend or backend languages.
Being good at coding while someone is watching aka "grace under pressure."
Being good at talking or explaining your reasoning while coding.

There are other behavioral & non-technical characteristics we look for (e.g. do they really want to work at a startup, are they a nice person), but these are the primary technical ones.

It turns out that screening for these well is really hard.

Our first shot

Initially, we had a pretty run-of-the-mill interview process:

Initial screen with me (technical founder) to gauge experience, background and interest
Live coding interview on the phone with an engineer
Onsite interview, mostly consisting of building a small app together

Let's talk about #2. We gave candidates a coding problem to solve with Repl.it. Here were the rules:

The interviewer stays on the phone the whole time.
The candidate can use any language they want.

The problem had a relatively straightforward recursive solution and a slightly more complex iterative solution.

Within the first ~ten interviews, we ran into a bunch of problems:

Unless a candidate figured out how to use recursion within the first 20 minutes, they were almost always doomed (even though an iterative solution exists!). We didn't actually care whether the candidate had prior experience with recursion. As a result, we ended up measuring the wrong thing.
People get incredibly nervous during interviews, doubly so when they have someone on the other end of the phone that's watching them code. I ran the process with a few of my friends who I thought highly of as engineers. Some of them couldn't get through the problem because their brains just froze while I was on the phone with them. They could all do the problem after the interview had ended without any hints. Clear red flag.
Even when you let people pick their language of choice, they sometimes can't remember how to do stuff and need to Google it. It makes zero sense to prevent them from doing this if measuring language familiarity isn't the primary goal.

Then, we get into the onsite interview. We actually did a pretty good job with this and although we've adapted it over time, it's been a very high signal way for us to evaluate candidates. Here's how it worked:

A brief sync with me (the founder) on background, interests, Verse's cultural values and other non-technical stuff
Working with two engineers (and sometimes me) on building a CLI tool incrementally

The original tool we had candidates build had a few interesting characteristics:

You read data out of a MongoDB database
The objects in the database were "linked lists" in that every time you changed an object, it would be duplicated and the old object would be put in the prev key. Something like this:

// Inital object
{ 'a': 1, 'b': 2 }

// Made an update
{ 
  'a': 5,
  'b': 8,
  'prev': {
    'a': 1,
    'b': 2
  },
}

This structure has the benefit (drawback?) of being pretty tricky to reason about. But, the task of building the CLI tool didn't require any meaningful pre-existing knowledge of linked lists.
As a result, we got a pretty strong sense of each candidates' ability to reason about and communicate complex concepts.
Pretty much every candidate had to debug lots of things, starting from connecting to Mongo with Python to details of how to operate on the weird linked list storage structure. Seeing the person think and work through the problems was very illuminating.
This really gave us a clear idea of the "practical" skills the candidates had, e.g. do they know how to check if a process is running with ps ?
Really strong candidates never got stuck. They could always figure a way out.

So far so good for the onsite.

Take two – the takehome

Clearly, our phone process had a lot of problems. We tried what seemed like a spectacular and popular idea: take home assignments.

We didn't really care if people Google'd stuff, so why not just send out a takehome? Sure, some candidates would cheat by just getting someone else to do it, but they'd probably be found out when we discussed the takehome with them.

Our takehome problem worked like this:

You're given a SQLite3 database consisting of click activity with cookie IDs and IPs for each click
All the clicks are of a single user session if they share either a cookie or IP at any point within the clicks
This means that if a click had (IP1, Cookie1), then another had (IP1, Cookie2), then another had (IP2, Cookie2), they would all be in the same session as (IP1, Cookie1). This is true even though (IP1, Cookie1) and (IP2, Cookie2) don't have any overlapping identifiers.
We had the candidates build some simple querying methods as a CLI tool on this dataset.

Unfortunately, this takehome approach didn't turn out to be useful. A few things that went wrong:

For better or for worse, we as an organization really care about speed of execution. As a consequence, we care about how fast engineers can build new stuff and debug existing code. We couldn't get any indication of how long candidates spent on the takehome. As a consequence, we had no idea how long it would took them to resolve issues that they ran into.
We had incredibly few candidates fail to complete the takehome! This could suggest that we were needlessly rejecting a ton of candidates previously, but...
We noticed a majority of candidates that did really well on the takehome didn't do nearly as well on the onsite. They were often missing pretty crucial practical software engineering skills (e.g. how to kill a process, how to check if an endpoint is working, etc.) as well as the ability to debug relatively simple issues quickly. There was often a night and day difference in candidate performance on the onsite and takehome.

In fact, the onsite performance for these candidates was significantly below par other candidates we'd qualified through the phone interview process, e.g. takehome-qualified candidates would frequently get stuck in resolving relatively simple bugs whereas phone interview-qualified candidates would solve them far more quickly.

At the end of the day, the fail rate for the onsite was too high for candidates that we qualified through the takehome. We realized the takehome provided very little signal to us.

Take three – The "Hang Up with Hints" Phone Interview

So, we brought back the phone interview, but in modified form. This is how we set it up:

The candidate can still use any language and is also encouraged to Google stuff throughout the interview. We just ask that they cite code that they find online.
The interviewer calls the candidate, walks through the problem, answers any questions and explains the rules.
The interviewer then hangs up. The candidate continues coding in a shared Repl.it instance. The interviewer observes the candidate code but doesn't stay on the phone.
The candidate can call back the interviewer with questions at any time.
The interviewer will call with a check-in at 30 mins. At this point, if the candidate hasn't figured out the basics of a solution, we try to provide a hint (e.g. "use recursion.")
The interviewer will then do a final wrap up at 60 mins.

The key is that the interviewer isn't on the phone with the candidate throughout but still watches them code. This has worked remarkably well for us. It solves a few issues that we found with other approaches:

Candidates were much, much less nervous than a traditional phone interview (we've asked!). Although the interviewer is still watching the person code, it's much less nerve wracking than having the interviewer on the phone throughout.
We still get a lot more signal in comparison to the takehome when it comes to debugging ability and the candidates' thought process when confronted with problems.
Providing a clear hint reduced the "you know it or you don't" nature of certain problems and still left enough time for strong candidates to build a solution. This was true for our problem that could be solved with recursion.
This approach is somewhat similar to a HackerRank problem. But, the difference is that because there's a live person involved, cheating is harder and, more importantly, we can ask questions if we want to get a better understanding of a candidate's approach or code.

In general, this system has worked well for us and for candidates that we've been able to get feedback from. Candidates who make it through the screen have performed fairly well on the onsite.

Caveats

There are still a few core difficulties with our phone interview process.

We can still screw it up entirely with a bad question. If we pick a LeetCode question, we'd just just be testing if the candidate knew to practice LeetCode. If the question is too language or technology specific, we'd just be testing if the candidate has exposure to the technology. Having a better phone interview experience doesn't do anything to solve these issues.
Although the new phone process gives us much more signal than the takehome, it's still less signal than being on the phone with the candidate the entire time and really understanding how they are reasoning through things. In our view, being able to better evaluate candidates who might get nervous in the traditional phone interview setting is a worthy exchange to losing some signal.
The interviewer has to be super attentive to the shared coding environment to get a clear understanding of how the candidate is working through the problem. If he/she tabs away and starts working on something else, this approach just turns into a HackerRank-like problem.

Until more of these "explain and hang up" phone interviews start cropping up everywhere, we're excited tout them as part of our relatively painless process at Verse :)