Do Coding Interviews Work?

I have recently come across some interesting information regarding coding interviews. If you are not familiar with coding interviews, these are interviews for technical people, usually software developers, to prove that they have the ability to code so they are sometimes referred to as programming interviews. These can be either taken as a computer-based test or frequently done as whiteboard exercises. They often take the form of brain teasing riddles or binary search questions. The premise is that these coding interviews, conducted in an arbitrary environment, are a good proxy for determining whether or not someone will perform well in the real world.

WhiteboardingAs with all things, instead of relying on our human instinct, which is riddled with cognitive biases, we must rely on science to understand true cause and effect. The science has spoken loud and clear; there is no relationship between coding interviews and performance on the job for software developers. Don’t believe me; here’s what Laszlo Bock, Senior Vice President of People Operations at Google had to say on the topic:

…everyone thinks they’re really good at it. The reality is that very few people are.

Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess…

It appears to me that very often the interviewer is much more concerned with showing the candidate how astute he or she is as opposed to finding out whether or not the candidate is a good fit for the position. I recently read a blog post that stated that candidates should spend a great deal of time preparing for these coding interviews, in the neighborhood of about 40 hours. While this might be what it takes to “ace” such an interview, it still begs the question of whether the coding interview is actual predictive of the candidate’s ability to function in the position. It is not.

InterviewThis is where the cognitive biases come in. It appears that there is a great deal of the illusion of control, which, as humans, we are highly susceptible to. We think that somehow we are able to ask some questions and magically be able to determine how one will perform on the job. I would expect there is a bit of confirmation bias because we are subject to cherry-picking our evidence to support our previously held views (i.e. coding interviews are effective) and a similar bias called choice-supportive bias which is the tendency to remember one’s own choices as better than they actually are. I am certain that a whole host of other biases can be brought forth which not only explain why we think coding interviews are effective when there is evidence to the contrary, but also the stubborn way in which these have continued to persist in spite of such evidence.

In my career I have taken a few of these interviews and I may have my own biases since I don’t recall ever getting a job offer after one of these interviews. I remember taking one many years ago on SQL and ETL. I had been doing SQL and ETL quite successfully for over a year and knew I could perform very well in the position.

QuizNevertheless, the test was taken not on my own computer, but a computer that I was wholly unfamiliar with, a laptop with a built in mouse. I remember that I had some frustration just with the configuration of the computer I was using. I also remember that the majority of the questions I could have easily answered had I been able to use reference materials like I would be able to do in the real world. It felt like the test was measuring how well I could fix my parachute after I had been thrown from the plane. It did not measure how I would perform on my job, but how well I had memorized simple syntax that is probably not worth memorizing.

I know there are those who will say that one should remember such commands, but given that the average programmer contributes five lines per day to the final product, does it really make that much sense? Perhaps it would be better to fill one’s mind with other more important things? What I do know is this – had I been offered the position I would have outperformed many who would happen to ace this test because I have a wealth of experience outside of the ability to memorize coding syntax.

In a recent blog post I wrote a tongue-in-cheek title, “Accenture Ends Annual Review (and Admits Earth Orbits the Sun)”. Of all my dozens of blogs (I have posted over 100 over the years), this was perhaps the most provocative of them all and certainly the most popular, with literally thousands of views. In this case it took literally decades to finally admit what science has taught us with respect to annual reviews. Therefore, I expect that coding interviews will be with us for some time to come, but at least I can look forward to the day when I write the blog “Company X abolishes the coding interview (and Admits Earth is Round).”

Leave a Reply