Last week I wrote about coding interviews and questioned whether they are the best method to predict future job success. There were strong opinions on both sides of the argument. Someone expressed the opinion that I was taking a quote from Laszlow Bock, senior vice president of people operations at Google, out of context, so I would like to begin by giving him the first word on what Google has discovered about interviews. Interestingly, he refers to cognitive biases, specifically confirmation bias (as I did in my post) as a reason traditional interviews are not good indicators of job performance.
In April 2015 he wrote an excellent article for Wired Magazine and I encourage everyone to take a look so that they can form their own conclusions. I will quote at length in order to make sure that Mr. Bock’s ideas are not taken out of context. In the article he stated:
In 1998, Frank Schmidt and John Hunter published a meta-analysis of 85 years of research on how well assessments predict performance. They looked at 19 different assessment techniques and found that typical, unstructured job interviews were pretty bad at predicting how someone would perform once hired.
Unstructured interviews have an r2 of 0.14, meaning that they can explain only 14 percent of an employee’s performance. This is somewhat ahead of reference checks (explaining 7 percent of performance), ahead of the number of years of work experience (3 percent).
The best predictor of how someone will perform in a job is a work sample test (29 percent). This entails giving candidates a sample piece of work, similar to that which they would do in the job, and assessing their performance at it. Even this can’t predict performance perfectly, since actual performance also depends on other skills, such as how well you collaborate with others, adapt to uncertainty, and learn.
I believe that the work sample test he refers to is analogous to the coding interview. If not, I will error on the side of caution since equating the two would mean that coding interviews are the most effective. Even so, the very best interview technique is only twice as effective as an unstructured interview in predicting job performance. I hope some statisticians can weigh in on the topic, but an r2 value of 0.29 does not seem to be very indicative of future job success. I vaguely remember from college that in physical sciences we look for much higher r2 values before we accept a hypothesis as proven.
Bock goes on to claim that using a combination of interview techniques in a “structured interview” is even more predictive (though he does not present any r2 values for the combination of techniques). And, believe it or not, I agree with him. He has presented some real scientific information that a structured interview process is better than an unstructured one, but I still wonder if we have committed a type 3 error (solving the wrong problem precisely).
Bock and I both agree that “the goal of our interview process is to predict how candidates will perform once they join the team,” but he also admits that doing things as he recommends is “hard to develop” and “a lot of work.” Even Bock states:
Full disclosure: I’m the Senior Vice President of People Operations at Google, and some of these (unstructured) interview questions have been and I’m sure continue to be used at the company. Sorry about that.
And this is where I would begin to question whether this process is one that everyone should follow. Of course, for highly desirable and highly capitalized companies like Google, Facebook, etc. the benefits likely outweigh the costs, but for the majority of companies that interview software developers, this may be a luxury they cannot afford. This overall hiring style is time consuming. There is not doubt that developers are willing to jump through hoops to work at Google or Facebook, but such is not the case for most companies where time is not a friend. If a majority of companies use this process they may find a much higher cost in potential candidates lost than the marginal effect of hiring better. This is the reality that many managers and companies face.
This begs the question of what should those who are not Google or Facebook, those without unlimited resources and legions of potential candidates, do to improve their hiring of software development professionals? The first thing is to be aware of the true costs and benefits associated with pursuing one path over another. Eschew convention and ask the hard questions like, “Given limited time and financial resources, is this the best use of either (or both)?” It appears that Bock agrees there is a high cost to more effective interviews, but for most companies is the higher cost justified? Is there a better way to allocate time and money? I believe there are things that should be considered and evaluated.
One thing I have suggested in the past is the concept of apprenticeship and creating a pipeline of talent from within a company. This is an essential aspect of good management and would reduce the need for interviewing new candidates. In my experience is that in software development in particular we often look for perfect fits and do not do a good job of creating a pipeline of candidates. This puts us “behind the eight ball” and increases the risk of hiring poorly, providing high pressure to make the right hire.
If we are truly looking for the best predictor of how well someone will preform on the job, then we could certainly achieve a much higher r2 value by actually putting someone on the job for a limited period (or at least give them an experience that would be as much like the job as possible). In response to my original blog, Duncan Campbell had perhaps the most insightful comment when he wrote:
The best way to find out if someone is good for a job is for them to do the job… which is why our “coding interview” is a real problem done in the candidate’s own time on their own PC followed by a code review.
I agree with Duncan, but it may even be cost effective to go further. Instead of spending many hours via interview, make the quick decision to bring the candidate into the company on a trial basis, perhaps two weeks at pay. We all seem to have projects that are less critical that would be a good proving ground for potential long-term employees. If the candidate doesn’t work out after two weeks, then part company. It may prove more cost effective for managers to actually do the work of managing existing people than spending a huge chunk of time in a laborious interview process.
Another possibility is the concept of contract to hire or using staffing agencies to do the heavy lifting. Full disclosure: I work for a company that has staffing as one of our offerings. Nevertheless, having another company take the time to screen candidates and having the employment-related risk owned by a third party does have its place in this discussion.
My original article, titled “Do Coding Interviews Work?”, was purposely open-ended. I have tried my best to present information to help people answer that particular question. The title was not “are there better ways to interview software developers?” for a reason. I am truly questioning any interviewing technique because of the high cost and the low correlation – even when interviews are conducted perfectly (or as least as perfectly as science has instructed us). I do not think that we will get rid of interviewing altogether, but it is important to know that for a great majority of companies there may be alternatives that, given all the costs and benefits, may be more effective ways of answering the larger question of how best to hire. I gave a few suggestions. I look forward to others.
Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Android | RSS
