John Sullivan wrote a great piece on ERE a few months ago, titled Five Ugly Numbers You Can’t Ignore. John’s article pointed out public research indicating fundamental flaws with the interviewing and assessment process used by most companies.

As a result of John’s article, I participated in a series of animated discussion on these ERE pages regarding the relative impact of increased interviewing accuracy on improving quality of hire. Now I know the academics among us get excited when they believe that better assessments directly correlate with increasing quality of hire, but according to the Recruiting Roundtable — a well-respected research group — research suggests this is not actually true.

In a recent public report it compared the impact nine variables had on improving quality of hire and time to hire. Interestingly, at least according to their research, accurate interviewing and assessments had no impact on improving quality of hire. Regardless, it is a gate to pass to get into the game. The top three for improving quality of hire were the need for a strong recruiter and hiring manager partnership, a clear understanding of job needs, and the recruiter’s ability to convert candidates at every step, from prospect to hire. This last point has to do with keeping the candidate engaged, overcoming concerns, presenting the job as a career move, negotiating offers, and keeping the competition at bay.

Leading some credence to the Recruiting Roundtable results is a report from Leadership IQ documenting a three-year survey it conducted with 5,247 managers covering more than 20,000 hires. The big conclusions — 46% of new hires fail within 18 months, with only 19% totally successful. The biggest surprise of them all was that the interviewing methodology used didn’t affect the results. I find this confusing, since I know that conducting an accurate assessment is a necessary, even though it’s not a sufficient aspect of improving quality of hire. The report went on to suggest that managers overvalued technical skills instead of evaluating other aspects of on-the-job performance, including motivation, emotional intelligence, coachability, and temperament. This alone indicates that the candidates were not interviewed properly, and to some degree, puts in doubt some of their other conclusions.

So while there is some data out there that contradicts published research from some of the top names in academia, it’s hard to believe that accurate assessments aren’t important, since without having a qualified and motivated candidate, you’ll wind up with a bad hire. Perhaps the problem is associated with curvilinearity, meaning once a threshold level of sufficient capabilities is met, recruiting skills takeover as being far more important in improving quality of hire.

The academic research does suggest that while a validated and structured interview is important, it might not be all not that important in the overall scheme of things. For example, the often-cited Schmidt and Hunter study reports that the combined correlation coefficient for a structured behavioral interview and GMA test is .63. In practical terms this means that only 36% (square of the correlation coefficient) of the candidate’s predicted on-the-job performance can be explained by these two factors, leaving 64% of job performance due to other factors. I’m surprised that more has not been made of this critical point. By itself, this might be the explanation as to why the Recruiting Roundtable and the Leadership IQ reports that interviewing accuracy has much less of an impact on quality of hire than would be expected.

Taking a different perspective entirely, in some cases, an accurate assessment can actually be counterproductive, especially when good people refuse to move forward until they make the determination the job offers a career move. This is why I suggest putting the necessary assessment process later in the hiring process to maximize the end-to-end conversion rate without compromising quality. Of course, this is a moot point when the supply of quality candidates exceeds demand, a rare situation in normal economic times.

Adding to the supply shortage dilemma, many of the best people — especially those with significant upside potential — are looking for career moves and learning opportunities. In these cases they might not have the requisite skills, knowledge, and abilities, and could be excluded for the wrong reasons. This relates to the classic potential vs. experience trade-off problem.

On another level, the relationship between interviewing accuracy and quality of hire is further distorted since they are separate and unequal tasks. If you are using the lens of maximizing selection accuracy as your primary objective, you might overlook the bigger challenge of hiring the best people possible including those who aren’t looking, those who have more potential than experience, those who have a different mix of skills, and those who have multiple offers. Each of these factors requires a rebalancing of the sourcing, recruiting, and selection process in order to maximize quality of hire. This is pretty much what the Recruiting Roundtable results indicated.

When supply is less than demand, a myopic maximize-assessment-accuracy objective leads to the potential for sub-optimization, or sacrificing the whole for the sake of one its parts. I experienced this problem firsthand early in my pre-recruiter career. Many, many years ago, in a place far, far away, I was involved with negotiating company transfer prices with a brake plant that wanted to sell spare parts to Ford and GM, rather than to an internal axle assembly plant, since it got a better prices by going rogue (external). This caused corporate earnings problems since the parent company not only made more money selling completed products, but worse, never had enough brakes to meet the demand for completed axles. It took six months to figure out the problem and develop an internal transfer pricing system to make sure the brake plant did the right thing. This is similar to having accurate assessments, but not enough good people to be interviewed. As a result, you’re left with assessing a population of people without any top performers in it, making the conclusions suspect. Some of the research mentions this as a potential problem with their data.

The way I see it, this apparent assessment vs. quality of hire controversy involves three big issues:

1. The fact that most assessments – even good ones – don’t cover the complete range of factors involved in measuring top performance. Some of these include subordinate and managerial fit, intrinsic motivation to do the work required, achievement of comparable results, trend and consistency of performance over time, and the ability to work with and influence teams of comparable size, level, and functional makeup. Measuring these multiple times and in multiple ways can increase assessment accuracy.

2. Ignoring the idea that the assessment is only a subset of the hiring process, not the complete hiring process, and that the linkages are not generally seamless. Just because someone is judged a top performer doesn’t mean the person will be hired. Problems here relate to recruiting skills, the hiring manager’s ability to attract a strong person, the career aspects of the job in comparison to competing opportunities, and the compensation.

3. Fundamental problems with how the interview and assessment process is implemented and how hiring decisions are made. Problems here generally involve lack of clarity with respect to the actual performance needs of the job, lack of hiring manager training, the use of a yes/no “add up the votes” decision-making process, not using evidence to make the decision, using a narrow band of selection criteria, and over-valuing presentation skills, affability, and intuition when making the decision, among others. Eliminating these is an essential aspect of the hiring process.

Given all of the survey evidence, the academic research, and my own personal experience of dealing with top performers and also-rans over the past 40 years, I would not discredit the necessity of a thorough interviewing and vetting process. However, I do believe that the traditional behavioral interview is far from the perfect solution, and could be a contributing factor preventing companies from improving quality of hire. There are interviewing and assessment solutions available that have been proven to be more accurate, but without better sourcing, a great recruiter, a clear understanding of job needs, and a strong recruiter/hiring manager partnership, you won’t be much better off. In this case, you’ll just be more confident you’re hiring someone in the half that makes the top-half possible.