Malcom Gladwell entertainingly speaks about mismatch problems: in essence, most hiring/selection practices attempt to evaluate performance in one area (the job, the task for which the person is being selected) through the use of seemingly related tests that don’t correlate well to performance in that area.
He uses concrete examples of sports, teaching and law, showing how selection criteria used to pick players in drafts, teachers for schools and candidates for law school don’t seem to correlate, or in some cases, negatively correlate with future performance.
It’s not hard to relate this quickly to the kinds of things we do when trying to hire developers, architects, managers. Unfortunately, the information that makes these studies compelling is good supporting data, and I don’t have that. Certainly part of the problem is that we don’t have a particularly good measure for quality when it comes to developers, even once they’ve been practicing their art — we don’t have a score that accurately reflects the skill of a developer at developing software.
That is, however, one of Gladwell’s final points — that our desire for certainty leads us to concrete measures that take us astray when subjective measures may, in fact, be more accurate, despite their subjectivity. Maybe all we really need is a score that aggregates the subjective opinions of those who have worked with you in the past.