Competence vs. Confidence

Do you feel confident?  Try this quick 2 minute Test of Confidence before continuing.

One of the first pieces I wrote in this blog, The Flow – Programming in Ecstasy, explored the factors that help us get into The Zone.  Most writing on this topic emphasizes the need for silence and lack of distractions.  While that is important, I learned of other important factors beyond the ability to concentrate: serenity (the absence of worries), direction (a sufficiently defined task and path to progress), confidence (to accomplish the task) and competence (again, to accomplish the task).

The last two factors, Confidence and Competence, weave their way through most articles I have written.  Despite confidence and competence being different in reality they are often confused.  So confidence and competence deserve some focus in their own right.

Confidence: the Perceived Proxy of Competence

Perhaps the definitions are obvious but let me state them.

Competence: the ability to perform a specific activity or job properly.

Self-confidence: self-assuredness in one’s personal judgement or ability.

Projected Confidence: ability to instill confidence in others of your competence; to instill trust in your competence.

As the third definition implies, it is natural for people to interpret a confident manner as evidence of competence.  People who project confidence are more likely to be perceived as competent and trusted by others.

It follows that lack of projected confidence is often perceived as lack of competence.

Is it accurate?  Is it fair?  Often, it is neither.  But it is the way people operate.

Perhaps it is because judgements of confidence are instinctual and subconscious making them easier than judging competence.  Judging competence needs an investment of time and energy plus it usually requires the cooperation (and honesty) of the person being judged.

As an important example, hiring staff is an important area in which I have found the conflation of competence and confidence to be both common and problematic.  At one place I worked the boss would send me technical candidates with glowing recommendations after meeting them some place or other.  I’d set up meetings and quickly determine the candidate to be technically weak despite their excellent presentation and confidence.  The boss was responding to the projected confidence and I was looking elsewhere.  Later, as a hiring manager I found myself doing exactly the same thing as my old boss!

The lesson for me was to be conscious of when I was reacting to a person’s confidence and when to actively investigate their competence.

Confidence tricks are a more extreme example of mis-perception though in this case the deception is deliberate and the con-artist is playing on a person’s greed or other characteristic.

The Paradox of Confidence

“One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision.” Bertrand Russell

The Dunning-Kruger Effect is working itself into pop-psychology (perhaps encouraged when it was awarded an Ig Noble prize?)  At the end of the last millennia Justin Kruger and David Dunning authored an article entitled “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”.

You have seen an extreme Dunning-Kruger Effect if you have watched the early rounds of any reality show like “Pop/American/Norwegian Idol” or “So You Think You Can Dance” or watched the spectacular failures replayed on YouTube.  It is the contestant who is so utterly incompetent that you wonder why they ever considered participating.  And these are simply awful…

Back in real reality, Dunning and Kruger did a series of experiments with students asking them to evaluate their performance relative to their peers.  The experiments involved tests of logical reasoning, English grammar and humor plus self-assessment of test performance.

Their first finding was that below-average students tended to over-estimate their competence relative to peers.  Their paper concluded that people with limited knowledge in a domain suffered a dual burden: not only was their performance poor, but their incompetence robs them of the ability to realize it.

They also found a “Burden of Expertise”.  Highly competent individuals also showed a systematic bias in their self appraisals but, unlike the below-average students, they underestimated their ability relative to their peers.  They attributed this to the False Consensus Effect: the assumption of a high competence person who performs well that assumes that everyone else would have performed well too.

Putting the two results together, they found an inverse correlation between confidence and competence referred to as the Paradox of Confidence by some.  That is the opposite of what most of us assume with our natural tendency to ascribe competence to confident people.  Put more cynically, the dumb get confident while the intelligent get doubtful.

A search for the Dunning-Kruger effect shows numerous claims of the effect for people’s ability or knowledge of driving, spelling, climate science, singing and many others tasks.

While labelling somebody as unaware of their incompetent is universally popular in the blogosphere, in my programming career I found the other end of the spectrum more compelling: that is, highly competent people who underestimated their capability.

Interestingly, there may be important cultural effects at play.  Studies in the USA find the negative correlation whereas in Europe the effect is more muted and in East Asian studies it reverses (that is, confidence and competence are positively correlated).  I grew up in Australia and worked for extended periods in Japan and the USA and my experiences certainly reflect this.

The Over-Confidence Test

Often the desire to appear competent impedes our ability to become competent because we are more anxious to display our knowledge than learn what we do not know.Madeleine de Souvré, marquise de Sablé (1599-1678)

Did you take the 2 minute Test of Confidence at the start of the article?  If your internal 90% confidence level is properly calibrated then you should be right 90% of the time.  On the 10 question quiz 94% of people should get 8 out of 10 right or better.

I did a lot worse – under 50%.  I only got 7 out of 10 when I repeated the test today, about 2 months after first taking it.  Granted, I am no expert on OPEC or the Nile River but my general knowledge is pretty good I think.  The point is not what I know but my (in-)ability to judge my own competence.

Software Effort Estimation

Estimating the length of the Nile River or the airspeed of an unladen swallow are hardly important in professional life but we are often engaged on topics we know little about.

For example, I wrote about the psychological biases that add to the intrinsic challenges of software effort estimation.  The two articles Software Effort Mis-Under-Estimations (part 1) and Biases create Software Effort Mis-Under-Estimations (part 2) cover some of the emotional terrain of estimation.

About a decade ago I started asking developers for a “90% estimate” with the clarification that 90% of the time the actual time should be no worse than the estimate.  It didn’t take long to notice that for software of reasonable complexity the 90% estimate was more like a 50% estimate and the blow-outs could still be massive.

I still think a “90% estimate” is a better request than a mid-point estimate or simply “how long will it take”.  However, studying confidence (or more accurately, false confidence) gives me a better understanding of the discrepancy.

Making Better Judgements

In background reading for this post I’ve seen a range of general advice for assessing other’s competence.  This is my distillation.

Awareness of limitations.  The competent know their limits.  As Dirty Harry put it, “A man’s got to know his limitations.”

Desire to learn.  Knowing the limits of one’s professional knowledge should provide a desire to learn and improve. The desire should be evident through curiosity, development of new skills and openness to new ideas.

Willingness to fail and learn from failure.  Competence grows by stepping outside one’s comfort zone, being prepared to risk failure and to learn from the experience.  This is not about recklessness but about calculated risk (neither grossly over- nor under-estimating risk).

If you would like to explore further by understanding more of the biases that effect judgements you could do worse than reading up on a list of biases that lead to over-confidence (they have great names like the planning fallacy, illusion of control, the wonderful Lake Wobegon effect and comparative optimism).

Let me close with a final warning.  I am no expert in psychology and the whole of this article may be tainted by the Dunning-Kruger Effect.  Fortunately my ignorance of my incompetence is bliss.

Andrew Hunt – Psygrammer


3 Comments on “Competence vs. Confidence”

  1. […] classified two posts as Advanced: Competence vs. Confidence and Questions of Dominance.  I can’t see a reason for the distinction.  Nor can I see […]

  2. Jeff says:

    I’ve always be fascinated by the tendency for people’s self-assessments to be taken as accurate. People who believe that they are highly intelligent or talented tend to be viewed as such by their peers, regardless of the reality. I worked with a programmer who did things as ludicrous as putting 108 ‘if’ statements in a single function who not only was convinced that she was an extraordinary programmer but managed to convince most of the staff, including management, of this, to the extent that people who disagreed with her were actually fired.

    • Samium Gromoff says:

      Jeff, I think that this wonderful term — reality distortion field — is reasonably close to capturing the essence of the phenomenon..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s