The OpenWater Blog

Best Practices for Awards Judging Criteria: Avoiding the Halo Effect

Are you confident that the questions you ask your judges adequately reflect the performance of the entry? Is your criteria structured in a way that ensures as much of an unbiased review as possible? Creating effective judging criteria for your awards program is not as straightforward as one might think.

There is a chapter in Daniel Kahneman’s New York Times Best Seller Thinking, Fast and Slow that describes an approach to conducting job interviews. While I was reading this chapter, it occurred to me that the method explained in the book is directly applicable to judging entries.

Kahneman elucidates that reviewers are often affected by what he calls The Halo Effect. Instead of analyzing a potential prospect on a set of core criteria, the reviewer often observes the prospect and comes to a summary conclusion of Pass/Fail. The Halo Effect is the aura surrounding what is being reviewed. In other words, if a prospect is incredibly social, handsome, and articulate, a reviewer is significantly more receptive to them even if they do not meet the qualifications for the position. By the end of the interview, the reviewer comes to a summary conclusion of hire / don’t hire. However, these summary conclusions over the entirety of a prospect, or entry in our case, are susceptible to The Halo Effect.

The solution is to avoid summary conclusions that allow even subconscious bias on the part of the reviewers to seep in. For many of our customers, they have multiple rounds of judging, where each round is simply “Pass – Forward to next round” or “Do not pass”. Another common question is, “What is your overall opinion of this entry?” These are examples of summary conclusions.

To avoid summary conclusions, your awards judging criteria should choose between three and six critical points that cover the quality of an entry. Then, you should ask the judges to score between 1 and 5. The total score of all criteria will be a better representation of the quality of the entry.

Some better examples:

“The concept, writing, and graphics of the piece are integrated and support the marketing message.” Score 1 to 5.

“The design reinforces the purpose, content, and organization of the piece.” Score 1 to 5.

“The marketing message is consistent and effective.” Score 1 to 5.

If you keep The Halo Effect in mind while creating your judging criteria, you will see a substantial improvement in the scoring of entries by your judges.

[button id=”” style=”” color=”orange” type=”large” size=”large” href=”http://www.getopenwater.com/subscribe-to-openwater-blog” align=”center” target=”_self”]Like what you’ve read? Click here to subscribe to this blog![/button]

photo: Lars0001 via wikipedia

Zack Schwartz

Zack is a generalist. His position here changes almost every day. He's been involved in pretty much every sector of the business ranging from sales and marketing to programming and product development. He enjoys being innovative and thinking of new ways to expand and improve the business.

Self Service Help and User Community Available 24/7

Oops! 4 Mistakes Associations Make When Planning Awards Programs