Psychometric Assessments: the Good, the Bad, the Ugly

I consider myself fortunate to have been mentored by some of the most renowned Organisational Psychologists. With their guidance, I’ve received incredible experience developing a variety of psychometric instruments, some of which have been incorporated into the recruitment and workforce development processes at large corporations.

For example, the Attitude Advisor; an assessment built for the UK’s Driving Standards Agency that evaluates a driver’s attitudes to risk-taking, or the Train Driver Aptitude Assessment designed for the UK’s Rail Safety and Standards Board, which measures a person’s suitability to become a train driver.

Under their guidance, I learned what it took to construct valid and reliable psychometric assessments. I learned the importance of measurement theory, and I learned (the hard way) that a deep understanding of statistical analysis is non-negotiable. I gained a real appreciation for the art that is survey design.

Psychometric assessments provide the psychological basis for a streamlined and dependable recruitment process for companies today. However, since the global assessment market lacks regulation and standardised review processes, they don’t always deliver what they were intended. This post will look at the disparity, power of assessments, and how they revolutionise organisations’ hiring processes.

Not all assessments are created equally

Although they all require it, not every assessment is put through a rigorous psychometric review process. Consequently, some assessments can yield inaccurate results. From a recruitment perspective, this is concerning as it can lead to poor hiring decisions as well as legal and ethical issues. I believe that most assessment developers have good intentions, yet due to poor understanding of the complex statistical processes involved in their design and validation, few are able to produce high-quality assessments that are accurate, fair, and focused on the things they are designed to measure.

This very situation has resulted in the growth of ‘amateur’ psychometric assessments. Often, these amateur assessments are pushed onto the recruitment market by developers with little awareness of the potential damage caused for the organisations that use them, plus the job candidates that complete them.

I’m looking to support the great psychologists who have spent their careers constructing valid and reliable instruments. These are people who have devoted their lives to building the integrity of our profession and reducing the negative stigma surrounding psychology and psychometric testing. I am also hoping this article will help recruiters and employers identify these ‘amateur’ assessments and understand the importance of choosing the right instruments.

What are psychometric assessments and why use them?

Measuring attributes like height, weight, and strength is reasonably simple. These are all physical and observable traits that you can assess objectively.

But what about factors that aren’t so easy to measure?

Traits such as personality, intelligence, attitude, and beliefs are important characteristics to assess and understand for identifying if a person is likely to fit within a job and perform their role successfully. Since a business is only as good as its people, understanding the human factor is important in recruiting and developing employees that fit with a company’s culture and work environment. Effective psychometric assessments seek to understand these hidden aspects of an individual and compare the results with the job requirements.

Broadly speaking, psychometric assessments are standardised instruments, exercises, or questionnaires that measure individual capabilities, cognitive functioning, behavioural preferences, and personality constructs. Well-designed assessments are perceived to be objective and predictive in filtering out candidates that aren’t a good match for jobs or organisations. If leveraged properly, these assessments can help fill every job vacancy with high-quality candidates.

Psychometric assessments in business

Employers want to know everything they can about potential employees, and they want that information to be accurate. A well-designed, administered, and interpreted psychometric assessment can provide a relatively accurate and insightful picture of a candidate’s suitability for a role.

Psychometric assessments are now used consistently by over 80% of the Fortune 500 companies and by over 75% of the Times Top 100 companies in the UK. This increased practice forms part of a broader trend towards the application of more structured, systematic, and competency-based recruitment and selection practices across industries. This development is the result of the cost of hiring the wrong employee; it can be astronomical.

Psychometric instruments are seen as good insurance against recruitment mistakes. Assessments help ‘weed out’ the bad choices and reveal the exceptional performers. They can also help you assess how good a ‘fit’ someone is to your organisation and work teams.

What’s the scientific evidence?

The accuracy with which psychometric assessments can predict individual job performance is well documented in the research literature. The strength of this relationship might be stronger than you initially think and gives weight to the argument of using assessments for recruitment and development purposes.

For example, a UK study looking at 56 papers reporting 283 samples (meta-analysis) found that assessments of specific cognitive abilities are strong, reliable, and valid predictors of both job performance and response to training (Bertua, Anderson & Salgado, 2005).

Operational validities were in the range of .5 -.6, meaning that cognitive assessments accounted for approximately 30% of the variance in job performance between candidates. It was also found that their ability to predict job performance is strongest for more complex roles, such as professional and managerial roles.

Psychology IS stats

You can be forgiven for thinking that a psychometric assessment is just a list of questions that somebody decided to write one day. Let me assure you there’s much more to it than that.

Modern psychometricians (what a title) are more statistician than psychologist. Psychometric assessment design involves a complex statistical process to ensure the assessment is accurate, fair, and only focuses on the things that it’s designed to measure. Assessments are constructed using an accepted set of rules and can take many years and several thousand trial subjects to produce.

One of the common pitfalls of psychometric assessments in the corporate environment is the erroneous belief that anyone can design, administer, and debrief them.

Adding to the problem is the fact that the global assessment market isn’t equally regulated. Anyone can write an assessment and supply it for use in a totally unrestricted manner, regardless of quality. The increased use of the Internet as an administration medium hasn’t really helped with this situation either.

Evaluating the quality of psychometric assessments is mainly concerned with three criteria: reliability, validity, and fairness. Validity asks the question ‘Is the assessment measuring what it says it measures?’. Reliability looks at how consistent assessment results are over time. Fairness asks the question ‘Does the assessment discriminate against anyone based on factors unrelated to what the assessment is suppose to measure — for example race, ethnic origin, nationality, religious a orientation or gender etc.?’

It’s more than just a pretty face

Even though the design of psychometric instruments is a very complex process, people still get away with their amateur attempts.

How is this possible?

Well, like most things, Rrst impressions count and we too often judge a book by its cover. What an assessment looks like is called face validity. As the name suggests, face validity relates to the physical appearance of an assessment rather than to what the assessment actually measures. This distinction is very important.

An assessment that seems, on the face of it, to measure what it claims to measure, has good ‘face validity.’ In other words, if I believe that an assessment looks legitimate, it will give me conRdence in the quality of the assessment and will help keep me motivated as I’m taking it. In terms of sales, an assessment with good face validity helps convince potential buyers (e.g., supervisors, HR staT, executives, etc.) to “buy in.”

However, what many do not realise is that even if an assessment has good face validity, it does not mean it is a good assessment and able to measure and predict what it claims.

On the flip side, an assessment that lacks face validity can still be relevant and useful, even if assessment takers poorly receive it. A good example of this is cognitive ability assessments. Many job applicants hold negative views and experiences with cognitive ability tests, questioning the applicability and face validity of these measures. Yet it is one of the best ways to predict job performance.

“Ultimately, face validity may be more a matter of public relations than psychometric soundness, but it seems important nonetheless.” (Cohen & Swerdlik, 2009, p. 176)

Legally, ethically, and statistically it is important to ensure you use an assessment that has good psychometric properties, beyond just a pretty face. An assessment with good face validity yet weak content, criterion, and/or construct validity, and poor reliability is useless.

What can we do?

Before initiating any psychometric testing, you need to produce two documents:

1. Job Description: Prepare a job description by carrying out a systematic job analysis. This analysis identifies the key tasks required for the job in question.
2. Person Specification: This document outlines the key characteristics a prototypical candidate needs in order to successfully complete the tasks described in the job description. The entire recruitment process needs to be geared towards assessing the characteristics listed in the person specification. Without it, the choice of any selection tool is a little more than random. When a client asks me, “Can you come and

   administer a suite of psychometric assessments for us?” I always ask to see the job description and person specification first so I can decide which suite of assessments would be best suited.

Once the person specification is written, you can then decide on the most appropriate assessments or measures. Sometimes, an interview, presentation, or other form of activity may be chosen in preference to a psychometric assessment. The key point here is that assessments aren’t just used for the sake of it when a better alternative selection tool exists.

Need Help?

I am often asked by companies to review the quality of psychometric assessments and make suggestions regarding the effectiveness of their recruitment process.

If you or your organisation have any similar questions or would like to know more about how to identify high-quality assessments, feel free to connect on LinkedIn.

Additionally, find out more about our intelligent hiring platform below.

INTELLIGENT HIRING PLATFORM

Your company is unique.

We use predictive insights to match you with candidates that are a great fit.

Let us do the heavy lifting, so you can accelerate the hiring of candidates who are most likely to succeed. 🚀

Find out more