Enhancing selection of the healthcare workforce: a programme of psychometric epidemiology research

Effective treatments in medicine will not translate into better patient outcomes unless delivered by a conscientious, competent, compassionate and professional workforce. Consequently, selecting tomorrow’s doctors is a high stakes issue.

At present there is wide variation in how medical schools select entrants. Academic performance in the early years of medical school may be predicted by prior educational attainment and ‘aptitude’ test scores, which evaluate intellectual ability. However, little is known about which factors predict later ‘real world’ performance in qualified doctors, though there may be some early indicators regarding which individuals may be at risk of future disciplinary action. It is not currently clear which outcomes are the most important to predict; some may be more closely associated with patient care than others (e.g Fitness to Practise issues versus written exam performance). Thus, it is important to develop an evidence base for existing selection strategies that takes into account the differing nature of outcomes.

Recently there has been an interest in testing non-academic attributes when choosing future doctors. These may predict professionalism and include such features as empathy and integrity. Failure to select on these factors may have serious consequences for patients; several scandals have highlighted the importance of compassion and conscientiousness in healthcare. These include the issues relating to poor care at Staffordshire Hospital (Francis report, 2013) and the abuses of disabled individuals at Winterbourne View care home. These situations have led to the requirement for 'Values-Based Recruitment' (VBR) in the NHS. VBR involves attempting to evaluate the extent to which an individual holds, and exhibits, the values in the NHS constitution (working together for patients, respect and dignity, commitment to quality of care, compassion, improving lives, and

everyone counts). There are a number of ways to test for relevant attributes (e.g. at interviews). However, there is emerging evidence from occupational psychology and initial findings in qualified doctors, that suggest Situational Judgment Tests (SJTs) may offer an efficient way of testing for traits associated with professionalism. SJTs evaluate knowledge of professional behaviour via a candidate’s responses to presented social situations. Though promising, at this point it is critical to establish what role SJTs should play in the selection of doctors,

and, more widely in health service recruitment. It may be that SJTs can be used to identify and exclude candidates that are at risk of unacceptable workplace behaviours. Moreover, SJTs may be less prone to ethnic bias when compared to tests of intellectual ability. Thus, the effective implementation of SJTs to NHS recruitment has implications for patient care and safety as well as the widening of access to the health professions.

The purpose of this research was to:

1. To understand how we might improve the selection of UK doctors by analysing information from the UK Medical Education Database (UKMED)
2. To develop the use of Situational Judgment Tests to select staff into Mental Health Services

This fellowship funded a programme looking at selection into healthcare. Specifically we used data from the UKMED, which is a large repository containing information on all UK doctors who entered medical school in 2007, join training schemes from 2012. By conducting data analysis we were able to say what factors at selection were likely to be associated with certain desirable, or undesirable, outcomes later on in a doctor’s career. For example, we looked at predictors related to future fitness to practice issues, and continue with this component of the work.

A separate strand was focussed on understanding how situational judgement tests (written tests which portray scenarios which challenge professionalism) could be used to improve staff selection for mental health services.

We found a number of factors predicted a medical student’s success at postgraduate clinical exams taking during training, after qualifying from medical school. These included how well they scored on parts of the UCAT, selection test for medical school. We also found that performance at the SJT used in the UCAT, before entering medical school, predicted the risk of disciplinary action, as a medical student.

We were also able to look at the factors which predicted both a doctor’s success at applying for training schemes, and how they did once they were in these.

We were able to develop and SJT that could predict how professional a mental health profession was in their practice, as rated by a supervisor. 

Assessing Professionalism in Mental Health Clinicians: The Development and Validation of a Situational Judgement Test

Aylott, L., Finn, G. M. & Tiffin, P. A., 13 Sept 2023, (Accepted/In press) In: BJPsych Open.

The cross-cutting edge: medical selection and education viewed through the lens of emotional intelligence

Tiffin, P. A. & Roberts, R. D., 6 Oct 2023, (E-pub ahead of print) In: Medical Education. 10 p.

New evidence on the validity of the selection methods for recruitment to General Practice training: a cohort study

Tiffin, P. A., Morley, E., Paton, L. W. & Patterson, F., 24 Nov 2023, (Accepted/In press) In: BJGP open.

Research output: Contribution to journal › Article › peer-review

Evidence-based appraisal of the role of SJTs in selection

Sahota, G., McLachlan, J., Patterson, F. & Tiffin, P. A., 10 Sept 2023, (Accepted/In press) In: Clinical Medicine.

Research output: Contribution to journal › Article › peer-review

Tiffin, P. A., Sanger, E., Smith, D., Troughton, A. & Paton, L. W., Situational judgement test performance and subsequent misconduct in medical students. Medical Education. 2022 (E-pub ahead of print)

Morgan, LJ, Finn GM & Tiffin PA (2021- Accepted). Are efforts to recruit to psychiatry closing the stable door after the horse has bolted? Knowledge and attitudes towards a career in psychiatry amongst secondary (high) school students: A UK-based cross-sectional survey. The Journal of Mental Health. (In press).

Tiffin PA & Paton LW. 2020. Differential attainment in the MRCPsych according to ethnicity and place of qualification between 2013 and 2018: a UK cohort study. The Postgraduate Medical Journal, Online advance version postgradmedj-2020-137913

Aylott L, Tiffin PA, Brown S. & Finn GM. 2020. Great expectations: Views and perceptions of professionalism amongst mental health services staff, patients and carers. Journal of Mental Health, Online advance version doi.org/10.1080/09638237.2020.1818195

Webster ES, Paton LW, Crampton PES & Tiffin, PA. 2020. Situational judgement test validity for selection: a systematic review and meta-analysis. Medical Education, 54(10), 888-902.

Tiffin PA, Paton LW, O'Mara D, et al. 2020. Situational judgement tests for selection: Traditional vs construct-driven approaches. Medical Education, 54(2), 105-15.

Tiffin PA, Paton LW. 2020. When I say … emotional intelligence. Medical Education, 54(7), 598-599.

Aslet M, Paton LW, Gale T & Tiffin, PA. 2020. Evaluating the recruitment process into UK anaesthesia core training: a national data linkage study of doctors' performance at selection and subsequent postgraduate training. The Postgraduate Medical Journal 2020, 96(1131), 14-20.

Tiffin PA, Paton LW. 2019. Does ‘online confidence’ predict application success and later academic performance in medical school? A UK-based national cohort study. BMJ Open, 9(12), e034437.

Aylott LME, Tiffin PA, Saad M, et al. 2019. Defining professionalism for mental health services: a rapid systematic review. Journal of Mental Health, 28(5), 546-65.

Tiffin PA, Paton LW. 2018. Artificial or intelligent? Machine learning and medical selection: possibilities and risks. MedEdPublish 7, [4], 35.

Tiffin PA, Orr J, Paton LW, et al. 2018. UK nationals who received their medical degrees abroad: selection into, and subsequent performance in postgraduate training: a national data linkage study. BMJ Open,8(7), e023060.

Tiffin PA, Alexander K, Cleland J. 2018. When I say ... fairness in selection. Medical Education, 52(12), 1225-1227.

Smith DT, Tiffin PA. 2018. Evaluating the validity of the selection measures used for the UK’s foundation medical training programme: a national cohort study. BMJ Open, 8(7), e021918.

Patterson F, Tiffin PA, Lopes S, et al. 2018. Unpacking the dark variance of differential attainment on examinations in overseas graduates. Medical Education 2018, 52(7), 736-46.

Paton LW, Tiffin PA, Smith D, et al. 2018. Predictors of fitness to practise declarations in UK medical undergraduates. BMC Medical Education, 18(1), 68.

Mwandigha LM, Tiffin PA, Paton LW, et al. 2018. What is the effect of secondary (high) schooling on subsequent medical school performance? A national, UK-based, cohort study. BMJ Open, 8(5), e020291.

Finn GM, Mwandigha L, Paton LW & Tiffin PA. 2018. The ability of 'non-cognitive' traits to predict undergraduate performance in medical schools: a national linkage study. BMC Medical Education,18(1), 93.

Fielding S, Tiffin PA, Greatrix R, et al. 2018. Do changing medical admissions practices in the UK impact on who is admitted? An interrupted time series analysis. BMJ Open, 8(10), e023274.

Tiffin PA, Paton LW, Mwandigha LM, et al. 2017. Predicting fitness to practise events in international medical graduates who registered as UK doctors via the Professional and Linguistic Assessments Board (PLAB) system: a national cohort study. BMC Medicine, 15(1), 66.

Tiffin PA*, Mwandigha LM, Paton LW, et al. 2016. Predictive validity of the UKCAT for medical school undergraduate performance: a national prospective cohort study. BMC Medicine, 14(1), 140.