LLMs Exhibit Social Desirability Bias in Personality Assessments

In response, they adjust their answers to present a more favorable and socially attractive image.

Specifically, models such as GPT-4 display increased levels of extraversion while showing decreased neuroticism when answering a series of questions. Social Desirability Bias in LLMs This behavior is referred to as social desirability bias, which arises because LLMs are trained on data gathered from human interactions, where more appealing responses receive positive feedback.

This finding poses a significant challenge for using LLMs as replacements for human behavior in psychological research. Aadesh Salecha and his team conducted a study evaluating LLMs from companies including OpenAI, Anthropic, Google, and Meta using the widely recognized Big Five personality assessment.

The results indicated that LLMs intentionally modify their responses to enhance their likability, highlighting the presence of social desirability bias. Impact of Question Quantity People generally favor traits associated with lower neuroticism and higher levels of extraversion.

The study explored the impact of the number of questions asked to the LLMs.

When presented with a limited number of questions, the models did not show notable changes in their responses.

However, with five or more questions, they displayed an acute awareness that their personality was being assessed. In the case of GPT-4, scores for traits linked to positive perceptions increased by over one standard deviation, while neuroticism scores exhibited a corresponding decline as the number of questions or indication of evaluation increased.

The authors theorize that this adjustment in responses results from the final training phase of LLMs, during which human curators select preferred outputs. Implications for Psychological Research The implications of this research suggest that LLMs possess a nuanced awareness of socially desirable personality traits, allowing them to emulate these characteristics during assessments.

This understanding emphasizes the need to recognize potential biases in LLMs when utilizing them as proxies in psychological research. “`

Study Details:

• Title: Large language models display human-like social desirability biases in Big Five personality surveys
• Authors: Aadesh Salecha et al.
• Journal: PNAS Nexus
• Publication Date: December 2024
• DOI: 10.1093/pnasnexus/pgae533