The Role of User Profile in Response Selection: Evidence from Experiments with Large Language Models

Current response selection tasks in conversational agents primarily focus on context-driven responses, without considering user-specific traits that could influence the choice of a response. These models often rely solely on the dialogue history to predict responses, missing potential improvements that could arise from integrating user characteristics. By incorporating additional information about the speaker into the response selection process, conversational models could select most relevant responses. In this work, we explore the impact of user traits on response selection in dialogue systems, using the Wikipedia Talk Page dataset as our primary source of data. We extract features related to the next speaker, and incorpo- rate these traits into the model’s input context to determine their influence on response selection. This method allows us to investigate whether adding speaker-specific information enhances the model’s ability to select accurate and coherent responses. To assess the impact of this approach, we evaluated the performance of two models: a baseline Siamese Recurrent Neural Network (RNN) and the LLaMA model. Both models were tested under two conditions: with and without the integration of user traits. This allowed us to systematically examine how incorporating user traits affects response selection in each case. The dataset used for this research consists of conversation threads from Wikipedia Talk Pages, representing open-domain discussions. We assessed the effectiveness of integrating user characteristics into the model through a series of experiments and evaluated improvements in response accuracy, showing the potential of incorporating speaker traits in enhancing conver- sational models.