Multifaceted variability in LLM-driven stock recommendations

초록

This study assesses the reproducibility of LLM-based stock recommendations by examining three variability sources: repetition variability, from issuing the same query repeatedly; rephrase variability, from using queries with identical intent but different wording; and system-prompt variability, from altering the model's embedded instructions. Using the ChatGPT and Claude APIs, we find that these variabilities are statistically significant. This finding suggests that, in practice, consistency can be improved by selecting only stocks that frequently appear across multiple identical and rephrased queries. Our results also underscore the critical role of prompt engineering, as tailoring system prompts to investor preferences enhances the relevance and reliability of LLM-based portfolios. However, a significant practical challenge is that LLM-based portfolios struggle to consistently outperform the market on a risk-adjusted basis.

키워드