Key Result — Goal-aware vs Goal-blind portfolio performance
Long–short portfolios formed on GPT-derived sentiment show meaningful predictability in both prompt regimes, but goal-aware sentiment yields larger pre-cutoff high–low spreads (1.552% vs 1.069% monthly) indicating apparent stronger in-sample economic content .
After the model knowledge cutoff, the difference between goal-aware and goal-blind long–short spreads disappears (post-cutoff spreads ≈ 2.269% vs 2.239%), consistent with the pre-cutoff advantage arising from information availability/internalized correlations rather than superior neutral measurement .
Table — Return Spreads from Quintile Portfolios
Table 1: Return Spreads from Quintile Portfolios Formed on Goal-Aware and Goal-Blind Sentiment Scores
Fama–MacBeth cross-sectional regressions: goal-blind scores have stable, significant predictive slopes both pre- and post-cutoff, while the incremental Diff (goal-aware − goal-blind) is positive and significant pre-cutoff but collapses to ~0 post-cutoff, rejecting equality across periods for Diff .
Out-of-sample expanding-window forecasts show goal-aware sentiment outperforms prior to cutoff but performs worse than goal-blind after cutoff (monthly average OOS R^2 series): the interaction (Goal-aware × Post-Cutoff) is negative and significant, documenting deterioration in real-time forecast accuracy for task-aware scoring .
Earnings/competition scores — predictive and OOS patterns
Competition-threat scores from goal-blind prompts negatively predict next-quarter EPS pre-cutoff (higher competition → lower EPS) and retain some relation post-cutoff, but the incremental predictive power from goal-aware prompts (Diff) is significant pre-cutoff and disappears post-cutoff, mirroring the sentiment/returns pattern .
Quarterly OOS R^2 series show both score types weaken after cutoff, with a sharper decline for goal-aware scores, indicating that task-aware components decay when model knowledge becomes stale .
Quant construction & evaluation methods (factor-style summary)
Score construction: LLM returns a continuous score in [-1,1] (sentiment or competition) from an earnings-call transcript; raw scores are transformed into percentiles within month–year (sentiment) or industry×year cohort (competition) for comparability .
Prompt variation (treatment): goal-blind prompt vs. goal-aware prompt that adds one sentence disclosing the downstream regression use; otherwise identical instructions and JSON numeric output format .
Backtest / evaluation: sample S&P 500 firms (Jan 2022–Dec 2024), model = GPT‑4o‑mini with knowledge cutoff 2023‑10‑01; evaluations include quintile portfolio sorts, Fama–MacBeth cross-sectional regressions, and expanding-window out-of-sample forecasts with OOS R^2 metrics .
Practical implication for quant workflows: when LLMs are used as measurement tools, keep prompts agnostic to downstream use and run strict out-of-sample tests to avoid purpose-conditioned measurement artifacts .
思维导图
深度阅读
元数据与概览(引言与报告概览)
报告标题为 “Seeing the Goal, Missing the Truth: Human Accountability for AI Bias.”,作者为 Sean Cao、Wei Jiang、Hui Xu,分别隶属 University of Maryland、Emory University 与 Lancaster University,最后更新日期为 2026-02-11,研究主题聚焦于“当人类披露下游目标时,LLM 在生成中间测量(如情绪、竞争强度)上如何产生目的导向的偏差”,并强调“该偏差源自人类在研究设计中对目标披露的责任而非模型架构本身”。