This paper empirically compares fitted dynamic programming (DP) and model-free reinforcement learning (DQN, A2C, PPO) on four finite-horizon dynamic pricing environments (single and multi-typology, with/without cumulative revenue constraints), evaluating revenue, stability, constraint satisfaction and computational scaling; results show DP is a stable, high-performing benchmark while RL (PPO/A2C) can approach DP given sufficient episodes but exhibits higher variance and slower convergence in low-data or higher-dimensional settings .
速读内容
Core conclusion: DP vs RL trade-offs
Fitted DP (model-based with demand estimation + Bellman recursion) yields consistently high and low-variance revenue across all environments, serving as a reliable benchmark .
RL (PPO, A2C) can learn competitive policies given large training budgets; PPO is most stable at low budgets while A2C improves more with more episodes; DQN shows slow convergence and high variance .
Computational trade-off: DP complexity scales badly with discretized multi-dimensional state space (O(T·|S|^2·|A|)), making it far slower in complex environments; RL scales with number of episodes and model cost (O(N_episodes·T·C_model)), often faster for high-dimensional settings .
Environment definitions and experimental setup
Four benchmark environments: Env1 single-typology (no constraint), Env2 two identical typologies (constraint at t=7), Env3 two heterogeneous typologies (constraint), Env4 single typology with constraint at t=5 .
Demand model: Poisson arrivals with intensity d_i(p,t)=max(0, α_i − β_i p)·(1 + t/T); DP estimates demand via linear regression on random-episode sales (excluding sold-out observations) .
Action discretization used so DP can enumerate prices; DP discretizes cumulative revenue for constrained cases (step sizes specified per env) .
Env 1 (single typology) — learning curves and summary
Fitted DP achieves highest and most stable revenue across all budgets; PPO/A2C approach DP as episodes increase; DQN underperforms with large variance early and later .
Implication: in low-dimensional, well-specified settings, model-based DP offers immediate advantage if demand can be reliably estimated .
DP remains top performer; PPO shows relatively small variance and better early performance; A2C improves with more episodes; DQN is unstable .
Violin plot before penalty step (t=7) shows DP concentrates mass above revenue target (highest median and %≥target), PPO/A2C closer to threshold but more dispersed; DQN often underachieves and triggers penalty more frequently .
DP again strong and stable even with heterogeneity; A2C shows steady improvement and can reach DP at large budgets; PPO is stable but sometimes lags in final revenue; DQN underperforms with large tails in distribution .
Violin plots show DP almost always meets constraint; RL methods show higher dispersion and occasional underperformance relative to DP .
Env 4 (single typology with constraint) & nonlinear-demand robustness (Env5)
Constrained single-typology: DP stable best; PPO most consistent RL baseline; A2C improves and can become competitive with enough episodes; DQN unstable .
Nonlinear-demand experiment (Env5) confirms qualitative conclusions: DP highest/stable; PPO stable early; A2C benefits from more data; DQN worst variability .
Training time and scaling
Empirical timings (Table 1) show DP run-time grows dramatically in multi-typology/constrained cases (orders of magnitude larger than RL at 2000 episodes), while RL methods scale roughly linearly with episodes; among RLs, PPO is slowest per episode .
Practical takeaway: use DP when a reliable demand model and moderate state dimension exist; use RL when state-action dimensionality explodes or model is unavailable and sufficient simulation budget exists .
思维导图
深度阅读
元数据与概览(引言与报告概览)
报告题目:A Comparative Study of Dynamic Programming and Reinforcement Learning in Finite-Horizon Dynamic Pricing。
作者:Lev Razumovskiy、Nikolay Karenin;作者署名与联系方式显示为 RAMAX Group 的电子邮件地址(在附录/末尾列出)。