NFQ-trained agent that continuously projects a hypothetical clearing price and optimizes both continuous LOB quotes and auction supply consistently outperforms AS and TWAP on mean returns in both synthetic and historical tests .
Empirical mean improvements (historical S&P500 sample): final NFQ achieves ≈+40.8% mean return vs AS and ≈+40.2% vs TWAP on average across tested tickers .
Method summary — MDP + Neural-Fitted Q (NFQ)
Formulation: finite-horizon MDP with two phases (continuous LOB and discrete closing auction); state vector encodes inventory, hypothetical clearing price H_t^cl, LOB depths, auction orders, cancellations, etc. .
Action space: (continuous phase) choose posted volume v_t and depth k_t; (auction) choose linear slope K_t^a and reference price S_t^a and cancellation flags; admissibility enforces v_t ≤ inventory and prices ≥ mid-price .
Learning: two separate DQN-style networks (continuous / auction phases) trained via Neural-Fitted Q-Iteration (replay buffers, target networks, Huber loss, ε-greedy schedule). Key training hyperparams: lr=3e-4, episodes=2000, batch=128, replay cap 50k .
Auction-clearing theory and projected clearing price algorithm
The paper proves existence/uniqueness of an estimated clearing price under mild continuity/Lipschitz conditions on exogenous supply functions and when agent's submitted slopes are positive (Theorem 2.1) .
During the continuous phase the agent constructs a hypothetical clearing price H_t^cl by treating outstanding LOB limit orders as blocks and solving an implied clearing equation (Algorithm 1), then smooths updates with a smoothing parameter T in practice .
Agent penalization: executions priced materially below the projected clearing price are penalized (regret term), and auction-side wrong-way executions are penalized via f^a to discourage buying when intended to sell .
Generative market model & calibration (for simulations)
Generative emulator (Algorithm 2) simulates Poisson arrivals for market takers, Pareto volumes for order sizes, Beta-distributed top-of-book volumes, geometric decay for deeper levels, and Bernoulli-driven auction events (market maker arrivals/cancellations, taker arrivals/cancellations) .
For synthetic mid-price generation the rough-Heston model is used (Euler-type discretization) to capture rough volatility features; parameters calibrated from literature (H, ρ, V0, θ, λ, ν) .
Numerical performance — benchmarks and results
Metric / Symbol
Initial NFQ
AS
TWAP
Final NFQ
Mean Return (generative)
3,536.5
9,411.1
4,592.5
12,169.2
Std Return
18,612.0
1,606.9
3,110.4
16,552.4
Mean Auction Reward
-5,399.2
(—)
(—)
3,290.1
Final NFQ increases mean return substantially vs benchmarks in both synthetic rough-Heston tests and historical S&P500-day tests; variance is higher due to simple NN architecture and large auction rewards/penalties .
Historical-data experiment (S&P500 tickers)
Symbol
hatσ
Initial NFQ
AS
TWAP
Final NFQ
Imp. vs AS
CAT
5.77e-4
8,469.4
9,691.3
9,774.4
15,823.5
+63.3%
PG
4.10e-4
8,225.2
9,537.4
9,713.6
14,743.2
+54.6%
GOOGL
5.90e-4
8,473.5
9,808.4
9,933.8
13,905.1
+41.8%
JPM
5.49e-4
8,293.7
9,497.7
9,417.0
12,655.1
+33.2%
MSFT
4.12e-4
8,454.2
9,630.6
9,465.6
10,687.4
+11.0%
Mean improvement (Final NFQ vs AS) ≈ +40.8% across symbols tested, supporting robustness across tickers .
Practical notes & limitations
NFQ networks used are simple MLPs (three hidden layers, width 16); authors note larger networks and better engineering would reduce variance but at higher compute cost .
Reward variance is high (especially due to large auction payoffs); results show mean improvement but also elevated tail risk—practitioners should consider stricter risk control/penalty calibration .
思维导图
深度阅读
以下为对论文“Learning Market Making with Closing Auctions”(Julius Graf & Thibaut Mastrolia,2026-01-27)的一次详尽、逐节解构式分析。全文力求覆盖报告每一重要论点、数学结构、关键假设、算法、图表与数值结果,并对模型和实验设计给出客观批评与可验证的解读。凡直接基于原文内容的结论或推断,句末加注页码溯源标识 [page::页码](页码对应原文分片的 page_idx 编号)。
一、元数据与总体概览(引言与报告要点)
报告元信息:题目“Learning Market Making with Closing Auctions”,作者 Julius Graf 与 Thibaut Mastrolia,所属机构为 UC Berkeley Department of Industrial Engineering and Operations Research,发布日期 2026-01-27,研究主题为:在含连续限价撮合(CLOB)与随后收盘拍卖的交易会话中,构建并训练强化学习(基于 Q-learning / Deep Q-Learning)的做市策略,显式把收盘拍卖纳入决策与奖励结构中。
该算法把当前未成交挂单当作若干线性供给/需求块投影到一个“虚拟拍卖”中并求解对应清算价 H_t^{cl},对 H 做指数平滑等更新(平滑参数 T、初值 H0 等),并在连续期用 H_t^{cl} 对提交低于该值的执行施加惩罚(即鼓励在连续期避免过早以低于未来清算价的价格成交)。算法细节在原文以伪代码表述并结合参数来自 Paul et al. 之类的参考进行校准。
Figure 1(基准策略—第 2000 次 episode,可视化 AS vs TWAP 行为,)
说明(图像描述):该图由四个子图组成:左上为库存随时间演化(AS 与 TWAP),右上为连续期提交的限价(AS price vs TWAP price),左下为连续期提交量随时间,右下为拍卖阶段提交的 K(拍卖强度)随时间(TWAP 与 AS 曲线)。作者指出 TWAP 在拍卖开始前未能完全清仓(top-left),而 AS 更积极在连续期清仓导致拍卖时库存更少或为零。
解读与趋势:图示显示 AS 在连续期快速降低库存并几乎在拍卖前清空(库存线到 0),而 TWAP 库存下降更平缓并在截面时间仍显著剩余,说明 AS 的连续期报价更激进或密集导致更多成交(左下的提交量显示 AS 在连续期初期挂出了大额量直到第 60 左右,然后投放降为 0;TWAP 持续小量暴露)。这支撑作者指出 TWAP 无法充分利用连续期的做市机会、从而在拍卖上暴露更多库存风险。
结构:列出 Initial NFQ、AS、TWAP、Final NFQ 的 mean return、std、median、mean final inventory、mean CLOB reward、mean auction reward,并列出相对改进百分比。
关键结果:Final NFQ 在 mean return 上为 12,169.2,明显高于 AS 的 9,411.1 与 TWAP 的 4,592.5;mean auction reward 从初始 NFQ 的 −5,399.2 变为 final NFQ 的 +3,290.1,显示训练后策略成功在拍卖期实现正收益。
解读:虽然 final NFQ 平均收益最好,但 std-return 较高(16,552),说明在高方差环境下策略表现波动大,回报有较大尾部风险/机会。作者也指出小网络与低惩罚参数会放大 variance(Remark 6.1)。
表4(历史数据实验:5 只股票在 100 episodes 上的均值表现,)
关键点:对 CAT、PG、GOOGL、JPM、MSFT 的 final NFQ 均优于 AS 与 TWAP,平均提升约 +40.7% vs AS 与 +40.2% vs TWAP(Mean 行)。这些结果支持 NFQ 在真实历史 mid price 路径下获得更高平均回报的结论,但并未给出风险/方差指标的详细对比(仅在生成实验提供 std)。