Research Ideas研究想法

RL Post-Training of Ternary Models as Discrete Optimization把三值模型的 RL 后训练当作离散优化问题

A ternary model stores each weight as −1, 0 or +1 times a per-row scale, so post-training it is a large-scale discrete optimization problem: which weights should change, and to which value? The standard approach keeps a hidden full-precision copy of every weight, trains it through a straight-through estimator, and rounds it in each forward pass. That copy and its optimizer state take far more memory than the 2-bit model itself, and most small updates never change a deployed weight. 三值模型的每个权重只能取 −1、0 或 +1，再乘一个逐行的 scale，所以它的后训练是一个大规模离散优化问题：该改哪些权重，改成哪个值？常规做法是给每个权重维护一份隐藏的全精度副本，用 straight-through estimator 训练，前向时再取整。这份副本和优化器状态占的内存远大于 2-bit 的模型本身，而且大部分微小更新改变不了部署的权重。

I am exploring zeroth-order search directly in the deployed format: candidate code edits are scored by forward rollouts and task rewards alone, with no full-precision copy and no backpropagation through the model. 我想探索直接在部署格式上做零阶搜索：候选的码修改只靠前向 rollout 和任务 reward 打分，不需要全精度副本，也不需要对模型反传。

Three questions are open. Which parts of the network deserve the search budget? How many codes should change at each step? How can we confirm that an edit is a real improvement and not rollout noise? The goal is to reach the same quality at a lower total rollout cost than evolution strategies on quantized weights. 现在有三个开放问题：搜索预算该分给网络的哪些部分？每一步该改多少个码？怎么确认一次修改是真的变好，而不是 rollout 噪声？目标是在同样的质量下，总 rollout 成本低于量化权重上的进化策略。

Key papers关键论文

• BitNet b1.58 2B4T Technical Report (Ma et al., 2025) — the ternary LLM that made this format practical. — 让三值大模型走向实用的代表工作。
• GXNOR-Net (Deng et al., Neural Networks 2018) — direct transitions between ternary states, without full-precision memory. — 不用全精度内存，直接在三值状态之间转移。
• Latent Weights Do Not Exist (Helwegen et al., NeurIPS 2019) — Bop flips binary weights directly, with no latent weights. — Bop 不用 latent 权重，直接翻转二值权重。
• PV-Tuning (Malinovskii et al., NeurIPS 2024) — alternates discrete and continuous updates for LLM compression. — 在大模型压缩中交替做离散更新和连续更新。
• Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity (Guo et al., 2024) — Fisher-informed coordinate selection for sparse zeroth-order fine-tuning. — 用 Fisher 选坐标，做稀疏的零阶微调。
• Quantized Evolution Strategies (Xu et al., 2026) — reward-based search in quantized weight space, and the direct baseline here. — 在量化权重空间上用 reward 搜索，是这里最直接的 baseline。

RL Training Dynamics on Low-Precision Models低精度模型上的 RL 训练动力学

In The Path Not Taken, we gave a parameter-space account of RL training: RL learns off the principal directions in weight space, and numerical precision hides many of its small updates. I want to continue this line on low-precision models, where precision is no longer a side effect: small updates reach the deployed weights only as discrete jumps. 在 The Path Not Taken 里，我们从参数空间解释了 RL 训练：RL 在权重空间中沿非主方向学习，而数值精度遮住了很多微小的更新。我想在低精度模型上继续这条线。到了这里，精度不再是副作用：微小的更新只能以离散跳变的形式落到部署的权重上。

This matters because RL training is sensitive to bias and noise. The model generates its own data, and signed advantages make each update a difference between two large terms, so terms unrelated to the reward can steer training. Existing results also point in different directions: QeRL finds that quantization noise raises policy entropy and helps exploration, while work on MXFP4 finds that quantization error severely degrades accuracy. 这件事重要，是因为 RL 训练对偏置和噪声很敏感。模型自己生成训练数据，带符号的优势又让每步更新变成两个大项之差，所以和 reward 无关的项也可能左右训练。已有结果的方向也不一致：QeRL 发现量化噪声会提高策略熵、有助于探索，而针对 MXFP4 的工作发现量化误差会严重降低准确率。

Three questions are open. Which reward-irrelevant biases does quantization amplify, and is that amplification general? In a direction with zero expected gradient, do discrete updates turn noise into lasting behavior change, or can later updates undo it? How do these effects change with bit width? 现在有三个开放问题：量化放大的是哪些和 reward 无关的偏置，这种放大是否普遍成立？在期望梯度为零的方向上，离散更新会不会把噪声变成持久的行为变化，后续更新还能不能撤销它？这些效应随位宽怎样变化？

Key papers关键论文

• The Path Not Taken: RLVR Provably Learns Off the Principals (Zhu et al., 2025) — the starting point for this line of work. — 这条线的起点。
• QeRL (Huang et al., ICLR 2026) — quantization noise raises policy entropy and helps exploration. — 量化噪声提高策略熵，有助于探索。
• Decomposing MXFP4 Quantization Error for LLM Reinforcement Learning (Li et al., 2026) — splits quantization error into bias, deadzone and an irreducible floor. — 把量化误差拆成偏置、死区和不可消除的下限。
• Overcoming Oscillations in Quantization-Aware Training (Nagel et al., ICML 2022) — quantized weights can oscillate between two grid points. — 量化权重可能在两个格点之间来回振荡。
• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (Ma et al., 2025) — discrete routing decisions can make RL training collapse. — 离散的路由决策可能让 RL 训练崩溃。
• Score Centering Stabilizes Off-policy Reinforcement Learning (Marek & Ryabinin, 2026) — traces RL instability under training-inference mismatch to drift. — 把训练-推理失配下的 RL 不稳定归因于 drift。