essay · 2026-09-15
Post-training
The research demonstrates that applying reinforcement learning uniformly across all task types leads to suboptimal performance because it ignores the fundamental difference between problems with ground-truth answers and those without. Specifically, extending RLVR beyond math and code is identified as "the hardest open problem in LLM training right now," highlighting a critical gap where current methods fail due to noisy or unverifiable reward signals. A key insight from recent surveys is that deriving a single policy gradient framework can unify pretraining, SFT, RLHF, and RLVR only if it accounts for the specific nature of each stage's feedback mechanism. For instance, while "Reinforcement Learning with Verifiable Rewards" works well for structured tasks like math or code generation, attempting to apply this same logic to messy real-world domains results in poor alignment without additional safeguards. The literature suggests that a frontier strategy involves using verifiable rewards strictly where they exist and switching to preference-based methods otherwise to maintain stability. This distinction is crucial because it prevents the model from optimizing for incorrect patterns wh