Paper tunes models to human preferences from comparisons rather than gradients
single source· 1 articles · confidence: medium · first seen 2026-09-15 20:00 UTC
What this means for you
Nothing to act on yet: no code, no weights and no evaluation date, so the reported win rates cannot be checked. The comparison is against other preference-tuning methods, not a fixed published baseline. If you work on preference optimisation, the convergence argument is the part worth reading.
An arXiv preprint (2609.19144, posted 15 September) proposes ComPO, a way to tune a language model to human preferences that takes a direction from each chosen-versus-rejected pair instead of optimising a preference loss on it. Direct alignment methods optimise such a loss; ComPO is zeroth-order, meaning it reads only the outcome of each comparison and never a gradient. The authors prove convergence for the offline version and report better length-controlled win rates than existing direct alignment methods across five model families. No evaluation date, harness or code release is given.
Key facts
- ·arXiv preprint 2609.19144, posted 15 September 2026, proposes Comparison-based Preference Optimization (ComPO). source
- ·ComPO is a zeroth-order method that extracts directional information from preference pairs without directly optimising a differentiable preference loss on them. source
- ·The paper establishes a convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. source
- ·An online variant retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. source
- ·Experiments cover Mistral, Llama, Gemma-2, Qwen3 and Gemma-3 models. source
- ·Improvements over existing direct alignment methods are reported as length-controlled win rates, with pair-level diagnostics offered as evidence consistent with mitigating likelihood displacement. source
What the sources say
- Hugging Face Daily Papers (research) — Sets out the method's convergence guarantees, an online variant, and head-to-head tests across five model families.
Sources
The original reporting. Follow these — they did the work.
- Hugging Face Daily PapersA Zeroth-Order Paradigm for LLM Preference Alignment2026-09-15