Fine-tuned 4B model edges past GPT-5.6 on a transit kiosk benchmark
single source · 1 articles · capability · confidence: medium · first seen 2026-09-08 20:00 UTC
A 4B model, tuned by training a small set of added weights rather than the whole network, edges past two GPT-5.6 tiers on a new transit-kiosk benchmark's deterministic scoring. MetroLLM-Bench, posted to arXiv, holds 955 cases over six metro systems of 37 to 414 stations; 238 are held out. The student takes 91.3 there, against 90.6 and 90.0 for the GPT-5.6 tiers and 84.6 for a rule-based baseline. Scaling it to 9B and 27B adds nothing. A separate 30B model leads the composite ranking, which adds eight semantic-quality components, six scored by a model judge. The results are the paper's own, with no evaluation date.
What this means for you
Nothing to buy: no API, no hosted endpoint, just a paper and a GitHub release. If you evaluate tool-calling agents, the harness is the useful part — it ships with the benchmark and the students, so you can rerun it. Serving configuration alone moved one comparison by 2.7 points, so distrust single-score gaps.
Key facts
- ·MetroLLM-Bench contains 955 cases across six real metro systems ranging from 37 to 414 stations, split 717 for training-data generation and 238 for held-out evaluation. source
- ·Tier 1 comprises 14 deterministic scoring components; Tier 2 comprises eight semantic-quality components, six of which use a language-model judge. source
- ·On the held-out partition, a 4B Qwen 3.5 student trained with PEFT scores 91.3 on Tier 1, against 90.6 and 90.0 for the GPT-5.6 tiers, 91.4 for GPT-5.4 at maximum reasoning effort, and 84.6 for a deterministic rule-based baseline. source
- ·The PEFT gain over the corresponding base model falls from +7.03 points at 2B to -0.91 at 27B, and 9B and 27B students give no further Tier 1 improvement over the 4B student at this training scale. source
- ·Muse Glimmer 30B leads the composite ranking, and serving configuration alone shifts the Qwen 3.5-to-3.8 Tier 1 comparison by 2.7 points. source
- ·The fine-tuned student has a 2.6 GB footprint at Q4_K_M precision; the benchmark, harness, reproduction guide and fine-tuned students are released on GitHub. source
What the sources say
- Hugging Face Daily Papers (research) — Publishes the case set, scoring tiers and trained students, and reports where a small tuned model lands.
Sources
The original reporting. Follow these — they did the work.
- Hugging Face Daily PapersMetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes2026-09-08