Eight separate model calls cost nearly five times the energy of one batch

single source· 1 articles · confidence: medium · first seen 2026-09-15 20:00 UTC

What this means for you

If you run multi-sample inference, the generation schedule matters as much as the candidate count: eight serial single-candidate calls cost 4.6–4.9x the GPU energy of one batched call of eight in this test. Report the schedule alongside N. Nothing to buy — this is a measurement result, not a model or an API.

A preprint on test-time scaling (generating several candidate answers per prompt, then combining them) reports that how candidates are produced matters more to energy and latency than how many there are. On 500 GSM8K prompts, raising the count from 1 to 8 improved accuracy by 8.4 percentage points for Phi-3-mini and 18.4 for Qwen2.5-1.5B. With the count fixed at 8, eight serial single-candidate calls on A100 GPUs used 4.64–4.86x the gross GPU-device energy and 5.77–6.12x the P95 latency of one batched call of eight. The authors argue evaluations should report the generation schedule and GPU-level systems metrics, not just candidate count and accuracy.

Key facts

  • ·Raising the number of generated candidates from 1 to 8 improved accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B across 500 GSM8K prompts. source
  • ·With the candidate count fixed at 8, eight serial single-candidate calls used 4.64–4.86x the gross GPU-device energy of one batched call of eight on A100 GPUs. source
  • ·The same comparison showed 5.77–6.12x the P95 latency for eight serial calls. source
  • ·The pattern held across three independently scheduled A100 nodes per model and in short-output SciQ experiments on V100 GPUs. source
  • ·The paper recommends reporting generation schedule and GPU-level systems metrics alongside candidate count and accuracy. source
  • ·The paper was posted to arXiv on 15 September 2026. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire