DATUM

TRL's async GRPO trainer can now sync a LoRA adapter, not the model

single source · 1 articles · research · confidence: medium · first seen 2026-09-10 00:00 UTC

TRL v1.14 ships LoRA support in AsyncGRPOTrainer, so an asynchronous reinforcement-learning run can train a LoRA adapter — a small set of extra weights trained instead of the whole model — and sync only that to vLLM instead of the roughly 3GB policy. A rank-1 adapter for a 1.5B model is a few megabytes. A project built on this runs the trainer and the vLLM replicas as separate Hugging Face Jobs on separate machines, passing adapters through a Storage Bucket mounted in every job, with a small proxy routing each rollout to the replica already holding its KV prefix (its cached context). It reports five runs taking 500 steps from 3h27m to 53m. Jobs are one container on one VM, capped at 8xH200.

What this means for you

If your GRPO run keeps the trainer and the inference replicas on one node because a shared filesystem or an NCCL group was the only easy way to sync them, this is a working split: separate jobs, a shared bucket, adapter-sized updates. The reported timings are one project's five runs, not a controlled comparison — measure on your own setup before planning around them.

Key facts

  • ·TRL v1.14 adds LoRA support to AsyncGRPOTrainer, landed in PR #7017 source
  • ·A rank-1 LoRA adapter for a 1.5B model is a few megabytes, against roughly 3GB for the full model source
  • ·Five runs on one recipe took 500 steps from 3h27m down to 53m source
  • ·The trainer and vLLM replicas run as separate Hugging Face Jobs on separate machines, with adapters moved through a Storage Bucket rather than NCCL source
  • ·A Hugging Face Job is one container on one VM, capped at 8xH200 per node source
  • ·vLLM can hold several adapters at once, so old rollouts finish on the policy they started with source

What the sources say

  • Hugging Face Blog — Engineering write-up of a distributed async GRPO setup, with adapter-sync mechanics and timings

Sources

The original reporting. Follow these — they did the work.

← the wire