Hugging Face adds GGUF loading to Transformers, starting on Apple Silicon

single source· 1 articles · confidence: high · first seen 2026-09-22 00:00 UTC

What this means for you

Nothing to migrate today: this is early, limited to Apple Silicon and Qwen3.5. The gain is one toolchain — GGUF checkpoints load through the same Transformers API as everything else, with llama.cpp's kernels underneath. Start at Q4_K_M, then measure the quality loss on your own workload rather than trusting the file sizes.

Hugging Face has added support for GGUF checkpoints to Transformers, so a model packaged for llama.cpp — the inference engine behind Ollama, LM Studio and Jan — can be loaded with `from_pretrained` and run locally without converting it. The implementation reuses llama.cpp's ggml kernels and trims overhead in `generate`. The first targets are Apple Silicon and the Qwen3.5 architecture. GGUF stores weights, tokenizer information and an optional chat template in one file, at selectable precision: Unsloth's Qwen3.5-4B is 8.42 GB unquantised, 3.53 GB at Q6_K, 3.14 GB at Q5_K_M and 2.74 GB at Q4_K_M. The post recommends starting at Q4_K_M and says the quality tradeoff depends on model and task.

Key facts

  • ·Transformers can now load GGUF checkpoints via from_pretrained, reusing llama.cpp's ggml kernels. source
  • ·Initial support targets Apple Silicon and the Qwen3.5 architecture. source
  • ·Unsloth's Qwen3.5-4B is 8.42 GB at BF16, 3.53 GB at Q6_K, 3.14 GB at Q5_K_M and 2.74 GB at Q4_K_M. source
  • ·Variants such as Q4_K_M mix tensor precisions, keeping sensitive tensors at higher precision while most weights are 4-bit. source
  • ·GGUF packages model weights, tokenizer information and an optional chat template into one file. source

What the sources say

  • Hugging Face Blog — Announces first-party GGUF loading in Transformers, with file sizes for one quantised model and setup advice.

Sources

The original reporting. Follow these — they did the work.

← the wire