llama.cpp Triple-Drops: Qualcomm NPUs, M3 Tunings, and NVFP4's Missing Scales
llama.cpp shipped three builds in under 45 minutes — headlined by runtime discovery of Qualcomm Hexagon NPUs — while a merged NVFP4 scale fix finally makes 4-bit speculative decoding work for Qwen 3.8-27B; vLLM 0.28.0 goes all-in on Kimi-K3 and Ollama brings Flash-Next to MLX.
No new frontier weights today. But the inference layer had one of its busiest afternoons in weeks: llama.cpp cut three builds in under 45 minutes (b10691 → b10693, 11:58–12:43 UTC), one of them extending the engine into territory it has never shipped before — runtime discovery of Qualcomm Hexagon NPUs. And a small merged fix to NVFP4 scale handling finally makes 4-bit speculative decoding actually work on Qwen 3.8-27B, the model the self-hosting world has already pulled 4.5M times.
The meta-trend, again: the tools are moving faster than the models. Today that’s literal — three engine builds in one afternoon, zero new weights.
🔥 Top Stories
1. b10693: llama.cpp learns to find Qualcomm’s NPU
The headline build adds device discovery and on-demand session creation for the Hexagon backend (#27785, co-authored with Qualcomm engineers): runtime detection of available NPU cores, lazy session allocation, and early rejection of non-existent devices at init.
What’s new: Every backend llama.cpp has shipped so far — CUDA, Metal, SYCL, Vulkan, WebGPU — targets GPU silicon. Hexagon is a tensor accelerator fused into a SoC, the class of hardware in Snapdragon phones and ARM edge boards. This is the first real step toward llama.cpp as a reference runtime for on-device LLMs on ARM edge hardware, not just servers, desktops, and Macs.
The other two builds in the trio are quieter but directly user-visible:
- b10692 — SYCL: split long rows in the TOP_K kernel instead of one workgroup per row (release). A throughput fix for Intel GPU users.
- b10691 — Metal: fixes a null-pipeline crash in the F16
mul_mat/mul_mat_idpath by failing closed when kernels are missing (release). If your Mac build has been segfaulting on mixed-precision layers, this is the fix.
2. The NVFP4 fix Qwen 3.8 was waiting for
PR #28000 (merged): dflash: pass missing NVFP4 scales to attention operations. The reporter tested the community DFlash2 draft model for Qwen3.8-27B and found the NVFP4 variant produced almost no accepted speculative tokens — the Q, K, V, and output projection scales simply weren’t reaching the graph ops. The BF16 and Q8 draft variants worked fine, which is exactly the kind of silent breakage that makes a “supported” quantization format quietly useless.
Why it matters — the ecosystem was already there: unsloth/Qwen3.8-27B-NVFP4 has 2.34M downloads. People were pulling the NVFP4 quant while the engine’s support was half-broken. Now that the scales flow through attention, NVFP4 — roughly 10GB for 27B weights — plus working speculative decoding is the most efficient 27B configuration a 12GB card can run.
Where the 27B fits today (weights + modest KV cache):
| Format | Approx size | 12GB card | 24GB card |
|---|---|---|---|
| FP8 (official) | ≈15GB | ✗ | ✓ |
| Q4_K_M (unsloth GGUF) | ≈16GB | ✗ | ✓ |
| NVFP4 (unsloth) | ≈10GB | ✓ (short context) | ✓✓ |
Also merged this cycle, for Apple users: fa-vec tunings for M3 Ultra (#27999) and M3 Pro (#27963, swept on an 18GB MacBook Pro) — hand-tuned attention tables per chip.
3. vLLM v0.28.0: a release built around Kimi-K3
vLLM 0.28.0 (Aug 26, 584 commits from 270 contributors) leads with a Kimi-K3 performance push: Decode Context Parallel (DCP) support, fused FlashKDA decode and prefill kernels, SiTU activation support for MegaMoE, GEMM-RS for sequence parallelism, and combined all-gathers reporting 1.5–3× kernel-level speedups.
Put in context: Kimi-K3 is a 2.8T-parameter multimodal LatentMoE — 896 experts, top-16, routed in a 3584-dim latent space, 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers (SGLang v0.5.17 release notes). SGLang shipped day-0 support on Aug 8; vLLM’s answer landed two weeks later as a full-stack performance effort, not just model files. That’s the serving race for frontier MoE in one release pair. On HuggingFace, Kimi-K3 sits at 11,089 likes / 2.79M downloads — #4 trending.
4. Ollama 0.33.1/0.33.2: Flash-Next lands on MLX, Claude Desktop stops dropping requests
- v0.33.1 (release, Aug 26): MLX: Qwen3.8 Flash Next support — the new
qwen4_expmultimodal family (HF) now runs on Macs via MLX. Also: mlxrunner structured output, and no more Metal GPU timeouts when loading models from slow storage. - v0.33.2 (release, Aug 27): dark mode restored, macOS single-instance handoff fixed, and the Claude Desktop proxy no longer interrupts in-flight requests when the model catalog updates.
The thread through 0.33.x: Ollama is becoming the local gateway for agentic clients, and the bug-fix cadence is tracking the integration’s real-world usage.
📊 Model Trends
No new weights crossed the threshold today, but the download curves tell the real story.
HuggingFace trending (top 5 by likes):
| Model | Likes | Downloads |
|---|---|---|
| black-forest-labs/FLUX.1-dev | 14,313 | 712K |
| deepseek-ai/DeepSeek-R1 | 13,598 | 2.25M |
| Qwen/Qwen3.8-27B | 13,300 | 4.51M |
| moonshotai/Kimi-K3 | 11,089 | 2.79M |
| stabilityai/stable-diffusion-xl-base-1.0 | 8,083 | 1.72M |
The Qwen 3.8 ecosystem, in numbers:
- FP8 overtook the base model. Qwen3.8-27B-FP8 has 5.13M downloads vs. the base’s 4.51M. For a model barely three weeks old, that’s a clear signal: FP8 is the de facto distribution format for self-hosters, and the BF16 checkpoint is now mostly a fine-tuning artifact.
- The GGUF is 2× the official. unsloth/Qwen3.8-27B-GGUF at 8.84M downloads — the quantized layer is where the actual users are.
- The abliterated wave. Six-plus uncensored/abliterated variants of a single 27B in its first weeks — orcarouter FP8 (302K), OBLITERATUS (726K), huihui-ai (1.62M), JonathanColetti (1.99M), HauhauCS (1.16M), Blackfrost-AI (280K). A single checkpoint spawning a derivative ecosystem is the fastest a model family has ever turned over.
- The throughput wave. DFlash2 speculative-decoding draft models — incoai (203K), z-lab (142K) — the community is now optimizing speed, not just footprint. Combined with today’s NVFP4 scale fix, that’s the 27B’s performance story for the next month.
- The ceiling, for reference: Qwen/Qwen3.8-2.4T-A95B (1,183 likes) — a 2.4T-parameter MoE with 95B active. Datacenter-only; it exists to make the 27B look like a bargain.
- New family on the rise: Qwen3.8-Flash-Next (4,336 likes, 122K downloads, updated Aug 27) — the
qwen4_expimage-text-to-text line that just landed in Ollama’s MLX runner.
Qwen org activity: No new 3.8 weights, but the whole repo suite got fresh commits today — Qwen3-TTS (⭐13.2K, streaming speech + voice design), Qwen3-VL (⭐19.9K), Qwen3-Omni (⭐4.0K, end-to-end omni-modal), Qwen-Image (⭐8.3K), qwen-code (⭐27.5K). The multimodal suite keeps moving even in the gaps between LLM releases.
⚙️ Engine Updates
llama.cpp — merged this cycle:
| PR | What | Why you’d care |
|---|---|---|
| #28000 | Pass NVFP4 scales to attention | NVFP4 speculative decoding actually works now |
| #27999 | fa-vec tunings, M3 Ultra | Hand-tuned Metal attention for M3 Ultra |
| #27963 | fa-vec tunings, M3 Pro | Same, swept on 18GB M3 Pro |
| #27969 | --tensor-read-lazy → --lazy-mode (-lzm) |
CLI cleanup, ggerganov |
| #27967 | Skip Hadamard copy when k_rot has no buffer |
Crash fix for Lightning Indexer context shifts |
| #27960 | op_alloc_size_may_expand in RPC |
Fixes RPC under-allocation for Metal/SYCL/WebGPU ops |
Still closed/unmerged: the MMVQ src1 activation cache (#28002) — caching quantized activations shared across Q/K/V projections, a real bandwidth win when it lands — and the Qwen4Exp MTP work (#27956) that yesterday’s briefing flagged as upcoming. Correction on the record: no MTP speedup ships in today’s builds.
SGLang v0.5.18 (release, Aug 22, 710 PRs): added Muse Glimmer and Intern-S2-Mobius to the supported-models list. Not a delta from yesterday, but it’s the current serving baseline for new multimodal AR models.
Landscape note: HuggingFace’s own TGI v3.3.7 (Dec 2025) changelog carries a “Maintenance mode” entry — worth knowing if you’re choosing a serving stack: the innovation is in llama.cpp, vLLM, SGLang, and Ollama.
📰 AI News
HN’s front page was light on AI today — only three stories survived the keyword filter. The two that matter:
- Claude Code now appends session URLs to commit messages and PR descriptions by default (issue #66504, HN). A one-line workflow change with real implications: AI-authored commits get a provable session trail in your git history for free.
- Gates: “The turbulent AI era is here. The choices we make now are critical” (HN, 10pts). Policy, not engineering — and the original gatesnotes.com URL currently 404s, so treat the thread as the source of record.
🔄 What Changed Since Last Scan
- llama.cpp: b10691, b10692, b10693 (all Aug 30) — new builds
- Hacker News: 3 new AI-filtered stories (Claude Code provenance, Gates note, one false-positive match)
- Qwen org: fresh commits across Qwen3-TTS / Qwen3-VL / Qwen3-Omni / Qwen-Image / qwen-code — no new 3.8 model weights
- HuggingFace trending: no new models entered the top-30
Sources
- llama.cpp builds: b10691 · b10692 · b10693
- llama.cpp PRs: #28000 · #27999 · #27963 · #28002 · #27956
- Ollama: v0.33.1 · v0.33.2
- vLLM: v0.28.0
- SGLang: v0.5.18 · v0.5.17
- Models: Qwen3.8-27B · FP8 · GGUF · NVFP4 · Flash-Next · 2.4T-A95B · Kimi-K3
- HN: 49498201 · 49497803
Collected & published: 2026-08-30 UTC