#local-llm

4 articles

2026-10-17
Tokens Per Second Is Two Benchmarks Wearing a Trench Coat
A single tokens-per-second score hides prompt ingestion, decoding, and the workload that actually matters. Benchmark both halves or stop comparing machines.
2026-10-16
Structured Output Is Not Truth: What a JSON Schema Can—and Cannot—Guarantee
A grammar can constrain a model's output shape. It cannot establish whether the answer matches the source. Here is a small, testable example.
2026-10-07
NVIDIA's Tri-Mode Model Doesn't Just Draft — It Switches Gears
NVIDIA's Nemotron-Labs-Diffusion unifies AR, diffusion, and self-speculation in one architecture. It can switch modes mid-generation. The paper claims 6× more tokens per forward than AR — but the real story is how tri-mode changes what hardware you need.
2026-04-04
Gemma 4 26B: Google Drops a MoE Monster With 4B Active Params
Google DeepMind released Gemma 4 today. The 26B A4B MoE scores 1441 on LMArena while burning just 4B active parameters. Here's what that means for local inference.

Tokens Per Second Is Two Benchmarks Wearing a Trench Coat