llama.cpp, vLLM, SGLang: Three Inference Engines, Three Completely Different Paths

Late last year I started moving some API traffic from OpenAI to self-hosted. The initial motivation was simple: cut costs. What I didn’t expect was discovering that different inference engines can perform differently by a factor of 5x on the same workload — and the gap isn’t about “who optimized better,” it’s about whether your traffic pattern matches the engine’s design assumptions.

Here’s what I learned from using or benchmarking three engines: llama.cpp, vLLM, and SGLang.

llama.cpp: Don’t think about serving yet. Just get it running.

llama.cpp has one design goal: make large models run on anything. Pure C/C++, no Python dependency, no CUDA toolchain. The compiled binary is about 50MB. You can copy it onto a USB stick and run it anywhere.

That philosophy shapes every technical choice. KV cache is basic paging. No continuous batching. Server concurrency tops out around 16 slots. It’s not built for high throughput — it’s built so your MacBook or Raspberry Pi doesn’t OOM when running a 7B model.

GGUF quantization is the most underrated part. Q4_K_M retains about 92% of model quality while compressing 3.5x. A 70B model shrinks to ~40GB — fits on a single RTX 4090. The model quality you get locally isn’t that far from cloud APIs.

Use it for: personal tinkering, local privacy-preserving inference, air-gapped environments, running an API for one or two friends. Don’t use it for: 20+ concurrent users. llama.cpp is not a mini vLLM. Their architectures are fundamentally different. “Upgrading” later is basically a rebuild.

Single-stream speed reference: RTX 4090 on Llama 3 70B Q4_K_M ≈ 45–55 tok/s. M4 Mac mini on 7B Q4 ≈ 35 tok/s. Enough for one person, not for a team.

vLLM: The safe bet for production

vLLM’s signature innovation is PagedAttention — borrowing the operating system’s virtual memory paging idea and applying it to KV cache. It slices cache into 16-token fixed blocks, allocates on demand, and doesn’t require contiguous memory. The result: the same VRAM budget holds 4–5x more concurrent requests.

But vLLM’s real moat isn’t the tech. It’s the ecosystem. Amazon Rufus serves 250 million customers on vLLM. LinkedIn runs 50+ generative AI use cases on it. Roblox uses it as their primary inference engine. Most production pitfalls have already been hit and documented. Search GitHub Issues and you’ll find answers.

My experience: when in doubt, choose vLLM. It has the thickest documentation of the three. Migrating from TGI takes about one to three days. Key parameters like max-num-seqs and gpu-memory-utilization have mature community-tested defaults.

One thing worth noting: vLLM’s prefix caching is optional (--enable-prefix-caching), and disabled by default. If you’re running Agent workloads where every request carries the same system prompt and tool definitions, you’re recomputing the same prefix every time without realizing it.

In January 2026, vLLM’s core maintainers founded Inferact, securing $150M in seed funding at an $800M valuation. They’ve committed to keeping Apache 2.0 unchanged. For teams evaluating build-vs-buy: the core dependency isn’t going anywhere.

SGLang: Built for Agents, not “universally faster”

SGLang’s headline feature is RadixAttention: a radix tree that organizes KV cache by token sequence. When a new request arrives, it walks the tree to find the longest matching prefix and reuses it directly. Unlike vLLM’s block-aligned approach, RadixAttention catches partial overlaps at any boundary.

This design shines for Agent workloads. Every Agent call carries a system prompt (500–2000 tokens), tool schemas (800–1500 tokens), and growing conversation history. SGLang computes the shared prefix once, then every subsequent instance hits the cache. 100 requests sharing a 2000-token prefix only store one copy — 99% VRAM savings on the shared portion.

But there’s a catch people miss: RadixAttention’s benefit isn’t free. It depends on your traffic having enough repeated prefixes. If your workload is code completion — where every context is different — it not only doesn’t help, it actually costs you, because the radix tree competes with model weights for the same VRAM budget.

Here’s real data. Spheron benchmarked both on a single H100 SXM5 with Llama 3.3 70B FP8 in June 2026:

Notice the first row: without prefix reuse, they’re essentially identical. SGLang’s advantage is entirely contingent on your traffic having repeated prefixes. No repeats, no speedup.

SGLang had multiple critical vulnerabilities surface in H1 2026, including a CVSS 9.8 unauthenticated RCE. If you’re on vLLM, you don’t need to track security patches as urgently. With SGLang, you do. It’s not a statement about quality — it’s about whether you have dedicated ops headcount.

The question more important than any benchmark: have you measured your prefix overlap rate?

Most teams pick a framework by running benchmarks. Throughput, latency, concurrency limits. Then they make a decision based on numbers.

But there’s one metric more decisive than all of them, and most people never measure it: what percentage of your traffic shares common prefix tokens.

This single number determines whether SGLang’s RadixAttention is worth the operational overhead. Below 40% overlap, vLLM and SGLang are equivalent — pick the one with better docs. Between 40% and 60%, SGLang starts showing real gains. Above 60%, SGLang delivers 37% lower TTFT and 30% lower cost.

How to measure: capture a few thousand production prompts, compute longest-common-prefix distributions across request pairs. If the median is several hundred tokens and covers most requests, seriously consider SGLang. If almost every prompt is unique, stick with vLLM — one less routing layer to maintain.

I genuinely believe this is the most underrated engineering insight in inference serving: spending an afternoon understanding your traffic structure matters more than a week of benchmark runs.

My recommended path

Not complicated. Three steps.

Step one: establish a production baseline with vLLM. When unsure, vLLM is the default. Thickest docs, easiest troubleshooting, most mature parameter defaults. Get it running, get monitoring in place.

Step two: measure prefix overlap rate. Run a month of real traffic, capture several thousand prompts, do prefix analysis.

Step three: if overlap rate consistently exceeds 60%, route Agent traffic to SGLang. Both engines speak the OpenAI-compatible API. Switching is literally changing a base URL. The real cost is rebuilding your benchmarks and retuning parameters.

For teams of three or more, the practical setup is a gateway layer that routes by model and workload type: llama.cpp for dev machines and edge devices, vLLM for general production traffic, SGLang for Agent and prefix-dense workloads. Three engines coexisting isn’t over-engineering — it’s the norm for mature teams in 2026.

Quick decision guide

One last thing

Inference engines iterate weekly. vLLM’s MRv2 is going default, SGLang is building hierarchical caching and prefill-decode disaggregation, llama.cpp is gradually adding server capabilities. Everything in this article might be outdated in three months.

But one thing doesn’t change much: the most important question in framework selection isn’t “which is fastest.” It’s “what does my traffic actually look like.” Figure that out, and you’re ahead of most teams.

Performance data sourced from Spheron, LLM Academy, n4n AI independent benchmarks (Apr–Jul 2026). Always re-benchmark on your target hardware with your actual traffic before making deployment decisions.

Need help setting up your inference pipeline? Follow “梦兽编程” on WeChat and send “推理框架” for our selection checklist and deployment SOP.

全栈之巅 · 梦兽编程 | rexai.top