Infrastructure / Developer Tools
vLLM vs SGLang vs TensorRT-LLM: 7 Inference Engines Tested Against MLPerf v6.0
By the NeuralWired Infrastructure Desk | Published July 22, 2026 | 12 min read
Your GPU bill went up again last month, and your throughput barely moved. That’s not a hardware problem anymore. It’s a software problem, and in 2026 the gap between a well-tuned inference stack and a default install is wide enough to change your infrastructure budget by double digits.
This guide ranks the best LLM inference optimization tools of 2026 for teams actually running models in production, not just testing them on a laptop. We pulled numbers from MLCommons’ newly released MLPerf Inference v6.0 suite, cross checked vendor claims against SemiAnalysis’s InferenceX benchmark platform, and flagged where the industry’s own benchmarking methods might be lying to you.
In this article
Why 2026 Changed the Inference Math
Two years ago, “serving an LLM” meant getting Llama 2 to answer chat prompts fast enough that users didn’t notice the wait. In 2026, the traffic looks nothing like that. Reasoning models like DeepSeek-R1 chew through multi-step chains of thought before producing a token. Agentic workloads fire off dozens of overlapping requests with shared context. RAG pipelines lean hard on prefix reuse. None of that fits the old single-turn chat benchmark.
That shift is exactly what MLCommons built its latest benchmark suite to capture, and it’s why picking an inference engine in 2026 is a genuinely different decision than it was in 2024.
What MLPerf Inference v6.0 Actually Measured
MLCommons released MLPerf Inference v6.0 on April 1, 2026, and it’s the most substantial rewrite of the suite in the benchmark’s history. Five of eleven datacenter tests were new or updated: a GPT-OSS 120B benchmark for math and coding reasoning, an expanded DeepSeek-R1 test with a speculative decoding scenario, a Meta-contributed recommender benchmark called DLRMv3, the suite’s first text-to-video generation test, and a new vision-language benchmark built on Shopify product catalog data.
“This is the most significant revision of the Inference benchmark suite that we’ve ever done.” Frank Han, Systems Development Engineering, Dell Technologies, MLPerf Inference Working Group Co-chair
The suite also introduced a new harness called LoadGen++, which lets submitters run benchmarks against a serving style software stack that looks a lot more like real production traffic than the older synthetic load generators. Twenty four organizations submitted results this round, including AMD, Google, NVIDIA, Oracle, Red Hat, and Lambda.
The infrastructure scale tells its own story. Multi-node submissions jumped 30% compared to the prior round in September 2025, and the largest submitted system used 72 nodes and 288 accelerators, four times the node count of the previous record. Ten percent of submitted systems now use more than ten nodes, up from just 2%.
“These partnerships were essential in ensuring that the tests include scenarios and workloads that represent the current state of the industry.” Miro Hodak, Senior Member of Technical Staff, AMD, MLPerf Inference Working Group Co-chair
Ultralytics contributed an upgraded YOLOv11 based object detection test to the edge category. Its founder framed the value of the whole exercise plainly.
“MLPerf Inference benchmarks play a vital role in driving transparency and accountability across the AI industry.” Glenn Jocher, CEO and Founder, Ultralytics
Why this matters for your engine choice
If you’re still benchmarking candidate engines against single-turn chat throughput, you’re testing for a workload that’s disappearing. MLPerf’s own suite has moved to reasoning and multi-node scenarios because that’s where production traffic actually lives now.
The 7 Engines, Compared
Here’s where most “best of” lists go wrong: they rank engines on raw tokens per second and call it a day. Workload shape matters more than any single number. A RAG pipeline and a batch summarization job want different things from the same GPU.
| Engine | Core approach | Best fit | Notable stat |
|---|---|---|---|
| vLLM | PagedAttention memory management, broad hardware support | General purpose, fastest path to production, multi vendor hardware | ~86,000 GitHub stars, releases roughly every two weeks |
| SGLang | RadixAttention prefix cache reuse | RAG, multi-turn chat, agentic loops with shared context | Frequently cited as the strongest option for prefix-heavy traffic |
| TensorRT-LLM | Compiles models into NVIDIA’s proprietary engine format | Single-vendor, NVIDIA-only deployments chasing peak throughput | Current version referenced in 2026 sources: v1.2.0 |
| llama.cpp | C/C++ engine, the backbone under most local tools | CPU and consumer-hardware inference, embedded deployments | 121,000 GitHub stars, 1,806 contributors |
| Ollama | llama.cpp backend (MLX reported on Apple Silicon in 2026) | Local developer workflows, prototyping | Reported at 172,000+ GitHub stars (unverified against GitHub directly) |
| Hugging Face TGI | Originally general purpose serving toolkit | Legacy deployments only, per reported status below | Reported to be in maintenance mode in 2026, unconfirmed against source repo |
| InferenceX benchmark stack | Not an engine; a continuously updated benchmark platform | Validating vendor claims before you commit to an engine | V2 launched February 2026 with GB300 NVL72 coverage |
A word on that Hugging Face TGI line, because it matters for anyone planning a migration. Multiple 2026 sources describe TGI shifting into maintenance mode with new users pointed toward vLLM, SGLang, and llama.cpp instead. We could not independently confirm this against Hugging Face’s own repository at the time of writing, so treat it as reported rather than settled. If you’re running TGI in production today, check the repository directly before you build a migration plan around a secondhand claim.
Worth noting too: 2026 is the year inference optimization became its own funded category rather than a side effect of the model layer. vLLM’s own creators, Woosuk Kwon, Simon Mo, and Ion Stoica, launched a company called Inferact in January 2026, backed by a16z, specifically to build what they call a universal inference layer across hardware and model architectures. When the people who solved the original memory bottleneck go out and raise venture money to solve it again commercially, that tells you how much money is riding on this layer of the stack.
Software Beats Hardware More Often Than You’d Think
Here’s the number that should reframe how you think about your next GPU purchase: in its own MLPerf v6.0 submission, Lambda found that NVIDIA’s Blackwell Ultra delivered 29% more throughput than the prior Blackwell generation on identical workloads. But the software stack alone, running on the exact same hardware, added another 9%. Lambda’s Smart Expert Routing technique cut P99 time-to-first-token by 31%.
Read that again. A software change on unchanged hardware moved the needle nearly a third as much as an entire GPU generation upgrade. If your team is budgeting for next year’s inference costs purely around which GPUs to buy, you’re solving half the problem.
This tracks with the broader MLPerf v5.1 data from September 2025, where NVIDIA’s GB300 NVL72 (Blackwell Ultra) delivered 45% more DeepSeek-R1 reasoning throughput than the prior GB200 NVL72 generation, according to HPCwire’s coverage of that round. AMD also made its first submissions that cycle, with a 4-node MI355X cluster posting a 3.4x throughput gain over the prior MI300X generation.
The Benchmark Trust Problem Nobody’s Talking About
Here’s our contrarian take, and it’s not ours alone: a lot of the “Engine X beats Engine Y by 40%” claims circulating in 2026 comparison articles might be measuring the wrong thing entirely.
A 2026 preprint from Google researchers Ashok Chandrasekar and Jason Kramberger modeled the client side of common LLM benchmarking tools using queueing theory, and found that single-process, asyncio-driven benchmarking clients suffer from Python’s GIL creating a queuing bottleneck of their own. As concurrency scales up, that bottleneck artificially inflates time-to-first-token and time-per-output-token measurements, according to the paper on arXiv. In plain terms: the tool measuring the engine can be slower than the engine it’s measuring, and that gap gets bigger exactly when you push concurrency higher, which is when it matters most.
Our read
This signals that a meaningful share of the public “vLLM vs SGLang vs TensorRT-LLM” benchmark posts circulating right now may be comparing benchmarking tool artifacts as much as real engine performance. Before you make a purchasing or migration decision off a single blog post, check whether the methodology discloses its process model and concurrency handling. If it doesn’t, treat the numbers as directional at best.
This isn’t a hypothetical concern either. Different sources genuinely disagree on the same matchups. One benchmark shows SGLang and LMDeploy topping 16,000 tokens per second against vLLM’s roughly 12,500 on Llama 3.1 8B, while another shows TensorRT-LLM leading at every concurrency level on Llama 3.3 70B running on H100. These aren’t necessarily contradictory since the model, hardware, and settings all differ, but it does mean a single “best overall” ranking is a simplification you should be skeptical of.
The safer move: validate any vendor benchmark against MLCommons’ public MLPerf dashboard or against SemiAnalysis’s InferenceX platform, which is continuously updated with support from Crusoe, CoreWeave, Nebius, TensorWave, Oracle, and Together AI. Crusoe’s CEO put the case for that kind of open, reproducible testing directly.
“At Crusoe, we believe being a great partner means empowering our customers with choice and clarity. That’s why we’re proud to support InferenceMAX, which provides the entire AI community with open source, reproducible benchmarks for the latest hardware.” Chase Lochmiller, Co-Founder and CEO, Crusoe
How to Actually Choose One
Skip the “best overall” instinct. Match the engine to the traffic pattern.
- Building a RAG or agentic product with heavy prefix reuse? SGLang’s RadixAttention scheduling is repeatedly cited as the stronger fit because it caches and reuses shared context instead of recomputing it.
- Need hardware portability across NVIDIA, AMD, TPUs, or Trainium? vLLM remains the broadest bet, with the fastest path from prototype to production and a release cadence that outpaces almost every other infra project in this category.
- All-in on NVIDIA and chasing maximum single-model throughput? TensorRT-LLM’s compiled engine format still wins on raw numbers once you’ve paid the setup cost.
- Running locally or on consumer hardware? llama.cpp is the substrate under nearly every local tool, including Ollama, and its 121,000 GitHub stars and 1,806 contributors reflect how entrenched it’s become.
- Still on Hugging Face TGI? Confirm its current status directly against the repository before you plan around secondhand reports of maintenance mode.
And one budgeting reality check: per-token inference costs for GPT-4-class performance have reportedly fallen roughly 1,000x over three years, down to around $0.40 per million tokens by some estimates. Total inference spending keeps rising anyway, because usage growth is outpacing the unit cost declines. Switching engines can absolutely cut your cost per token. It won’t necessarily shrink your total bill, and any plan that assumes otherwise is setting up a budget conversation you’ll lose later.
Frequently Asked Questions
What is the best LLM inference engine in 2026?
There’s no single “best” engine, it depends on workload. vLLM offers the broadest hardware support and fastest path to production. SGLang leads on prefix-heavy workloads like RAG and multi-turn chat via RadixAttention. TensorRT-LLM delivers the highest raw throughput on NVIDIA-only deployments once compiled.
What is the difference between vLLM and TensorRT-LLM?
vLLM is an open source, hardware-portable engine (NVIDIA, AMD, TPU, Trainium) using PagedAttention for memory efficiency and fast iteration. TensorRT-LLM is NVIDIA’s proprietary engine that compiles models into optimized TensorRT engines for maximum single-vendor throughput, at the cost of flexibility and longer setup time.
Is Hugging Face TGI still maintained in 2026?
Multiple 2026 sources report TGI has moved into maintenance mode, with Hugging Face directing new projects toward vLLM, SGLang, and llama.cpp instead. This is reported by secondary sources and should be confirmed directly against Hugging Face’s own repository before being treated as final.
What is MLPerf Inference and why does it matter for LLM serving?
MLPerf Inference is MLCommons’ independently audited, vendor-neutral benchmark suite for AI system performance. Its April 2026 v6.0 release added GPT-OSS 120B, expanded DeepSeek-R1 reasoning, and a new serving-style LoadGen++ harness, making it the most credible public reference point for comparing real-world inference performance.
How much does LLM inference cost in 2026?
Estimates vary, but per-token costs for GPT-4-class performance are widely reported to have fallen roughly 1,000x over three years, to around $0.40 per million tokens. Total inference spending continues rising anyway, because usage growth is outpacing these per-token cost declines.
Where This Goes Next
The engine layer stopped being a free add-on to the model layer somewhere around January 2026, when vLLM’s own founders decided it was worth its own venture-backed company. Expect more of that: standalone inference infrastructure businesses raising money on the premise that the serving stack, not the model weights, is where the next round of margin gets won or lost.
Watch three things over the next 6 to 18 months. First, whether Hugging Face TGI’s reported maintenance mode status gets confirmed or walked back, since that will settle a live migration debate for a lot of teams. Second, whether InferenceX and MLPerf’s LoadGen++ harness push more vendors toward disclosing their actual benchmarking methodology, given the Google GIL findings. Third, whether reasoning-model workloads like DeepSeek-R1 and GPT-OSS 120B keep pulling multi-node deployment further into the mainstream, the way the 30% jump in multi-node MLPerf submissions suggests they already are.
Pick your engine for the traffic you actually have, not the traffic you had two years ago. And test any benchmark claim, including the ones in this article, against your own workload before you bet a production budget on it.
Want infrastructure stories like this before anyone else covers them?
Subscribe to The Neural Loop at neuralwired.com/newsletter
Subscribe to The Neural Loop at neuralwired.com/newsletter