vLLM vs TensorRT-LLM vs HF TGI vs LMDeploy, A Deep Technical Comparison for Production LLM Inference

Production LLM Serving: A Comparative Analysis

Production Large Language Model (LLM) serving has evolved into a complex systems problem, shifting focus from the generate() loop to optimizing for tokens per second, tail latency, and cost per million tokens on available GPU resources. This comparison analyzes four prominent inference stacks: vLLM, NVIDIA TensorRT-LLM, Hugging Face Text Generation Inference (TGI v3), and LMDeploy.

The ideal model of LLM inference often clashes with technical realities; achieving peak throughput frequently necessitates trade-offs in latency, and managing the KV cache efficiently is crucial for cost-effective scaling. Inefficient KV cache handling can lead to significant memory waste and performance bottlenecks, impacting the overall cost of serving millions of tokens.

Key Insights

• TensorRT-LLM H100 Performance: NVIDIA’s TensorRT-LLM reaches over 10,000 output tokens/s on H100 GPUs with FP8 precision, achieving up to 4.6x higher throughput compared to A100.
• PagedAttention: vLLM’s PagedAttention treats the KV cache like paged virtual memory, reducing fragmentation and improving concurrency.
• TGI v3 Long Prompt Optimization: Hugging Face TGI v3 achieves up to a 13x speedup on long prompts (over 200,000 tokens) compared to vLLM through chunking and prefix caching.

Working Example

# Example of using vLLM with Ray Serve (simplified)
from ray import serve
from vllm import LLM, SamplingParams

llm = LLM(model="facebook/opt-125m")

@serve.deployment
class MyLLM:
    def __init__(self, llm: LLM):
        self.llm = llm

    def __call__(self, prompt: str):
        sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=100)
        return self.llm.generate(prompt, sampling_params)

serve.run(MyLLM.deploy(llm))

Practical Applications

• High-Volume Chatbots: NVIDIA TensorRT-LLM is ideal for high-volume, low-latency chatbot applications requiring maximum throughput on NVIDIA GPUs.
• RAG Pipelines: Hugging Face TGI v3 excels in Retrieval-Augmented Generation (RAG) pipelines with long context windows, leveraging its prefix caching for significant speedups.

References:

• https://nvidia.github.io/TensorRT-LLM/blogs/H100vsA100.html
• https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html
• https://huggingface.co/docs/text-generation-inference/en/conceptual/chunking
• https://www.marktechpost.com/2024/12/10/hugging-face-releases-text-generation-inference-tgi-v3-0-13x-faster-than-vllm-on-long-prompts/
• https://huggingface.co/posts/Narsil/601808386353996
• https://github.com/InternLM/lmdeploy
• https://lmdeploy.readthedocs.io/