KubeCon + CloudNativeCon India 2026

As (LLMs) continue to grow in size and demand, single-node inferencing quickly becomes a bottleneck for performance, scalability, and cost. While vLLM has become popular for efficient LLM serving on a single node, it does not fully address the challenges of distributed inferencing across multiple GPUs and nodes in Kubernetes environments.

This talk introduces llm-d, a emerging cloud-native project designed to enable distributed LLM inferencing on Kubernetes. We will cover why vLLM gained popularity and the limitations when scaling beyond a single node. We will explore how llm-d goes a step further by enabling multi-node, multi-GPU inferencing with cloud-native primitives.

Attendees will learn how llm-d fits into modern Kubernetes platforms, how it improves scalability and resource utilization. The session focuses on practical architecture, design trade-offs, and real-world use cases rather than theory with a demo on how llm-d distributes load.