KubeCon + CloudNativeCon India 2026

The surge of Generative AI and large language models (LLMs) is introducing new operational challenges for Kubernetes platforms. Unlike traditional APIs, GenAI traffic is highly variable, token-driven, and cost-sensitive, requiring new approaches to routing, security, and observability.

This session, presented by the project maintainers, examines Envoy AI Gateway, a CNCF-hosted open source project, and its role as a Kubernetes-native control plane for GenAI workloads.. We’ll break down an end-to-end architecture that uses Envoy AI Gateway to manage and govern traffic across multiple LLM backends both self-hosted and cloud-based while enforcing policies such as token-aware rate limiting, authentication, and dynamic model selection.

Attendees will leave with practical insights into designing resilient, scalable GenAI platforms on Kubernetes, and an understanding of how AI-aware gateways fit into modern cloud-native infrastructure.