KubeCon + CloudNativeCon India 2026

We built a research testbed to validate ML workload scalability on Kubernetes with Kubeflow and Cilium. During 500-node stress tests, GPUs sat idle 60% of the time while pods waited for available resources.

Through controlled experiments, we isolated the cause: Kubeflow's pipeline scheduler and Cilium's network-aware pod placement make conflicting decisions. Kubeflow schedules pods without considering network topology. Cilium optimizes networking but can't move scheduled pods. Result? GPUs unused while the scheduler searches for placement that won't happen.

This talk shares our systematic investigation, diagnostic methodology, and scheduling constraints that resolved it. Lab tests show GPU utilization improved from 40% to 85%. You'll see the problem reproduced live, understand why it's hard to detect, and get tested Kubernetes configs. This matters for anyone planning distributed training with Kubeflow or similar orchestrators on network-optimized clusters.