Running AI/ML Workloads on Kubernetes: GPUs, Scaling and Reliability

Beyond Standard CPU Microservices

For years, Platform Engineers have perfected the art of running stateless CPU microservices on Kubernetes. The formula is well understood: pack multiple pods onto standard VMs, use the Horizontal Pod Autoscaler (HPA) based on CPU utilization, and rely on Kubernetes to seamlessly reschedule pods if a node dies.

When migrating AI/ML workloads to Kubernetes, teams often try to apply this same formula. The result is usually disastrous: staggering cloud costs, models crashing from Out-Of-Memory (OOM) errors, and scheduling failures.

AI workloads behave fundamentally differently than web APIs:

1. They require specialized hardware (GPUs). An NVIDIA L4 or A100 is significantly more expensive and scarce than a standard vCPU.
2. They are extremely resource-hungry. Model inference requires loading massive weights into GPU memory (VRAM). You cannot easily "bin-pack" multiple Large Language Models (LLMs) onto a single small GPU.
3. They scale slowly. A web API container might start in 500ms. An LLM inference container might take 2-3 minutes to pull a 15GB model layer from cloud storage into VRAM.

Designing a Kubernetes platform for AI workloads requires separating CPU traffic from GPU processing and rethinking how we handle scheduling and autoscaling.

---

Separating CPU and GPU Workloads

The most fundamental architectural decision for an AI platform is isolating the expensive GPU workloads from the standard API services.

If you run your API gateways, observability agents, and web backends on nodes equipped with $10,000 GPUs, you are wasting massive amounts of capital.

To solve this, we create dedicated Node Pools in GKE.

1. CPU Node Pool: Standard VMs (e.g., e2-standard-4) running ingress controllers, auth services, and standard APIs.
2. GPU Node Pool: Accelerated VMs (e.g., g2-standard-12 with an L4 GPU) dedicated entirely to model inference and ML training workers.

Taints and Tolerations

Creating the pool isn't enough. We must prevent standard CPU pods from accidentally scheduling on the GPU nodes. We do this using Kubernetes Taints.

When provisioning the GPU node pool (e.g., via Terraform), we apply a taint:

hcl
resource "google_container_node_pool" "gpu_pool" {
  name = "l4-gpu-pool"
  
  node_config {
    machine_type = "g2-standard-12"
    
    guest_accelerator {
      type  = "nvidia-l4"
      count = 1
    }

    taint {
      key    = "nvidia.com/gpu"
      value  = "present"
      effect = "NO_SCHEDULE"
    }
  }
}

Now, no pod will schedule on this node unless it explicitly tolerates the taint.

---

Scheduling the AI Workload

To run a model serving workload (like vLLM or Triton) on our GPU nodes, the Pod specification must include three critical elements:

1. Tolerations: To bypass the taint we set above.
2. Node Selectors / Affinity: To ensure it only lands on the GPU nodes and doesn't try to schedule on the CPU pool.
3. Resource Requests: To request the physical GPU from the Kubernetes device plugin.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference-service
spec:
  replicas: 2
  template:
    spec:
      # 1. Tolerate the GPU taint
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Equal"
        value: "present"
        effect: "NoSchedule"
        
      # 2. Force scheduling onto the GPU node pool
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
        
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:latest
        resources:
          limits:
            # 3. Request the physical GPU device
            nvidia.com/gpu: 1
            memory: "32Gi"
          requests:
            nvidia.com/gpu: 1
            cpu: "4"
            memory: "32Gi"

---

The HPA Illusion: Why AI Autoscaling is Hard

A common mistake is assuming the standard Kubernetes HPA will magically handle AI scaling.

If you configure an HPA to scale an LLM deployment when CPU usage hits 70%, it will likely fail. Model inference is almost entirely constrained by GPU VRAM and Compute capability, not CPU.

Furthermore, if the HPA detects a spike and requests 5 new pods, the cluster autoscaler will attempt to provision 5 new GPU nodes. Provisioning a GPU node, pulling a massive container image, and loading a 20GB model into memory can take several minutes. By the time the pods are ready, the user who initiated the request has probably timed out.

Better Strategies for AI Scaling

1. Custom Metrics: Do not scale based on CPU. Scale based on custom metrics like queue_depth or concurrent_requests exposed by the inference server.
2. Overprovisioning (Pause Pods): If you require low latency, you may need to intentionally run dummy "pause" pods that hold GPU capacity in reserve. When a real workload needs to scale, it preempts the pause pod, eliminating the node provisioning time.
3. Batch vs. Online: Separate batch processing from online inference. Online inference requires immediate availability (costly overprovisioning). Batch processing can tolerate spin-up times, allowing for aggressive scale-to-zero architectures.

---

Reliability: Preventing Disruption

GPUs are physical hardware, and hardware fails. Furthermore, node upgrades will forcibly evict your pods. Because AI models take so long to initialize, an unexpected eviction can cause severe service degradation.

You must implement PodDisruptionBudgets (PDBs) to ensure Kubernetes doesn't evict all your model instances simultaneously during a cluster upgrade.

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-inference-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: llm-inference-service

This guarantees that the platform orchestration will always leave at least one model replica running, even during maintenance windows.

Conclusion

Running AI workloads on Kubernetes forces Platform Engineers to reconsider fundamental assumptions about scheduling, scaling, and resource allocation. By strictly isolating GPU hardware, utilizing proper taints and affinities, and abandoning naive CPU-based autoscaling in favor of metric-driven scaling and careful capacity planning, you can build an AI infrastructure that is both highly resilient and cost-effective.