Private GKE Networking: The Problems You Discover After Going Production

Public vs. Private Kubernetes

When evaluating managed Kubernetes offerings like Google Kubernetes Engine (GKE), one of the most critical decisions is the networking architecture. In development, a public cluster—where worker nodes have public IP addresses—is easy to use. But in an enterprise environment, exposing your compute layer directly to the internet is a severe security risk.

The industry standard for production workloads is a Private Cluster. In a private GKE cluster:

1. Worker nodes only have internal IP addresses.
2. The control plane can be locked down to specific authorized networks.
3. Outbound internet access must be explicitly configured via NAT.

However, moving to a private networking model introduces profound architectural complexities that catch many engineering teams off guard.

The IP Exhaustion Problem

The most common failure mode I've encountered when teams adopt GKE is IP address exhaustion. Kubernetes is incredibly thirsty for IP addresses. Unlike traditional VMs where one server equals one IP, a Kubernetes node hosts dozens of Pods, and each Pod requires its own unique IP.

In GKE (specifically VPC-native clusters), IP addresses are allocated using Alias IPs. This requires you to define three distinct ranges:

1. Primary Subnet Range: For the Node IP addresses.
2. Secondary Range (Pods): For the Pod IP addresses.
3. Secondary Range (Services): For the ClusterIPs of Kubernetes Services.

Why Did My Node Pool Fail to Scale?

Imagine you configure a /24 secondary range for your Pods, yielding 256 IP addresses. You assume this is plenty for your small microservice application.

A few months later, traffic spikes, and the Horizontal Pod Autoscaler (HPA) triggers a node pool scale-up. The scale-up fails. Why?

By default, GKE allocates a /24 block (256 addresses) to every single node to ensure there are enough IPs for the maximum number of pods per node (default is 110). If your entire Pod secondary range is a /24, your cluster is hard-limited to exactly one node.

How to Prevent IP Exhaustion

To fix this, you must carefully calculate your IP ranges before cluster creation (as secondary ranges cannot easily be expanded once the cluster is heavily utilized without creating new subnets).

If you expect 50 nodes, and you keep the default 110 pods/node, you need 50 * 256 IPs. You would need a /18 or /17 secondary range for Pods.

Alternatively, you can optimize IP usage by reducing the max-pods-per-node setting during node pool creation:

bash
gcloud container node-pools create optimized-pool \
    --cluster my-private-cluster \
    --max-pods-per-node 30 \
    --region us-central1

By lowering max-pods-per-node to 30, GKE only allocates a /26 (64 IPs) to each node, drastically reducing your IP consumption.

Outbound Connectivity: Cloud NAT and PGA

In a private cluster, your nodes have no public IPs. This means they cannot pull images from Docker Hub, they cannot download apt packages, and your applications cannot communicate with external APIs (like Stripe or Twilio).

1. Cloud NAT

To provide internet access to private nodes, you must provision a Cloud NAT gateway attached to a Cloud Router in the cluster's region.

hcl
resource "google_compute_router" "router" {
  name    = "gke-router"
  region  = "us-central1"
  network = google_compute_network.vpc.id
}

resource "google_compute_router_nat" "nat" {
  name                               = "gke-nat"
  router                             = google_compute_router.router.name
  region                             = google_compute_router.router.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}

2. Private Google Access (PGA)

What if your private nodes need to talk to Google Cloud services like Cloud Storage (GCS) or BigQuery? Routing this traffic through Cloud NAT is inefficient and incurs unnecessary egress costs.

By enabling Private Google Access on your subnet, GKE nodes can reach Google APIs using Google's internal backbone, completely bypassing the public internet and Cloud NAT.

Private DNS and Internal Load Balancing

Once your workloads are secure inside a private cluster, they need to communicate with other internal systems—perhaps a database in another VPC or an on-premise legacy API.

Service Networking & VPC Peering

Managed services like Cloud SQL utilize a specialized form of VPC Peering called Private Services Access. When deploying private GKE alongside Cloud SQL, ensure your GKE cluster and the Cloud SQL instance are peered to the same VPC, and that the firewall rules allow ingress from the GKE Pod secondary range, not just the Node primary range.

Internal Load Balancers (ILB)

If you want to expose a Kubernetes service to other VMs within your VPC (without exposing it to the internet), use an Internal Load Balancer. In GKE, this is triggered via an annotation on your Service:

yaml
apiVersion: v1
kind: Service
metadata:
  name: internal-api
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: internal-api

This provisions a Google Cloud ILB, mapping an internal IP from your primary subnet directly to your Kubernetes service.

A Troubleshooting Checklist for Private GKE

When connectivity fails in a private GKE environment, the layers of abstraction make debugging difficult. Use this checklist to isolate the issue:

1. Verify Pod IP Allocation: Run kubectl get pods -o wide to ensure Pods are receiving IPs from the expected secondary range.
2. Check Node Subnet Status: Run gcloud compute networks subnets describe <subnet-name> to check if the secondary ranges are exhausted.
3. Validate Cloud NAT: If Pods cannot reach the internet, check the Cloud NAT logs. Ensure the NAT gateway is attached to the correct region and subnet.
4. Inspect Firewall Rules: GCP firewalls apply at the network level. Ensure rules allow traffic from the Pod CIDR, not just the Node CIDR, if you are using VPC-native routing.
5. DNS Resolution: If internal services cannot be resolved, check the kube-dns logs and verify if Cloud DNS private zones are correctly linked to the VPC.

Conclusion

Private GKE networking transforms Kubernetes from a simple compute orchestrator into a complex piece of cloud infrastructure. The shift from public to private requires a deep understanding of IP address management, NAT routing, and VPC peering.

By carefully planning your CIDR blocks, minimizing IP waste per node, and strategically using Cloud NAT and Internal Load Balancers, you can build a network architecture that is both highly secure and highly scalable.