Building Production-Ready GKE Platforms with Terraform

The Problem with "Click-to-Deploy" Clusters

When organizations first adopt Kubernetes, the journey usually begins in the cloud console. An engineer clicks through the Google Kubernetes Engine (GKE) wizard, selects some node types, enables a few add-ons, and within ten minutes, a cluster is running.

This works perfectly for the first cluster. But in an enterprise environment, a single cluster is never enough. You need separate clusters for Development, Staging, and Production. You need clusters in different regions for disaster recovery. You need isolated clusters for sensitive workloads.

Suddenly, that "easy" click-to-deploy process becomes an operational nightmare:

• Drift: The Staging cluster was updated to Kubernetes 1.29, but Production is still on 1.28.
• Security: Someone manually added a firewall rule to a cluster for debugging and forgot to remove it.
• Knowledge Silos: Only one engineer knows exactly which settings were toggled when the Production cluster was created.
• Disaster Recovery: If a region goes down, rebuilding the exact cluster configuration from memory is impossible.

As Platform Engineers, our goal is to eliminate these failure modes by treating infrastructure as code (IaC). Terraform turns infrastructure changes into reviewable, version-controlled configuration. The real benefit appears when the same platform pattern has to be reproduced across environments.

---

Why Terraform Belongs at the Platform Foundation

Terraform should manage the foundation of your platform. This includes:

• The VPC network and subnets
• Cloud IAM roles and Service Accounts
• The GKE cluster control plane
• Node pools and autoscaling configurations
• Workload Identity configurations

However, a common anti-pattern is trying to use Terraform to manage everything, including Kubernetes deployments and Helm charts. While the kubernetes and helm Terraform providers exist, coupling application delivery to infrastructure provisioning slows down developers and bloats your Terraform state.

The Golden Rule: Use Terraform to build the cluster. Use GitOps (like ArgoCD or Flux) to deploy applications into the cluster.

---

Designing Reusable Terraform Modules

To avoid copying and pasting Terraform code for every new cluster, we design reusable modules. A well-designed module exposes only the necessary variables, hiding the complexity of the underlying implementation.

Here is an example of a streamlined module structure:

bash
platform-infrastructure/
├── modules/
│   ├── network/
│   │   ├── main.tf
│   │   └── outputs.tf
│   └── gke-cluster/
│       ├── main.tf
│       ├── node_pools.tf
│       └── variables.tf
└── environments/
    ├── dev/
    │   └── main.tf (consumes modules)
    └── prod/
        └── main.tf (consumes modules)

The Network Module

Before you can create a private GKE cluster, you need a robust network foundation. This module should provision the VPC, private subnets, and secondary IP ranges for Kubernetes Pods and Services.

hcl
# modules/network/main.tf
resource "google_compute_network" "vpc" {
  name                    = "${var.environment}-vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "gke_subnet" {
  name          = "${var.environment}-gke-subnet"
  region        = var.region
  network       = google_compute_network.vpc.name
  ip_cidr_range = var.primary_cidr

  secondary_ip_range {
    range_name    = "pods"
    ip_cidr_range = var.pods_cidr
  }

  secondary_ip_range {
    range_name    = "services"
    ip_cidr_range = var.services_cidr
  }
}

The GKE Module

The cluster module consumes the network outputs. Notice how we disable the default node pool. In production, default node pools are dangerous because they cannot be easily modified or deleted without destroying the entire cluster.

hcl
# modules/gke-cluster/main.tf
resource "google_container_cluster" "primary" {
  name     = "${var.environment}-cluster"
  location = var.region
  network  = var.network_id
  subnetwork = var.subnet_id

  # We can't create a cluster with no node pool defined, but we want to only use
  # separately managed node pools. So we create the smallest possible default
  # node pool and immediately delete it.
  remove_default_node_pool = true
  initial_node_count       = 1

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = var.master_cidr
  }

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }
}

Managing Node Pools as Separate Resources

By separating node pools from the cluster resource, we can upgrade, resize, or replace worker nodes without touching the control plane. This is critical for zero-downtime infrastructure maintenance.

hcl
# modules/gke-cluster/node_pools.tf
resource "google_container_node_pool" "general_compute" {
  name       = "general-compute"
  location   = var.region
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = var.min_nodes
    max_node_count = var.max_nodes
  }

  node_config {
    machine_type = var.machine_type
    
    # Use a custom service account, NOT the compute engine default!
    service_account = google_service_account.gke_nodes.email
    
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]
  }
}

---

Security and IAM Considerations

A production-ready platform must follow the principle of least privilege.

1. Custom Service Accounts: Never use the default Compute Engine service account for your GKE nodes. Create a dedicated service account (gke-node-sa) with only the permissions necessary to pull container images and write logs/metrics.
2. Workload Identity: Do not export Service Account keys for your applications. Enable Workload Identity (as shown in the cluster config above) to allow Kubernetes Service Accounts to act as Google Cloud IAM Service Accounts seamlessly.
3. Private Clusters: Ensure enable_private_nodes = true is set. Your worker nodes should not have public IP addresses. Outbound internet access should be routed through Cloud NAT.

---

The CI/CD Workflow for Infrastructure

To truly realize the benefits of IaC, Terraform must be executed through a CI/CD pipeline, not from a developer's laptop.

A common pattern that works well for platform teams is:

1. A developer opens a Pull Request modifying environments/prod/main.tf to increase the max_nodes variable.
2. GitHub Actions runs terraform plan. The output is posted as a comment on the PR.
3. Senior engineers review the plan to ensure no destructive changes are being proposed.
4. Once the PR is merged into the main branch, a CD pipeline runs terraform apply.

This creates a perfect audit trail of every infrastructure change.

---

Common Failure Modes & Trade-offs

Failure Mode: State Lock Contention If you have a large engineering team, you will eventually experience Terraform state lock contention. To mitigate this, break your Terraform state into logical boundaries. Don't put your VPC, Database, and GKE cluster in the same state file. Use Terraform data sources to pass information between them.

Trade-off: Terraform vs. Config Connector Google provides Config Connector, which allows you to manage GCP resources using Kubernetes YAML. While this sounds appealing for teams fully bought into Kubernetes, I've found that Terraform's rich ecosystem, planning phase, and robust state management make it the superior choice for foundational infrastructure. Save Config Connector for application-specific resources (like a Pub/Sub topic needed by a microservice).

---

Production Readiness Checklist

Before moving your Terraform-managed GKE platform into production, verify:

• [ ] Remote state is configured with a locking mechanism (e.g., GCS bucket with versioning enabled).
• [ ] The default Compute Engine service account is completely unlinked from the cluster.
• [ ] Workload Identity is enabled and configured.
• [ ] Nodes are private and internet egress flows through Cloud NAT.
• [ ] Terraform runs exclusively through a CI/CD pipeline; human access to the state bucket is read-only.
• [ ] IP ranges (Primary, Pods, Services) are properly sized for future growth to prevent IP exhaustion.

Conclusion

Building a Kubernetes platform is much more than deploying a cluster. It's about designing a repeatable, secure, and auditable foundation. By leveraging Terraform with modular design, strict IAM boundaries, and separated node pools, you transform infrastructure from a manual liability into an automated asset.

The initial investment in writing solid Terraform modules pays dividends every time you need to stamp out a new environment, recover from a failure, or pass a compliance audit.