Posts

Terraform State Management, Locking, and Backups: A Production Deep Dive

Image
Terraform State Management, Locking, and Backups: A Production Deep Dive Terraform state is an operational database, not a disposable artifact. This deep dive covers S3 backends, S3 and DynamoDB locking, encryption, IAM, backups, migrations, state surgery, CI concurrency, and incident runbooks. TL;DR Terraform state management should be treated like production data management: isolate state by blast radius, store it in a remote backend, enable locking, encrypt it, version it, and rehearse restore procedures before an outage. For AWS teams, the modern S3 backend can use native S3 lock files, while older estates may still need DynamoDB locking during migration. The strongest designs combine least-privilege IAM, S3 Versioning, KMS controls, CI concurrency, state migration discipline, and documented runbooks for stuck locks, accidental overwrites, and state surgery. Terraform State Is a Database, Not a File Use a remote backend, lock state, and back it up. The production version is mo...

Operators vs Helm: What Platform Teams Should Use When

Image
Operators vs Helm: What Platform Teams Should Use When Helm and Operators solve different Kubernetes lifecycle problems. This deep dive gives platform teams a practical decision model, failure modes, migration path, and working YAML examples. TL;DR Platform teams should choose Helm when the job is packaging, parameterizing, installing, upgrading, and rolling back mostly static Kubernetes resources. Choose an Operator when the system needs a controller that watches live cluster state, reconciles drift, writes status, manages finalizers, or encodes domain-specific upgrade logic. The strongest platforms often use both: Helm to install controllers and default resources, Operators to run ongoing lifecycle automation. The decision boundary is not complexity alone. It is whether ownership must continue after the initial apply. The Real Decision Is Runtime Ownership The usual Operators vs Helm debate gets framed as "simple versus complex." That is useful for first-pass triage, b...

Terraform CI/CD Pipelines with GitHub Actions: A Production-Grade Deep Dive

Image
Terraform CI/CD Pipelines with GitHub Actions: A Production-Grade Deep Dive Terraform automation is not just a YAML wrapper around plan and apply. This guide designs a production GitHub Actions pipeline with OIDC, saved plans, state locking, reviews. TL;DR A production Terraform CI/CD pipeline should separate read-only pull request validation from protected apply jobs, preserve the reviewed plan, use remote state locking, and authenticate to AWS with GitHub OIDC instead of long-lived access keys. GitHub Actions can run fmt, validate, plan, policy, and drift checks before reviewers approve changes. The apply stage should run through a protected environment, a concurrency group, least-privilege IAM, and observability hooks so infrastructure changes remain auditable, reproducible, recoverable, and tied to clear ownership. The Hard Part Is Not Running Terraform in CI The simplest Terraform pipeline is a trap: checkout the repository, install Terraform, run terraform apply -auto-approv...