Terraform CI/CD Pipelines with GitHub Actions: A Production-Grade Deep Dive
Terraform CI/CD Pipelines with GitHub Actions: A Production-Grade Deep Dive
Terraform automation is not just a YAML wrapper around plan and apply. This guide designs a production GitHub Actions pipeline with OIDC, saved plans, state locking, reviews.
TL;DR
A production Terraform CI/CD pipeline should separate read-only pull request validation from protected apply jobs, preserve the reviewed plan, use remote state locking, and authenticate to AWS with GitHub OIDC instead of long-lived access keys. GitHub Actions can run fmt, validate, plan, policy, and drift checks before reviewers approve changes. The apply stage should run through a protected environment, a concurrency group, least-privilege IAM, and observability hooks so infrastructure changes remain auditable, reproducible, recoverable, and tied to clear ownership.
The Hard Part Is Not Running Terraform in CI
The simplest Terraform pipeline is a trap: checkout the repository, install Terraform, run terraform apply -auto-approve, and hope the state backend, cloud credentials, and reviewers all line up. That workflow is easy to demo and unsafe to operate. Production Terraform CI/CD pipelines with GitHub Actions need to answer harder questions:
- Who can create a plan, and who can apply it?
- Which identity mutates cloud infrastructure?
- How do reviewers know the applied changes match the reviewed plan?
- What prevents two deployments from writing to the same state at the same time?
- Where do drift, policy, cost, and rollback signals show up?
This deeper version keeps the same goal but moves the design toward a pipeline that can survive production realities: short-lived credentials, remote state, approval gates, plan artifacts, concurrency controls, and deployment telemetry.
HashiCorp documents the Terraform automation shape as plan on pull request and apply on the main branch or equivalent release path. GitHub documents OIDC as the way for workflows to request short-lived cloud credentials instead of storing long-lived secrets. AWS further recommends constraining GitHub OIDC trust policies by repository, organization, branch, or other subject claims so that unrelated workflows cannot assume the same role.
That gives us the operating principle:
Design rule: Pull request jobs should explain infrastructure changes. Protected apply jobs should mutate infrastructure.
Pipeline Architecture
A mature Terraform workflow usually has two separate execution paths.
The pull request path is read-heavy:
- Check formatting with
terraform fmt -check. - Initialize the working directory with backend access appropriate for planning.
- Validate syntax and provider configuration.
- Generate a plan.
- Convert the plan to JSON for review, policy checks, and cost analysis.
- Publish a concise PR comment or artifact.
The apply path is write-heavy:
- Runs only after merge, release, or manual approval.
- Uses a protected GitHub environment such as
production. - Assumes a cloud role through OIDC with a tight trust policy.
- Acquires the Terraform backend lock.
- Applies a saved plan or regenerates a plan under controlled conditions.
- Emits deployment and audit evidence.
Here is the high-level flow:
pull_request
-> fmt / validate / plan
-> plan artifact + PR summary
-> human and policy review
push to main or workflow_dispatch
-> protected environment approval
-> GitHub OIDC token
-> AWS STS AssumeRoleWithWebIdentity
-> terraform apply
This split matters because Terraform is not just a build command. It is a stateful controller for real infrastructure. The state backend is a coordination point, the provider credentials are production privileges, and the plan is the reviewable contract between desired configuration and real-world changes.
Repository Layout for Repeatable Automation
Terraform pipelines become fragile when every workflow step has to guess where configuration lives. Use a predictable layout and make each environment explicit:
infra/
modules/
ecs-service/
network/
live/
dev/
backend.hcl
main.tf
terraform.tfvars
prod/
backend.hcl
main.tf
terraform.tfvars
.github/
workflows/
terraform-pr.yml
terraform-apply.yml
The important design choice is not the exact folder name. It is that each root module has one state backend, one environment identity boundary, and one pipeline concurrency group. A monorepo can still run multiple roots, but each root should be treated as a separate deployment unit.
Use -chdir or workflow working-directory consistently:
terraform -chdir=infra/live/prod init -backend-config=backend.hcl
terraform -chdir=infra/live/prod plan -var-file=terraform.tfvars -out=tfplan
terraform -chdir=infra/live/prod show -json tfplan > tfplan.json
HashiCorp's CLI docs call out that terraform init prepares the working directory by accessing the configured backend, downloading modules, and installing provider plugins. That means your CI job must treat initialization as an environment-specific step, not a generic preamble.
A Pull Request Workflow That Produces Reviewable Plans
The PR workflow should use the smallest permissions it needs. For a typical plan workflow, it needs to read repository contents and, if you post a PR comment, write to pull requests. If it assumes an AWS role to read data sources or refresh state, it also needs id-token: write.
name: terraform-pr
on:
pull_request:
paths:
- "infra/**"
- ".github/workflows/terraform-pr.yml"
permissions:
contents: read
pull-requests: write
id-token: write
concurrency:
group: terraform-pr-${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
plan:
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: infra/live/dev
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v4
with:
terraform_version: "1.14.6"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v6.1.0
with:
role-to-assume: arn:aws:iam::123456789012:role/github-terraform-plan
aws-region: eu-west-1
- name: Terraform fmt
run: terraform fmt -check -recursive
- name: Terraform init
run: terraform init -backend-config=backend.hcl -input=false
- name: Terraform validate
run: terraform validate -no-color
- name: Terraform plan
run: terraform plan -input=false -no-color -out=tfplan
- name: Export plan JSON
run: terraform show -json tfplan > tfplan.json
- name: Upload plan artifacts
uses: actions/upload-artifact@v4
with:
name: terraform-plan-${{ github.event.pull_request.number }}
path: |
infra/live/dev/tfplan
infra/live/dev/tfplan.json
retention-days: 7
This workflow intentionally does not apply. It answers "What would change?" without letting unmerged code mutate shared infrastructure. The saved binary plan is useful for traceability, and the JSON form is useful for downstream policy checks.
There is a nuance: for untrusted fork pull requests, granting cloud access is risky. Many teams run a minimal fmt and static validation workflow for forks, then require a trusted maintainer-triggered plan for workflows that need cloud credentials. The exact pattern depends on repository trust, cloud account exposure, and whether your Terraform configuration reads live data sources during planning.
Plan Artifacts, JSON, and PR Comments
Terraform plan output is built for humans, but raw output can be noisy. A practical PR comment should summarize the blast radius:
- Number of resources to create, update, replace, or delete.
- Module path and resource addresses for destructive actions.
- Whether the plan touches IAM, security groups, public network paths, databases, or production services.
- A link to the full artifact.
You can generate a small plan summary from terraform show -json:
jq -r '
[.resource_changes[]?.change.actions | join(",")] as $actions
| {
create: ($actions | map(select(. == "create")) | length),
update: ($actions | map(select(. == "update")) | length),
delete: ($actions | map(select(. == "delete")) | length),
replace: ($actions | map(select(. == "delete,create" or . == "create,delete")) | length)
}
' tfplan.json
For a stricter review gate, inspect individual changes:
jq -r '
.resource_changes[]
| select(.change.actions | index("delete") or index("create"))
| "\(.address) -> \(.change.actions | join(","))"
' tfplan.json
That is a foundation for policy-as-code. You can block changes that delete production databases, open 0.0.0.0/0 on sensitive ports, modify privileged IAM policies, or replace a stateful service without an explicit exception. The point is not to remove human review. It is to make the risky parts visible before the reviewer scrolls through hundreds of lines.
Saved Plans and the Apply Contract
HashiCorp's plan docs describe saved plans as a way to ensure Terraform applies the same changes that were approved, even when the process spans machines or time. In automation, that property is valuable, but you need to respect the tradeoffs.
A saved plan captures decisions made at plan time. It may include provider-derived values and can contain sensitive data. Treat it like a sensitive artifact:
- Keep short retention.
- Restrict artifact access.
- Avoid publishing the binary plan outside the deployment system.
- Prefer JSON redaction for public comments.
- Regenerate the plan if provider credentials, variables, module versions, or backend configuration changed.
For production, many teams choose one of two patterns.
The strict pattern applies the reviewed saved plan:
terraform init -backend-config=backend.hcl -input=false
terraform apply -input=false tfplan
The fresh-plan pattern regenerates the plan after merge, posts the new summary, and requires environment approval:
terraform init -backend-config=backend.hcl -input=false
terraform plan -input=false -out=tfplan
terraform show -json tfplan > tfplan.json
terraform apply -input=false tfplan
The strict pattern maximizes review-to-apply fidelity. The fresh-plan pattern handles late drift and changes introduced by merge order. If you use the fresh-plan pattern, make the new plan visible in the apply run logs or deployment summary before approval.
OIDC: Stop Storing Long-Lived AWS Keys in GitHub
GitHub Actions supports OpenID Connect so workflows can request short-lived credentials from a cloud provider. AWS IAM supports an OIDC identity provider for https://token.actions.githubusercontent.com, and the official AWS credentials action can exchange the GitHub token for AWS role credentials.
The workflow side is small:
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v6.1.0
with:
role-to-assume: arn:aws:iam::123456789012:role/github-terraform-prod-apply
aws-region: eu-west-1
role-session-name: terraform-${{ github.run_id }}
The IAM trust policy is where the security boundary lives. Keep it specific:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:example-org/platform:ref:refs/heads/main"
}
}
}
]
}
For production applies, consider tying the subject to a GitHub environment instead of only a branch when your deployment process uses environment protection rules. The main failure mode is an overly broad trust policy such as repo:example-org/*:* paired with a powerful IAM policy. That combination turns every compromised workflow in the organization into a possible cloud deployment identity.
Use separate roles for planning and applying:
github-terraform-plan
-> read state
-> read cloud resources needed by data sources
-> no broad create/update/delete
github-terraform-prod-apply
-> mutate only the services managed by the prod root module
-> pass only approved task roles and execution roles
-> assume only from protected main/environment workflows
Least privilege for Terraform is hard because providers often need wide read access and resource-specific write access. Still, splitting plan and apply identities lowers blast radius and makes CloudTrail easier to reason about.
Remote State, Locks, and Concurrency
Terraform automation is not production-ready until state is remote and locked. Local state in a GitHub Actions runner disappears after the job and can leak into artifacts if mishandled. Remote state gives every run a shared source of truth; locking prevents two runs from writing at once.
The backend is environment-specific:
# infra/live/prod/backend.hcl
bucket = "example-platform-terraform-state"
key = "prod/ecs-service/terraform.tfstate"
region = "eu-west-1"
encrypt = true
use_lockfile = true
Then align GitHub Actions concurrency with the same deployment unit:
concurrency:
group: terraform-prod-ecs-service
cancel-in-progress: false
Backend locking protects Terraform state. GitHub concurrency protects the broader deployment workflow. You want both because not every conflict is a state write. Two production applies can also race on image deployment, service stabilization, smoke tests, or post-deploy hooks.
For pull request workflows, cancel-in-progress: true is useful because a newer commit makes the previous plan stale. For production applies, cancellation is a more delicate choice. Many teams prefer not to cancel an apply once it starts; instead, queue or manually control production deploys so the state lock is not abandoned mid-change.
Protected Environments for Apply
GitHub environments add a human and policy boundary around deployment jobs. A job that references an environment must pass that environment's protection rules before it can run and before it can access environment secrets. Even when you use OIDC instead of secrets, the approval gate is still useful because the apply job is the state mutation point.
name: terraform-apply
on:
push:
branches:
- main
paths:
- "infra/**"
workflow_dispatch:
inputs:
environment:
description: "Terraform environment"
required: true
default: "prod"
permissions:
contents: read
id-token: write
concurrency:
group: terraform-prod-ecs-service
cancel-in-progress: false
jobs:
apply:
runs-on: ubuntu-latest
environment: production
defaults:
run:
shell: bash
working-directory: infra/live/prod
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v4
with:
terraform_version: "1.14.6"
- uses: aws-actions/configure-aws-credentials@v6.1.0
with:
role-to-assume: arn:aws:iam::123456789012:role/github-terraform-prod-apply
aws-region: eu-west-1
role-session-name: terraform-prod-${{ github.run_id }}
- name: Terraform init
run: terraform init -backend-config=backend.hcl -input=false
- name: Terraform plan
run: terraform plan -input=false -out=tfplan
- name: Terraform apply
run: terraform apply -input=false tfplan
The environment should require reviewers for production and restrict deployment branches or tags. If the repository is public, plan for the availability differences of environment features across GitHub plans. If the repository is private or internal, confirm that the organization's plan supports the rules you want to enforce.
Provider Plugin Caching Without Poisoning the Pipeline
Terraform initializes providers and modules during init. Caching can reduce repeated downloads, but provider caches should be keyed by the lock file and operating system. Do not share a broad mutable cache across unrelated roots.
- name: Cache Terraform providers
uses: actions/cache@v4
with:
path: ~/.terraform.d/plugin-cache
key: terraform-${{ runner.os }}-${{ hashFiles('infra/**/.terraform.lock.hcl') }}
restore-keys: |
terraform-${{ runner.os }}-
- name: Configure Terraform plugin cache
run: |
mkdir -p ~/.terraform.d/plugin-cache
cat > ~/.terraformrc <<'EOF'
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
EOF
Caching is an optimization, not a correctness requirement. The .terraform.lock.hcl file is the correctness mechanism because it records provider selections. If a cache produces surprising provider behavior, remove the cache first and prove the workflow works without it.
Why We Built the Pipeline This Way
The design deliberately separates four concerns that are often collapsed into one YAML file.
First, reviewability. The plan is the artifact reviewers care about. The pipeline should turn it into a readable summary and keep the full plan available for inspection. Reviewers should not approve production changes by reading only a green checkmark.
Second, identity. A GitHub workflow is not a person, and a pull request workflow should not receive the same cloud privileges as a production deployment workflow. OIDC lets IAM decide which repository, branch, and environment can assume which role.
Third, coordination. Terraform state is a shared mutable record. Backend locking and GitHub concurrency are not optional polish; they are the controls that prevent conflicting applies.
Fourth, operational recovery. A good pipeline leaves evidence: plan JSON, apply logs, CloudTrail role sessions, deployment status, and service events. When a deploy fails, the team should know whether the failure came from Terraform graph execution, IAM permissions, provider behavior, or application startup.
This is why "just automate apply" is the wrong target. The target is a controlled change-management loop that still moves fast.
Failure Modes to Design For
Stale Plans
A plan generated before another infrastructure change may no longer represent reality. Use state locking, short artifact retention, and either apply the exact saved plan quickly or regenerate the plan before production approval.
Broad OIDC Trust Policies
The AWS trust policy must not allow arbitrary repositories or branches to assume production roles. Restrict token.actions.githubusercontent.com:sub and aud, and keep IAM permissions scoped to the resources managed by the workflow.
Secret Leakage in Plan Output
Terraform marks many sensitive values, but providers and custom outputs can still surprise you. Avoid posting raw full plans into PR comments. Prefer summarized output and keep full artifacts access-controlled.
Destructive Changes Hidden in Noise
Large plans can hide a single dangerous replacement. Parse JSON plans and highlight deletes, replacements, public exposure, IAM privilege changes, and database mutations.
Apply Cancellation
Canceling stale PR plans is usually fine. Canceling production applies can leave partial infrastructure changes that must be reconciled. Use environment approvals and deployment queues for production instead of aggressive cancellation.
A Minimal Policy Gate
You do not need a full policy platform to catch the first class of mistakes. A small JSON check can block deletes until the team adopts something more formal:
#!/usr/bin/env bash
set -euo pipefail
plan_json="${1:-tfplan.json}"
deletes="$(jq -r '
[
.resource_changes[]?
| select(.mode == "managed")
| select(.change.actions | index("delete"))
| .address
]
| .[]
' "$plan_json")"
if [[ -n "$deletes" ]]; then
echo "Refusing to continue. Terraform plan includes managed resource deletes:"
echo "$deletes"
exit 1
fi
echo "No managed resource deletes found."
That script is intentionally conservative. In real environments, you will want exception files, owner annotations, or policy-as-code. But even this small guard prevents a common failure mode: reviewers miss a destructive action in a long plan.
Observability and Audit Trail
Treat every apply as a deployment event. Useful telemetry includes:
- GitHub run ID and commit SHA.
- Terraform root module and workspace or backend key.
- Assumed AWS role ARN and session name.
- Plan summary counts.
- Resource addresses changed.
Add a deployment summary at the end of the workflow:
{
echo "## Terraform apply summary"
echo
echo "- Commit: ${GITHUB_SHA}"
echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Root: infra/live/prod"
echo "- Backend key: prod/ecs-service/terraform.tfstate"
echo "- AWS role session: terraform-prod-${GITHUB_RUN_ID}"
} >> "$GITHUB_STEP_SUMMARY"
This is not just cosmetics. When production changes create unexpected behavior, incident responders need to connect symptoms to the exact infrastructure run that changed the environment.
Practical Rollout Plan
Do not jump straight from manual Terraform to fully automated production applies. Move in layers:
- Add
fmt,init,validate, and read-onlyplanon pull requests. - Add plan JSON summaries and destructive-change detection.
- Move cloud credentials from static GitHub secrets to OIDC roles.
- Configure remote state locking for every root module.
- Add protected environments and concurrency groups.
- Enable apply for a non-production environment.
- Promote the same pattern to production with required reviewers.
- Split application deployment, such as ECS task definition updates, into a dedicated job or workflow.
This sequence lets the team improve review quality before it grants automation the ability to mutate production.
Frequently Asked Questions
Q: How do I build a Terraform CI/CD pipeline with GitHub Actions?
A: Start with a pull request workflow that runs terraform fmt -check, terraform init, terraform validate, and terraform plan. Publish a plan summary for review, then run terraform apply only from a protected environment after merge or explicit approval. Use remote state, state locking, OIDC credentials, and workflow concurrency for each deployment unit.
Q: Should Terraform apply run on every pull request?
A: No. Pull request workflows should normally be read-only because they run before code is trusted and reviewed. Use PR workflows to produce plans and policy signals, then reserve apply for protected branches, release workflows, or manual dispatch events with environment approval.
Q: Why use GitHub OIDC for Terraform on AWS?
A: OIDC removes the need to store long-lived AWS access keys as GitHub secrets. The workflow requests a short-lived token, AWS validates it against an IAM OIDC provider and trust policy, and the job receives temporary role credentials scoped to the workflow's repository, branch, or environment.
Q: How do I prevent concurrent Terraform applies?
A: Use a remote backend that supports state locking and configure a GitHub Actions concurrency group per environment or Terraform root module. Backend locking protects the state file, while GitHub concurrency prevents overlapping deploy workflows from racing on approvals, smoke tests and post-deploy steps.
Related Internal Guides
- Terraform Security Best Practices for AWS IAM
- Drift Detection and Remediation
- Using Terraform GitOps Bridge Modules
Resources
- Original source post: Terraform CI/CD Pipelines with GitHub Actions
- HashiCorp: Automate Terraform with GitHub Actions
- HashiCorp: setup-terraform GitHub Action
- HashiCorp: Initialize the Terraform working directory
- HashiCorp: Create a Terraform plan
- GitHub Docs: OpenID Connect
- GitHub Docs: Workflow syntax for GitHub Actions
- GitHub Docs: Managing environments for deployment
- AWS IAM: Create a role for OpenID Connect federation
Comments
Post a Comment