Terraform State Management, Locking, and Backups: A Production Deep Dive

Terraform state is an operational database, not a disposable artifact. This deep dive covers S3 backends, S3 and DynamoDB locking, encryption, IAM, backups, migrations, state surgery, CI concurrency, and incident runbooks.

TL;DR

Terraform state management should be treated like production data management: isolate state by blast radius, store it in a remote backend, enable locking, encrypt it, version it, and rehearse restore procedures before an outage. For AWS teams, the modern S3 backend can use native S3 lock files, while older estates may still need DynamoDB locking during migration. The strongest designs combine least-privilege IAM, S3 Versioning, KMS controls, CI concurrency, state migration discipline, and documented runbooks for stuck locks, accidental overwrites, and state surgery.

Terraform state platform with CI runners, S3 backend, lock file, DynamoDB migration path, KMS encryption, versioned backups, and restore workflow

Terraform State Is a Database, Not a File

Use a remote backend, lock state, and back it up. The production version is more demanding. Terraform state is not just a cache and it is not just a JSON file. It is the database Terraform uses to map configuration addresses such as module.vpc.aws_subnet.private["az1"] to real cloud objects. If that mapping is wrong, Terraform can plan to recreate, delete, or forget infrastructure that is otherwise healthy.

That changes the operating model. You do not "store a state file"; you operate a small but critical state platform. The platform needs:

Durable storage with version history.
Exclusive writes during plan, apply, state mv, state rm, and imports.
Encryption at rest and in transit.
IAM boundaries around each environment and root module.
A restore process that is tested before the first incident.
CI/CD concurrency that matches the backend lock instead of fighting it.

HashiCorp's S3 backend documentation now describes S3-native locking through use_lockfile, recommends bucket versioning for recovery, and marks DynamoDB-based locking as deprecated but still configurable for migration from older Terraform versions. That one detail is important because many estates still have mixed Terraform versions across laptops, CI runners, Terragrunt wrappers, and golden build images.

Design rule: choose the state backend first, then design the Terraform repository, IAM roles, CI jobs, and incident runbooks around that backend's failure modes.

The State Architecture to Aim For

A production AWS state backend usually has four layers:

S3 bucket: stores *.tfstate objects under environment-specific prefixes.
Locking mechanism: S3 lock files for current Terraform S3 backend usage, or DynamoDB during legacy migration windows.
KMS and IAM boundary: restricts who can read, write, decrypt, and mutate state prefixes.
Operational controls: versioning, lifecycle, CloudTrail/S3 data event visibility where required, CI concurrency, and restore runbooks.

The backend should normally live outside the infrastructure it controls. For example, keep the state bucket in a tooling, security, or platform account, then let environment-specific Terraform roles access only their own prefixes. Do not create the production state bucket in the same Terraform root module that depends on it. Bootstrap it once with a separate root module or an account vending workflow, then treat changes to the backend itself as privileged platform work.

# bootstrap/backend-state/main.tf
resource "aws_s3_bucket" "terraform_state" {
  bucket = "example-platform-terraform-state"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.terraform_state.arn
      sse_algorithm     = "aws:kms"
    }

    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket                  = aws_s3_bucket.terraform_state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_kms_key" "terraform_state" {
  description             = "KMS key for Terraform state objects"
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

This is not a complete account baseline, but it encodes the state-specific defaults that matter: versioning for rollback, SSE-KMS for key-level access control, and public-access blocking because state should never be web-addressable. S3 enables SSE-S3 by default for new objects, but state files often contain sensitive values and deserve explicit KMS policy review. HashiCorp also notes that the S3 backend can encrypt state at rest when encryption is enabled, while AWS documents that KMS-protected reads and writes need kms:Decrypt and kms:GenerateDataKey permissions.

S3 Backend Configuration with Current Locking

For a modern S3 backend, enable use_lockfile. Terraform stores state at the configured bucket and key, and uses a sibling lock object when the backend lock is active.

terraform {
  required_version = ">= 1.10.0"

  backend "s3" {
    bucket       = "example-platform-terraform-state"
    key          = "prod/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:us-east-1:111122223333:key/00000000-1111-2222-3333-444444444444"
    use_lockfile = true
  }
}

The version constraint is not decorative. If one engineer uses a Terraform binary that understands use_lockfile and another uses an older runner that does not, the team can end up with inconsistent backend behavior. Pin the CLI version in .terraform-version, CI setup, container images, and pre-commit checks before migrating shared state.

During migration, you can configure both S3 and DynamoDB locking arguments because HashiCorp supports that path for older Terraform versions that only understand DynamoDB locking. Keep the dual-lock period short and document the cutover:

terraform {
  required_version = ">= 1.10.0"

  backend "s3" {
    bucket         = "example-platform-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    use_lockfile   = true

    # Deprecated. Retain temporarily only while older runners are being removed.
    dynamodb_table = "terraform-state-locks"
  }
}

The migration caveat is simple:

All runners are current and pinned: use S3 lock files.
Some runners are older: keep DynamoDB locking until they are retired.
Mixed fleet during cutover: configure both only long enough to prove that no old runner remains.
New estate: avoid new DynamoDB lock tables unless an explicit compatibility requirement forces them.

If you still need DynamoDB, the table must use a partition key named LockID with type String, matching the S3 backend requirement.

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Do not confuse Terraform's lock ID with an application-level deployment ID. The lock ID is a nonce used by Terraform to ensure the unlock operation targets the right lock. If a run fails to unlock, Terraform prints the lock ID; terraform force-unlock needs that exact value.

IAM: Least Privilege for State Is Prefix-Specific

The worst state IAM policy is s3:* on the whole state bucket. It lets any Terraform job read every environment, overwrite unrelated roots, and often delete state accidentally. The practical pattern is one role per trust boundary and one prefix per root module or environment.

For S3 lock files, the S3 backend documentation calls out an important permission split: Terraform needs s3:GetObject and s3:PutObject on the state object, and if use_lockfile is set it also needs s3:GetObject, s3:PutObject, and s3:DeleteObject on the .tflock object. It does not need s3:DeleteObject on the state file itself.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisStatePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-platform-terraform-state",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "prod/network/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadWriteOnlyThisStateFile",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::example-platform-terraform-state/prod/network/terraform.tfstate"
    },
    {
      "Sid": "ManageOnlyThisLockFile",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::example-platform-terraform-state/prod/network/terraform.tfstate.tflock"
    },
    {
      "Sid": "UseStateKmsKey",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/00000000-1111-2222-3333-444444444444"
    }
  ]
}

For read-only planning roles, be careful. A speculative plan still needs to read state and often refresh real infrastructure. If the plan must not write state, run with terraform plan -refresh=false only when you understand the tradeoff, or use a role that can read state but cannot apply cloud mutations. Some providers and workflows still need temporary local files, plugin caches, and data source access. Security is not just state IAM; it is the full set of cloud permissions the plan can exercise.

State Contains Secrets Even When Outputs Are Sensitive

Terraform's sensitive = true is a display control, not a promise that a value never reaches state. Provider schemas often store resource attributes needed for diffing and replacement decisions. Passwords, private endpoints, generated tokens, certificate material, connection strings, and provider-returned metadata can all appear in state depending on the resource.

That has three consequences:

Do not commit local state. Treat terraform.tfstate, terraform.tfstate.backup, *.tfplan, and terraform.tfstate.d/ as secrets.
Do not over-share remote state outputs. terraform_remote_state can couple systems and expose more than intended.
Do not use state as a secret store. Prefer references to AWS Secrets Manager, SSM Parameter Store, Vault, or generated credentials that the provider can manage without exposing unnecessary values.

Repository guardrails should be explicit:

# Terraform state and local execution artifacts
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log

# Backend config can include sensitive values if teams use partial config files
backend.override.hcl
*.backend.hcl

For state access reviews, ask the same question you would ask for a production database: who can read it, who can write it, who can decrypt it, who can restore old versions, and who can delete historical versions?

Versioning, Backups, and Restore Are Different Things

S3 Versioning is the baseline. AWS documents that versioning preserves, retrieves, and restores multiple variants of an object, and that versioning-enabled buckets can recover from accidental deletion or overwrite. For Terraform state, that gives you a version history every time Terraform writes the state object.

Versioning alone is not a complete backup strategy:

It does not prove that anyone knows which version is good.
It does not create cross-account or cross-region isolation by itself.
It can retain bad versions as faithfully as good ones.
It can become expensive if lifecycle rules are never reviewed.
It does not replace an incident runbook.

At minimum, define retention for noncurrent versions and decide whether state needs replication to a separate security account. For highly regulated environments, consider S3 Object Lock or protected backup copies, but test Terraform compatibility and operational friction before enabling irreversible controls on an active state bucket.

A state backup runbook should be boring:

# 1. Capture the current backend state before risky work.
terraform state pull > "state-backups/prod-network-$(date -u +%Y%m%dT%H%M%SZ).tfstate"

# 2. Record the current remote S3 object version.
aws s3api list-object-versions \
  --bucket example-platform-terraform-state \
  --prefix prod/network/terraform.tfstate \
  --query 'Versions[?IsLatest==`true`].[VersionId,LastModified,Size]' \
  --output table

# 3. Inspect the local snapshot without editing it by hand.
terraform show -json "state-backups/prod-network-20260613T230453Z.tfstate" \
  | jq '.values.root_module.resources[] | {address, type, name}'

Restore should be slower than backup. Pause CI applies. Tell the team which state path is frozen. Identify the exact S3 version to restore. Copy it to a quarantine file first:

aws s3api get-object \
  --bucket example-platform-terraform-state \
  --key prod/network/terraform.tfstate \
  --version-id "3HL4kqtJlcpXrof3Hf..." \
  restored-prod-network.tfstate

terraform show restored-prod-network.tfstate

Only after review should you replace remote state. In many cases, the safer path is to use terraform state push from a controlled workstation or a locked break-glass CI job. Follow it with a plan:

terraform state push restored-prod-network.tfstate
terraform plan -lock-timeout=10m -out=restore-check.tfplan
terraform show restore-check.tfplan

If the plan wants to recreate critical resources, stop. The restore may be wrong, the real infrastructure may have drifted, or provider versions may be interpreting state differently.

State Migration Without Drama

Changing backend configuration is not the same as moving resources. Terraform's init command has two flags that matter:

terraform init -migrate-state attempts to copy existing state to the new backend.
terraform init -reconfigure discards the previous backend configuration for the working directory and does not migrate state.

Use migration when adopting a backend or moving state between backend locations. Use reconfiguration when the state is already where the new configuration says it is, or when you intentionally want to reconnect without copying.

Example migration from local state to S3:

terraform {
  backend "s3" {
    bucket       = "example-platform-terraform-state"
    key          = "prod/app/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

# Before changing backend settings, take a local backup.
cp terraform.tfstate "terraform.tfstate.pre-s3.$(date -u +%Y%m%dT%H%M%SZ)"

# Initialize and copy state into the configured backend.
terraform init -migrate-state

# Validate that the remote backend is authoritative.
terraform state list
terraform plan -lock-timeout=10m

For non-interactive CI migrations, avoid hidden prompts. Run migration from a controlled terminal first, or use a dedicated migration job with peer approval. Do not bury terraform init -migrate-state -force-copy in a generic pipeline template where every branch can trigger it.

Workspaces vs Directories: Choose Blast Radius Over Convenience

Terraform CLI workspaces are separate state instances within the same working directory. HashiCorp explicitly positions them as useful for specific cases, such as parallel copies of the same configuration, but recommends alternatives for complex deployments that need separate credentials and access controls.

That distinction matters for state layout:

infra/
  live/
    prod/
      network/
        backend.tf   # key = prod/network/terraform.tfstate
      eks/
        backend.tf   # key = prod/eks/terraform.tfstate
    staging/
      network/
        backend.tf   # key = staging/network/terraform.tfstate
      eks/
        backend.tf   # key = staging/eks/terraform.tfstate
  modules/
    vpc/
    eks/

Use separate directories or root modules when environments differ by account, region, credentials, compliance boundary, or owner. Use workspaces for temporary copies of nearly identical infrastructure when shared backend access is acceptable.

The dangerous workspace pattern is this:

locals {
  environment = terraform.workspace
}

resource "aws_db_instance" "main" {
  identifier = "app-${local.environment}"
  # ...
}

It is convenient until someone runs terraform workspace select prod from a laptop with broader AWS credentials than intended. For production, directories with separate backend configuration and IAM roles make the blast radius visible in code review.

State Surgery: Move, Remove, Import, and Push Carefully

State surgery is any operation where you manipulate the state mapping directly. Sometimes it is the right tool: refactoring modules, renaming resources, importing brownfield infrastructure, removing abandoned objects from management, or repairing provider address changes.

The safe order is always:

Freeze applies for the affected root module.
Pull and archive current state.
Make the configuration change.
Run the state command with locking.
Run a plan and verify no unintended create/destroy action.
Record the command in the pull request or incident ticket.

Renaming a resource without replacement:

terraform state pull > state-backups/before-rename.tfstate

terraform state mv \
  'aws_security_group.app' \
  'module.network.aws_security_group.app'

terraform plan -lock-timeout=10m

Removing a resource from Terraform management without destroying the real object:

terraform state pull > state-backups/before-rm.tfstate

terraform state rm 'aws_iam_role.legacy_breakglass'

terraform plan -lock-timeout=10m

HashiCorp documents that state rm makes Terraform forget the object; the remote object continues to exist. That is exactly why a follow-up plan may try to create a replacement and fail due to name conflicts. Use removed blocks when you want reviewable removal as part of normal plan/apply flow. Use state rm when you are deliberately separating an object from the state database.

Import has its own failure mode. The CLI import command binds an existing remote object to an address in state, but it does not generate configuration. Newer import block workflows can make imports reviewable, but the core invariant remains: each remote object should map to one Terraform resource address.

import {
  to = aws_s3_bucket.logs
  id = "example-prod-access-logs"
}

resource "aws_s3_bucket" "logs" {
  bucket = "example-prod-access-logs"
}

terraform plan -generate-config-out=generated-imports.tf
terraform apply

Never edit remote state JSON by hand unless HashiCorp support or a documented recovery procedure leaves no alternative. State JSON is easy to break and hard to reason about under provider schema changes.

CI Concurrency Complements Backend Locking

Terraform state locking prevents concurrent state writers. CI concurrency prevents waste, noisy failures, and partially overlapping environment operations. You want both.

For GitHub Actions, use a concurrency group per state path or environment. For apply jobs, usually do not cancel an in-progress run; let it finish or fail cleanly so Terraform can release its lock.

name: terraform-prod-network

on:
  push:
    branches: [main]
    paths:
      - "infra/live/prod/network/**"

concurrency:
  group: terraform-prod-network
  cancel-in-progress: false

jobs:
  apply:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    environment: prod
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.10.5

      - name: Init
        working-directory: infra/live/prod/network
        run: terraform init -input=false

      - name: Plan
        working-directory: infra/live/prod/network
        run: terraform plan -input=false -lock-timeout=10m -out=tfplan

      - name: Apply reviewed plan
        working-directory: infra/live/prod/network
        run: terraform apply -input=false -lock-timeout=10m tfplan

The backend lock is still the source of truth. The concurrency group is a queueing policy. If a human runs Terraform from a laptop, the backend lock still protects the state object. If two CI workflows target different roots, separate concurrency groups let them proceed independently.

Avoid -lock=false in CI. It exists for exceptional workflows and is dangerous in shared state. Also avoid blanket cancel-in-progress: true on apply workflows. Cancellation can interrupt Terraform after it acquired the lock, leaving the team with a failed run and a manual unlock decision.

Failure Modes and Runbooks

Stuck Lock

Symptoms:

Error acquiring the state lock
A previous runner was canceled or lost network connectivity.
No current process appears to be applying the same root module.

Runbook:

# 1. Verify no active CI run or human apply is still alive.
# 2. Capture the lock error output, including Lock ID and path.
# 3. Check recent state object and lock object activity.
aws s3api list-object-versions \
  --bucket example-platform-terraform-state \
  --prefix prod/network/terraform.tfstate

# 4. Unlock only your own abandoned lock.
terraform force-unlock LOCK_ID_FROM_ERROR

# 5. Immediately run a read-only plan.
terraform plan -lock-timeout=10m

HashiCorp's state locking documentation is blunt: force unlock can cause multiple writers if someone else still holds the lock. Make the human verification step mandatory.

Accidental State Overwrite

Symptoms:

A plan suddenly wants to create many existing resources.
terraform state list is unexpectedly short.
An S3 object version changed around the time of a failed migration or manual state push.

Runbook:

# Freeze writes first.
aws s3api list-object-versions \
  --bucket example-platform-terraform-state \
  --prefix prod/network/terraform.tfstate \
  --query 'Versions[*].[VersionId,IsLatest,LastModified,Size]' \
  --output table

aws s3api get-object \
  --bucket example-platform-terraform-state \
  --key prod/network/terraform.tfstate \
  --version-id "$CANDIDATE_VERSION" \
  candidate.tfstate

terraform show candidate.tfstate

Restore only after you understand why the overwrite happened. If the root cause is an incorrect backend key, fix that first or the next apply can corrupt the same path again.

Split-Brain Backend Configuration

Symptoms:

Two directories or branches point to the same key.
Production and staging both mutate env:/default/....
One job uses S3 lock files and another still uses only DynamoDB.

Runbook:

rg -n 'backend "s3"|key\\s*=|workspace_key_prefix|dynamodb_table|use_lockfile' infra/
terraform version
terraform init -backend=false

Then pin one backend configuration per root module, remove duplicate keys, and migrate state explicitly. Do not rely on workspace naming conventions to fix a confused backend layout.

Provider Upgrade State Drift

Symptoms:

A provider upgrade changes computed attributes.
A no-code-change plan wants replacement.
State JSON schema or provider source address changes.

Runbook:

terraform init -upgrade
terraform providers
terraform state pull > state-backups/before-provider-upgrade.tfstate
terraform plan -lock-timeout=10m -out=provider-upgrade.tfplan
terraform show provider-upgrade.tfplan

Upgrade providers in small batches. Keep the state backup and lock discipline even when no HCL changed.

A Practical Production Checklist

Use this as the review checklist for each Terraform root module:

Backend key is unique and matches the root module's blast radius.
S3 bucket has Versioning enabled.
State is encrypted with an reviewed KMS key policy.
Terraform version is pinned in CI and developer tooling.
use_lockfile = true is enabled for current S3 backend usage.
DynamoDB locking is either absent or documented as a temporary compatibility bridge.
IAM grants write access only to the exact state and lock paths.
CI has concurrency per state path and does not cancel active applies by default.
terraform init -migrate-state is used only in approved migration workflows.
State surgery commands require backup, peer review, and a follow-up plan.
Restore has been tested from an S3 object version in a non-production root.

The best Terraform state management is intentionally uneventful. Engineers should rarely think about locks, restores, and backend keys during routine work because the platform defaults already handle them. When an incident does happen, the runbook should tell the team exactly which lock to inspect, which version to restore, which CI queue to pause, and which plan output proves recovery is safe.

Frequently Asked Questions

Q: What is the safest AWS backend pattern for Terraform state management? A: Use an S3 backend with bucket versioning, explicit encryption, prefix-scoped IAM, and backend locking. For current Terraform S3 backend usage, prefer use_lockfile = true; retain DynamoDB locking only for older Terraform versions or a documented migration period.

Q: Do I still need DynamoDB for Terraform S3 state locking? A: Not for new current-version S3 backend designs. Terraform's S3 backend supports S3 lock files, and HashiCorp marks DynamoDB locking as deprecated while preserving it for compatibility and migration. Mixed fleets should pin Terraform versions before removing the old table.

Q: How do I back up and restore Terraform state? A: Enable S3 Versioning, take manual terraform state pull snapshots before risky operations, and document restore from a specific S3 object version. A restore should freeze writes, inspect the candidate state, push it from a controlled context, and verify with a no-surprise plan.

Q: Are Terraform workspaces safe for production environments? A: Workspaces can be useful for short-lived copies of the same configuration, but they share a working directory and backend configuration. For production, staging, separate AWS accounts, or separate teams, prefer distinct root modules with separate backend keys and IAM controls.

Q: When should I use Terraform state surgery commands? A: Use state mv for refactors where an existing object should move to a new address, state rm when Terraform should forget an object without destroying it, and import workflows for existing infrastructure. Always back up state first, serialize writes, and run a plan immediately after.

Search This Blog

DevOpsDreams