Terraform in Production: State, Modules, CI/CD and Drift

Running Terraform in production means small, isolated root modules with remote, encrypted, locked state; versioned modules with validated inputs; and every change going through a pull request that shows the plan, passes policy checks and is applied by CI with short-lived credentials. Nobody applies from a laptop, nobody fixes things in the console, and drift is detected on a schedule. The rest is discipline about blast radius: keep what a single apply can destroy small enough that a bad plan is an inconvenience, not an incident.

Key takeaways

• One root module and one state file per component per environment. Size them so you would be comfortable if a single apply went wrong.
• Use the S3 backend with use_lockfile = true, KMS encryption and bucket versioning. DynamoDB locking is deprecated.
• Pin Terraform, providers (commit .terraform.lock.hcl) and modules. Upgrade them deliberately, one at a time.
• Refactor with moved, import and removed blocks so state changes are reviewed in a pull request, not typed into a terminal.
• Plan on the pull request, apply on merge, with OIDC roles split between plan and apply, a guard against unexpected deletes, and policy checks on the plan JSON.
• Schedule terraform plan -detailed-exitcode for drift, and treat the state bucket as a secrets store.

Terraform or OpenTofu

Terraform moved to the Business Source License in August 2023, and HashiCorp is now part of IBM. OpenTofu is the MPL-licensed fork, governed under the Linux Foundation. At the language level they remain largely compatible, and everything in this guide works on both. Their newer features are diverging: Terraform has added Stacks, list resources with terraform query and provider-defined actions, while OpenTofu has client-side state encryption and its own additions. Pick one per organization. Running both doubles the tooling and the upgrade work for no benefit. As of early October 2026 the latest releases are Terraform 1.16 and OpenTofu 1.13.

Layout and blast radius

The most expensive Terraform mistake is a single state file that owns everything. One bad apply can then touch networking, databases and IAM at once, plans get slow, and every engineer contends for the same lock.

infrastructure/
  modules/
    vpc/
    rds-postgres/
    ecs-service/
  envs/
    prod/
      networking/       main.tf, backend.tf, terraform.tfvars
      database/
      services/
      iam/
    staging/
      networking/
      database/
      services/
  global/
    dns/
    org-policies/

Each directory under envs/ is a root module with its own state, its own pipeline and, ideally, its own IAM role. Split by rate of change and by consequence:

Pass values between layers through outputs and data sources, or through a parameter store, rather than hard-coding IDs. Reading another layer's state with terraform_remote_state works but couples you to its internal layout and requires read access to its whole state.

State: encrypted, versioned, locked

terraform {
  required_version = "~> 1.16"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
 
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "networking/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:us-east-1:111122223333:alias/terraform-state"
    use_lockfile = true
  }
}
 
provider "aws" {
  region = "us-east-1"
 
  default_tags {
    tags = {
      environment = "production"
      component   = "networking"
      managed_by  = "terraform"
      repository  = "acme/infrastructure"
    }
  }
}

use_lockfile = true uses S3 conditional writes to create a lock object next to the state. It became generally available in Terraform 1.11, and the DynamoDB arguments are deprecated; during a migration you can enable both, then delete the table. default_tags stamps every resource the provider creates, which is how cost allocation and "who owns this" questions get answered later.

The state bucket deserves more care than most production data:

• Versioning on. A corrupted or accidentally overwritten state file is recovered by restoring the previous object version.
• Public access blocked and TLS required through bucket settings and a bucket policy.
• Access limited to the Terraform pipeline roles and a small break-glass group. State contains every attribute of every resource, including values Terraform marks as sensitive. sensitive = true hides a value from plan output; it does not keep it out of state.
• Bootstrapped separately. The bucket and KMS key that hold state are created once, by a tiny root module with local state that you keep in a safe place, or by your account vending process.

Commit .terraform.lock.hcl. It records the exact provider versions and checksums, so CI and every engineer use the same provider builds. Generate hashes for every platform your team uses:

terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_arm64

Modules with a small, validated interface

A good module has few required inputs, safe defaults, validation that fails before any API call, and outputs for everything a caller might need.

variable "name" {
  type        = string
  description = "Short environment name used in resource names."
 
  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{1,20}$", var.name))
    error_message = "name must be lowercase letters, digits and hyphens, 2 to 21 characters."
  }
}
 
variable "cidr_block" {
  type        = string
  description = "VPC CIDR, between /16 and /20, from the network allocation plan."
 
  validation {
    condition = (
      can(cidrnetmask(var.cidr_block)) &&
      tonumber(split("/", var.cidr_block)[1]) >= 16 &&
      tonumber(split("/", var.cidr_block)[1]) <= 20
    )
    error_message = "cidr_block must be a valid CIDR between /16 and /20."
  }
}
 
variable "az_count" {
  type    = number
  default = 3
}
 
output "vpc_id" {
  value = aws_vpc.this.id
}
 
output "private_subnet_ids" {
  value = aws_subnet.private[*].id
}

Call modules by version, never from a moving branch:

module "vpc" {
  source = "git::https://github.com/acme/terraform-modules.git//vpc?ref=v2.4.1"
 
  name       = "prod"
  cidr_block = "10.40.0.0/16"
  az_count   = 3
}

For third-party modules, pin a registry version or a commit SHA you have reviewed. A module runs with your credentials; treat it like any other dependency. Extract a module when you have written the same thing two or three times, not before. A generic module designed on day one usually has thirty inputs and no users.

Test modules before they reach an environment

terraform test runs .tftest.hcl files against a module. With a mock provider, plan-level tests need no cloud credentials and run in seconds:

mock_provider "aws" {}
 
variables {
  name       = "test"
  cidr_block = "10.0.0.0/16"
  az_count   = 3
}
 
run "one_private_subnet_per_az" {
  command = plan
 
  assert {
    condition     = length(aws_subnet.private) == 3
    error_message = "Expected one private subnet per availability zone."
  }
}
 
run "rejects_tiny_cidr" {
  command = plan
 
  variables {
    cidr_block = "10.0.0.0/28"
  }
 
  expect_failures = [var.cidr_block]
}

Add terraform fmt -check, terraform validate and tflint with the provider's ruleset, and you catch most mistakes before a plan ever runs against a real account.

Refactor without touching state by hand

Renaming a resource, switching from count to for_each, adopting an existing resource or handing one to another state used to mean terraform state mv and terraform import typed into a terminal. Do it in code instead, so the change is reviewed and repeatable across environments:

moved {
  from = aws_instance.web[0]
  to   = aws_instance.web["api"]
}
 
import {
  to = aws_s3_bucket.logs
  id = "acme-prod-logs"
}
 
removed {
  from = aws_cloudwatch_log_group.legacy
 
  lifecycle {
    destroy = false
  }
}

moved tells Terraform the object at the old address is the one at the new address. import adopts an existing resource during the next apply; terraform plan -generate-config-out=generated.tf drafts its configuration for you. removed with destroy = false drops a resource from state without deleting it, which is how you hand it to another root module.

Secrets: keep them out of variables and, where possible, out of state

Never put secrets in .tfvars files or variable defaults. Prefer designs where Terraform never sees the secret at all:

resource "aws_db_instance" "orders" {
  identifier                  = "orders-prod"
  engine                      = "postgres"
  instance_class              = "db.r7g.large"
  allocated_storage           = 100
  storage_encrypted           = true
  username                    = "orders_admin"
  manage_master_user_password = true
  deletion_protection         = true
  backup_retention_period     = 14
  final_snapshot_identifier   = "orders-prod-final"
 
  lifecycle {
    prevent_destroy = true
  }
}

manage_master_user_password has RDS generate the password and store it in Secrets Manager, so it never passes through Terraform. Where you must pass a secret, use ephemeral resources and write-only arguments (Terraform 1.10 and 1.11), which keep values out of plan and state for providers and resources that support them. deletion_protection and prevent_destroy are two independent locks on the same door, one in the cloud API and one in Terraform. Use both on anything that holds data.

CI/CD: plan on the pull request, apply on merge

The workflow below plans every pull request that touches production networking, posts the plan as a comment, and applies on merge after an environment approval. It uses two OIDC roles: a plan role that can read everything but change only the state lock, and an apply role that can change infrastructure and is only available to the protected production-infra environment. Our GitHub Actions guide explains the pinning and OIDC setup in detail, and the cloud IAM guide covers scoping the roles.

name: terraform-prod-networking
 
on:
  pull_request:
    paths: ["envs/prod/networking/**", "modules/**"]
  push:
    branches: [main]
    paths: ["envs/prod/networking/**", "modules/**"]
 
permissions: {}
 
concurrency:
  group: terraform-prod-networking
  cancel-in-progress: false
 
defaults:
  run:
    working-directory: envs/prod/networking
 
jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    permissions:
      contents: read
      id-token: write
      pull-requests: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: 1.16.5
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: arn:aws:iam::111122223333:role/terraform-plan-prod
          aws-region: us-east-1
      - run: terraform fmt -check -recursive
      - run: terraform init -input=false -lockfile=readonly
      - run: terraform validate
      - run: terraform plan -input=false -lock-timeout=5m -out=tfplan
      - name: Install conftest
        working-directory: ${{ runner.temp }}
        run: |
          curl -sSfL -o conftest.tar.gz https://github.com/open-policy-agent/conftest/releases/download/v0.71.0/conftest_0.71.0_Linux_x86_64.tar.gz
          echo "765dfefdf0730693d7541ea0147e10d530a54ed257bbfec055ff34c837ffdf72  conftest.tar.gz" | sha256sum -c -
          tar -xzf conftest.tar.gz conftest && sudo mv conftest /usr/local/bin/
      - run: terraform show -json tfplan > plan.json && conftest test plan.json --policy ../../../policy
      - name: Comment plan
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
        run: |
          terraform show -no-color tfplan > plan.txt
          { echo "Plan for envs/prod/networking"; echo; echo "~~~text"; head -c 60000 plan.txt; echo "~~~"; } > comment.md
          gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body-file comment.md
 
  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-24.04
    timeout-minutes: 60
    environment: production-infra
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: 1.16.5
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: arn:aws:iam::111122223333:role/terraform-apply-prod
          aws-region: us-east-1
          role-duration-seconds: 3600
      - run: terraform init -input=false -lockfile=readonly
      - run: terraform plan -input=false -lock-timeout=5m -out=tfplan
      - name: Refuse unexpected deletes
        env:
          ALLOW_DESTROY: ${{ vars.TF_ALLOW_DESTROY }}
        run: |
          deletes=$(terraform show -json tfplan | jq '[.resource_changes[] | select(.change.actions | index("delete"))] | length')
          if [ "$deletes" -gt 0 ] && [ "$ALLOW_DESTROY" != "true" ]; then
            echo "Plan deletes or replaces $deletes resources. Review and set TF_ALLOW_DESTROY to proceed."
            exit 1
          fi
      - run: terraform apply -input=false -lock-timeout=5m tfplan

Three trade-offs to make consciously:

• Re-plan on merge versus apply the reviewed plan. This workflow re-plans on main, because the plan from the pull request can be stale by the time it merges. The delete guard stops the most dangerous kind of surprise. Tools such as Atlantis instead apply the exact plan from the pull request before merging. HCP Terraform, Spacelift and env0 offer managed versions of either model.
• A plan is not read-only. terraform plan runs provider code and data sources from the branch being planned, including external data sources that execute commands. Anyone who can open a pull request that triggers a plan can run code with the plan role's credentials. Keep that role read-only, keep fork pull requests away from it, and require review of workflow changes.
• Credentials can expire mid-apply. Some resources take a long time to create or modify. Set the role session duration (and the role's maximum) longer than your slowest apply, or the apply fails halfway and leaves state locked.

Policy as code on the plan

Policies that run against terraform show -json output catch problems in what will actually change, regardless of how the module was written. A conftest (Open Policy Agent) rule that blocks SSH open to the internet:

package main
 
import rego.v1
 
deny contains msg if {
  some rc in input.resource_changes
  rc.type == "aws_vpc_security_group_ingress_rule"
  rc.change.after.cidr_ipv4 == "0.0.0.0/0"
  rc.change.after.from_port <= 22
  rc.change.after.to_port >= 22
  msg := sprintf("%s opens SSH to the internet", [rc.address])
}

Checkov and Trivy's configuration scanning cover hundreds of common misconfigurations out of the box; conftest is where your organization's own rules live: required tags, approved regions, encryption everywhere. Our cloud security posture guide covers detecting the same problems in what is already deployed.

Detect drift on a schedule

Run a plan for every root module nightly. -detailed-exitcode turns the result into something a pipeline can act on:

set +e
terraform plan -input=false -lock=false -detailed-exitcode -out=drift.tfplan > drift.log 2>&1
code=$?
set -e
 
case "$code" in
  0) echo "No drift." ;;
  2) echo "Drift detected"; terraform show -no-color drift.tfplan | head -200; exit 2 ;;
  *) echo "Plan failed"; cat drift.log; exit 1 ;;
esac

Route exit code 2 to the owning team as an issue or an alert. Then decide case by case: if the manual change was right, codify it; if not, the next apply reverts it. Drift that nobody looks at becomes the reason a routine apply does something surprising months later.

What goes wrong in production

count index shifts. Removing the first element of a list used with count shifts every index, and Terraform plans to destroy and recreate everything after it. Use for_each with stable keys for any collection that can change, and moved blocks to migrate.

A harmless-looking change forces replacement. Some attributes cannot be updated in place, so changing them means destroy and create: a database identifier, a subnet's CIDR, a launch template property. The plan says "must be replaced". The delete guard and prevent_destroy exist for exactly this. Read every replacement line in a plan.

Provider upgrades change behavior. Major provider releases rename arguments, change defaults and alter how resources are read. Upgrade providers in a dedicated pull request, read the upgrade guide, and plan every environment before applying anywhere.

A stuck lock. A canceled job or expired credentials can leave the lock in place. Confirm no apply is running, then terraform force-unlock <LOCK_ID>. Never make force-unlock a step in the pipeline.

Console hotfixes. Someone fixes an incident by hand, and the next unrelated apply silently reverts it, re-creating the incident. The drift job surfaces it the next morning; the runbook should say to codify incident fixes before closing the incident.

-target becomes a habit. Targeted applies leave the rest of the configuration unapplied and the state out of step with the code. Reserve them for recovery, and follow every targeted apply with a full plan.

The state file grows until plans crawl. Thousands of resources in one state mean slow refreshes, API rate limits and a giant blast radius. Split along the component boundaries above. Many teams then manage what runs inside Kubernetes with a GitOps controller instead; our GitOps guide covers where that boundary usually sits.

Frequently Asked Questions

How should I structure Terraform for multiple environments?

Use a separate root module, with its own state, for each component in each environment, for example envs/prod/networking and envs/prod/database. Shared logic lives in versioned modules that every environment calls with different inputs. This keeps the blast radius of any single apply small and makes it obvious which environment a change touches.

Should I use Terraform workspaces for environments?

Usually not for production versus staging. Workspaces share one configuration and backend, so the only thing separating production from staging is which workspace is selected, which is easy to get wrong. Separate directories with separate state, credentials and pipelines give you real isolation; workspaces fit short-lived copies such as per-branch test environments.

How do you store Terraform state securely?

Use a remote backend with encryption at rest, versioning and locking, such as an S3 bucket with a KMS key, versioning and use_lockfile enabled. Restrict access to the people and pipelines that run Terraform, because state contains resource attributes and often secrets in plain text. Never commit state files to Git.

Do I still need DynamoDB for Terraform state locking?

No. Since Terraform 1.11, the S3 backend supports native locking with use_lockfile = true, which writes a lock object next to the state file, and the DynamoDB arguments are deprecated. You can enable both during a migration, then remove the DynamoDB table.

Should I use Terraform or OpenTofu?

Both work for production. Terraform is developed by HashiCorp, now part of IBM, under the Business Source License, and integrates with HCP Terraform. OpenTofu is the open-source MPL-licensed fork with features such as client-side state encryption. Pick one per organization based on licensing and tooling, because their newer features are diverging.

How do you detect drift in Terraform?

Run terraform plan with the -detailed-exitcode flag on a schedule for every root module. Exit code 0 means no changes, 2 means the real infrastructure differs from the code, and 1 means an error. Alert on 2, then either codify the manual change or let the next apply revert it.

Need help with Terraform?

We build and run infrastructure as code for teams that need changes to be reviewable, repeatable and reversible: layout, state, modules, pipelines and policy. If we built it, we are on the pager for it, and you start with a call with the engineer who would do the work.

• Cloud Architecture and Infrastructure
• Startup Tech Consulting
• Talk to an engineer