Questions
Implement a secure Terraform CI/CD pipeline with plan reviews, apply approvals, and state protection.
The Scenario
Your team manually runs terraform apply from laptops. Problems:
- No review process - anyone can apply anything
- No audit trail of who changed what
- State locking conflicts when multiple people work
- Credentials scattered on developer machines
- “Works on my machine” issues with different Terraform versions
The Challenge
Design and implement a CI/CD pipeline that enforces code review, requires approval for production changes, maintains state integrity, and provides full audit trails.
A junior engineer might auto-approve all applies, store AWS credentials as plain environment variables, skip the plan output review, or allow direct pushes to main. This leads to unapproved infrastructure changes, credential exposure, and no meaningful review process.
A senior engineer implements a multi-stage pipeline: automated plan on PR, plan output posted for review, manual approval gates for production, OIDC authentication (no long-lived credentials), state locking, and comprehensive audit logging.
Architecture Overview
PR Created
│
▼
┌─────────────────┐
│ Format Check │ ← terraform fmt -check
│ Validation │ ← terraform validate
│ Lint (tflint) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Security Scan │ ← tfsec, checkov
└────────┬────────┘
│
▼
┌─────────────────┐
│ Terraform Plan │ ← Posts plan to PR
└────────┬────────┘
│
▼
Code Review
(Human reviews plan output)
│
▼
PR Merged
│
▼
┌─────────────────┐
│ Terraform Plan │ ← Re-plan on main
└────────┬────────┘
│
▼
Manual Approval (prod)
│
▼
┌─────────────────┐
│ Terraform Apply │
└─────────────────┘GitHub Actions Implementation
# .github/workflows/terraform.yml
name: Terraform CI/CD
on:
pull_request:
branches: [main]
paths:
- 'terraform/**'
push:
branches: [main]
paths:
- 'terraform/**'
permissions:
id-token: write # For OIDC
contents: read
pull-requests: write
env:
TF_VERSION: "1.6.0"
TF_WORKING_DIR: "terraform"
jobs:
# Stage 1: Validation
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Terraform Format Check
run: terraform fmt -check -recursive
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Init
run: terraform init -backend=false
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Validate
run: terraform validate
working-directory: ${{ env.TF_WORKING_DIR }}
# Stage 2: Security Scanning
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: tfsec
uses: aquasecurity/tfsec-action@v1.0.0
with:
working_directory: ${{ env.TF_WORKING_DIR }}
soft_fail: false
- name: Checkov
uses: bridgecrewio/checkov-action@v12
with:
directory: ${{ env.TF_WORKING_DIR }}
framework: terraform
soft_fail: false
# Stage 3: Plan
plan:
needs: [validate, security]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Configure AWS Credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Terraform Init
run: terraform init
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Plan
id: plan
run: |
terraform plan -no-color -out=tfplan 2>&1 | tee plan.txt
echo "plan<<EOF" >> $GITHUB_OUTPUT
cat plan.txt >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
working-directory: ${{ env.TF_WORKING_DIR }}
continue-on-error: true
- name: Post Plan to PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const plan = `${{ steps.plan.outputs.plan }}`;
const truncated = plan.length > 60000
? plan.substring(0, 60000) + '\n\n... (truncated)'
: plan;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## Terraform Plan Output
\`\`\`hcl
${truncated}
\`\`\`
**Review the plan carefully before approving the PR.**`
});
- name: Save Plan Artifact
uses: actions/upload-artifact@v4
with:
name: tfplan
path: ${{ env.TF_WORKING_DIR }}/tfplan
- name: Check Plan Status
if: steps.plan.outcome == 'failure'
run: exit 1
# Stage 4: Apply (only on main, with approval)
apply:
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
needs: [plan]
runs-on: ubuntu-latest
environment: production # Requires manual approval
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Download Plan
uses: actions/download-artifact@v4
with:
name: tfplan
path: ${{ env.TF_WORKING_DIR }}
- name: Terraform Init
run: terraform init
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Apply
run: terraform apply -auto-approve tfplan
working-directory: ${{ env.TF_WORKING_DIR }}OIDC Authentication (No Long-Lived Credentials)
# In AWS account - create OIDC provider for GitHub
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}
# Role that GitHub Actions will assume
resource "aws_iam_role" "github_actions" {
name = "GitHubActionsTerraformRole"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Federated = aws_iam_openid_connect_provider.github.arn
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
}
StringLike = {
# Only allow from specific repo and branch
"token.actions.githubusercontent.com:sub" = "repo:company/infrastructure:ref:refs/heads/main"
}
}
}
]
})
}Atlantis Alternative
# atlantis.yaml
version: 3
projects:
- name: production
dir: terraform/environments/prod
workspace: default
autoplan:
when_modified:
- "*.tf"
- "../modules/**/*.tf"
enabled: true
apply_requirements:
- approved
- mergeable
# Commands in PR:
# atlantis plan
# atlantis apply (requires approval)Multi-Environment Pipeline
# .github/workflows/terraform-multi-env.yml
jobs:
plan:
strategy:
matrix:
environment: [dev, staging, prod]
runs-on: ubuntu-latest
steps:
- name: Plan ${{ matrix.environment }}
run: |
cd terraform/environments/${{ matrix.environment }}
terraform plan -out=tfplan
apply-dev:
needs: plan
if: github.ref == 'refs/heads/main'
environment: dev # Auto-approve
steps:
- run: terraform apply
apply-staging:
needs: apply-dev
environment: staging # Manual approval
steps:
- run: terraform apply
apply-prod:
needs: apply-staging
environment: production # Requires 2 approvers
steps:
- run: terraform applyState Protection
# backend.tf
terraform {
backend "s3" {
bucket = "terraform-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "terraform-locks"
# Additional protection
kms_key_id = "arn:aws:kms:us-east-1:123:key/abc"
}
}# S3 bucket policy - only CI/CD can modify state
{
"Statement": [
{
"Sid": "DenyDeleteExceptFromCI",
"Effect": "Deny",
"Principal": "*",
"Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
"Resource": "arn:aws:s3:::terraform-state/*",
"Condition": {
"StringNotEquals": {
"aws:PrincipalArn": "arn:aws:iam::123:role/GitHubActionsTerraformRole"
}
}
}
]
} Pipeline Security Checklist
| Control | Implementation |
|---|---|
| No long-lived credentials | OIDC with short-lived tokens |
| Plan review | Post plan to PR comments |
| Apply approval | GitHub Environments with reviewers |
| Branch protection | Require PR, no direct push to main |
| State encryption | S3 SSE-KMS |
| State access logging | S3 access logs + CloudTrail |
| Audit trail | GitHub Actions logs + AWS CloudTrail |
Practice Question
Why should Terraform plans be re-run on the main branch before applying, even if they passed during PR review?