DeployU
Interviews / Cloud & DevOps / Database passwords are visible in terraform.tfstate. Implement secure secrets management.

Database passwords are visible in terraform.tfstate. Implement secure secrets management.

practical Security Interactive Quiz Code Examples

The Scenario

A security audit revealed that database passwords are stored in plaintext in your Terraform state:

$ terraform state pull | jq '.resources[] | select(.type == "aws_db_instance") | .instances[].attributes.password'
"SuperSecretProd123!"

$ cat terraform.tfvars
db_password = "SuperSecretProd123!"

The state file is in S3 (encrypted at rest), but:

  • Anyone with state access can read passwords
  • Passwords are in version control (tfvars)
  • No rotation mechanism exists
  • Same password used across environments

The Challenge

Implement a secrets management strategy that removes secrets from state, enables rotation, and follows security best practices.

Wrong Approach

A junior engineer might just enable S3 encryption and call it secure, hardcode secrets in environment variables, or use terraform.tfvars with .gitignore. These approaches still expose secrets in state files, CI/CD logs, and don't solve the rotation problem.

Right Approach

A senior engineer removes secrets from Terraform entirely using AWS Secrets Manager or HashiCorp Vault, uses data sources to reference secrets without storing them in state, implements automatic rotation, and ensures secrets never appear in logs or version control.

Step 1: Create secrets outside Terraform

# Create secret manually or via separate secure process
aws secretsmanager create-secret \
  --name "prod/database/password" \
  --secret-string "$(openssl rand -base64 32)" \
  --description "Production database password"

Step 2: Reference secrets in Terraform

# data.tf - Read secret without storing in state
data "aws_secretsmanager_secret" "db_password" {
  name = "prod/database/password"
}

data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = data.aws_secretsmanager_secret.db_password.id
}

# main.tf - Use the secret
resource "aws_db_instance" "main" {
  identifier     = "production-db"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.r6g.large"

  username = "admin"
  # Password comes from Secrets Manager - NOT stored in state
  password = data.aws_secretsmanager_secret_version.db_password.secret_string

  # Better: Use IAM authentication instead of passwords
  iam_database_authentication_enabled = true
}

Step 3: Enable automatic rotation

resource "aws_secretsmanager_secret_rotation" "db_password" {
  secret_id           = data.aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rotate_secret.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

Strategy 2: HashiCorp Vault

# provider.tf
provider "vault" {
  address = "https://vault.company.com"
  # Auth via VAULT_TOKEN env var or other method
}

# data.tf
data "vault_generic_secret" "db_creds" {
  path = "secret/data/production/database"
}

# main.tf
resource "aws_db_instance" "main" {
  identifier = "production-db"
  engine     = "postgres"

  username = data.vault_generic_secret.db_creds.data["username"]
  password = data.vault_generic_secret.db_creds.data["password"]
}

Strategy 3: AWS SSM Parameter Store

# For less sensitive config, SSM is simpler/cheaper
data "aws_ssm_parameter" "db_password" {
  name            = "/prod/database/password"
  with_decryption = true
}

resource "aws_db_instance" "main" {
  # ...
  password = data.aws_ssm_parameter.db_password.value
}

Strategy 4: Generate Secrets in Terraform (Rotate via Replacement)

# Generate random password - stored in state but rotatable
resource "random_password" "db" {
  length           = 32
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

# Store in Secrets Manager
resource "aws_secretsmanager_secret" "db_password" {
  name = "prod/database/password"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id     = aws_secretsmanager_secret.db_password.id
  secret_string = random_password.db.result
}

# To rotate: terraform taint random_password.db && terraform apply

Strategy 5: IAM Authentication (Eliminate Passwords)

# Best approach: No passwords at all!
resource "aws_db_instance" "main" {
  identifier = "production-db"
  engine     = "postgres"

  # Enable IAM auth
  iam_database_authentication_enabled = true

  # Still need master password for initial setup
  username = "admin"
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
}

# Create IAM policy for database access
resource "aws_iam_policy" "rds_connect" {
  name = "rds-db-connect"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = "rds-db:connect"
        Resource = "arn:aws:rds-db:${var.region}:${var.account_id}:dbuser:${aws_db_instance.main.resource_id}/app_user"
      }
    ]
  })
}

Preventing Secrets in State

# Use lifecycle to prevent sensitive data in state
resource "aws_db_instance" "main" {
  # ...

  lifecycle {
    ignore_changes = [password]
  }
}

# Mark outputs as sensitive
output "db_connection_string" {
  value     = "postgresql://${aws_db_instance.main.username}@${aws_db_instance.main.endpoint}"
  sensitive = true
}

CI/CD Pipeline Security

# .github/workflows/terraform.yml
jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1

      # Never echo secrets
      - name: Terraform Plan
        run: terraform plan
        env:
          # Secrets from GitHub Secrets, not tfvars
          TF_VAR_vault_token: ${{ secrets.VAULT_TOKEN }}

      # Mask any accidental secret exposure
      - name: Mask Secrets
        run: |
          echo "::add-mask::${{ secrets.DB_PASSWORD }}"

State File Protection

# backend.tf
terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "prod/database/terraform.tfstate"
    region         = "us-east-1"

    # Encryption at rest
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789:key/abc-123"

    # Encryption in transit (default)
    # Access logging
    # Versioning for recovery

    dynamodb_table = "terraform-locks"
  }
}

# Restrict state bucket access
resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "DenyUnencryptedObjectUploads"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:PutObject"
        Resource  = "${aws_s3_bucket.terraform_state.arn}/*"
        Condition = {
          StringNotEquals = {
            "s3:x-amz-server-side-encryption" = "aws:kms"
          }
        }
      }
    ]
  })
}

Secrets Management Comparison

MethodSecrets in StateRotationComplexityCost
tfvars fileYesManualLowFree
Environment varsYesManualLowFree
Secrets Manager data sourceNoAutoMedium~$0.40/secret/month
VaultNoAutoHighVault license
SSM Parameter StoreNoManualLowFree (standard)
IAM AuthN/AAutomaticMediumFree

Practice Question

Why does using a data source to read from Secrets Manager keep the secret out of Terraform state?