Questions
Database passwords are visible in terraform.tfstate. Implement secure secrets management.
The Scenario
A security audit revealed that database passwords are stored in plaintext in your Terraform state:
$ terraform state pull | jq '.resources[] | select(.type == "aws_db_instance") | .instances[].attributes.password'
"SuperSecretProd123!"
$ cat terraform.tfvars
db_password = "SuperSecretProd123!"
The state file is in S3 (encrypted at rest), but:
- Anyone with state access can read passwords
- Passwords are in version control (tfvars)
- No rotation mechanism exists
- Same password used across environments
The Challenge
Implement a secrets management strategy that removes secrets from state, enables rotation, and follows security best practices.
A junior engineer might just enable S3 encryption and call it secure, hardcode secrets in environment variables, or use terraform.tfvars with .gitignore. These approaches still expose secrets in state files, CI/CD logs, and don't solve the rotation problem.
A senior engineer removes secrets from Terraform entirely using AWS Secrets Manager or HashiCorp Vault, uses data sources to reference secrets without storing them in state, implements automatic rotation, and ensures secrets never appear in logs or version control.
Strategy 1: AWS Secrets Manager (Recommended for AWS)
Step 1: Create secrets outside Terraform
# Create secret manually or via separate secure process
aws secretsmanager create-secret \
--name "prod/database/password" \
--secret-string "$(openssl rand -base64 32)" \
--description "Production database password"Step 2: Reference secrets in Terraform
# data.tf - Read secret without storing in state
data "aws_secretsmanager_secret" "db_password" {
name = "prod/database/password"
}
data "aws_secretsmanager_secret_version" "db_password" {
secret_id = data.aws_secretsmanager_secret.db_password.id
}
# main.tf - Use the secret
resource "aws_db_instance" "main" {
identifier = "production-db"
engine = "postgres"
engine_version = "15.4"
instance_class = "db.r6g.large"
username = "admin"
# Password comes from Secrets Manager - NOT stored in state
password = data.aws_secretsmanager_secret_version.db_password.secret_string
# Better: Use IAM authentication instead of passwords
iam_database_authentication_enabled = true
}Step 3: Enable automatic rotation
resource "aws_secretsmanager_secret_rotation" "db_password" {
secret_id = data.aws_secretsmanager_secret.db_password.id
rotation_lambda_arn = aws_lambda_function.rotate_secret.arn
rotation_rules {
automatically_after_days = 30
}
}Strategy 2: HashiCorp Vault
# provider.tf
provider "vault" {
address = "https://vault.company.com"
# Auth via VAULT_TOKEN env var or other method
}
# data.tf
data "vault_generic_secret" "db_creds" {
path = "secret/data/production/database"
}
# main.tf
resource "aws_db_instance" "main" {
identifier = "production-db"
engine = "postgres"
username = data.vault_generic_secret.db_creds.data["username"]
password = data.vault_generic_secret.db_creds.data["password"]
}Strategy 3: AWS SSM Parameter Store
# For less sensitive config, SSM is simpler/cheaper
data "aws_ssm_parameter" "db_password" {
name = "/prod/database/password"
with_decryption = true
}
resource "aws_db_instance" "main" {
# ...
password = data.aws_ssm_parameter.db_password.value
}Strategy 4: Generate Secrets in Terraform (Rotate via Replacement)
# Generate random password - stored in state but rotatable
resource "random_password" "db" {
length = 32
special = true
override_special = "!#$%&*()-_=+[]{}<>:?"
}
# Store in Secrets Manager
resource "aws_secretsmanager_secret" "db_password" {
name = "prod/database/password"
}
resource "aws_secretsmanager_secret_version" "db_password" {
secret_id = aws_secretsmanager_secret.db_password.id
secret_string = random_password.db.result
}
# To rotate: terraform taint random_password.db && terraform applyStrategy 5: IAM Authentication (Eliminate Passwords)
# Best approach: No passwords at all!
resource "aws_db_instance" "main" {
identifier = "production-db"
engine = "postgres"
# Enable IAM auth
iam_database_authentication_enabled = true
# Still need master password for initial setup
username = "admin"
password = data.aws_secretsmanager_secret_version.db_password.secret_string
}
# Create IAM policy for database access
resource "aws_iam_policy" "rds_connect" {
name = "rds-db-connect"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = "rds-db:connect"
Resource = "arn:aws:rds-db:${var.region}:${var.account_id}:dbuser:${aws_db_instance.main.resource_id}/app_user"
}
]
})
}Preventing Secrets in State
# Use lifecycle to prevent sensitive data in state
resource "aws_db_instance" "main" {
# ...
lifecycle {
ignore_changes = [password]
}
}
# Mark outputs as sensitive
output "db_connection_string" {
value = "postgresql://${aws_db_instance.main.username}@${aws_db_instance.main.endpoint}"
sensitive = true
}CI/CD Pipeline Security
# .github/workflows/terraform.yml
jobs:
terraform:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
# Never echo secrets
- name: Terraform Plan
run: terraform plan
env:
# Secrets from GitHub Secrets, not tfvars
TF_VAR_vault_token: ${{ secrets.VAULT_TOKEN }}
# Mask any accidental secret exposure
- name: Mask Secrets
run: |
echo "::add-mask::${{ secrets.DB_PASSWORD }}"State File Protection
# backend.tf
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "prod/database/terraform.tfstate"
region = "us-east-1"
# Encryption at rest
encrypt = true
kms_key_id = "arn:aws:kms:us-east-1:123456789:key/abc-123"
# Encryption in transit (default)
# Access logging
# Versioning for recovery
dynamodb_table = "terraform-locks"
}
}
# Restrict state bucket access
resource "aws_s3_bucket_policy" "terraform_state" {
bucket = aws_s3_bucket.terraform_state.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "DenyUnencryptedObjectUploads"
Effect = "Deny"
Principal = "*"
Action = "s3:PutObject"
Resource = "${aws_s3_bucket.terraform_state.arn}/*"
Condition = {
StringNotEquals = {
"s3:x-amz-server-side-encryption" = "aws:kms"
}
}
}
]
})
} Secrets Management Comparison
| Method | Secrets in State | Rotation | Complexity | Cost |
|---|---|---|---|---|
| tfvars file | Yes | Manual | Low | Free |
| Environment vars | Yes | Manual | Low | Free |
| Secrets Manager data source | No | Auto | Medium | ~$0.40/secret/month |
| Vault | No | Auto | High | Vault license |
| SSM Parameter Store | No | Manual | Low | Free (standard) |
| IAM Auth | N/A | Automatic | Medium | Free |
Practice Question
Why does using a data source to read from Secrets Manager keep the secret out of Terraform state?