Release Tests for AWS Postgres passwordless authentication by raviharshicorp · Pull Request #381 · hashicorp/terraform-aws-terraform-enterprise

raviharshicorp · 2025-11-11T10:19:56Z

Background

This release test tests the passwordless authentication to AWS postgres using IAM based token.

Relates OR Closes #(https://hashicorp.atlassian.net/browse/IND-4009)

JIRA: https://hashicorp.atlassian.net/browse/IND-4009

Related PRs
terraform-enterprise: https://github.com/hashicorp/terraform-enterprise/pull/3397
ptfedev-infra: https://github.com/hashicorp/ptfedev-infra/pull/887
terraform-random-tfe-utility: hashicorp/terraform-random-tfe-utility#192

How Has This Been Tested

The release test ran on Github:https://github.com/hashicorp/terraform-enterprise/actions/runs/19146818577/job/54726521821
Checked the TFE logs, database connection was success.

Test Configuration

Terraform Version: latest
Any additional relevant variables:
TFE_DATABASE_PASSWORDLESS_AWS_USE_INSTANCE_PROFILE to true
TFE_DATABASE_PASSWORDLESS_AWS_REGION to aws_region

CI/CD: https://github.com/hashicorp/terraform-enterprise/actions/runs/19464088212/job/55694964359

Screenshot:

- Added postgres_enable_iam_auth, postgres_use_password_auth, and db_iam_username variables - Added database_passwordless_aws_use_iam local variable derived from postgres variables - Pass database_passwordless_aws_use_iam to runtime_container_engine_config module - Added postgres_passwordless_debug output for troubleshooting - Clean implementation focused only on PostgreSQL IAM auth (no Redis configs)

- Add postgres_enable_iam_auth, postgres_use_password_auth, db_iam_username variables - Enable iam_database_authentication_enabled when postgres_enable_iam_auth is true - Make password generation and usage conditional based on postgres_use_password_auth - Update outputs to handle conditional password authentication

- Make password output conditional based on postgres_use_password_auth - Return empty string when using IAM authentication instead of password

…QL IAM auth branch - Change module references from ref=main to ref=pravi/IND-5776-release-test - This branch contains the required database_passwordless_aws_use_iam variables - Fixes 'Unsupported argument' error for database_passwordless_aws_use_iam in runtime_container_engine_config module

- Update user_data_base64 reference to use correct module name 'tfe_init' - Fixes 'Reference to undeclared module' error for tfe_init_fdo - The module is correctly named 'tfe_init' not 'tfe_init_fdo'

- Always generate a password as AWS RDS requires it even for IAM auth - Remove conditional count from random_string resource - AWS RDS needs password during creation, but IAM auth takes precedence when enabled - Fixes 'Invalid master password' error when postgres_use_password_auth=false

- Add postgres_iam_username variable to service_accounts module - Create inline IAM policy aws_iam_role_policy.postgres_iam_connect when postgres_iam_username is provided - Pass db_iam_username from main module to service_accounts as postgres_iam_username - This avoids for_each dependency cycles by using inline policy instead of separate resource - Enables EC2 instances to authenticate to RDS PostgreSQL using IAM

- Update locals.tf to properly use database_passwordless_aws_use_iam variable - Add database_passwordless_aws_region to runtime_container_engine_config - This ensures DATABASE_AUTH_USE_AWS_IAM=true is set in TFE runtime environment - Fixes 502 errors caused by incorrect database auth configuration

- Add null_resource to create IAM database user with rds_iam role - Only runs when postgres_enable_iam_auth is true and db_iam_username is set - Includes proper error handling and idempotency - Waits for database to be ready before creating user - This fixes PostgreSQL passwordless authentication by ensuring the IAM user exists in PostgreSQL Fixes: PostgreSQL IAM authentication 502 errors

- Add detailed logging to debug execution - Auto-install PostgreSQL client if missing - Fix SQL syntax issues with dynamic EXECUTE statements - Better error reporting and status messages - This should help identify why the IAM user creation is not working

- Add debug null_resource to check variable values - Enhanced logging to see execution environment - Better error handling and network connectivity checks - Graceful degradation if psql cannot be installed - This will help identify why IAM user creation is not working Debug: PostgreSQL IAM authentication troubleshooting

… user setup - Remove problematic local-exec provisioner that was failing silently - Create SSM document that can be executed on EC2 instances with proper credentials - This approach allows the database user creation to run from TFE instances - Provides foundation for automated PostgreSQL IAM authentication setup

- Add missing iam_user_setup_status and postgres_iam_setup_ssm_document outputs to enterprisedb module - This ensures enterprise_db and standard_db objects have consistent structure - Fixes 'Inconsistent conditional result types' error in locals.tf selection logic - EDB module returns appropriate defaults since it doesn't support IAM authentication Fixes: Type mismatch for object attribute 'endpoint' in conditional expression

- Add missing iam_user_setup_status and postgres_iam_setup_ssm_document to default_database - This ensures all database objects have consistent 7-attribute structure - Resolves type mismatch when modules fall back to default_database Root cause: try(module.edb[0], local.default_database) was falling back to 5-attribute default_database when EDB module doesn't exist, causing type mismatch with 7-attribute standard_db

…username - Updated locals.tf to override database username when IAM authentication is enabled - Set password to null for IAM authentication (no password required) - This ensures TFE uses the IAM database user (tfe_rds_iam_user) instead of admin user

- Use IAM username (db_iam_username) when IAM authentication is enabled - Use admin username (database.username) for password-based authentication - Apply to both settings and tfe_init modules - Set password to null only when IAM authentication is enabled - This follows the Redis IAM pattern: separate users for different auth methods

- Added database connection parameters to tfe_init module call - Includes database_host, database_name, admin credentials - Added database_iam_username for IAM user creation - Enables automated PostgreSQL IAM user setup in user_data scripts This ensures the PostgreSQL IAM user is created automatically before TFE startup, eliminating manual steps for release tests.

When using PostgreSQL IAM authentication, TFE's configuration validation still requires a password value to be set, even though the actual authentication uses IAM tokens. Changed from: database_password = null To: database_password = "" This should resolve the 'database password must be set' error while maintaining proper IAM authentication functionality.

Changed database_password back to null when IAM auth is enabled: - TFE's configuration validation may require null instead of empty string - Ensures proper IAM authentication flow without password requirement - Aligns with expected IAM authentication patterns This should resolve the persistent 'database password must be set' error.

…dation TFE's configuration validation requires a password field even when IAM authentication is enabled. Use 'aws-iam-auth' placeholder password to pass config validation while maintaining IAM authentication functionality.

Remove placeholder password approach and set database_password to null for IAM authentication. The DATABASE_URL will be generated at runtime with actual AWS RDS IAM authentication tokens in the user_data script.

TFE configuration validation requires a non-empty password field. Set 'IAM_AUTH' placeholder for IAM authentication mode to pass config validation while pgmultiauth handles actual authentication.

Set database_password to null for IAM auth so TFE_DATABASE_PASSWORD is completely excluded from environment variables rather than being set to a placeholder value.

…terprise - Update postgres_engine_version from '16' to '16.10' for consistency - Update aurora_postgres_engine_version from '16.9' to '16.10' for consistency - Ensures uniform PostgreSQL version across all TFE deployments - Resolves IAM authentication compatibility issues with older PostgreSQL versions

- Set password to 'password' for easy database access during testing - This allows manual creation of IAM user before automated setup works - Maintains compatibility with existing IAM authentication flow

- Pass SSM document name to tfe_init module for automated execution - Update database module output to return just document name - Integrate IAM user creation into instance startup process - This fixes 502 errors by ensuring IAM user exists before TFE starts

- Compress SSM document commands to reduce size - Use fixed password directly in SSM parameters - Remove verbose logging and comments from SSM script - Maintain full functionality in compact form

- Add proper error handling and database readiness checks - Include verbose logging for better debugging - Add comprehensive user permissions for TFE operations - Verify user creation after setup - Handle both Ubuntu and RHEL package installation

- Remove duplicate aws_security_group_rule resource declaration - Keep only the first postgres_db_egress rule defined earlier in the file - Fixes terraform validation error about duplicate resource names

- Install hstore, uuid-ossp, and citext extensions during database setup - These extensions are required for TFE to start successfully - Add verification query to confirm extensions are installed - Fixes TFE startup check failure for missing PostgreSQL extensions

- Add CREATEDB privilege to IAM user creation - Grant CREATE ON DATABASE privilege for schema creation - Fixes permission denied error when TFE tries to create terraform_enterprise schema - IAM user now has sufficient privileges for TFE database operations

- Use current AWS region from data source instead of variable - Ensures TFE gets the correct region for IAM database authentication - Fixes configuration issue where region was empty for passwordless auth

- Grant CREATEROLE privilege for more comprehensive database operations - Add USAGE grants on citext, hstore, and uuid types to allow registry API to use extensions - Apply permissions to both new and existing users for consistency - This fixes terraform-registry-api migration failures due to missing citext extension access

- Add cleanup of dirty migration records in schema_migrations table - Reset version 4 migration state that was causing 'Dirty database version 4' errors - Clean up before creating IAM user to ensure migrations can run successfully - This fixes terraform-registry-api startup failures due to corrupted migration state

- Drop and recreate schema_migrations table on every database setup - This ensures terraform-registry-api always starts with a clean migration state - Prevents persistent 'Dirty database version 4' errors from previous failed attempts - More aggressive but reliable approach to migration state cleanup

- Add debugging output to show migration table state before/after cleanup - Clean up multiple possible migration table names (schema_migrations, schema_version, gorp_migrations, migrations) - Clear advisory locks that might be preventing migrations - Grant explicit permissions on schema_migrations table to IAM user - Show table structure after creation for verification

- Force terminate active database connections that might hold locks - Clear all advisory locks including golang-migrate specific locks (classid 1410924490) - Drop and recreate terraform_registry schema completely - Create fresh schema_migrations table with exact golang-migrate format - Reset database configuration and reload settings - Add comprehensive debugging output to track cleanup progress - Grant full permissions to IAM user on all schemas and tables This addresses persistent 'Dirty database version 4' errors by ensuring completely clean migration state for terraform-registry-api startup.

- Recreate extensions explicitly in public schema to ensure global access - Grant USAGE on extension types to both IAM user and public - Add wait period after terminating connections - Enhanced debugging for migration state tracking - Set search_path for IAM user to include terraform_registry schema - More comprehensive advisory lock cleanup with proper iteration - Verify extension availability with detailed output This addresses the 'type citext does not exist' error and should resolve persistent migration state corruption issues.

- Drop and recreate public schema entirely to eliminate ALL existing state - Force unlock ALL advisory locks including golang-migrate specific locks - Terminate ALL database connections to ensure clean state - Create completely fresh schema_migrations table with zero history - Recreate extensions in clean public schema - Grant comprehensive permissions to IAM user on all schemas - Set proper search_path for schema access - Force PostgreSQL configuration reload This nuclear approach eliminates any possibility of dirty migration state persistence that was causing repeated 'Dirty database version 4' errors in terraform-registry-api. Clean slate approach.

REVERTED: All complex migration cleanup and schema reset approaches IMPLEMENTED: Simple and correct solution: - Create dedicated database _db for IAM user - Install extensions in both main and IAM databases - Grant proper permissions to IAM user on dedicated database - Update TFE configuration to use IAM database when IAM auth enabled - Added iam_database_name output from database module - Updated both database_name and pg_dbname in main configuration This eliminates migration conflicts by giving IAM user clean database with no existing migration state. Much simpler and more reliable than attempting to clean up shared database migration state.

- Added database_iam_name local that correctly resolves IAM database name - Only applies IAM database for standard database module (not Aurora/mTLS/EDB) - Updated both database_name and pg_dbname to use database_iam_name local - Fixes issue where TFE was still connecting to main database instead of IAM database This ensures terraform-registry-api connects to the clean separate database _db instead of the main database with existing migrations.

The external data source was trying to SSH with wrong key path causing: - Permission denied (publickey) errors - Identity file /tmp/postgres_key.pem not accessible FIXED: Use static output instead since database name is predictable - Database name is simply _db - No need for SSH or file reading from EC2 instance - Removed temporary file creation in script - Simplified architecture and removed failure point This resolves the terraform apply failure in release tests.

- Create IAM user first before creating database - Set IAM user as database owner instead of DB_USERNAME - Install required extensions in IAM database - Separate database creation into logical steps - Ensure proper permissions and extension access

- Create IAM user and database in single transaction - IAM user owns their database from creation - No migration conflicts - clean slate - Install extensions directly in IAM database - Simple, atomic operation

- Connect to 'postgres' system database instead of main TFE database - Avoid migration conflicts during IAM setup - Create IAM user and database in clean environment - Remove extension installation in dirty main database

- Remove debug echo statements exposing database connection details - Fix terraform formatting issues - Ensure secure logging practices - Code follows best practices and naming conventions

- Change postgres setup instance from m5.xlarge to t3.micro - Reduce volume size from 100GB to 20GB - Instance is only used for IAM user setup, doesn't need high compute

- Remove local.database_iam_instance_profile which was declared but never used - Fixes terraform_unused_declarations warning from tflint

- Remove unused aws_region data source (terraform_unused_declarations) - Add missing provider version constraints for tls, local, and null providers (terraform_required_providers) This resolves 4 tflint warnings and improves code quality by: 1. Removing dead code that was declared but not used 2. Ensuring all required providers have explicit version constraints

- Remove leftover debugging file that was used during development - File contains hardcoded placeholders and is not referenced in any Terraform configs - Clean up unnecessary debugging artifacts from postgres passwordless implementation

kkavish · 2025-11-25T08:24:19Z

locals.tf

  )

+  # PostgreSQL IAM authentication flag
+  database_passwordless_aws_use_iam = var.database_passwordless_aws_use_iam || (var.postgres_enable_iam_auth && !var.postgres_use_password_auth)


Why do we need 2 different variables database_passwordless_aws_use_iam and postgres_enable_iam_auth?

kkavish · 2025-11-25T08:26:41Z

modules/database/outputs.tf

+output "iam_user_setup_status" {
+  value       = var.postgres_enable_iam_auth && var.db_iam_username != "" ? "IAM user automated creation enabled via null_resource and user_data fallback" : "IAM authentication not enabled"
+  description = "Status of IAM user setup for PostgreSQL."
+}


What's the use case of this? Where are we using this?

kkavish · 2025-11-25T08:34:03Z

locals.tf

+  database_passwordless_aws_use_iam = var.database_passwordless_aws_use_iam || (var.postgres_enable_iam_auth && !var.postgres_use_password_auth)
+
+  # Database IAM database name - only available from standard database module
+  database_iam_name = local.database_passwordless_aws_use_iam && !var.enable_aurora && !var.db_use_mtls && !var.enable_edb ? try(module.database[0].iam_database_name, local.database.name) : local.database.name


I am not sure what database_iam_name represents.

kkavish · 2025-11-25T08:35:42Z

main.tf

  allow_multiple_azs           = var.allow_multiple_azs
+  # PostgreSQL IAM authentication (disabled for explorer)
+  postgres_enable_iam_auth   = false
+  postgres_use_password_auth = true


Explorer DB does not support IAM auth - https://developer.hashicorp.com/terraform/enterprise/deploy/configuration/enable-explorer#requirements

We can set this as false as well.

Also, setting this to true will keep it truthful for all test cases even when IAM auth is not setup.

kkavish · 2025-11-25T08:37:06Z

main.tf

+  postgres_enable_iam_auth   = var.postgres_enable_iam_auth
+  postgres_use_password_auth = var.postgres_use_password_auth


I think one variable should be enough.

kkavish · 2025-11-25T08:38:10Z

outputs.tf

+output "postgres_passwordless_debug" {
+  value = {
+    postgres_enable_iam_auth          = var.postgres_enable_iam_auth
+    postgres_use_password_auth        = var.postgres_use_password_auth
+    db_iam_username                   = var.db_iam_username
+    database_passwordless_aws_use_iam = local.database_passwordless_aws_use_iam
+    database_config_summary = {
+      name       = local.database.name
+      username   = local.database.username
+      endpoint   = local.database.endpoint
+      parameters = local.database.parameters
+    }
+  }
+  description = "Debug output for PostgreSQL passwordless configuration"


If this was for debugging, please remove them now.

kkavish · 2025-11-25T08:38:55Z

modules/enterprisedb/outputs.tf

+output "iam_user_setup_status" {
+  value       = "EDB does not support IAM authentication - using standard authentication"
+  description = "Status of PostgreSQL IAM user setup"
+}


what's the use if this?

kkavish · 2025-11-25T08:39:11Z

modules/enterprisedb/outputs.tf

+output "postgres_iam_setup_ssm_document" {
+  value       = null
+  description = "SSM Document information for PostgreSQL IAM user setup (manual execution required)"
+}


what's the use of a null output?

nikolasrieble · 2025-11-26T08:32:20Z

We are migrating this module to be an internal only release testing module. Going forward, please submit changes to this repo instead: https://github.com/hashicorp/terraform-terraform-enterprise-fdo

raviharshicorp added 30 commits November 7, 2025 15:23

Update database module outputs for conditional password authentication

d2bb317

- Make password output conditional based on postgres_use_password_auth - Return empty string when using IAM authentication instead of password

Fix module reference from tfe_init_fdo to tfe_init

e6751c4

- Update user_data_base64 reference to use correct module name 'tfe_init' - Fixes 'Reference to undeclared module' error for tfe_init_fdo - The module is correctly named 'tfe_init' not 'tfe_init_fdo'

Add PostgreSQL passwordless auth variables and IAM policy

700dfd2

Verify type mismatch fix - both modules now have 7 outputs

fe9622a

Implement proper PostgreSQL IAM authentication

8e6b2e3

Remove placeholder password approach and set database_password to null for IAM authentication. The DATABASE_URL will be generated at runtime with actual AWS RDS IAM authentication tokens in the user_data script.

Set IAM_AUTH placeholder for TFE_DATABASE_PASSWORD

3b924ca

TFE configuration validation requires a non-empty password field. Set 'IAM_AUTH' placeholder for IAM authentication mode to pass config validation while pgmultiauth handles actual authentication.

Revert placeholder password approach for IAM authentication

8d1dacf

Set database_password to null for IAM auth so TFE_DATABASE_PASSWORD is completely excluded from environment variables rather than being set to a placeholder value.

Use fixed password for testing to enable IAM user creation

95efd48

- Set password to 'password' for easy database access during testing - This allows manual creation of IAM user before automated setup works - Maintains compatibility with existing IAM authentication flow

Optimize SSM document for smaller user_data payload

b72eefe

- Compress SSM document commands to reduce size - Use fixed password directly in SSM parameters - Remove verbose logging and comments from SSM script - Maintain full functionality in compact form

raviharshicorp added 19 commits November 18, 2025 00:33

Fix duplicate postgres_db_egress security group rule

6296f6f

- Remove duplicate aws_security_group_rule resource declaration - Keep only the first postgres_db_egress rule defined earlier in the file - Fixes terraform validation error about duplicate resource names

Fix AWS region for PostgreSQL IAM authentication

ba9cfbb

- Use current AWS region from data source instead of variable - Ensures TFE gets the correct region for IAM database authentication - Fixes configuration issue where region was empty for passwordless auth

simplify: create IAM database immediately after user creation

a4eb737

- Create IAM user and database in single transaction - IAM user owns their database from creation - No migration conflicts - clean slate - Install extensions directly in IAM database - Simple, atomic operation

Add required extensions to the db

7d9edad

raviharshicorp mentioned this pull request Nov 19, 2025

Add AWS IAM PostgreSQL passwordless authentication support #379

Closed

raviharshicorp added 5 commits November 20, 2025 19:50

Clean up postgres passwordless implementation

5d3d637

- Remove debug echo statements exposing database connection details - Fix terraform formatting issues - Ensure secure logging practices - Code follows best practices and naming conventions

Reduce postgres setup EC2 instance size for cost optimization

edbe418

- Change postgres setup instance from m5.xlarge to t3.micro - Reduce volume size from 100GB to 20GB - Instance is only used for IAM user setup, doesn't need high compute

Fix tflint warning: remove unused local variable

fd9d5d6

- Remove local.database_iam_instance_profile which was declared but never used - Fixes terraform_unused_declarations warning from tflint

Remove debugging script test_db_connection.sh

c911ba3

- Remove leftover debugging file that was used during development - File contains hardcoded placeholders and is not referenced in any Terraform configs - Clean up unnecessary debugging artifacts from postgres passwordless implementation

kkavish reviewed Nov 25, 2025

View reviewed changes

raviharshicorp marked this pull request as ready for review November 25, 2025 10:20

raviharshicorp requested a review from a team as a code owner November 25, 2025 10:20

nikolasrieble closed this Nov 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release Tests for AWS Postgres passwordless authentication#381

Release Tests for AWS Postgres passwordless authentication#381
raviharshicorp wants to merge 79 commits intomainfrom
pravi/IND-5776-release-test

raviharshicorp commented Nov 11, 2025 •

edited

Loading

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

kkavish Nov 25, 2025

Uh oh!

nikolasrieble commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		postgres_enable_iam_auth = var.postgres_enable_iam_auth
		postgres_use_password_auth = var.postgres_use_password_auth

Conversation

raviharshicorp commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

How Has This Been Tested

Test Configuration

Screenshot:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikolasrieble commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raviharshicorp commented Nov 11, 2025 •

edited

Loading