Can't connect to RDS from FARGATE service

[kateile]

Hi guys, I am very new to AWS and AWS CDK in general, I am trying to deploy my stack which consists of POSTGRES(Auora), REDIS(Elastcache) and FARGATE. With few knowledge I have gathered from the internet I have managed to deploy db, cache with exception for Fargate. I can't start service because I cant connect to database.

I believe there is something I am missing on how to allow my fargate to connect to database and I just can't figure it how.

Here is my code how I do it.

import * as cdk from 'aws-cdk-lib';
import * as elc from 'aws-cdk-lib/aws-elasticache';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as rds from 'aws-cdk-lib/aws-rds';
import * as ecs_patterns from 'aws-cdk-lib/aws-ecs-patterns';
import * as ecr_assets from 'aws-cdk-lib/aws-ecr-assets';
import {Construct} from 'constructs';
import * as sm from 'aws-cdk-lib/aws-secretsmanager';
import * as logs from "aws-cdk-lib/aws-logs";
import * as path from 'path';

enum Stage {
    LIVE = "live",
    TEST = "test"
}

export class CdkStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        const stage: Stage = Stage.TEST

        const vpc = new ec2.Vpc(this, `${stage}-vpc`, {
            vpcName: `${stage}-vpc`,
            maxAzs: 3, // Default is all AZs in region
            natGateways: 0,
            subnetConfiguration: [
                {
                    subnetType: ec2.SubnetType.PUBLIC,
                    name: 'public-subnet'
                },
                {
                    subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
                    name: 'private-subnet'
                }
            ]
        });

        // const fargateSecurityGroup = new ec2.SecurityGroup(this, `fargate-${stage}-security-group`, {
        //     vpc: vpc,
        //     allowAllOutbound: true
        // })

        // create a security group for aurora db
        const dbSecurityGroup = new ec2.SecurityGroup(this, `db-security-group-${stage}`, {
            vpc: vpc, // use the vpc created above
            allowAllOutbound: true, // allow outbound traffic to anywhere
        })

        const dbPort = 5432

        // allow inbound traffic from anywhere to the db
        dbSecurityGroup.addIngressRule(
            ec2.Peer.anyIpv4(),
            ec2.Port.tcp(dbPort), // allow inbound traffic on port 5432 (postgres)
            'allow inbound traffic from anywhere to the db on port 5432'
        )

        const secret = sm.Secret.fromSecretNameV2(this, "prod", "prod")
        const dbName = "project"

        const databaseCluster = new rds.DatabaseCluster(this, `database-${stage}-cluster`, {
            clusterIdentifier: `${stage}-db`,
            engine: rds.DatabaseClusterEngine.auroraPostgres({
                version: rds.AuroraPostgresEngineVersion.VER_15_2
            }),
            credentials: rds.Credentials.fromSecret(secret),
            instances: 1,
            defaultDatabaseName: dbName,
            port: dbPort,
            instanceProps: {
                vpc: vpc,
                instanceType: new ec2.InstanceType('serverless'),
                autoMinorVersionUpgrade: true,
                publiclyAccessible: true,
                securityGroups: [dbSecurityGroup],
                vpcSubnets: vpc.selectSubnets({
                    subnetType: ec2.SubnetType.PUBLIC, // use the public subnet created above for the db
                }),
                enablePerformanceInsights: true,
            },
            deletionProtection: false,
            cloudwatchLogsRetention: logs.RetentionDays.ONE_WEEK,
        });

        cdk.Aspects.of(databaseCluster).add({
            visit(node) {
                if (node instanceof rds.CfnDBCluster) {
                    node.serverlessV2ScalingConfiguration = {
                        minCapacity: 1,
                        maxCapacity: 1,
                    }
                }
            },
        })

        const subnetsForRedis = vpc.publicSubnets.map((subnet) => subnet.subnetId)

        const redisSubnetGroup = new elc.CfnSubnetGroup(
            this,
            `redis-cluster-private-subnet-group-${stage}`,
            {
                cacheSubnetGroupName: `redis-private-subnet-group-${stage}`,
                subnetIds: subnetsForRedis,
                description: `redis subnet group for ${stage}`
            }
        )

        const redis = new elc.CfnCacheCluster(this, `${stage}-cache-cluster`, {
            cacheNodeType: "cache.t2.micro",
            engine: "redis",
            numCacheNodes: 1,
            clusterName: `${stage}-redis-cache`,
            cacheSubnetGroupName: redisSubnetGroup.ref,
            vpcSecurityGroupIds: [vpc.vpcDefaultSecurityGroup],
        });

        redis.addDependency(redisSubnetGroup)

        const taskIamRole = new cdk.aws_iam.Role(this, `app-${stage}-role`, {
            roleName: `app-${stage}-role`,
            assumedBy: new cdk.aws_iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
        });

        const taskDefinition = new ecs.FargateTaskDefinition(this, `${stage}-task`, {
            taskRole: taskIamRole,
        });

        secret.grantRead(taskIamRole)

        const root = path.join(__dirname, '..', '..');
        const imageAsset = new ecr_assets.DockerImageAsset(this, "image", {
            directory: root,
            file: "Dockerfile",
        })

        const image = ecs.ContainerImage.fromDockerImageAsset(imageAsset);

        taskDefinition.addContainer(`${stage}-container`, {
            image: image,
            portMappings: [{
                containerPort: 7373,
                protocol: ecs.Protocol.TCP
            }],
            logging: ecs.LogDrivers.awsLogs({
                streamPrefix: `${stage}-task`,
                logRetention: logs.RetentionDays.ONE_WEEK,
            }),
            secrets: {
                "DB_PASSWORD": ecs.Secret.fromSecretsManager(secret, "password"),
                "DB_USER": ecs.Secret.fromSecretsManager(secret, "username"),
            },
            environment: {
                "DB_HOST": databaseCluster.clusterEndpoint.hostname,
                "DB_PORT": databaseCluster.clusterEndpoint.port.toString(),
                "DB_NAME": dbName,
                "REDIS_ADDRESS": `${redis.attrRedisEndpointAddress}:${redis.attrRedisEndpointPort}`,
            },
        });

        const cluster = new ecs.Cluster(this, `${stage}-cluster`, {
            clusterName: `${stage}-cluster`,
            vpc: vpc,
            containerInsights: true,
        })

        // Create a load-balanced Fargate service and make it public
        new ecs_patterns.ApplicationLoadBalancedFargateService(this, `fargate-service-${stage}`, {
            cluster: cluster,
            cpu: 1024, // Default is 256
            memoryLimitMiB: 2048, // Default is 512
            taskDefinition: taskDefinition,
            desiredCount: 1,
            serviceName: `${stage}-app`,
            assignPublicIp: true,
            publicLoadBalancer: true,
            circuitBreaker: {
                rollback: true
            },
            //securityGroups: [fargateSecurityGroup],
            maxHealthyPercent: 200,
            minHealthyPercent: stage === Stage.TEST ? 0 : 100, // speed up deployment in dev testing
            openListener: true,
        })

        //https://stackoverflow.com/questions/68314584/allow-connections-from-ecs-to-an-existing-rds-database
        databaseCluster.connections.allowFrom(cluster, ec2.Port.tcp(dbPort), "Allow from fargate cluster")
    }
}

Using the above codes my console stuck at

Stack CdkStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)

and when I look into service logs see service prints there is database connection issue.

For now I want db to be accessable from anywhere so that I can seed my database from my local machine. But after that only fargate should have access. How to do that?

[peterwoodworth]

Hey @kateile, thanks for submitting this discussion. You ask a good question

I'm not super familiar with building this kind of use case, so I don't have an answer off the top of my head. Let me ask a few questions to help me in my investigation for this:

Could you please let me know in exactly which context you saw the following message? Was this after cdk deploy?

Stack CdkStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)

Additionally, what exactly are the logs showing when you say there is a database connection issue?

Thanks!
-Peter

[peterwoodworth]

Based on this example it looks like you might want to set the db connection info as environment variables for your task image

[kateile]

Thanks @peterwoodworth for response. My problem was that my fargate service could not connect to database . But after so much searching I manage to solve that using security groups like below.

 const fargateSecurityGroup = new ec2.SecurityGroup(this, `fargate-${stage}-security-group`, {
            vpc: vpc,
            allowAllOutbound: true
        })

const dbSecurityGroup = new ec2.SecurityGroup(this, `db-security-group-${stage}`, {
            vpc: vpc, // use the vpc created above
            allowAllOutbound: true, // allow outbound traffic to anywhere
        })

 const dbPort = 5432
        const appPort = 7373
        const redisPort = 6379

        // allow inbound traffic from anywhere to the db
        fargateSecurityGroup.addIngressRule(
            ec2.Peer.ipv4("0.0.0.0/0"),
            ec2.Port.tcp(appPort), // allow inbound traffic on port 5432 (postgres)
            'allow inbound traffic from anywhere to the db on port 5432'
        )

        dbSecurityGroup.addIngressRule(
            fargateSecurityGroup,
            ec2.Port.tcp(dbPort), // allow inbound traffic on port 5432 (postgres)
            'allow inbound traffic from anywhere to the db on port 5432'
        )

and then use fargateSecurityGroup and dbSecurityGroup in ApplicationLoadBalancedFargateService and DatabaseCluster respectively.

Don't know if that is the best solution. But it works for me

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.