Fargate service initializing task in target with wrong availability zone

[wdolek]

We use ApplicationLoadBalancedFargateService (currently 2.59.0) to deploy our dockerized application, however ECS decides to initialize task and assign target which is not enabled for ALB, getting error:

Target is in an Availability Zone that is not enabled for the load balancer

Health check endpoint is never reached by ALB, causing ECS to discard task after some time - this repeats indefinitely when we set desiredCount > 2 (ECS tries to run task in every availability zone obviously).

We have following availability zones:

• eu-west-1a both public and private subnet
• eu-west-1b both public and private subnet
• eu-west-1c contains only private subnet

ALB should use a and b obviously, but ECS uses c as well.

This issue is described here: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-load-balancer-errors/

If a service uses a load balancer and starts a task located in an Availability Zone that the load balancer isn't configured to use, then the task never passes the health check and the task is stopped.

Fargate service setup looks like this - we don't configure ECS nor ALB directly, my understanding was this should be done automatically:

// configuration and variables omitted

const vpc = Vpc.fromLookup(this, id, {
  vpcId: 'vpc-00000000000000000'
});

const cluster = new Cluster(this, "AcmeCluster", {
  vpc: vpc,
  clusterName: 'Acme-Cluster'
});

const service = new ApplicationLoadBalancedFargateService(this, serviceId, {
  cluster: cluster,
  cpu: cfg.taskManagement.cpu, // Default is 256
  desiredCount: cfg.taskManagement.desiredCount, // Default is 1
  memoryLimitMiB: cfg.taskManagement.ram, // Default is 512
  publicLoadBalancer: true, // Default is false,
  // ...
  taskImageOptions: {
    image: ContainerImage.fromRegistry(asset.imageUri),
    logDriver: new AwsLogDriver({
      streamPrefix: 'Acme-Api',
      logGroup: logGroup
    }),
  }
});

service.targetGroup.configureHealthCheck({
  path: "/",
  interval: Duration.seconds(30),
  healthyHttpCodes: "200",
  healthyThresholdCount: 2,
  unhealthyThresholdCount: 3,
  timeout: Duration.seconds(5),
});

// don't drop long running incoming calls too early (they happen during peak load)
// default 60
// https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
service.loadBalancer.setAttribute(
  "idle_timeout.timeout_seconds",
  "120"
);

// speed up target deregistraion, moving from `draining` to `unused` state faster
// default 300
// https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay
service.targetGroup.setAttribute(
  "deregistration_delay.timeout_seconds",
  "20"
);

// custom route53 registraion
let loadBalancerTarget = new LoadBalancerTarget(service.loadBalancer);
let recordTarget = RecordTarget.fromAlias(loadBalancerTarget);

new ARecord(this, 'AcmeDns', {
  target: recordTarget,
  recordName: cfg.domainNamePrefix,
  zone: cfg.hostedZone
});

I found somewhat related tickets:

• Subnet selection returns more than one per AZ #3126
• ecs-patterns/ApplicationLoadBalancedFargateService fails to deploy in VPC with multiple subnets in same AZ #5892

I tried to add:

const lb = service.loadBalancer.node.defaultChild as CfnLoadBalancer
lb.subnets = vpc.selectSubnets({ onePerAz: true, subnetType: SubnetType.PUBLIC}).subnetIds

... but ECS was still using wrong availability zone.

We have several other projects where we use Fargate, but this is the first time I'm seeing such behaviour. I compared this CDK with other projects but couldn't spot any difference. What could be causing this?