ECS Patterns, ApplicationLoadBalancedFargateService with health check startPeriod

[wdolek]

Our service requires some time to start (populate caches, compile at runtime, download base configuration...). When scale out occurs, we want to give new instance time before it's used by LB.

This is how we setup our instance currently:

import * as ecsPatterns from "aws-cdk-lib/aws-ecs-patterns";

// ...

const fargateService =
  new ecsPatterns.ApplicationLoadBalancedFargateService(
    this,
    "FargateService",
    {
      serviceName: `AcmeSvc`,
      desiredCount: 3,
      cpu: 512,
      memoryLimitMiB: 4096,
      publicLoadBalancer: true,
      healthCheckGracePeriod: Duration.seconds(60),
      taskImageOptions: {
        // ...
      },
      // ...
    }
  );

const scalableTarget = fargateService.service.autoScaleTaskCount({
  minCapacity: 3,
  maxCapacity: 4,
});

scalableTarget.scaleOnCpuUtilization("CpuScaling", {
  targetUtilizationPercent: 50,
});

// don't drop long running incoming calls too early (they happen during high load)
// default 60
// https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
fargateService.loadBalancer.setAttribute("idle_timeout.timeout_seconds", "300");

// speed up target deregistraion, moving from `draining` to `unused` state faster
// default 300
// https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay
fargateService.targetGroup.setAttribute("deregistration_delay.timeout_seconds", "20");

fargateService.targetGroup.configureHealthCheck({
  path: "/ready",
  interval: Duration.seconds(30),
  healthyHttpCodes: "200",
  healthyThresholdCount: 2,
  unhealthyThresholdCount: 2,
  timeout: Duration.seconds(4),
});

Code snippet above does not work as expected. Despite we use healthCheckGracePeriod, it looks like traffic is let to instance immediately (despite /ready is not returning 200) making requests queue up until service can handle them. This leads to spikes in overall throughput of instances altogether (portion of incoming requests are suddenly hung on new instance which is just starting and initializing).

We tried to use slow_start.duration_seconds to ease up start of new instance like this:

// linearly increase number of requests by load balancer after instance starts
// https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#slow-start-mode
fargateService.targetGroup.setAttribute("slow_start.duration_seconds", "60");

... which helped, but it doesn't seem to be correct solution. Once instance is warmed up, it should be able to serve full load, we don't need to linearly increase traffic.

Apparently, there should be another way how to achieve desired behaviour using health check startPeriod, but I can't figure out how to set it using ApplicationLoadBalancedFargateService, nor I couldn't figure out how to use escape hatch and configure this property using CfnTaskDefinition.

Notice HealthCheck used by fargateService.targetGroup.configureHealthCheck is of type aws_elasticloadbalancingv2.HealthCheck and not aws_ecs.HealthCheck.

[wdolek]

Discovering more documentation, seems that startPeriod is not relevant for ELB (which we use with our Fargate)

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html

If you do not use an Elastic Load Balancing, we recommend that you use the startPeriod in the task definition health check parameters. For more information, see Health check.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.