Respect retryer set in ECS Waiters

[TopherGopher]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
The TasksRunning/TasksStopped waiters do not appear to respect the max retry configuration provided.

Version of AWS SDK for Go?
github.com/aws/aws-sdk-go-v2 v1.6.0
github.com/aws/aws-sdk-go-v2/service/ecs v1.5.0

Version of Go (go version)?
go1.15.3

To Reproduce (observed behavior)
In the following code snippet, you'll see that the max attempts is set to 1, yet the waiter does not respect that. A note that I've decreased the timeouts to almost nothing to exacerbate the issue. In production, we have these numbers set really high.

	ecsManager := ecs.NewFromConfig(ktm.AWSConf, func(options *ecs.Options) {
		options.Retryer = retry.AddWithMaxAttempts(retry.NewStandard(), 1)
	})
	startWaiter := ecs.NewTasksRunningWaiter(ecsManager)
	err := startWaiter.Wait(ctx, &ecs.DescribeTasksInput{
		Cluster: &ktm.Cluster,
		Tasks:   []string{ktt.TaskARN},
	}, time.Second, func(o *ecs.TasksRunningWaiterOptions) {
		o.LogWaitAttempts = true
		o.MinDelay = time.Nanosecond
		o.MaxDelay = time.Nanosecond
	})

Output:

SDK 2021/06/28 15:47:13 DEBUG attempting waiter request, attempt count: 1
SDK 2021/06/28 15:47:13 DEBUG attempting waiter request, attempt count: 2
SDK 2021/06/28 15:47:14 DEBUG attempting waiter request, attempt count: 3
SDK 2021/06/28 15:47:14 DEBUG attempting waiter request, attempt count: 4
SDK 2021/06/28 15:47:14 DEBUG attempting waiter request, attempt count: 5
SDK 2021/06/28 15:47:14 DEBUG attempting waiter request, attempt count: 6
SDK 2021/06/28 15:47:14 DEBUG attempting waiter request, attempt count: 7
time="2021-06-28T15:47:14-06:00" level=error msg="Error while waiting for task to start" err="exceeded max wait time for TasksRunning waiter" name=The_Joker src=ktecs

Expected behavior
With the above snippet, I would expect the retry max to be respected but it is not. Instead an internal number of some sort is consumed and that retry number does not appear to be alterable. I can see that the timeout is respected though, so that's good.

[jasdel]

Thanks for reaching out @TopherGopher. From your example it looks like you're looking to limit the number of attempts the TaskRunningWaiter checks for the target state, before giving up. The configuration of waiters in v2 abstracted out number of attempts, in favor the maximum duration to wait for.

The API client's Retryer operates on individual API operations calls. The waiter treats API operation calls as opaque across per wait attempt. This means that API operation retry attempt are evaluated separate from waiter wait attempts. The two's configuration do not overlap. An API operation can be retried the maximum number of retry attempts, and this will have no impact on the waiter's wait attempt, other than decrementing the total wait time.

The v2 SDK changed how it determines how long to run for, making the number of wait attempts an internal aspect of the waiter. Part of the reasoning behind this was that waiters can have significantly different wait time between attempt defined by the service team. From the application's perspective, the duration the waiter is running is generally the most important. Whereas the number of attempts the waiter makes is an internal behavior. This is the reason the v2 SDK's waiters take a wait duration in the function signature. The wait duration limits how long the waiter will run. Similar to a Context with timeout, but as a value the SDK can use directly without channels/goroutines.

If you're looking to ensure the waiter only makes a single attempt the TasksRunningWaiterOptions option struct has a member Retryable that takes a function to customize the waiters determination if the waiter is retryable or not. This function is separate from timing, and only operates on the response from the service.

If you're looking to control the number of attempts the waiter makes to something other than 1, this is a bit more complicated. Because, currently the implementation of the waiter retryable function is not exported. This means you wouldn't be able to wrap the SDK's internal wait retryable logic with your own. We're open to reviewing if these functions should stay private or not.

[TopherGopher]

The configuration of waiters in v2 abstracted out number of attempts, in favor the maximum duration to wait for.

• Ahhhh - I gotcha - so it will always respect the duration specified in the second argument over anything else

Thank you very much for that help @jasdel - much appreciated!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.