Skip to content

✨ Spot instances fallback - WaitingTimeout #5062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

✨ Spot instances fallback - WaitingTimeout #5062

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a70a04
api: add WaitingTimeout in SpotMarketOptions
Prucek Jul 18, 2024
56e58ed
ec2: implement spot waiting timeout logic
Prucek Jul 18, 2024
433ce71
docs: update spot with WaitingTimeout
Prucek Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions api/v1beta1/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ limitations under the License.
package v1beta1

import (
"time"

"k8s.io/apimachinery/pkg/util/sets"

clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
Expand Down Expand Up @@ -280,6 +282,9 @@ type SpotMarketOptions struct {
// +optional
// +kubebuilder:validation:pattern="^[0-9]+(\.[0-9]+)?$"
MaxPrice *string `json:"maxPrice,omitempty"`
// WaitingTimeout defines the maximum time the user is willing to wait for acquiring a Spot VM instance, after that it will fallback to a regular on-demand instance.
// +optional
WaitingTimeout time.Duration `json:"waitingTimeout,omitempty"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new field only needs to be added to the latest API version, so v1beta2.

Please can you remove it from here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a "roundtripping" fuzzy test that will complain but you can "store" and "restore" the value.

}

// EKSAMILookupType specifies which AWS AMI to use for a AWSMachine and AWSMachinePool.
Expand Down
2 changes: 2 additions & 0 deletions api/v1beta1/zz_generated.conversion.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions api/v1beta2/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package v1beta2

import (
"strings"
"time"

"k8s.io/apimachinery/pkg/util/sets"

Expand Down Expand Up @@ -426,6 +427,9 @@ type SpotMarketOptions struct {
// +optional
// +kubebuilder:validation:pattern="^[0-9]+(\.[0-9]+)?$"
MaxPrice *string `json:"maxPrice,omitempty"`
// WaitingTimeout defines the maximum time the user is willing to wait for acquiring a Spot VM instance, after that it will fallback to a regular on-demand instance.
// +optional
WaitingTimeout time.Duration `json:"waitingTimeout,omitempty"`
}

// EKSAMILookupType specifies which AWS AMI to use for a AWSMachine and AWSMachinePool.
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/controlplane.cluster.x-k8s.io_awsmanagedcontrolplanes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1378,6 +1378,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: The name of the SSH key pair.
Expand Down Expand Up @@ -3439,6 +3445,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: The name of the SSH key pair.
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: The name of the SSH key pair.
Expand Down Expand Up @@ -2353,6 +2359,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: The name of the SSH key pair.
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachinepools.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: |-
Expand Down Expand Up @@ -851,6 +857,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: |-
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,12 @@ spec:
description: MaxPrice defines the maximum price the user is willing
to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after that
it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: SSHKeyName is the name of the ssh key to attach to the
Expand Down Expand Up @@ -1047,6 +1053,12 @@ spec:
description: MaxPrice defines the maximum price the user is willing
to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after that
it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: SSHKeyName is the name of the ssh key to attach to the
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachinetemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,12 @@ spec:
description: MaxPrice defines the maximum price the user
is willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the
user is willing to wait for acquiring a Spot VM instance,
after that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: SSHKeyName is the name of the ssh key to attach
Expand Down Expand Up @@ -981,6 +987,12 @@ spec:
description: MaxPrice defines the maximum price the user
is willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the
user is willing to wait for acquiring a Spot VM instance,
after that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: SSHKeyName is the name of the ssh key to attach
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/infrastructure.cluster.x-k8s.io_awsmanagedmachinepools.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: |-
Expand Down Expand Up @@ -847,6 +853,12 @@ spec:
description: MaxPrice defines the maximum price the user is
willing to pay for Spot VM instances
type: string
waitingTimeout:
description: WaitingTimeout defines the maximum time the user
is willing to wait for acquiring a Spot VM instance, after
that it will fallback to a regular on-demand instance.
format: int64
type: integer
type: object
sshKeyName:
description: |-
Expand Down
9 changes: 9 additions & 0 deletions docs/book/src/topics/spot-instances.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ spec:
instanceType: ${AWS_NODE_MACHINE_TYPE}
spotMarketOptions:
maxPrice: ""
waitingTimeout: ""
sshKeyName: ${AWS_SSH_KEY_NAME}
```
Users may also add a `maxPrice` to the options to limit the maximum spend for the instance. It is however, recommended not to set a maxPrice as AWS will cap your spending at the on-demand price if this field is left empty, and you will experience fewer interruptions.
Expand All @@ -35,6 +36,14 @@ spec:
maxPrice: 0.02 # Price in USD per hour (up to 5 decimal places)
```

Users may also add a `waitingTimeout` to the options to limit waiting time for the Spot instance. After exceeding this timeout a regular ec2 instance will be acquired.
```yaml
spec:
template:
spotMarketOptions:
waitingTimeout: "10m" # Time in minutes
```

## Using Spot Instances with AWSManagedMachinePool
To use spot instance in EKS managed node groups for a EKS cluster, set `capacityType` to `spot` in `AWSManagedMachinePool`.
```yaml
Expand Down
26 changes: 23 additions & 3 deletions pkg/cloud/services/ec2/instances.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/utils/ptr"

infrav1 "sigs.k8s.io/cluster-api-provider-aws/v2/api/v1beta2"
Expand Down Expand Up @@ -663,9 +664,28 @@ func (s *Service) runInstance(role string, i *infrav1.Instance) (*infrav1.Instan
}
}

out, err := s.EC2Client.RunInstancesWithContext(context.TODO(), input)
if err != nil {
return nil, errors.Wrap(err, "failed to run instance")
var out *ec2.Reservation
var err error
if i.SpotMarketOptions != nil && i.SpotMarketOptions.WaitingTimeout > 0 {
pollError := wait.PollUntilContextCancel(context.TODO(), i.SpotMarketOptions.WaitingTimeout, true, func(ctx context.Context) (bool, error) {
out, err = s.EC2Client.RunInstancesWithContext(ctx, input)
if err != nil {
return false, errors.Wrap(err, "failed to run instance")
}
return true, nil
})
if pollError != nil {
input.InstanceMarketOptions = nil
out, err = s.EC2Client.RunInstancesWithContext(context.TODO(), input)
if err != nil {
return nil, errors.Wrap(err, "failed to run instance")
}
}
} else {
out, err = s.EC2Client.RunInstancesWithContext(context.TODO(), input)
if err != nil {
return nil, errors.Wrap(err, "failed to run instance")
}
}

if len(out.Instances) == 0 {
Expand Down
Loading