Skip to content

Make subscriptionID mutable #4612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Make subscriptionID mutable #4612

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d8d3acc
Make subscriptionID mutable
dlipovetsky Mar 1, 2024
e8a948c
Make subscriptionID immutable in AzureManagedControlPlane
dlipovetsky Mar 12, 2024
46bfbaf
Make AzureCluster subscriptionID conditionally mutable
dlipovetsky Mar 12, 2024
b9b52e5
fixup! Make AzureCluster subscriptionID conditionally mutable
dlipovetsky Mar 13, 2024
7ddaaa9
fixup! Make AzureCluster subscriptionID conditionally mutable
dlipovetsky Mar 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions api/v1beta1/azurecluster_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,6 @@ func (c *AzureCluster) ValidateUpdate(oldRaw runtime.Object) (admission.Warnings
allErrs = append(allErrs, err)
}

if err := webhookutils.ValidateImmutable(
Copy link
Contributor

@jackfrancis jackfrancis Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than allow mutability, should we simply allow changes from "" to "<non-empty string>"

Copy link
Contributor

@nojnhuh nojnhuh Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other option might be to bring back the global config only for sub ID.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

That allows the field to be updated at most one time.

What happens if the value is wrong, e.g. by mistake?

Can the user fix the mistake? For example, can they pause the Cluster, remove the AzureCluster resource (removing all finalizers by hand!), fix the subscriptionID, and re-create it?

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other option might be to bring back the global config only for sub ID.

I think we're discussing whether making this field mutable introduces risk to users.

Let's please remember that the global config is (and always has been) mutable.

So, making the field write-many does not introduce more risk for users.

Moreover, making the field write-once allows users to migrate away from the write-many global config, actually reducing risk.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other information can I provide to help us decide between write-once and write-many?

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the goals are:

  1. Allow users who previously relied on the manager credential bootstrap for the subscription ID to migrate to CAPZ that no longer uses this credential.
  2. Prevent users from inadvertently changing the subscription ID.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a solution that meets both (1) and (2) perfectly.

  • Write-once is a problem for (1) if the user makes a mistake, but helps with (2).
  • Write-many works for (1), but does not help with (2).

How about something unusual, like "conditional" mutability. For example, if the user wants to change subscriptionID, they have to first set an annotation on AzureCluster. When the annotation is present, subscriptionID is mutable. After the user sets subscriptionID, they remove the annotation. At that point, subscriptionID is again immutable.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above idea is in 46bfbaf.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have consensus, I'll of course update the docs in this PR.

field.NewPath("Spec", "SubscriptionID"),
old.Spec.SubscriptionID,
c.Spec.SubscriptionID); err != nil {
allErrs = append(allErrs, err)
}

if err := webhookutils.ValidateImmutable(
field.NewPath("Spec", "Location"),
old.Spec.Location,
Expand Down
1 change: 0 additions & 1 deletion api/v1beta1/azuremanagedcontrolplane_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@ func (mw *azureManagedControlPlaneWebhook) ValidateUpdate(ctx context.Context, o
old interface{}
new interface{}
}{
{field.NewPath("Spec", "SubscriptionID"), old.Spec.SubscriptionID, m.Spec.SubscriptionID},
{field.NewPath("Spec", "ResourceGroupName"), old.Spec.ResourceGroupName, m.Spec.ResourceGroupName},
{field.NewPath("Spec", "NodeResourceGroupName"), old.Spec.NodeResourceGroupName, m.Spec.NodeResourceGroupName},
{field.NewPath("Spec", "Location"), old.Spec.Location, m.Spec.Location},
Expand Down
4 changes: 2 additions & 2 deletions api/v1beta1/azuremanagedcontrolplane_webhook_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1930,7 +1930,7 @@ func TestAzureManagedControlPlane_ValidateUpdate(t *testing.T) {
wantErr: false,
},
{
name: "AzureManagedControlPlane SubscriptionID is immutable",
name: "AzureManagedControlPlane SubscriptionID is mutable",
oldAMCP: &AzureManagedControlPlane{
Spec: AzureManagedControlPlaneSpec{
AzureManagedControlPlaneClassSpec: AzureManagedControlPlaneClassSpec{
Expand All @@ -1949,7 +1949,7 @@ func TestAzureManagedControlPlane_ValidateUpdate(t *testing.T) {
},
},
},
wantErr: true,
wantErr: false,
},
{
name: "AzureManagedControlPlane ResourceGroupName is immutable",
Expand Down
6 changes: 0 additions & 6 deletions api/v1beta1/azuremanagedcontrolplanetemplate_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,12 +88,6 @@ func (mcpw *azureManagedControlPlaneTemplateWebhook) ValidateUpdate(ctx context.
if !ok {
return nil, apierrors.NewBadRequest("expected an AzureManagedControlPlaneTemplate")
}
if err := webhookutils.ValidateImmutable(
Copy link
Contributor

@jackfrancis jackfrancis Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nojnhuh what would the practical side-effect be of actually changing from one sub id to another (see my comment above about potentially scoping the change to just "from empty string" changes)?

The sub id is not part of the actual ManagedCluster CRD, so would CAPZ simply create a new AKS cluster in the new sub, assuming that the existing Azure creds are valid w/ equivalent privileges across both subscriptions?

Ref: https://azure.github.io/azure-service-operator/reference/containerservice/v1api20231102preview/#containerservice.azure.com/v1api20231102preview.ManagedCluster_Spec

Copy link
Contributor

@nojnhuh nojnhuh Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I suspect would happen.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, from my experience, CAPZ will fail to reconcile an AzureManagedControlPlane with an empty subscriptionID (although the field is optional in the API!).

capz-controller-manager-667d8ff489-drt8x E0208 00:25:43.520687       1 controller.go:329]  "msg"="Reconciler error" "error"="error creating AzureManagedControlPlane default/dlaksdkp27: failed to reconcile AzureManagedControlPlane service managedcluster: failed to create resource dlaksdkp27/dlaksdkp27 (service: managedcluster): Code=\"LinkedAuthorizationFailed\" Message=\"The client has permission to perform action 'Microsoft.Network/virtualNetworks/subnets/join/action' on scope '/subscriptions/REDACTED/resourceGroups/dlaksdkp27/providers/Microsoft.ContainerService/managedClusters/dlaksdkp27', however the linked subscription 'resourceGroups' was not found. \"" "AzureManagedControlPlane"={"name":"dlaksdkp27","namespace":"default"} "controller"="azuremanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AzureManagedControlPlane" "name"="dlaksdkp27" "namespace"="default" "reconcileID"="REDACTED"

I thought it would make sense to make subscriptionID mutable everywhere, but if we're concerned about behavior for AzureManagedControlPlane specifically, I think the field could remain immutable, because an empty (or invalid) value never worked to begin with.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since AzureManagedControlPlane can not be successfully reconciled when subscriptionID is empty, I don't see a need to make it mutable in that resource. I'll update the PR to make it immutable.

Copy link
Contributor Author

@dlipovetsky dlipovetsky Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e8a948c

field.NewPath("Spec", "Template", "Spec", "SubscriptionID"),
old.Spec.Template.Spec.SubscriptionID,
mcp.Spec.Template.Spec.SubscriptionID); err != nil {
allErrs = append(allErrs, err)
}

if err := webhookutils.ValidateImmutable(
field.NewPath("Spec", "Template", "Spec", "Location"),
Expand Down
4 changes: 2 additions & 2 deletions api/v1beta1/azuremanagedcontrolplanetemplate_webhook_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,14 @@ func TestControlPlaneTemplateUpdateWebhook(t *testing.T) {
wantErr: false,
},
{
name: "azuremanagedcontrolplanetemplate subscriptionID is immutable",
name: "azuremanagedcontrolplanetemplate subscriptionID is mutable",
oldControlPlaneTemplate: getAzureManagedControlPlaneTemplate(func(cpt *AzureManagedControlPlaneTemplate) {
cpt.Spec.Template.Spec.SubscriptionID = "foo"
}),
controlPlaneTemplate: getAzureManagedControlPlaneTemplate(func(cpt *AzureManagedControlPlaneTemplate) {
cpt.Spec.Template.Spec.SubscriptionID = "bar"
}),
wantErr: true,
wantErr: false,
},
{
name: "azuremanagedcontrolplanetemplate location is immutable",
Expand Down