[DevSettings] Add support for selective Chef Attributes updates + mark cluster/slurm/reconfigure_timeout as updatable

[gmarciani]

Description of changes

Add support for selective ExtraChefAttributes updates.
Accept updates for the Chef attribute cluster/slurm/reconfigure_timeout.
The new Chef attribute is introduced in aws/aws-parallelcluster-cookbook#3087.

Use the new attribute in the existing cluster update executed by test_slurm, to validate that the update would succeed.

User Experience

[UC1] Setting reconfigure_timeout > 300, should succeed

# From
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"

# To
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"
  Cookbook:
    ExtraChefAttributes: |
      { "cluster" : { "slurm" : { "reconfigure_timeout": 600 } } }

# Result
{
  "cluster": {
    "clusterName": "simple-0107-3",
    "cloudformationStackStatus": "UPDATE_IN_PROGRESS",
    "cloudformationStackArn": "arn:aws:cloudformation:us-east-1:319414405305:stack/simple-0107-3/a00668f0-ec0d-11f0-b576-0affc7d57d0d",
    "region": "us-east-1",
    "version": "3.15.0",
    "clusterStatus": "UPDATE_IN_PROGRESS",
    "scheduler": {
      "type": "slurm"
    }
  },
  "changeSet": [
    {
      "parameter": "DevSettings.Cookbook.ExtraChefAttributes",
      "requestedValue": "{ \"cluster\" : { \"slurm\" : { \"reconfigure_timeout\": 600 } } }\n",
      "currentValue": "-"
    }
  ]
}

[UC2] Setting reconfigure_timeout to non integer value, should fail

# From
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"

# To
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"
  Cookbook:
    ExtraChefAttributes: |
      { "cluster" : { "slurm" : { "reconfigure_timeout":"xyz" } } }

# Result
{
  "configurationValidationErrors": [
    {
      "level": "ERROR",
      "type": "ExtraChefAttributesValidator",
      "message": "Invalid value in DevSettings/Cookbook/ExtraChefAttributes: attribute 'cluster/slurm/reconfigure_timeout' must be an integer."
    }
  ],
  "message": "Invalid cluster configuration."
}

[UC3] Setting reconfigure_timeout < 300, should fail

# From
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"

# To
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"
  Cookbook:
    ExtraChefAttributes: |
      { "cluster" : { "slurm" : { "reconfigure_timeout": 200 } } }

# Result
{
  "configurationValidationErrors": [
    {
      "level": "ERROR",
      "type": "ExtraChefAttributesValidator",
      "message": "Invalid value in DevSettings/Cookbook/ExtraChefAttributes: attribute 'cluster/slurm/reconfigure_timeout' must be greater than 300."
    }
  ],
  "message": "Invalid cluster configuration."
}

[UC4] Setting a non existing attribute, should fail

# From
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"

# To
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"
  Cookbook:
    ExtraChefAttributes: |
      { "cluster" : { "slurm" : { "reconfigure_timeoutX": 600 } } }

# Result
{
  "message": "Update failure",
  "updateValidationErrors": [
    {
      "parameter": "DevSettings.Cookbook.ExtraChefAttributes",
      "requestedValue": "{ \"cluster\" : { \"slurm\" : { \"reconfigure-timeoutX\": 600 } } }",
      "message": "The following ExtraChefAttributes fields cannot be updated: cluster.slurm.reconfigure_timeoutX. Revert the non-updatable ExtraChefAttributes fields to their original values.",
      "currentValue": "-"
    }
  ],
  "changeSet": [
    {
      "parameter": "DevSettings.Cookbook.ExtraChefAttributes",
      "requestedValue": "{ \"cluster\" : { \"slurm\" : { \"reconfigure_timeoutX\": 600 } } }",
      "currentValue": "-"
    }
 

[UC5] Setting a non updatable attribute, should fail

# From
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"

# To
DevSettings:
  AmiSearchFilters:
    Owner: "447714826191"
  Cookbook:
    ExtraChefAttributes: |
      { "cluster" : { "slurm" : { "reconfigure_timeout": 600 }, "in_place_update_on_fleet_enabled" : "false"} }

# Result
{
  "message": "Update failure",
  "updateValidationErrors": [
    {
      "parameter": "DevSettings.Cookbook.ExtraChefAttributes",
      "requestedValue": "{ \"cluster\" : { \"slurm\" : { \"reconfigure_timeout\": 600 }, \"in_place_update_on_fleet_enabled\" : \"false\"} }\n",
      "message": "The following ExtraChefAttributes fields cannot be updated: cluster.in_place_update_on_fleet_enabled. Revert the non-updatable ExtraChefAttributes fields to their original values.",
      "currentValue": "-"
    }
  ],
  "changeSet": [
    {
      "parameter": "DevSettings.Cookbook.ExtraChefAttributes",
      "requestedValue": "{ \"cluster\" : { \"slurm\" : { \"reconfigure_timeout\": 600 }, \"in_place_update_on_fleet_enabled\" : \"false\"} }\n",
      "currentValue": "-"
    }
  ]
}

Tests

• Unit tests (they cover the changes)
• Manually verified the user experience (see above UX)
• [ONGOING] E2E test test_slurm (using custom cookbook with changes from [DevSettings] Expose a new Chef attribute cluster/slurm/reconfigure_timeout. aws-parallelcluster-cookbook#3087), where we update the attribute cluster/slurm/reconfigure_timeout.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[hanwen-cluster]

Why is this change needed?

[gmarciani]

The IGNORE policy on a param basically says that the decision to accept/reject an update is on its subsection.
This is required to accept an update from a config where Cookbook section is missing to a config where there is Cookbook/ExtraChefAttributes with updatable attributes.

Example.

# From
DevSettings:
... (no CookbookSection)

# To
DevSettings:
  Cookbook:
    ExtrachefAttributes: |
      {"cluster": {"slurm": {"reconfigure_timeout": 600 }}}