Add conditional set-me in config

[Ajarmar]

Warning

This is public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request
• business confidential information, such as customer names

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• kind/adr

What does this PR do / why do we need this PR?

Adds conditional set-me config that will warn the user if the option is not set and the condition is fulfilled.

• Fixes [5] Improve config validation using conditional "set-me"s #1110

Additional information to reviewers

• The condition after set-me-if- has to be a valid yq4 condition. I added the parentheses because I felt it made it easier to read, but they are not strictly necessary unless there are multiple conditions.
• I tried adding this for secrets as well but it seems that set-mes in the secrets file are not validated at all currently, so it seemed out of scope for this task.
• I checked if it would be possible to add conditions to the json schemas added in this PR too, and it seems possible but it has to be manually added to the schemas (i.e. it can't just see that an option is conditional in the config and skip schema validation based on that). I didn't add it yet since that PR is not merged yet; if it merges before this one I could add it but otherwise I think it's more appropriate to create a separate task for it. Example for how such a conditional would look in the schema:

  properties:
    ingressNginx:
      properties:
        controller:
          properties:
            service:
              type: object
              if:
                properties:
                  enabled:
                    const: true
              then:
                properties:
                  type:
                    enum:
                      - ClusterIP
                      - NodePort
                      - LoadBalancer
                      - ExternalName
              properties:
                enabled:
                  type: boolean
                type:
                  type: string

• When the conditional is fulfilled, the option is changed to set-me only in the temporary config file, not in the actual config file. Not sure what the desired outcome would be here but this was more natural with how I implemented it, open for feedback on this.
• I didn't add any migration scripts for this as most set-me options should already be dealt with for existing environments, so it wouldn't make much of a difference. Open for feedback on this as well.

Screenshots

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • The logs do not show any errors after the change
• Network Policy checks:
  • Any changed pod is covered by Network Policies
  • The change does not cause any dropped packages in the NetworkPolicy Dashboard
• Pod Security Policy checks:
  • Any changed pod is covered by Pod Security Admission
  • Any changed pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Bug checks:
  • The bug fix is covered by regression tests

[robinAwallace]

I like this change 👍 Could you add unit tests for it? Then I would be happy to approve it 🙂

[Ajarmar]

I pushed unit tests for the conditional set-me functionality. I think some of them perhaps should be split up a bit more, but I wasn't sure about the exact granularity we wanted. Please take a look. (You can try running the tests with make unit/bin/conditional-set-me/main.)

I also noticed that some of the old tests are still failing, but I think I will not have time to fix those before I go on vacation.

[aarnq]

This looks good, though I get three "Do you want to continue" which is a bit too much, can we make it so that it is only printed once after conditionals, validate + schema validate?

[Zash]

I didn't add it yet since that PR is not merged yet

That PR has been merged.

[Ajarmar]

This looks good, though I get three "Do you want to continue" which is a bit too much, can we make it so that it is only printed once after conditionals, validate + schema validate?

I'm surprised you got three, I only got two (one after validate and one after schema validate). Now there should only be one, unless I missed the thing that made you get three.

Will work on the conditionals for the schema now.

[Zash]

LGTM

[aarnq]

Do the dashes need to be escaped?

[Ajarmar]

Maybe not. In any case, this function is not used anymore after the config template rework, so I'll just remove it.

[aarnq]

Or thanos.

[robinAwallace]

This looks very nice! Just one nit pick, but it LGTM! 🥳

[robinAwallace]

Nit: could you extract the common parts for sc/wc and run them after the if-statements?

Something like this?

Suggested change

	if [[ $1 == "sc" ]]; then
	check_config "${config_template_path}/common-config.yaml" \
	"${config_template_path}/sc-config.yaml" \
	"${config_template_path}/secrets.yaml"
	yq_merge "${config_template_path}/common-config.yaml" \
	"${config_template_path}/sc-config.yaml" \
	> "${template_file}"
	check_conditionals "${config[config_file_sc]}" "${template_file}"
	validate "${config[config_file_sc]}" "${template_file}"
	schema_validate "${config[config_file_sc]}" "${config_template_path}/schemas/config.yaml"
	check_conditionals "${secrets[secrets_file]}" "${config_template_path}/secrets.yaml"
	validate "${secrets[secrets_file]}" "${config_template_path}/secrets.yaml"
	schema_validate "${secrets[secrets_file]}" "${config_template_path}/schemas/secrets.yaml"
	elif [[ $1 == "wc" ]]; then
	if [[ $1 == "sc" ]]; then
	check_config "${config_template_path}/common-config.yaml" \
	"${config_template_path}/sc-config.yaml" \
	"${config_template_path}/secrets.yaml"
	yq_merge "${config_template_path}/common-config.yaml" \
	"${config_template_path}/sc-config.yaml" \
	> "${template_file}"
	config_to_validate="${config[config_file_sc]}"
	elif [[ $1 == "wc" ]]; then
	...
	else
	...
	fi
	check_conditionals "${config_to_validate}" "${template_file}"
	validate "${config_to_validate}" "${template_file}"
	schema_validate "${config_to_validate}" "${config_template_path}/schemas/config.yaml"
	check_conditionals "${secrets[secrets_file]}" "${config_template_path}/secrets.yaml"
	validate "${secrets[secrets_file]}" "${config_template_path}/secrets.yaml"
	schema_validate "${secrets[secrets_file]}" "${config_template_path}/schemas/secrets.yaml"