[cinder-csi-plugin] Update ignore-volume-az docs

[stephenfin]

What this PR does / why we need it:

This option is only useful if you are manually configuring volume AZs using the topology Storage Class parameter. Update docs to reflect this and suggest (likely better) alternatives for working around AZ mismatches.

Which issue this PR fixes(if applicable):
fixes #2185

Special notes for reviewers:

#2865 makes the topology code easier (IMO) to grok, if anyone doubts my testing notes from #2185 (comment).

Release note:

NONE

[mandre]

This is now much clearer what ignore-volume-az does. Also nice to have instructions for what to do when pods won't schedule in multi-AZ environments.
/lgtm

[kayrus]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kayrus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kayrus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rptaylor]

@stephenfin thanks for this clarification!
Do you know how this relates to the --feature-gates=Topology=false setting that is applied by the Helm chart?
https://github.com/kubernetes/cloud-provider-openstack/blob/master/charts/cinder-csi-plugin/templates/controllerplugin-deployment.yaml#L72

Mentioned here: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/features.md#topology

If you have --feature-gates=Topology=false is it still necessary to also set --with-topology=false for the node and controller plugins (.Values.csi.plugin.controllerPlugin.extraArgs, .Values.csi.plugin.nodePlugin.extraArgs) ?

[stephenfin]

@stephenfin thanks for this clarification! Do you know how this relates to the --feature-gates=Topology=false setting that is applied by the Helm chart? master/charts/cinder-csi-plugin/templates/controllerplugin-deployment.yaml#L72

Mentioned here: master/docs/cinder-csi-plugin/features.md#topology

If you have --feature-gates=Topology=false is it still necessary to also set --with-topology=false for the node and controller plugins (.Values.csi.plugin.controllerPlugin.extraArgs, .Values.csi.plugin.nodePlugin.extraArgs) ?

No, it's not. Setting that to false disables it at the provisioning stage. However, note that feature flags are not kept around forever and this will eventually be removed from the provisioner. This was the rationale for adding the --with-topology flag. When it is, these charts will break. We likely need to update them sooner rather than later.