terraform test: New version with local helm chart support

[def-]

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[def-]

@bobbyiliev I'm wondering why this doesn't work: https://buildkite.com/materialize/test/builds/99223#01951a82-de29-48b1-979f-897baf998e06

error: getting semver of materialize: db error: ERROR: CLUSTER "mz_system" has no replicas available to service request

I'm explicitly setting the defaultReplicationFactor in helm_values, I assume that should work together with use_local_helm_chart.
I'm passing in the operator version here: https://github.com/MaterializeInc/materialize/pull/31527/files#diff-83d8f0169fbca62050b7506c763a1ce53697a5979da1d3759d83cb5a48e3f3eaR163 but I'm actually wondering if that will use the operator from the current source code how I could specify that.

This is somehow related to #31452, but I don't understand how we can both use a new operator version that sets the defaults to 0, but doesn't accept me to override them. cc @SangJunBak maybe you have an idea too

[SangJunBak]

@def- from first glance, it seems like defaultReplicationFactor isn't scoped properly! If you look in values.yaml, it's scoped as clusters.defaultReplicationFactor. I am curious... If we specify clusters.defaultReplicationFactor, would we need to specify clusters.sizes and other variables in the clusters scope?

[def-]

I tried that but no difference still:

error: getting semver of materialize: db error: ERROR: CLUSTER "mz_system" has no replicas available to service request

[bobbyiliev]

The clusters section is under the operator key in the values file:

https://github.com/MaterializeInc/materialize/blob/main/misc/helm-charts/operator/values.yaml#L11

[def-]

So is it true that if I set anything in operator I have to override everything?

[bobbyiliev]

No, this should not be the case. It will override only the values that you are modifying:

https://helm.sh/docs/chart_template_guide/subcharts_and_globals/#overriding-values-from-a-parent-chart

Are you seeing a different behavior?

[def-]

I didn't get it into a working state, I'm not sure what I'm doing wrong with the helm_values or if they don't work. I just manually alter the cluster now and that works for me. So can be merged in this state.