[DOC] - conda_store resource settings appear to be invalid

[satra]

Preliminary Checks

• This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/nebari-dev/nebari/discussions

Summary

This section doesn't work with the current schema and tf file:

https://www.nebari.dev/docs/explanations/custom-overrides-configuration#conda-store-worker

[viniciusdc]

Hey @satra thanks for opening this, indeed this is incorrect. Here is a breakdown of the current supported variables:
https://github.com/nebari-dev/nebari/blob/d680ca8294b9f29e38dbffb93a2c7f5e9188393d/src/_nebari/stages/kubernetes_services/__init__.py#L242-L248
I need to find why that was pushed to our docs, but in the meantime, we will be including that option soon through this PR nebari-dev/nebari#3094
Regarding the allocation of resources to conda-store, right now the central pod that hosts the worker does not have any limits on its resources, meaning that it will use whatever resources are available from the node (unitol Kubernetes considers that pod harmful, which could lead to eviction)

[satra]

thanks @viniciusdc - happy to try the PR if you think it's stable enough for trying.

[viniciusdc]

I have just updated the schema to be easier to expand in the future. I will be testing this during the afternoon. However, I would recommend holding off on testing this if you plan to push to your direct production cluster, as this would require installing from main, which could lead to other issues, and rolling back would be difficult. We should have a new release next week, and I would like to include those updates in it. But can't promise

If you are eager to test how this would "work", you can modify this file under your stages folder:

https://github.com/nebari-dev/nebari/blob/d680ca8294b9f29e38dbffb93a2c7f5e9188393d/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/worker.tf#L68-L137

That PR will allow overriding the number of replicas, the node selector, and include the missing resources block in the deployment. You can change those manually in that file e.g

container {
          name  = "conda-store-worker"
          image = "${var.conda-store-image}:${var.conda-store-image-tag}"

          args = [
            "conda-store-worker",
            "--config",
            "/etc/conda-store/conda_store_config.py"
          ]
	...
	resources {
              limits = {
                cpu    = "2"
                memory = "4Gi"
              }
              requests = {
                cpu    = "1"
                memory = "2Gi"
          }
	...
}

Same with the selector:

node_selector_term {
                match_expressions {
                  key      = var.node-group.key
                  operator = "In"
                  values = [
                    "worker"
                  ]
                }
              }

then when deploying use --disable-render to avoid overrides from the nebari internal logic:

nebari deploy -c nebari-config.yaml --disable-render