jsonnet support for cluster templates

[fsgh42]

I'd like to use jsonnet for cluster templates.

I already got this to work here: main...fsgh42:omni:feat/jsonnet-support
I'd like to ask if support is desired and if I should create a PR for this?
Might solve #1750, at least partially.

If support is desired, you can implement this dumb or thoroughly, the commit I linked shows the dumb way.
I'd prefer the thorough way and already gave it a shot - but that blows up the size of the change, as aside from the required code changes you need to change/add tests. It adds the ability to add more file formats in the future, though.

[smira]

I think this is cool, I would like to have a wider discussion on what is the approach we would like to have in the end.

E.g. do we want jsonnet or something else, support multiple, or allow Go templates, etc.

[fsgh42]

I am aware jsonnet is not everyone's darling, but I'll try to make a case for it:

• templating: jsonnet itself has a layered approach and all templating is handled by the language itself, e.g. omnictl wouldn't have to take care of any of this. How templates are structured is left to the user, they can be arbitrarily complex. All we enforce is that what is surfaced at rendering time matches the spec of cluster template objects.
• extensible: go-jsonnet allows for implementing functions that can be called from jsonnet like std.native("funcName")(param1, param2, ... ). If we ever want to add something like this, we can.
• rendered result is always compatible to YAML: Every JSON document rendered is immediately valid YAML. We can add this on top of what we already have without changing any existing code logic.
• It solves the issue of inline cluster manifests, as data can be loaded via import or importstr by jsonnet. Adding to the native functions argument, in another project I implemented native rendering of helm charts, which would enable users to do something like this: std.native("renderHelm")(chartRepo, chartName, chartVersion) and directly embed this into cluster templates. This essentially requires us to compile in the helm binary, not sure if we care. Just listing it as a potential argument. Calling external binaries is also possible that way, of course, even though it is against the idempotent concept of jsonnet.
• good reputation and battle tested: grafana tanka is a similar project, that solves a similar issue on the kubectl level.
• jsonnet it is a "true" data templating language, not just string substitution.

The only downside I see really is that the go-jsonnet project has not seen many releases after it's completion. It does not appear completely inactive, though.

[smira]

My point wasn't that jsonnet is bad, we can support several solutions, but rather the overall direction is not clear for me when it comes to details. If we get something in, we can't take it out, so we need to think before making any change.

[xvzf]

We've started using jsonnet to template the configuration before running a cluster template sync. In an initial MVP we planned on using kustomize to generate all resources but were hindered by the lack of apiVersion, kind and metadata.name fields not being available on all resources.

@smira it might be great to start supporting the KRM as general schema. This way, commodity tooling (be it Kustomize, Helm or even Tanka) can be used to generate plain YAML which is then piped into the omnictl cluster template.

Might be too much to ask for, but a kubernetes-compatible API would be gold and allow gitops control patterns out of the box with e.g. fluxcd. Thinking of e.g. https://github.com/kcp-dev/kcp here ;-)