Switch to `go tool` for tool dependency management

[m1kola]

There is a proposal for go to receive support for tool dependency management which might land as soon as Go 1.22: golang/go#48429

Once it is available and our codebase is updated to the corresponding Go version we should switch to native support.

At the moment of writing we use github.com/bwplotka/bingo for tool pinning.

[yuriolisa]

Can I sign up for this one?

[m1kola]

@yuriolisa thanks for your interest in this. Unfortunately from golang/go#48429 it looks like we will have to wait a longer. As far as I understand this feature didn't make it into Go 1.22 and will likely not be release in 1.23.

[m1kola]

It seems like this will be released in 1.24 with chis about 2 months away: golang/go#48429 (comment)

[m1kola]

It seems like all the relevant changes got merged in Go and the issue is now closed. We should be able to use it once we upgrade to 1.24.

[m1kola]

Removed good first issue label because it is more complicated than it seems :)

I've been experimenting with 1.24 tool management in this PR.

It seems like it is possible to achieve what we currently have with bingo:

1. We can have a separate modfile for each tool so their dependencies won't conflict (as of today - our tools do have conflicts between themselves).
2. Go 1.24 will handle tool installation and caching between projects (in the same way it caches modules between projects).
3. We will be able to to continue using make the same way as we currently do

What will change:

1. Instead of bingo CLI we will have to use Go CLI. And this is where bingo is easier:

   # Can just run this in the project root
   bingo get sigs.k8s.io/controller-tools/cmd/controller-gen@new-version 
   

   versus go 1.24:

   # Have to specify path to to the right module
   go get -modfile tools/controller-gen.mod -tool sigs.k8s.io/controller-tools/cmd/controller-gen@new-version 
   

   We can probably do some makefile magic for this, but still.
   Also - if we accidentally run go mod tidy again one of these tools modfiles - go mod tidy will add a bunch of dependencies from the main module, I think.
2. Go 1.24 currently doesn't support build tags for tools. I think it is currently recommended to use go run for tools with build tags, but I haven't tried this out yet.

It is also possible to use tool management with the main go.mod in the repo root, but in this case we will have same dependency hell as with tools.go: there will be dependency conflicts between tools themselves and our codebase as everything will be sharing the same graph.

[m1kola]

1. We can have a separate modfile for each tool so their dependencies won't conflict (as of today - our tools do have conflicts between themselves).

It is also possible to use tool management with the main go.mod in the repo root, but in this case we will have same dependency hell as with tools.go: there will be dependency conflicts between tools themselves and our codebase as everything will be sharing the same graph.

Want to clarify these bits. When I'm referring to tools.go I mean the pattern where people create a go file with blank imports and with a build tag such as tools which is excluded from the build process of the main codebase by default.

This tricks go to add these dependencies into module's mod file. This works well (with some caveats which are not important here) when there are no conflicts between the codebase and tools. When there are some dependencies shared between codebase and tools - it might be a problem because there will be a single dependency graph between tools and the codebase. Or even if the tools have shared dependencies. Go will try to resolve dependencies, but in some cases it will throw hands up in the air. At the moment of writing, It is the case for our repos where we use a lot of dependencies from k8s and some of the tools such as kustomize also use them and can't agree on the version. Last week, when I was playing with it kustomize was conflicting with some other tool.

Anyway - looking into the future, Go 1.24 will help to avoid having to use tools.go pattern by adding a new directive into mod file and tooling around it.

However, by default Go uses module's go.mod file for managing tools. This means that it also shares dependency the graph between the codebase and all the tools. And we will have the same issue as with tools.go pattern here.

It is possible to specify a custom modfile for each tool and this is what I did in my experimental PR. This way codebase doesn't share the dependency graph with tools and tools between themselves do not share the dependency graph. It works in the same way as Bingo, but UX is not as nice because:

1. Requires developers to know which modfile to use for each tool or come up with tooling\scripts around it
2. There is currently no support for build tags for tools. There was a suggestion to use go run somewhere (can't find it right now)

See the message above for examples.

[camilamacedo86]

Hi @m1kola

Based on the above description, I understand that it would not be a good thing.
So, can we not close this one?

[m1kola]

As of Go 1.24 - I think it is probably better to keep bingo. After Go 1.24 release there was a lot feedback for go tool from the community. I haven't followed this space very closely, but I'm hopeful that go tool will evolve to better serve more use cases (including operator-controller's).

Feel free to close this for now.