[feature] A more effective go dependency model.

[kcross-ctoken]

Currently the golang dependency model is based around the mod files.
While this is not incorrect it is not very effective.
Golang modules are really just a external dependency list but when working with a monorepo everything is internal.
Go does its internal dependency checking using
pacakges -> object files -> binaries dependency tree.
This works well for testing because the test binary can consist of different object files so it knows what to build and if it needs to retest according to the package-> object file cache it has.

This enables go to have 2 different dependency trees from a single mod file

1. testing (test binary)
2. application/executable

Currently we can not model this in moon so We are having to artifically breakup all our packages into mod files to enable moon to properly detect what needs to be released.

Im going to suggest here that the golang dependency plugin goes a step further and looks at each package in the project and which binaries need them then work out what needs to be tested for a test binary ... or released for a release binary. The two dependency lists are not the same.

[milesj]

What does "look at each package" mean exactly? Besides mod files, what other metadata is there?

[kcross-ctoken]

well the thing is that go can use its ast directly.
I bumped this out in a couple hours
https://github.com/kcross-ctoken/go-depend
This lists the test package dependencies or the binary package dependencies.
We could put this into the wasm plugin to work out what to build or test ... also more importantly what to release.

[milesj]

This requires parsing every .go source file right? That's a huge upfront cost that moon would have to constantly run.

This is probably something that requires a daemon. This is why we only parse manifest/lockfiles.

[kcross-ctoken]

You would be surprised on how quick this is ... like miliseconds for a large codebase.
Go optimised its language for parsing so its blindingly fast.
So fast that we have go run ... which pretends its a scripting language but it really just compiles everything up front.

[milesj]

Right but that's when it's written in Go. It needs to be written in Rust for moon, which means we would have to build our own Go AST parser.

[kcross-ctoken]

hmm I thought we were using webasm for it ... which means you just need to recompile the ast parser into webasm.
also we could cache the results ... just like a task.

[milesj]

The toolchains are WASM plugins, but the Go toolchain source is written in Rust here: https://github.com/moonrepo/plugins/tree/master/toolchains/go/src

So the AST parsing would need to be implemented there.

[kcross-ctoken]

hmm ok why not write the go toolchain in go and compile it to wasm ... seems like a odd or more likely time constrained choice.
Go's std lib parsing and ast is very good. Basically the whole build chain is available programmatically in go.
Suppose I could look at porting that over to golang wasm.

[milesj]

Because you would have to reimplement all the moon PDK/WASM APIs in Go, and it's just not worth it as a non-Go developer.

Is there no go subcommand that returns deps/AST info?

[kcross-ctoken]

hmmm ... not sure let me check

[kcross-ctoken]

BTW the pdk stuff is already availble in go
https://github.com/extism/go-pdk

[milesj]

That's just for the extism piece, there's also all this:

• https://github.com/moonrepo/moon/tree/master/crates/pdk
• https://github.com/moonrepo/moon/tree/master/crates/pdk-api
• https://github.com/moonrepo/moon/tree/master/crates/pdk-test-utils

[kcross-ctoken]

hmmm ... im kind interested in this area. Go -> wasm and deps.
So I might try a side project around this to do a wasm golang deps plugin in wasm.

Anyway.
Thanks 😄.
Ive liked my moon experience once I got the deps stuff done. I just think it needs a more integrated dependency with golang to really make a game changer for golang mono repo projects.

[milesj]

moon has been designed so that all of the built-in toolchains can be overridden, so if you want to build your own Go toolchain in Go, that's totally possible.