Replacing wit/deps/

[lann]

With the addition of "nested packages" it is now possible to represent multiple packages in a single WIT file. We'd like to use this feature to set a new convention for the on-disk structure of WIT packages that are managed by wasm-pkg-tools:

• As a prerequisite, a "package lock" file will be introduced which will represent all of a project's (transitive) WIT package dependencies, e.g. wkg.lock (exact path TBD)
• The "locked" dependencies will be fetched and merged into a single wit/resolved.wit file (name to be bikeshod) with a boilerplate comment at the top e.g. "this file is generated; edits will be overwritten, etc"
• The resolved.wit file should be "deterministicishly" generated from the wit.lock, but will still likely be committed to source control in many cases to avoid typical registry auth / CI tooling issues. To help with the potential synchronization issue this can cause, there should be some kind of validation command e.g. wkg wit pull --check (?) to validate that the resolved.wit is up-to-date with the wit.lock.

[yoshuawuyts]

I'm curious if you could further elaborate why we need to check in a generated wit file if we also have a lock file? From what I gather it sounds like under this design each dependency will be referenced three times in each project:

• once by-version when imported
• once more in the lock file
• a third time in the generated deps file

In Node.js we used to vendor deps before lock files were added. But that basically stopped being necessary after lock files were added. So I guess I'm wondering what is different in this model that we would need both a lock file and vendoring?

[yoshuawuyts]

Okay, so what you're saying is that this is basically just a formalization of how wit-deps already works today - but probably works a little better since we now have nested packages? Am I reading that right?

[macovedj]

In the Packaging SIG yesterday there was a bit of a discussion about how anything that can be represented by a deps folder can now be represented with nested packages. There are also people who find the deps folder difficult to work with, and like the idea of everything being in a single file. I don't think a final decision was made, but it was certainly considered that potentially we should recommend that various tools transition at whatever rate makes sense to prefer using nested packages over a deps folder. Hope that context helps!

[lann]

I should also add that this proposal is only about generic wit tooling (i.e. wkg wit or wherever that ends up living). Language-specific tools like cargo-component or jco may choose to follow this convention or may choose to do something completely different.

In contrast, we do want to standardize the "package lock" file across toolchains.

[lukewagner]

Agreed that it's not clear whether the generated WIT file should or shouldn't be checked in by convention (I can think of arguments in both directions.)

But just to check if the following aspects of the proposal are uncontroversial:

• Once there has been a depsolve that produces a lock file and the WIT package binaries have been fetched, a single-file WIT is generated (deterministically) from these package binaries as the next step of the overall build pipeline.
• This generated WIT file takes the place of the deps/ directory. Even if the WIT file is not checked in, it is generated and stored somewhere in the local project directory in a predictable location so that the developer can find it to see the "ground truth" of the world they are compiling against.
• This generated WIT file, an output directory, and a target world name (which itself could be optional if we add unnamed worlds) are the sole inputs to wit-bindgen and thus wit-bindgen doesn't need to know anything about registries, directory structure, the current directory, wit/, or deps/.

Does anything sound off about any of that?

[lukewagner]

One thing that wasn't touched on in the above proposal is the relation between the generated WIT file and any user-authored WIT. My understanding is that there are two cases:

• The developer is authoring a component that targets a world already published somewhere (e.g. wasi:http/proxy) and so there is no user-authored WIT (just the registry-resolved name of the world).
• The developer authors some WIT that is not published to a registry but is instead committed along with the source code (and thus the name of this user-authored world is irrelevant, making it a good use case for unnamed worlds). However, this user-authored WIT may refer to registry-published interfaces or world names (via import, export, use or include).

In the former case, I was imagining there is a wkg wit command that takes a world name and then spits out the generated single-file WIT file containing the named world as a "root" from which all transitive dependencies are found and included.

In the latter case, I imagine that there is a different wkg wit command that takes the filename of the user-authored world and a (one-day optional) world name and then spits out the generated single-file WIT file containing the user-authored world as the "root" from which all transitive dependencies are found and included.

Does that sound right?

[lann]

I can see the generic workflow having two separate steps:

• Any time you update dependencies, you regenerate the lock file and wit/resolved.wit
  • This could happen explicitly (wkg wit resolve?) or implicitly (wkg wit add <dependency>?)
• In order to produce the actual input to wit-bindgen et al, you would further bundle everything in wit/*.wit, which would include that resolved.wit and/or user-authored WIT(s).
  • It might make sense to keep this simpler "glob a single directory" bundling step in wit-bindgen itself

[lukewagner]

Oh, I see. So is the idea that, if you have a user-defined world, you end up with two WIT files: the one you wrote and the generated one with all your WIT dependencies and then you feed them both into wit-bindgen?