#layout(auto)

[nickpdemarco]

This (draft) PR makes significant changes to Zngur's output format in support of a new #layout(auto) feature.

Please see the description of the new output format in book/src/standalone_output.md, and the implementation of #layout(auto) in zngur-auto-layout/README.md.

At a high level, we're compile the user's crate to a .rlib that has size and offset constants baked into the object format under specific link names. This allows us to support cross-compiling workflows, since we don't need to execute a binary to determine these values. Most of the object file complexity is handled by the object crate, but it still is a significant increase in complexity for this project scope. That said, IMO the value add for #layout(auto) (with the obvious possibility of making this the default) is worth the complexity.

I hope to use this PR to get feedback on the architecture and hopefully simplify it without losing the benefits.

[HKalbasi]

The object format trick looks interesting, but I wonder if there were an easier solution. #58 uses -Z print-type-sizes which looks simpler, at least in terms of LOC, and there is also the rustdoc way. What is the benefit of this approach relative to the others?

About adding #layout(auto) and eventually making it the default, my idea was to keep the zng file simple for cases where there are challenges in integrating things with the build system, and introduce a second IDL (e.g. autozng) which integrates with the compiler and supports automatic layouts, method lists, trait implementation and more, that generates a simple zng file (or multiple ones, one per target). I also mentioned this in #58. We can eventually make the second IDL the default and discourage creating zng files by hand, treating them like a compiler artifact.

By the way, can you please clarify which parts of code/docs are AI generated, if any? I have no objections against using AI, I just want to know.

[nickpdemarco]

What is the benefit of this approach relative to the others?

This was mostly an experiment to see if layout(auto) was possible to implement using only stable Rust features. I also think it's somewhat simpler from a cache invalidation perspective, since tracking layout information in a sidecar file (I think that's what #58 does) is a bit more error prone. Here, we rely on cargo's (presumably robust) dependency tracking by embedding the layout info in the object file.

my idea was to keep the zng file simple for cases where there are challenges in integrating things with the build system, and introduce a second IDL (e.g. autozng) which integrates with the compiler and supports automatic layouts...

That seems reasonable. I could see this (or #58) easily changing to fit that model.

By the way, can you please clarify which parts of code/docs are AI generated, if any?

Most of this PR. But I've read/reviewed all of it.

[HKalbasi]

This was mostly an experiment to see if layout(auto) was possible to implement using only stable Rust features.

Making this work with stable compiler seems valuable. Will look at your approach more closely. Can we extend this approach to finding out available functions and modules using stable?

I also think it's somewhat simpler from a cache invalidation perspective, since tracking layout information in a sidecar file (I think that's what #58 does) is a bit more error prone. Here, we rely on cargo's (presumably robust) dependency tracking by embedding the layout info in the object file.

I think I didn't understand this part. If one can afford the compiler generation, they can generate the .zng file for every compilation and work directly with the .autozng file, treating .zng as an intermediate compiler artifact and cache it.

[nickpdemarco]

If one can afford the compiler generation, they can generate the .zng file for every compilation and work directly with the .autozng file, treating .zng as an intermediate compiler artifact and cache it.

Right - but then there's the (minor?) overhead of something like cargo:rerun-if-changed=main.autozng, whereas baking the layout info into the object file gets Cargo's cache invalidation automatically.