How does Carbon plan to handle shared/dynamic libraries and their interfaces (headers/module files in C++)

[LouChiSoft]

Hi, I'm hoping this is the right place to discuss it but I am happy to be redirected if it's not. I've been keeping an eye on Carbon from a distance and this would be my first proper in-depth discussion about something within the language.

I'm here because I feel like when it comes to making and sharing dynamic libraries I don't think any language has quite got it right. For some background most of my opinion comes from my professional work where I work with confidential IP so any code we share has to be in the form of fully compiled code. But some of it will apply to more generic situations as well.

I mostly work in C++ where creating a shared binary ends up being more complex than I feel like it needs to be. There are three main issues I have:

• You end up in a situation where you have to mark functions with various macros for visibility which also change between platforms.
• Then if you work with confidential IP like I do you might end up having to wrap your C++ code with a C API and use pointers since without it you would have to provide a header with the full class declaration in it with private members included, which we obviously can't do.
• Finally the last issue I have, which is minor but still annoying any time it happens, is that by having separate interfaces and implementations there is an overhead of ensuring that any change to a function handle is made to both files.

So here is my idea/proposal to improve the process of making shared libraries (Disclaimer: not a compiler engineer, what I suggest might not be possible. These are just ideas from a C++ developer):

• When compiling your source code, the compiler is going to know which classes, functions, etc. are going to be visible outside the shared library. This map of public classes and functions could then be used to generate some kind of interface structure. That structure can then be used when importing the library. This solves or otherwise prevents point 1 and 3 by knowing that the interface matches the compiled code. And if when you use it you don't see a function that you should be able to see then you know you have made a mistake in the creation of the library itself
• Enable the "hiding" of private data in classes. We use pointers to hide this data, but if there was a way to split a class into "public" and "private" sections that the compiler knows about when a consumer of the library instantiates the class but otherwise obfuscates the private information from the user. I realise that this one would have ABI implications, and I think I remember seeing that Carbon is going to implement some kind of ABI for boundaries between libraries and executables, so maybe this is something that could be considered. This would solve point 2, or most of the issues that I have personally ran into with point 2 and also enable more complex API designs for libraries where needed.
• Finally, while not specifically tied to any 1 point from the issues list. Embed this information in the library itself. If a Carbon written/compiled library had this information embedded as functions that information can then be used elsewhere for ease of consumption. Some examples would be during compilation there would be a pre-compiled list of available APIs that the compiler can check before returning an invalid name error. Similarly an LSP or some other kind of IntelliSense can look through the list of libraries being linked and if they are a Carbon compiled library take the libraries interface and make it available to the user. And finally, it could be used for runtime loading of libraries. I don't know if Carbon plans on supporting runtime loading of libraries in the standard library. But it could be an advantage over the C++ standard library if it did and would also be able to call __GET__LIBRARY__API__() or something similar when you load it and the user would be able to print a list of classes and functions you can use now it's loaded. Combine this with clever generics work and you could probably setup a whole plugin system with that information.

Thanks for reading. Again, if this isn't the right place please feel free to direct me to the right place. All feedback is welcome. 🙂

[jonmeow]

Hi! There are some interesting thoughts here, but to be transparent, we're still working on the 0.1 language. The roadmap has some timeline for this, and it's uncertain when we're going to dig more into the ABI. It's possible early versions of the language (i.e., 0.1 and 0.2) won't need to be able to make .so ABIs; C++ interop should give us some flexibility there by letting us use a .so on C++ ABIs.

We do plan on having some way to define a stable ABI for things like .so files, although the shape of that isn't well defined (nor what language features would be allowed on the stable ABI). It might be file-level with limited language functionality inside, or perhaps it would be done through attributes on functions. Whether that has an effect of hiding private data isn't clear, and may have limits. Whatever that ends up, it will probably look a little like what you suggest: the compiler will need to know what's intended to be visible outside the shared library, because that's the stable ABI.

I would suspect, though, that whatever is built would end up being maintained by developers. That could take the form of autogenerated code. In particular if there's a separate stable API based on a private API, it might make sense to have a tool that generates the stable API based on annotations in the private API file. That's something possible with C++, though could be a little inconvenient; Carbon's trying to do a little more to support language-aware tooling.

Anyways, we expect some distinction for calls exposed on ABI boundaries, it's just too early to speak with much confidence about how that'll look.

[LouChiSoft]

Hi, thanks for your feedback. I appreciate the project is in very early stages at the moment, still an experiment based on the roadmap. But I still think that while nothing needs to be set in stone right now, thinking about it can't hurt.

It's possible early versions of the language (i.e., 0.1 and 0.2) won't need to be able to make .so ABIs; C++ interop should give us some flexibility there by letting us use a .so on C++ ABIs.

This is probably a good starting, one of Carbon's goals is high quality interoperability so having that as a stop gap measure while a proper Carbon solution would be useful. It also means popular languages like Vulkan, SDL, etc. would be easy to make bindings for.

We do plan on having some way to define a stable ABI for things like .so files, although the shape of that isn't well defined (nor what language features would be allowed on the stable ABI). It might be file-level with limited language functionality inside, or perhaps it would be done through attributes on functions.

Again, I've never done any compiler/toolchain development so I can't say how hard it would be. But my suggestion would be to support as many language features in a .so as is reasonable. I can make the educated guess that that would lead to a very complex "boundary" ABI and so maybe there would be have to be some concessions made to not make it impossible to maintain. But, as a high level example, I remember watching a talk on Carbon a while ago talking about how the compiler handles generics by converting them into essentially functions that the compiler calls to convert the input generic + type into the output specific. Maybe it would be possible to include these functions as what C++ would label a constexpr function that are used during dynamic linking of libraries to enable the use of generics from compiled libraries. Something that as far as I know is impossible to do with C++.

I would suspect, though, that whatever is built would end up being maintained by developers

I'm not exactly certain what is meant by this. By this do you mean the interface that comes with the library? If so, I feel like auto-generated should be the preferred method. Like I said in the original post, the compiler should be able to build the public interface from the source code. If there is something missing or something available that shouldn't be in the auto-generated code I personally think that should be indicative of the source code having a mistake. That being said, I do still think that there is value in having the interface embedded in the library itself. Not only would it mean that as long as you have the .so you have a matching interface by calling a function that returns that information, but other languages could potentially use that information to make it easier to call functions from that library.

Take Python as an example, if you had a a .so with the API and presumably some kind of information on the ABI embedded in the library then you could load the .so and Python could auto-generate bindings for your library. I know something like this is getting way ahead of myself and that if something like this did exist it would be years from now. It's more about demonstrating use-cases for robust API and ABI information embedded in the .so to enable things not just within the Carbon ecosystem but the wider programming ecosystem. Python is a fast growing language and a lot of popular python packages are C/C++ packages with Python bindings. And if Carbon is looking to improve and surpass C++ in a lot of cases, then I don't think it's unreasonable to look at the wider uses of C++ like high performance Python packages.

Anyways, we expect some distinction for calls exposed on ABI boundaries, it's just too early to speak with much confidence about how that'll look.

I look forward to reading about what the ABI boundaries might look like whenever those conversations come up. And, of course, if you need anymore wacky ideas for dynamic library support I am happy to think of some more 😂

[jonmeow]

Regarding supporting as many language features as possible, it might be worth looking at C++: it's had a very stable ABI. This was discussed in P1863, "ABI: Now or Never". That paper suggested C++ was taking a 5-10% aggregate performance hit as a consequence of ABI stability.

For Carbon, that paper is just risks we'd like to avoid. Maybe we'll need to make compromises [ed: e.g., stability over performance for simplicity], but all we have right now is essentially a goal to avoid the same performance penalties.

To be sure, I think you make good points, and I'll in particular try to keep the idea of embedding source in the .so in mind when we get around to working on this. It's just hard to really delve in right now because we're busy building the foundations.

[LouChiSoft]

That paper suggested C++ was taking a 5-10% aggregate performance hit as a consequence of ABI stability.

Wow, I new there were issues with performance I didn't realise it could get that high.

For Carbon, that paper is just risks we'd like to avoid. Maybe we'll need to make compromises [ed: e.g., stability over performance for simplicity], but all we have right now is essentially a goal to avoid the same performance penalties.

I think I am starting to understand the goals a bit better and just how complex this might be. I could easily see multiple "tiers" of ABI compatibility depending on the features you need for your library.

To be sure, I think you make good points, and I'll in particular try to keep the idea of embedding source in the .so in mind when we get around to working on this. It's just hard to really delve in right now because we're busy building the foundations.

I completely understand that for a project still so deep in it's infancy that ABIs aren't high priority right now. I would like to keep track of when ABI features are beginning to be designed. Is the Discord the best place to be for when this starts happening?

[danakj]

Is the Discord the best place to be for when this starts happening?

Yes, but if that's too much to follow along with, we have just started publishing summaries weekly here: https://github.com/carbon-language/carbon-lang/discussions/categories/last-week-in-carbon (there's also an RSS feed of them)

[LouChiSoft]

The last week in carbon posts should do nicely, thanks for bringing them to my attention