Possible to deserialise an AST?

[mlyoung101]

I was just wondering if it's possible to deserialise an AST. I know there's an ASTSerializer, but I don't see an equivalent ASTDeserializer.

My use case is that I'm writing a language server, and I'd like to be able to cache compilations, in a similar way to what clangd does.

I know something like this is pretty non-trivial in C++, but is there any interest or plans? I imagine we could do something very similar to ASTSerializer, just with inverse functions (e.g. something like expectObject() instead of readObject()).

[MikePopoloski]

It would be nice to have for sure but it's not implemented yet. Deserialization is much harder than serialization because you have to correctly create the C++ objects and link them back up to each other, deduplicate types, make sure you're handling all of the private state of the nodes and not just the public stuff we're currently serializing, find places where we assume we'll have access to syntax nodes, etc.

[mlyoung101]

Ahh yes I understand. In terms of writing out the private state, does this mean that every node would need to be refactored to be a friend class of some sort of ASTSerializer? That would probably be pretty substantial. Also out of interest, could you elaborate on what's involved specifically in deduping types?

[MikePopoloski]

Doesn't necessarily have to be a friend if each AST type can also serialize the private stuff and then knows how to recreate itself from the deserialized data, but I doubt that is the case currently.

Different classes of AST nodes are de-duped, basic types being one of them. When types are serialized they're generally done in a way that would make it hard to recover that (like we just serialize some string instead of all of the type details, though there are various options controlling the output now).

It's all doable with some work though, just many things to solve. Ax another example, basically all string memory (for symbol names and whatnot) is backed by the original file loaded into the SourceManager. That won't be there when you deserialize, so you need some other mechanism for making sure all that data you need gets allocated somewhere. Diagnostics are based on source locations that point into those files as well, so you'll need some other way of representing source locations, and diagnostics involving these deserialized nodes could probably no longer include source file line context in the message.

[mlyoung101]

Cool, thank you so much for the detailed description, that is super helpful. Sadly I don't think I have the capacity to take on a project of this scale at the moment, but I'll keep it on the backburner (assuming no one else picks it up) as I would eventually like to contribute to Slang.

It seems to me like the most difficult thing to manage is the loading of source code into memory from the deserialised parse tree. For symbol names, we could probably make a copy of it when we write it out, but deserialising and storing that copy when we load it back in again might be tricky. I think writing this deserialiser would break a lot of the underlying assumptions that Slang relies on (i.e. having an entire document loaded into memory with valid source locations).