Remove support for JSON serialization format

[JukkaL]

Now that we have a new binary fixed format for serialization, it would be good to eventually drop the old JSON based serialization code, since maintaining duplicate implementations slows down development, and there is a risk that the implementations will get out of sync.

At work we have some tools that consume the JSON based cache format, and it would be non-trivial to update them to directly use the binary format. As a workaround, we could provide a tool that converts a binary cache into JSON files that resemble the current JSON format (they don't need to be 100% compatible since the format isn't documented anywhere).

To implement this, we can make the binary format "self-describing", i.e. it would have enough redundancy that we can write a simple generic parser that reads arbitrary data serialized using the format and converts it into JSON, and that doesn't need to know too many details of each possible object type that can be serialized (to simplify maintenance).

Here is one possible to way to make the format self-describing:

• Each value/object is serialized as <type tag><data>, where the format of data depends on the type tag. The type tag is a 8-bit integer.
• Simple values are also encoded as a type tag followed by data. These include integers, strings, booleans, floats and None.
• We can have similar generic encoding for lists and dicts, such as <type tag for list><number of items><arbitrary value>....
• AST nodes would also have distinct type tags, so that it would be possible to mix simple values and AST nodes.
  • I would propose that AST nodes and types would be encoded using a format like this: <type tag><field tag><arbitrary value>...<end tag>. Field tags are also 8-bit integers that map to JSON keys. The end tag is a reserved 8-bit number that is distinct from all type tags.

Now we can implement a generic parser. It needs to know how to parse all simple values and container values, and it must have a mapping from valid type tags for AST nodes (including type objects) and the field tags to strings.

I think that this format would also make it easy to implement lazy deserialization. We can easily find the end of an arbitrary object in a serialized byte stream, without having to deserialize it. Instead of deserializing a FuncDef, for example, we could just find the and of the serialized representation and put the serialized byte string into a symbol table.

cc @ilevkivskyi

[JukkaL]

The main changes to the current [de]deserialization code would be these:

• Add type tags before simple values.
• Add constants for each field type (e.g. KIND: Final = 56; 56 is the field tag value). The JSON key name can be derived automatically from the constant name (KIND -> "kind").
• When reading/writing fields, pass the field tag (e.g. write_int_field(buf, KIND, o.kind)).
• Add end tags when serializing objects.

This would a little less efficient that the current format, but I don't expect the performance to degrade much, since all the new features can be compiled to fast machine operations (no new dynamic allocations etc. that would slow things down). If this enables lazy deserialization, I'd expect overall performance to improve significantly.

[ilevkivskyi]

TBH I don't like this. IMO we should simply drop the JSON format at some point, period. More details:

• First and foremost, I don't want to make any design decisions based on benefits of a single corporate user. Dropbox already benefits a lot from work of mypy contributors (myself included). Do we all must look at and maintain some boilerplate for years, so they can benefit even more?
• People who need to inspect the cache files can simply load them, it is as simple as from mypy.nodes import MypyFile; from native_internal import Buffer; tree = MypyFile.read(Buffer(data)). A requirement to inspect mypy cache files while not having mypy available looks very niche.
• I am not so sure performance impact will be negligible.
• I don't see how <end tag> will help with lazy deserialization. You can't simply scan the byte string looking for this magic byte (it may appear accidentally in other serialized values). IMO it can be implemented in a simpler (and probably more efficient) way by storing total byte size in front of each symbol:
  • Write 8 placeholder bytes, and record current position
  • Serialize content of a symbol (FuncDef, Var, etc)
  • Write the difference between current position and recorded position in the placeholder 8 bytes

[JukkaL]

If there's strong opposition to this, we could implement and maintain the JSON converter internally at Dropbox. However, we have no reliable way of knowing ahead of time whether other mypy users are parsing the JSON files and would have to change their workflows. I remember discussing the things we do with cache files at conferences, and it's possible that others have borrowed the idea or came up with it by themselves. In any case I volunteer to implement the required functionality and I don't think it would add a significant maintenance burden (but clearly it would have some impact, even if small).

I think the field tags are redundant and can be dropped from the proposal. I also think that the remaining bits would be useful when implemeting lazy deserialization. We could also use the "quickly skip some objects during deserialization" functionality to implement a low-tech incremental parser that will skip serialized nodes until reaching line N.

I am not so sure performance impact will be negligible.

If we'd drop the field tags and only keep the type tags, which could useful for lazy deserialization, there might not be any impact. This at least will be easy to measure once we have a prototype implementation.

People who need to inspect the cache files can simply load them

I hadn't thought about this. I will experiment with this. If it looks like this can support the use cases we have at work, we can document this new approach in mypy release notes and perhaps add some simple example under misc/ that does something like this. In that case the JSON converter wouldn't be needed.

I don't see how will help with lazy deserialization.

It will be used for tracking the nesting depth of objects. The byte value of the end tag only needs to not overlap with field tags (or type tags if we drop field tags). Serialized simple values such as integers and strings can have arbitrary binary data with no reserved bytes. We just need a way to skip until the end of the deserialized data for each possible data type (i.e. simple types, container types, and object types). All object types are similar, but without the end tag we don't know when the object ends without some extra metadata (which would require additional maintenance and thus wouldn't be ideal). I can explain this is more detail if this is still unclear.

cc @msullivan who mentioned potential impact to debugging issues with incremental mode.

[ilevkivskyi]

However, we have no reliable way of knowing ahead of time whether other mypy users are parsing the JSON files and would have to change their workflows.

I think when we will flip the default to use fixed format cache, and then later will emit deprecation warnings if people use --json-cache, they will likely start complaining. Then we will see how much it is needed.

I can explain this is more detail if this is still unclear.

Yeah, I understand this "galloping mode" read. But whether this is really needed or we can just insert total size for few selected classes depends on the details of the proposal. I see you commented in a dedicated issue, we can discuss it there.

If we'd drop the field tags and only keep the type tags, which could useful for lazy deserialization, there might not be any impact.

Adding just the type tags may be OK, this really depends on the details of how it will be implemented (e.g if adding type tags will be "automated" for primitive types etc).

If there's strong opposition to this, we could implement and maintain the JSON converter internally at Dropbox

It is not like I am putting my foot down, but I think we need to weight all options before committing to anything here. Btw yet another possible option may be to have a separate file that will have a list of attribute names for each class in the order they appear in cache. This file would be kept up to date by people who are interested in parsing the cache files without loading (assuming this will be actually needed).

[A5rocks]

BTW Trio relies on JSON serialization to quickly check whether an attribute is publicly exported in mypy's view. While we can do this by calling mypy directly, this is significantly slower than populating cache and then pulling from it! I think we can use this workaround though:

it is as simple as from mypy.nodes import MypyFile; from native_internal import Buffer; tree = MypyFile.read(Buffer(data))

But FYI this might be more popular than you expected. (Though I also only did this because I've contributed to mypy and knew the internals)

[JukkaL]

We'll add sufficient type tags (i.e start tags) and end tags so that we can find the end of a serialized object with a generic algorithm that doesn't understand all the specifics of serialized objects (see #17924 (comment) for context). It would be good to have this for the next mypy public release (release tracked at #19964), but not absolutely essential.

We can possibly use MypyFile.read(Buffer(data)) as a replacement for the current JSON serialization format (see discussion above). This should be documented at least in the release notes.

However, we still have the option of writing a binary cache to JSON converter, which would perhaps only support a subset of the data exposed in the current JSON format, but enough for common use cases. I think the most interesting bit is symbol table data -- we could start by providing an export tool that dumps module and class symbol tables, with kinds of nodes (e.g variable of function), cross_refs and some other relevant information (maybe also static types).

[JukkaL]

I started prototyping and it doesn't look hard to write a tool that calls MypyFile.read(...) on a fixed format cache file and then converts the result to JSON that mostly looks like the current JSON cache format. This looks like a reasonable solution, and we can allow the tool to be maintained on a best effort basis (i.e. not all information added to the symbol might be exported). Once we have this tool, I think we can drop support for the JSON serialization format after a reasonable deprecation period.

[JukkaL]

Now we have a tool to convert .data.ff files to JSON (#20071). The tool will still need to be updated to support binary meta cache files, once mypy supports them (meta are files are still JSON currently, but we should switch them to use the binary format as well).