LYD_LYB no longer works.

[choppsv1]

I recently updated to the latest libyang master and LYD_LYB format now no longer works for us.

2025/10/22 06:40:14 MGMTD: libyang: Different current LYB context modules hash compared to the one stored in the LYB file (0xe20d2077 != 0xeb21d279).

This is FRR, and so we have a suite of daemons and a central management process mgmtd. The daemons send YANG tree results back to mgmtd using LYB format for efficiency. If I had to guess based on the above message though, it seems like this has been broken by a recent libyang version.

All of the FRR daemons are compiled together against the exact same installed version of libyang. They all run on the same machine using the same libyang shared library. The expectation was that we should be able to use and benefit from libyang binary format for the increased efficiency -- we certainly use binary messaging internally for the same reason that it of course will work with these build and run restrictions.

Is there something else we need to do now to have this work?

[michalvasko]

Sorry about that, the LYB format has recently been optimized (a lot, based on our measurements it generates data with the same length or even smaller than protobuf) but some new limitations were also added, which we did not think would restrict the usage too much. One of those is that the transmitting and receiving end must have the exact same context hash (very fast check, instead of verifying all the required modules are present). What the hash consists of is documented. Is that a problem for you?

[choppsv1]

Looks like it, yeah. The only restriction we currently place on what modules are loaded is that the central management daemon, mgmtd must load all the modules used by all the daemons, the other daemons only load what they need to support for receiving their config and sending oper-state. So each daemon will only load a subset of the modules that mgmtd loads.

[michalvasko]

Then I do not think I can help you. If you want to take advantage of the optimized LYB, you will have to have the same context (we have solved this in sysrepo by sharing a single context by all the processes on a system) or you can switch to the JSON format, which should be similarly efficient to the old LYB format if you use the shrink flag when printing it. That is what we also did in sysrepo for cases when having the same context proved too restrictive.

[choppsv1]

I'm pretty sure I don't know what you mean here by "the same context". As a libyang user a context is something I create by calling ly_ctx_new() so it seems impossible to share that among different process. Indeed I have one daemon (a restconfd) which even creates multiple contexts in a single process. So you must mean something else.

What are the actual limitations here? It shouldn't be that hard to implement a highly efficient binary format (much more efficient than a packed JSON!) w/o having to load all the same modules in exactly the same order. :(

[michalvasko]

it seems impossible to share that among different process

Well, it requires some advanced features but I assure you it is possible and we have it working. A single context is created and written into a binary file, which is then mmap()'d by all the processes on the same absolute address so that the pointers can be used directly.

What are the actual limitations here?

LYB relies on schema node hashes instead of global SIDs (like CBOR), for example, so it is more flexible and easier to use. The disadvantage is that it requires the YANG modules to be the same with the same schema nodes to prevent possible hash collisions and matching incorrect nodes when parsing the data. That is why they were being stored in the LYB file and checked when parsing it. This requires additional space and time for every print/parse so we optimized it the way I mentioned, by using a context hash instead. And since the focus of LYB is precisely this, to be small and fast, I still think it was the right decision.

If you look at other binary formats, there are always limitations like this and ours is still a good trade-off, in my opinion. Protobuf, for example, requires the data to be known and exactly the same during compilation already while we allow run-time changes of the YANG context. CBOR has its long list of SIDs and I am not even sure how efficient is the encoding of actual values, have not looked at the details yet. But LYB packs the data on bit-level and uses quite aggressive variable-size values. So all-in-all, I think our LYB has quite a good design.

It shouldn't be that hard to implement a highly efficient binary format

So it actually is pretty hard and there are always some restrictions.

However, there may be a simple solution that you may be okay with. There can a flag be added (preferably for both print and parse) that puts the responsibility of keeping the parse and print context the same within the required restrictions on the user. I just would not want to be in the position of someone debugging an issue in case the restrictions are broken unintentionally.

[choppsv1]

I'm still not sure I understand what "keeping the context the same". Does your new LYB actually require they be in the same memory locations? If so I can't imagine that working for us (or really anyone other than CESNET internal users who are actually in full control of things), no one else is going to be able to be that dependent on things they don't understand or can't control.

I mean if the new LYB requires using shared memory for the libyang data structures, why bother with a binary format at all? You've already paid such a huge complexity cost you might as well just pass the data around raw in share memory buffers and skip the conversion to LYB.

This is really too bad, as storing things and passing them around in binary format was seen as a really big win that we got with libyang -- sad that we've lost it.

[michalvasko]

Does your new LYB actually require they be in the same memory locations?

No, this is not related to LYB. It only requires the context to have the same hash. To rephrase the hash description, have exactly the same set of YANG modules, in the same implemented/imported state and with the same features enabled.

[choppsv1]

Why all those ordering requirements? Is it a by-product of choosing a non-commutative hash function? Even so since you aren't constantly regenerating the context hash, but just using it to validate sameness why not just rerun the full hash function when something in it changes (e.g.,, when a new module was loaded) -- making module load time a tiny bit faster does not seem worth the price being paid for it.

[michalvasko]

I am not sure we understand each other, there are no ordering requirements, it is about the exact same YANG context. Hash is cached in the context and updated on every context change already.

The change I am talking about is that, BEFORE, the list of required YANG modules with their revisions and features was written at the beginning of every LYB file. Then, when parsing it, it was used to check that the YANG context includes all the modules with the same revisions and features. NOW only the context hash is written into the LYB file, which is simply compared to the parsing context and it must match, otherwise you get the error you see.

So, it is unfortunate your use-case does not work anymore but the justification for the changes is solid, in my opinion. I have also suggested a way of making it work for you.

[choppsv1]

I am not sure we understand each other, there are no ordering requirements, it is about the exact same YANG context. Hash is cached in the context and updated on every context change already.

I'm going from the doc on the hash function:

/**
 * @brief Get the hash of all the modules in the context. Since order of the modules is significant,
 * even when 2 contexts have the same modules but loaded in a different order, the hash will differ.
 *
...

The change I am talking about is that, BEFORE, the list of required YANG modules with their revisions and features was written at the beginning of every LYB file. Then, when parsing it, it was used to check that the YANG context includes all the modules with the same revisions and features. NOW only the context hash is written into the LYB file, which is simply compared to the parsing context and it must match, otherwise you get the error you see.

This does not track with how it works for us prior to this version. The daemons for sure do not load all the modules that mgmtd does, but LYB works. I suspect it's working b/c the receiver mgmtd has a superset of the modules the sender of LYB has.

So, it is unfortunate your use-case does not work anymore but the justification for the changes is solid, in my opinion. I have also suggested a way of making it work for you.

I don't understand the fix. If it means things continue to work for us then that's great. If it means we have to have exactly the same context that is not as simple and wasn't required previously.

Thanks,
Chris.

[choppsv1]

why not put a modules-used array at the tail of LYB encoding of the modules used in the encoding, then (index, module-scoped-schema-node-hash-key) which could easily fit in a integer if you need that, to refer to schema nodes in the encoding. Then you don't have this super restrictive contexts must be identical requirement, you only have to have the modules used loaded on the receiver which is a totally reasonable restriction.

Requiring e.g., the RIP daemon load every single other protocols set of YANG modules (the set of which completely depends on how the system was built and installed by the user) is pretty sub-optimal.

[michalvasko]

I'm going from the doc on the hash function:

You are right, that was a completely unnecessary restriction so I have fixed it, the order of the modules does not matter anymore for the hash.

This does not track with how it works for us prior to this version. The daemons for sure do not load all the modules that mgmtd does, but LYB works. I suspect it's working b/c the receiver mgmtd has a superset of the modules the sender of LYB has.

Yes, there was a change. And it obviously worked for you because of the reasons mentioned.

I don't understand the fix. If it means things continue to work for us then that's great.

Yes, my suggestion is to add a new flag that would skip checking the parsing context completely. Meaning it would work for you just like before but if it happens that you make some changes and the required YANG modules are not loaded in the same state as when printing the LYB data, you will not be notified by the non-matching hash. Most likely, you will get an error saying that a hash did not match a schema node during parsing but there is a small chance the hash will actually match a wrong schema node and the parsing will create invalid data and most likely crash during parsing.

why not put a modules-used array at the tail of LYB encoding of the modules used in the encoding, then (index, module-scoped-schema-node-hash-key) which could easily fit in a integer if you need that, to refer to schema nodes in the encoding.

Just to answer you, hashes are unique only within one level (parent - all siblings), not within the whole module, which allowed us to use only uint8_t for the hashes.

you only have to have the modules used loaded on the receiver which is a totally reasonable restriction.

That is still the actual restriction but it is inefficient to fully check it, which is exactly the optimization we did. And it made the restriction more severe, all the modules must be the same, but I did not really expect problems. Seems I was wrong.

[choppsv1]

I don't understand the fix. If it means things continue to work for us then that's great.

Yes, my suggestion is to add a new flag that would skip checking the parsing context completely. Meaning it would work for you just like before but if it happens that you make some changes and the required YANG modules are not loaded in the same state as when printing the LYB data, you will not be notified by the non-matching hash. Most likely, you will get an error saying that a hash did not match a schema node during parsing but there is a small chance the hash will actually match a wrong schema node and the parsing will create invalid data and most likely crash during parsing.

Well that's better than nothing, so we'll take it. :)

I still have concerns for the versions that are now released that will break deployed running code.