Meta data updates does not update during kobo sync

[joggs]

Describe the bug/problem
Meta data updates does not update during kobo sync. you always find a missing description etc and changes it in calibre, and then want to sync it the calibre-web-> kobo way. but it does not get updated.

To Reproduce
Steps to reproduce the behavior:

1. have a fully synced kobo/calibre-web setup
2. edit the metadata for a book in calibre
3. sync with kobo
4. the changes are not updated

Logfile
Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

Expected behavior
the updated metadata should be synced to the kobo

Environment (please complete the following information):

• OS: docker linux(unraid)
• Python version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
• Calibre-Web version: 0.6.21 Beta
• Docker container: [LinuxServer]:
• Special Hardware:

Additional context
same results using reverse proxy(traefik) as when using direct access.

[OzzieIsaacs]

Your obersation is comprehensible. This is a limitation of the sync protocol and the kobo reader. As long as the book Id (more precise the uuid of the book in the calibre database) isn't changed, a new sync doesn't change anythin on the kobo side.
I think there is already another issue about this

[joggs]

Thank you for your quick answer. I could not find a previous issue regarding this, but I'll have another look, as there could be ideas of workarounds. Maybe I'll try to modify the uuid directly in the database to see what happens... or if you could get an editable column with uuid in calibre? :-) I have to dive deeper into calibre and calibre-web

[fancl20]

I think the root cause is same as #2509. My understanding is we don't have to use the uuid of the book in the calibre database as the id in the kobo protocol. A computed id (from the content of metadata) can trick kobo think it's a different book.

Though I don't know how difficult the change can be :/

[smellslikemoose]

Still an issue in Feb 2024, wondering if there’s been any progress?

[FerdiZwanink]

Hiya, any progress on this issue? My work around so far was t make sure all metadata is complete before sync. Created a clear 'zero point' by continually forcing the whole database to sync, which is tedious to say the least. :D

[tsheinen]

I'd like to prefix this with "I've done no testing at all just some vibes-based libnickel reverse engineering". I haven't fully followed through how Kobo processes these but I did see some potentially telling strings in the part of the library that handles the server response.

It looks like the sync protocol does have an instruction, "ChangedProductMetadata", to invalidate/force resync book metadata? The JSON format it expects is unclear to me right now but I think this is possible -- it's just a matter of understanding the protocol better. I'll keep poking around libnickel and see if I can figure out in more detail how to do this.

[tsheinen]

Okay, actually it was really straightforward!

I did a brief patch to emit ChangedProductMetadata (tsheinen@a4781a6) for every synced book and did observe metadata set in the calibre-web UI propagating to Kobo. There seems to be minor differences in the metadata format between a new book and changing metadata but I don't think it's actually a problem because it just ignores extra fields.

syslog had one of these for each synced book

Aug 25 18:44:04 nickel: (  4440.204 @ 0x2078ab0 / ui.warning) void LibraryParser::parseChangedMetadata(const QVariantMap&) Ignoring unrecognized metadata "Categories"

[jmarmstrong1207]

Okay, actually it was really straightforward!

I did a brief patch to emit ChangedProductMetadata (tsheinen@a4781a6) for every synced book and did observe metadata set in the calibre-web UI propagating to Kobo. There seems to be minor differences in the metadata format between a new book and changing metadata but I don't think it's actually a problem because it just ignores extra fields.

syslog had one of these for each synced book

Aug 25 18:44:04 nickel: (  4440.204 @ 0x2078ab0 / ui.warning) void LibraryParser::parseChangedMetadata(const QVariantMap&) Ignoring unrecognized metadata "Categories"

Would this also cover book cover changes as well?

[jmarmstrong1207]

@tsheinen If it also works reliably, you definitely should pull request the patch to this repository!

[tsheinen]

in my testing it works reliably -- I just haven't had the time to put together a good patch. I did a quick and dirty patch to test how kobo responds to that sync command but it's super lazy -- just emitting a changed metadata command for everything in the library. A proper change would compare the metadata change timestamp (which i believe is present in the calibre database?) to the last kobo sync time (which i believe is present in the sync token) and only emit the command as necessary.

I haven't done any testing or investigation on if this command also includes covers. I would imagine it would? Although I'm not sure if it'll just do it automatically or if it'll require some playing with CoverImageId. It seems like in the existing implementation it just reuses the book uuid but it's unclear to me if this correlation is important or just a way to implicitly guarantee that each cover uuid is in fact unique. I can imagine a few plausible ways libnickel would handle this (or who knows, it could be different, this is just some educated guessing)

• ChangeProductMetadata does cache invalidation and changing the cover will Just Work
• cache fetch uses CoverImageId and it just needs to be unique (idk hash the cover?)
• cache fetch uses Some Other Information (in particular i'm curious if it keys on RevisionId for anything)

tl;dr idk i'll get a patch done eventually but i need to be able to clear an evening to write a good patch and test it and that's depressingly difficult

[tsheinen]

Relevant note: ChangedProductMetadata must be combined with NewEntitlement to propagate. It's unclear to me what the difference between NewEntitlement and ChangedEntitlement since everything I've tested seems to work fine with NewEntitlement

[tsheinen]

I can confirm the second case is true. Kobo's cover cache is based on CoverImageId. Changing metadata does not inherently update the cover but if you use that to update CoverImageId in metadata it will cache miss and re-fetch the cover.

To summarize my thoughts so far:

• It's unclear what ChangedEntitlement does. I've seen hints, although not tested, of it being used to remove books from the Kobo. I believe that calibre-web should just simplify the logic and send NewEntitlement instead.
• ChangedProductMetadata should probably be unconditionally sent since it's not a huge issue to just send more data over multiple sync requests -- and I think the book-keeping necessary to not do so is error-prone.
• CoverImageId will probably need to become derived from hash and we can store the hash -> book id mapping in KoboSyncedBooks?

and I think that's all the parts I need to actually put together a PR