Writer should allow to create _adhoc_ Xapian Title index entries

[kelson42]

Since years already, but the tendency gets stronger, we publish ZIM files which work like a single Web Page application. Based on a heavy usage of Javascript and Vue.js, these ZIM files (and their correspongins scrapers) are significantly different from traditional ones:

• One article is not one HTML page
• Only one page and lots of json based resources load dynamically
• Not so many real HTML links A DOM nodes
• Everything is loaded dynamically via Javascript calls

This leads to many challenges, but the most important one been the following:

• We still want to allow user to search via the suggestion feature (based on Xapian Title index). Unfortunately this is not straight because (1) there is only one page and all the parameters of the query have to be put in the URL fragment (2) traditional ZIM redirects can not handle URL fragments, see Add support for redirect with fragments #1001.

For the moment, the solution (I should write the "workaround") is to create an HTML redirection for each entry we want to appear in the suggestions. But these has a few disadvantages:

• This creates an additional round-trip between the ZIM and the browser and this creates a bad UX (slow and visual flash effect)
• This can create millions of "heavy" entries in the ZIM files
• This slows down the ZIM writting process.

A better solution, would be to have ZIM redirections handling the URL fragment as well. See #1001. Unfortunately this is not that easy and actually I believe not even the best solution, because this is not really what we need. We don't need a new article to which an page can link (with fragment) to. What we need is to be able to have suggestions supporting URL fragments... and we need all the suggestions pointing to the same page (remember: single page application).

What I propose is to:

• Be able to add entries in the Xapian Title index, independently of any new ZIM entry
• Add support of URL fragements to the Xapian Title index.

It could take the form of a something like:
zim::writer::Creator::addSuggestion(const std::string &path, const std::string &title, const std::string &fragment)

In term of feature introduction, this should be considered:

• This would need no change at reader side, beside having the latest ZIM library (to support URL fragments, so I guess older reader will just don't handle the fragment)
• Once the feature implemented we can immediatly release and use it to create ZIM file
• Probably a ZIM format number minor incrementation
• We should implement the feature quickly and start only to release ZIM file with it, once all the readers have been released using the latest ZIM library (will take a bit)

My proposal has a drawback. Only readers supporting the Xapian indexes will be able to fully enjoy the ZIM files using this feature.

This proposal does not compete with #1001, this is two different things (but yes, a bit of overlapping topic)

[kelson42]

@benoit74 @rgaudin @Jaifroid I would like your validation before implementation
@veloman-yunkan We discussed the feature already. Not more urgent than maintenance fixes or even BitTorrent support in libkiwix. But considering (1) pretty straight forward (2) we need to wait before releasing ZIM files using the feature, it would be good to not wait too much with the implementation. So when you are motivated, just move on it please.

[rgaudin]

I understand the approach of fixing a suggestion/search issue at the index level, though…

Your proposal is suggestion-specific and eliminates search results for those ZIMs (or there would be no advantage). I'm not sure if it's on purpose or not but although there are certainly cases where search results have no value, I'm sure there are where it does. Dropping it should be explicit and argued at least.

Not using entries at all means loosing the IndexData feature which makes sense in those ZIMs (although we're not leveraging it much I believe). To be considered.

• This would need no change at reader side, beside having the latest ZIM library (to support URL fragments, so I guess older reader will just don't handle the fragment)

I don't think that's possible given querying the suggestion engine currently returns a SearchResultSet which is a collection of Entry.

We'd need to return something different that includes this additional payload, and then the reader would need to combine the entry's path and that fragment when changing the webview because fragment management is client-side.

• Probably a ZIM format number minor incrementation

I don't know xapian so may be completely wrong but if we're changing the structure of the listing/titleOrdered/v1 we'd have to change its mimetype as well and parsing of it would need to be updated… not a backward compatible change in that case… but I trust implementation will be smarter if you came to that conclusion after discussing it with @veloman-yunkan

[Jaifroid]

Thanks for including me in this discussion -- this is clearly a real problem for SPA-based ZIMs and the proposed solution looks pragmatic to me (I leave the low-level technicalities to @rgaudin ).

A few notes from the reader/WASM side:

On Xapian performance in the WASM port -- I have implemented suggestions in openzim/javascript-libzim but haven't yet integrated them into Kiwix JS / Kiwix PWA. For context: Kiwix JS uses a custom pure-JS implementation of the ZIM file reader for most operations, which is highly optimized and fast. For Xapian-based search we use the WASM port of libzim, which introduces two sources of latency: loading the WASM module itself, and instantiating the Xapian code and its indices. This is why for title search we currently use our own fast binary search on the title-ordered listing rather than Xapian. Whether Xapian-based suggestions would be significantly faster than full-text search (they don't need to extract snippets) is honestly unknown and would need testing, but for large archives the WASM overhead is a real concern regardless. It is however far less of a concern for small ZIMs, which most SPAs are likely to be?

On reader-side changes -- Fragment navigation would require new code on the reader side in general, but there's an existing mechanism in that may be leverageable. In WARC-based (Zimit) ZIMs, the URL index effectively stores a Response formatted as a ZIM URL, including query strings and fragments (? - not sure if the hash gets stripped in WARC-style ZIMs). If the Xapian suggestions API returned a complete URL (path + fragment combined) that is findable in the URL index (rather than a bare path), we could then route it through that same logic (if I've understood correctly). Each navigation in such ZIMs is stored, and is replayable as the response usually to a click in the Browsertrix scraper. So the key question for the API design is: does the suggestion result expose the full ZIM URL for each page or section navigated to as a single string, or path and fragment as separate fields? The former might make reader-side integration easier, at least for Zimit ZIMs.

Changes to the suggestion API (return type, call signature, or both) are otherwise manageable on the binding side in javascript-libzim provided they are clearly documented.

On concrete SPA ZIM examples -- it would be useful to have a few named ZIMs for testing purposes, particularly for validating WASM behaviour. I presume we're talking mostly about warc2zim ZIMs (Zimit)?

It's also worth noting that not all SPAs use hash-based routing for navigation -- some rely on query parameters (e.g. ?page=3&section=configuration) or pure JS state, which fragments wouldn't cover for navigation-through-search purposes (they are handled for normal navigation, when clicking on links, in warc-style ZIMs, as mentioned). Whether that's in scope for this proposal or a separate concern is worth clarifying.

[Jaifroid]

An example of a non-Zimit Single Page Application is Kubectl Docs. Coincidentally it has been causing issues in the PWA which I've fixed.

But this gives me a clearer view of the issue above, at least for well-defined non-Zimit scrapers that know what content they are expecting. In the case of this devdoc ZIM, it's not obvious how the system outlined by @kelson42 would work, because literally there is one single very long HTML page loaded in a div, and a set of navigation controls in another div that merely jump around in that page. One might as well just use in-page search (equivalent of browser's Ctrl/Cmd + F) to search for contents on the single page. Users are just going to use the well-defined navigation links in any case rather than search to navigate.

Do we have more complex SPA ZIMs which use JS to show and hide parts of a single underlying HTML page, i.e. a true "application"? If so, I somehow doubt simple navigation via hash fragments will be how "subpages" are displayed.

[benoit74]

@Jaifroid this issue is not about Zimit at all, it is mostly only about all other scrapers.

You can for instance have a look at:

• mwoffliner: we have "fake" entries (e.g. if you search for geography nairobi, which leads to #Geography section of Nairobi article)
• youtube: we have same kind of "fake" entries for each video so that we can perform suggestion search.

I'm not really convinced anymore by the solution outlined by @kelson42, at least I need more info on why implementing #1001 is so hard.

As @rgaudin said, I'm quite sure we will have situations where we want a real redirect to store data in full-text search index as well. This is probably what we need for Gutenberg ZIMs for instance (which are moving to a SPA approach).

And I don't really like the fact we are adding yet another scenarii (suggestions without matching entries) to something which is already complex to comprehend (at least for me: suggestion search, full-text search, redirect, alias, ...).

Since we will have to modify readers anyway, I wonder if we should not prefer to do it only once with a "final" solution.

Can we develop in issue #1001 the implementation issues we have to make a better informed decision?

[veloman-yunkan]

For the moment, the solution (I should write the "workaround") is to create an HTML redirection for each entry we want to appear in the suggestions. But these has a few disadvantages:

• This creates an additional round-trip between the ZIM and the browser and this creates a bad UX (slow and visual flash effect)

• This can create millions of "heavy" entries in the ZIM files

• This slows down the ZIM writting process.

I am in favor of optimizing the current workaround by enhancing our redirect entries so that additional parameters can be stored with them (as proposed in #1001). This won't even require a major change to the ZIM format, as ZIM directory entries are equipped with an unused field parameter which suspiciously seems as if being there with such a use case in mind. Then every such HTML redirection can be added as an extended ZIM redirect which will appear to the libzim clients as a quasi-item for which dynamically generated HTML content will be returned. Later, readers that decide to process such extended redirects more efficiently are free to do so.

[kelson42]

On Xapian performance in the WASM port -- I have implemented suggestions in openzim/javascript-libzim but haven't yet integrated them into Kiwix JS / Kiwix PWA. For context: Kiwix JS uses a custom pure-JS implementation of the ZIM file reader for most operations, which is highly optimized and fast. For Xapian-based search we use the WASM port of libzim, which introduces two sources of latency: loading the WASM module itself, and instantiating the Xapian code and its indices. This is why for title search we currently use our own fast binary search on the title-ordered listing rather than Xapian. Whether Xapian-based suggestions would be significantly faster than full-text search (they don't need to extract snippets) is honestly unknown and would need testing, but for large archives the WASM overhead is a real concern regardless. It is however far less of a concern for small ZIMs, which most SPAs are likely to be?

@Jaifroid The non-support of Xapian index in Kiwix JS is a time bomb. This was a problem few years ago already, but the problem is only going to continue to grow. Same "music" for non-libzim based ZIM reading.

[kelson42]

As @rgaudin said, I'm quite sure we will have situations where we want a real redirect to store data in full-text search index as well. This is probably what we need for Gutenberg ZIMs for instance (which are moving to a SPA approach).

Redirects, whatever their style have never been fulltext indexed as they have no content (body).

@benoit74 Can you please give one current and one hypothecial scenario where this approach would not do the job for a single page app?

And I don't really like the fact we are adding yet another scenarii (suggestions without matching entries) to something which is already complex to comprehend (at least for me: suggestion search, full-text search, redirect, alias, ...).

This approach is exactly what the problem requests: you just want arbitrary suggestions (without creating an item). This feature will probably anyway be implemented as this is just what is needed in most of the case... and pretty sure the most lightweight scenario.

Since we will have to modify readers anyway, I wonder if we should not prefer to do it only once with a "final" solution.

This is not conceived as a temporary solution.

[benoit74]

Redirects, whatever their style have never been fulltext indexed as they have no content (body).

Indeed, and I realize only now that this is in fact a problem I've completely oversight in #1001

@benoit74 Can you please give one current and one hypothecial scenario where this approach would not do the job for a single page app?

In gutenberg scraper, we create one "thing" (mini-html for now) per book in the ZIM. Each book should be on both Xapian index. With the mini-html workaround, we create a normal entry with both a title and the book content as full-text index data so that both are indexed: https://github.com/openzim/gutenberg/blob/40fde7632b7e440834905ed352217f98274443f6/scraper/src/gutenberg2zim/export.py#L702-L725

In youtube scraper, we add videos and playlists, and each of them have their descriptions in the full-text index via same mechanism.

In freecodecamp we could (should?) add each exercise description to the full-text index. In maps we could add a description about each place in the full-text index.

you just want arbitrary suggestions (without creating an item)

This is absolutely not what I've requested in #1001, this is the solution you proposed.

This is not conceived as a temporary solution.

Once/if #1001 is implemented, I doubt we will continue to use both approaches in scrapers, but you're right that this is pure speculation.