Why articles with titles `Paris` or `Miri` don't make it into suggestion search

[benoit74]

See openzim/mwoffliner#2230

I have two tests ZIM:

• build with node-libzim (supposed to have Paris title): tests_en_mwoffliner_2025-04.zip (replace .zip with .zim extension)

• build with python-libzim python-scraperlib (supposed to have both Paris and Foo Paris titles, but only Foo Paris show up when you search Par): tests.zip (replace .zip with .zim extension)

I'm really not sure whether this is a bug or a feature, but since @kelson42 consider this might be a bug in mwoffliner, better to ask 🤣

[rgaudin]

Mind sharing your pylibzim generating code?

[benoit74]

Sure. It assumes you have 3 HTML files from https://github.com/user-attachments/files/19645767/tests_en_mwoffliner_2025-04.zip available (just need to extract them with zimdump).

from zimscraperlib.zim import Creator
from pathlib import Path
from zimscraperlib.zim import metadata

creator = Creator(Path("tests.zim"), "index").config_metadata(
    std_metadata=metadata.DEFAULT_DEV_ZIM_METADATA
)

# start creator early to detect any problem early as well
creator.start()

creator.add_item_for("index", "Main Page", fpath=Path("/home/benoit/Repos/openzim/mwoffliner/output/dump/index"), is_front=True )

creator.add_item_for("Paris", "Paris", fpath=Path("/home/benoit/Repos/openzim/mwoffliner/output/dump/Paris"), is_front=True )
creator.add_item_for("Foo Paris", " Foo Paris", fpath=Path("/home/benoit/Repos/openzim/mwoffliner/output/dump/Paris"), is_front=True )

creator.finish()

[benoit74]

(Btw, I used python-scraperlib, not python-libzim, sorry)

[rgaudin]

from zimscraperlib.zim import Creator
from pathlib import Path
from zimscraperlib.zim import metadata

creator = Creator(Path("tests_reg.zim"), "index").config_metadata(
    std_metadata=metadata.DEFAULT_DEV_ZIM_METADATA
)

# start creator early to detect any problem early as well
creator.start()

creator.add_item_for("index", "Main Page", content="<h1>Hello from index</h1>", mimetype="text/html", is_front=True )
creator.add_item_for("Paris", "Paris", content="<h1>Hello from Paris</h1>", mimetype="text/html", is_front=True )
creator.add_item_for("Maris", "Maris", content="<h1>Hello from Maris</h1>", mimetype="text/html", is_front=True )
creator.add_item_for("Paris_Love", "Paris Love", content="<h1>Hello from Paris_Love</h1>", mimetype="text/html", is_front=True )
creator.add_item_for("Foo Paris", "Foo Paris", content="<h1>Hello from Foo Paris</h1>", mimetype="text/html", is_front=True )

creator.finish()

from zimscraperlib.zim import Archive
from libzim.suggestion import SuggestionSearcher

zim = Archive("tests_reg.zim")

search_string = "Paris"
suggestion_searcher = SuggestionSearcher(zim)
suggestion = suggestion_searcher.suggest(search_string)
suggestion_count = suggestion.getEstimatedMatches()
print(f"there are {suggestion_count} matches for {search_string}")
print(list(suggestion.getResults(0, suggestion_count)))
there are 2 matches for Paris
['Paris_Love']

pylibzim also cant find Paris in the suggestion but as you can see, it returns an incoherent number of results = 2 but only has a single result to give…

[benoit74]

Thanks a lot! Would love to be able to see what is inside the Xapian index (openzim/overview#50)

[rgaudin]

for num in range(1, 5):
    entry = zim._get_entry_by_id(zim.all_entry_count - num)
    pathlib.Path(entry.path.replace("/", "_")).write_bytes(bytes(entry.get_item().content))
    print(entry.path)

[benoit74]

Looks like redirects are impacted as well, at least a redirect with Bamako title seems to not be searchable either

[kelson42]

@mgautierfr @veloman-yunkan If confirmed, this is a major issue/bug/regression. This is TOP prio that we understand here what is going on.

[veloman-yunkan]

The bug was introduced by #616 and is caused by the optimization that doesn't separately record the title of entries for which the title matches their ZIM path (excluding the namespace). This results in zim::writer::Dirent::getRealTitle() (called by zim::writer::XapianHandler::indexTitle()) to return an empty string.