Control encoding used for content in search_index.json

[awanlin]

Hi, new here so if I'm in the wrong place feel free to redirect me 👍

I'm wondering if there is any way of controlling the encoding used for the content in the search_index.json file that's created by the Search plugin?

Some context: we ran into this backstage/backstage#19584 issue and what we found was that one of our README.md file was using UTF-16 LE encoding which got into the search_index.json and that lead to some errors down stream in our Backstage instance. Backstage is a framework for building developer portals that uses MkDocs and mkdocs-material as part of its TechDocs feature.

[squidfunk]

Thanks for asking. We're currently not planning to allow for changing the character encoding of the search index, as this would create the need for additional configuration not only inside MkDocs, but also for where you host your site and how the frontend application consumes the search index. UTF-8 is the canonical and most efficient encoding. To my knowledge, UTF-16 is used in runtimes due to its easier consumability (at the expense of additional storage), but is not an encoding that should be used for files.

That being said, we could (or should?) emit a warning regarding encodings, but the semantics and modalities must first be worked out, and it's something that should go into MkDocs, not Material for MkDocs, as it impacts all themes and plugins. Thus, I recommend to raise this to the maintainers of MkDocs ☺️