Clarify assumptions on character encoding in the pynxtools/NeXus/HDF ecosystem

[mkuehbach]

Plugins in and pynxtools use YAML and HDF5 and python libraries to interact with these technologies. What happens if users write strings, i.e. NeXus NX_CHAR characters using characters that are not part of ASCII or UTF8?. We have in various places decoding assumptions made like decode("utf8") but if these raise UnicodeConversion errors need to be catched.

• YAML v1.1 supports https://yaml.org/spec/1.1/current.html#id868742
• HDF5 supports at most UTF8 https://support.hdfgroup.org/documentation/hdf5-docs/hdf5_topics/UsingUTF-8EncodinginHDF5Apps.html
• NeXus is NX_CHAR is vague wrt on character encoding but gives a preference what to use, inspired by its applications rooted heavily on HDF5 https://manual.nexusformat.org/datarules.html

[lukaspie]

The function decode_or_not (from the definition submodule), which is the one we are mostly using, raises a UnicodeDecodeError. We should see that we wrap this properly for different use cases in pynxtools. For the NOMAD parsing, we should not raise, but rather log an error message and continue the parsing. For other funcionality, it may be okay to raise.