Fix for non-ascii characters in text annotations of open-ephys files

[MarinManuel]

Open Ephys (binary, not sure about other format) files can contain text information that contains non-ascii characters. When trying to open such a file, the code fails with error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 38: ordinal not in range(128)

This fix detects the UnicodeDecodeError and falls back to decoding byte by byte use utf8 encoding.

As far as I can tell, this change does not affect any of the existing tests. Let me know if we want to add a test with a file with non-ascii characters.

[alejoe91]

Thanks @MarinManuel !

Would it make sense to always decode to be sure? (also to avoid the try-except?)

What characters were causing issues?

[MarinManuel]

My initial thought was also to always decode with utf8, but I did not want to risk breaking anything, so I choose the safer approach. I also assume the .astype("U") is faster than looping, but I don't know if that matters really.

I had some text strings with the characters µ and Δ that were creating issues. I initially raised an issue with Open ephys (see thread on discord if you are interested) before I pinpointed the issue here.

[zm711]

I also prefer to avoid a bunch of try-except blocks if possible. I'm not sure how just always decoding would break but to be fair I don't do much with unicode vs utf8 stuff.

[MarinManuel]

That's fair, but unfortunately, removing the try blocks causes issues because in some instances the array contains a bunch of numbers which should not be decode()ed.

As an alternative to the try blocks, I propose using conditional tests then to only use decode() if the array contains strings, otherwise use astype("U") as before

[zm711]

Is this a previous typo on our part? Why are we looking at channels here if we are checking for metadata? I'm not super familiar with this format so would love to know what is going on here? I'm just brainstorm if there is a simpler way to make this work and noticed this pattern that seems off?