how to detect invalid byte in utf8(and other Encodings)

[John0King]

I ask it on stackoverflow a few days before https://stackoverflow.com/questions/73812569/detect-invalid-byte-in-utf8/73813339

when you random read a 1kb byte of a large stream, there no (good) way to detect the start of the valid byte

I'm wonder that is that the Decoder or Encoder of the Encoding class lack of this capability ?

this is the question on stackoverflow

---

 I'm implement a text read that read from end to start, I face two problems

the stream can only read forward
can not guarantee the first byte is a start of char if I get a 1kb buffer from the stream end.
I'm looking into the Decoder class and the DecoderFallbackBuffer , but I didn't find a way to detect the invalid bytes.

```C#
void Doo() {
     var b = new Span<byte>(Encoding.UTF8.GetBytes("你好啊，兄弟们哼哼哼😂"));
     var b3 = b[2..];
     WriteLine(Encoding.UTF8.GetString(b3));
     WriteLine(decoder.FallbackBuffer.Remaining);
     while (decoder.FallbackBuffer.Fallback(b3.ToArray(), 0))
     {
         var r = decoder.FallbackBuffer.Remaining;
         WriteLine(decoder.FallbackBuffer.Remaining);
         b3 = b3[1..]; // move forward
     }
     WriteLine(Encoding.UTF8.GetString(b3));
 }
 Doo()
�好啊，兄弟们哼哼哼😂
 0
 1
System.ArgumentException: Recursive fallback not allowed for bytes \xE5 \xA5 \xBD \xE5 \x95 \x8A \xEF \xBC \x8C \xE5 \x85 \x84 \xE5 \xBC \x9F \xE4 \xBB \xAC \xE5 \x93 .... (Parameter 'bytesUnknown')
  + System.Text.DecoderFallbackBuffer.ThrowLastBytesRecursive(byte[])
  + System.Text.DecoderReplacementFallbackBuffer.Fallback(byte[], int)
  + Submission#7.Doo()

Is this possible in C# ?
What do I need to detect the first valid byte's position?

[Clockwork-Muse]

You can't just read from a random segment of a byte stream and get valid output, you have to read from the start. Even ignoring the issue of multi-bute characters, Unicode allows for multi-character sequences, which means finding a valid starting byte isn't sufficient. You'd have to find the start of the entire stream.

Why are you reading from the end? Why aren't you reading from the start anyways?

[John0King]

will , at least utf-8 can detect the invalid byte and use a FallbackBuffer to replace it with ? or throw , but the problem is Decoder and Fallback and FallbackBuffer don't give you a way to report those invalid bytes. I expect a callback on Decoder.GetChars so I can detect the valid byte start.

[Clockwork-Muse]

... why? What - larger picture - are you trying to do?
Why are you trying to get the start of the next valid byte (which isn't sufficient for many real-world scenarios)?

[John0King]

I'm writting/appending a big json that part of the IEnumerable<T> or IAsyncEnumerable<T> like log . every time it's done writing it will be a valid json file (end with ]), and when you append it next time, you have to remove the ] or maybe add a , before append other object. so I endup to read last 1kb buffer from the Stream and decode it to char[] then , I need to remove the white spce, finding the ] , and I may need to then finding the [ if it's an empty json array or it's completely empty.

// case1
[     ] \r\n
// case 2
no character or white spce

// case 3
[
.......,
{ .... }
]

It will waste more time if I read it from start to end.

[Clockwork-Muse]

You're making life hard for yourself, because you're misunderstanding the problem domain.
You don't actually care about invalid bytes, because you don't need to read that many. You only need to read a maximum of two characters, which are both exactly one byte. One of these can be simply ignored outright (the terminating ]). When you reopen the file, you want to open it two bytes from the end, and just manually read the byte at the cursor. It will be one of two categories:

• [ - a previously new/empty file.
• anything else (almost certainly }) - the end of the previous object.

Advance one byte. When you write the next log object, if the file was not previously empty append a comma, then write the object. You should be able to just write a JSON stream to the file stream at that point, without even needing to work with a manual UTF-8 encoding stream (and certainly without the fallback encoder). Then write the array terminator when you go to close the file.

Note, however, that many logging providers don't write a "valid" JSON object in these cases, instead writing a JSON object per line, saving them from having to backtrack like this (especially due to potential error cases if the system shuts down suddenly).
And come to that, are you sure you want to be writing your own, as opposed to using one of the many third-party providers?

[canton7]

To add to @Clockwork-Muse, even if you're not using UTF-8 and are using a multi-byte encoding like UTF-16 or UTF-32, you're still looking at it backwards.

If you want to strip a trailing ,, don't find the bytes representing the final character, turn them into a char, then see if that char is ,. Start with the char ,, turn that into the 1 or more bytes of your chosen encoding, and then see if the file ends with those bytes.

E.g. with UTF-16, if you want to see whether the file ends with a comma, start by turning , into its UTF-16 bytes 0x00 0x2C, and then see if the file ends with the bytes 0x00 0x2C and strip them if so.

That said, you should seriously consider the strategy of writing one JSON object per line of your file (make sure your serializer is configured not to output newlines), and simply ignore the surrounding [ and ]. This has many benefits including:

1. When appending an object, you don't need to faff around with [ and ]
2. When parsing the file, you can deserializer each row at a time, in isolation. Just scan forward to the next newline (or use a StreamReader etc) and deserialize that. Much more resource-friendly.
3. You can remove the first/last N records from a file easily, without needing to parse the whole file as JSON.