lib/tutf8e: refactoring for flb_encoding (WIP)

[bluebike]

WIP: in process of enable encoding first few changes to tutf8e encoding library.

---

Summary

• separate encoding* functions removed.
• length_functions return TUTF8E_SAME if no changes needed in string
• multiple modes modes for handling invalid chars:
  • KEEP: (default) just encode byte to utf8
  • IGNORE: skip character
  • FAIL: fail if invalid chars
  • REPLACEMENT: replace invalid character with unicode 0xFFFD
  • QUESTION: relace invalid characters with '?' char.
• tests functions made for invalid char handling.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[bluebike]

@nigels-com would like to see some comments of my changes...
Mainly I tried to make this handle invalid chars + get info is string doesn't need changes (optimization).

[nigels-com]

I'll take a closer look.

Seems like a significant rework, some further discussion of the goals and rationale would be helpful, I think.

[nigels-com]

My first question from a fluent-bit perspective is if there is a realistic need for all the modes?
The expectation is to configure this per input? Which mode is the intended default?
From an embedded perspective, I'd like to only link the code/mode that is needed for a particular application.

[bluebike]

@nigels-com Probably overkill having different modes.
But different users of fluent-bit could have different requirements.
Having different modes doesn't really add that much code I think, mostly test code.

Anyway. Just failing with bad chars is not enough (for me).
Also way to see that is encoding really needed is important for me, because in our case probably 99.9% of all messages are just plain ASCII and don't really need encoding/decoding.

[nigels-com]

@bluebike You're saying that the advantage of REPLACEMENT or QUESTION modes are that they'll flag message streams that are not actually encoded as expected, but at least the ASCII parts will succeed. Do you feel that KEEP and IGNORE are also important modes?

[nigels-com]

I'm not convinced the TUTF8E_SAME mode is worth the additional complexity. Have you been able to measure any actual throughput, memory or CPU improvements?

[nigels-com]

As an aside, I filed some issues over on the tutf8e repo, as reminders to myself.

nigels-com/tutf8e#8
nigels-com/tutf8e#6

[bluebike]

Ok.. .did some changes...

• *olen used as giving output buffer length + returning length of output utf8 string.
• some comments fixed.
• removed extra strlen
• added test for too short output buffer returning TUTF8E_TOOLONG

[nigels-com]

@bluebike Please take a look at PR #2326 to see if this arrangement strikes a good balance of features and "tinyness".

Rather than the various iconv modes, it adds support for a UTF-8 proxy for each invalid input character.
If the proxy is NULL, the API will continue erroring for invalid input.

[bluebike]

(this goes nowhere)