Not detecting unicode symbols in fp-telegram

[FeroxGraxaim]

Good morning!
I'm trying to make my BOT detect unicode symbols lilke fractions in a TRegExpr expression:
'((?:\d+\s?(?:1/8|1/4|3/8|1/2|5/8|3/4|7/8|\¼|\½|\¾|\⅛|\⅜|\⅝|\⅞)|\d+(?:[.,]\d+)?))#',

But the unicode symbols are not detected in a message, but are detected in a string that I put the symbol on it manually.
So for example I put a message with ¼" and it's not converting to milimiters, I need to put 1/4" to work.

How to solve this problem?

[Al-Muhandis]

Hi!
Thanks for the feedback.
Theoretically, such symbols are supported by the fp-telegram. But I didn't fully understand the problem. You may be accessing the immediate API response in a raw data (/u...), or the TRegExpr function is not working properly. If you can provide at least a part of the code where the problem is present, or ideally send me a minimal example of the problem, I could try to help you.

[FeroxGraxaim]

Thanks for answering!
The code that reads the message text is:

procedure TMainClass.DetectMessage(ASender: TObject; AMessage: TTelegramMessageObj);
var
  MsgText, Response, AiReply: string;
  RepMsgID: integer;
  JsonResponse: TJSONData;
  AiMsg:   TJSONObject;
  choices: TJSONArray;
  i: integer;
label
  Send, Mesure;
begin
  try
    MsgText  := AMessage.Text;
    WriteLn('Raw message: ', AMessage.Text);
    for i := 1 to Length(AMessage.Text) do
  writeln(Format('Char %d: %s (ord=%d hex=%x)', [i, AMessage.Text[i], Ord(AMessage.Text[i]), Ord(AMessage.Text[i])]));
    RepMsgID := AMessage.MessageId;

    if CanBeFun(AMessage.ChatId, True) then
    begin
      Response := FunnyMsgs.FunnyMessage(MsgText);
      if Response <> 'null' then
        goto Send
        else goto Mesure;
    end;
    Mesure:
    Response := Mesurement.ConvertValueStr(MsgText);
    if Response <> 'null' then
      goto Send
    else
      Exit;

    Send:
      Bot.sendMessage(Response, pmDefault, False, nil, RepMsgID, True);
  except
    On E: Exception do
    begin
      writeln('Error: ' + E.Message);
      Bot.sendMessage(FATAL_ERR, pmDefault, False, nil, RepMsgID, True);
      Halt(1);
    end;
  end;
end;                         

While AMessage is a TTelegramMessageObj.

I put a debug on the code as you can see, and it returns this value:

Raw message: 72?"
Char 1: 7 (ord=55 hex=37)
Char 2: 2 (ord=50 hex=32)
Char 3: ? (ord=63 hex=3F)
Char 4: " (ord=34 hex=22)

The "?" should be one of the unicode fraction symbols, but as you can see in the char code, it shows the "?" code instead, so the problem is not with the console, but with the fptelegram message reading.
The problem is not with TRegExpr because I tested with a custom variable that I put the symbol, and it detected.

Link for the repository:
https://github.com/FeroxGraxaim/imperial-metric-telegram-bot

[Al-Muhandis]

1. As far as I understood your code, it is a typical error when working with UTF8 in Lazarus. You cannot fit an emoji code point into a single-byte Char. You cannot use indexed byte access to enumerate code points in a UTF8 string. You can use, for example, UTF8Pos and similar UTF8 string functions.
2. This is not related to the problem, but I strongly recommend that you never use goto. Exit is ok. Goto is not.

[FeroxGraxaim]

It's not an emoji, it's a fraction symbol like ⅛ for example.
And it doesn't work through fp-telegram, but works when I put it directly in the variable, using ReadLn or in the coding by itself.
I tested on Linux terminal, and the symbol doesn't show in the debug line, but in any other case I use the symbols, it shows on Linux terminal with no problem.

What I wanna make, is to detect the fraction symbol to convert imperial to metric units.

[Al-Muhandis]

AMessage.Text[i] do not addressing to code point/symbol. It is addressing to byte/ansii charcter.
⅛ etc is 3-byte UTF8 character. Not ACSII

[FeroxGraxaim]

Yes, but with or without this debug, the ⅛ symbol and other similar symbols are not being detected.
And I discovered that it's a problem with JSON, I made a simple program without fp-telegram to read BOT messages and it also didn't detect the UTF8 symbols.
But in the TFPHTTPClient test, it showed the character correct code, but not converted it to the symbol and put a "?" instead.

[Al-Muhandis]

Can You send me [minimal] example? And how to produce the problem. I'll double check everything.