Gawk formatting issues whenever autocomplete is triggered

[SpiderUnderUrBed]

The issue is I get Invalid multibyte data detected. There may be a mismatch between your data and your locale each time autocomplete is invoked with ble.sh

Summary:

$ ble summary

The summery subcommand is not avalible:

[spiderunderurbed@daspidercave:~]$ ble summary
ble (ble.sh): unrecognized subcommand 'summary'.
[ble: exit 2]

[spiderunderurbed@daspidercave:~]$ ble --help
usage: ble [SUBCOMMAND [ARGS...]]

SUBCOMMAND
  # Manage ble.sh
  attach  ... alias of ble-attach
  detach  ... alias of ble-detach
  update  ... alias of ble-update
  reload  ... alias of ble-reload
  help    ... Show this help
  version ... Show version
  check   ... Run unit tests

  # Configuration
  opt     ... alias of bleopt
  bind    ... alias of ble-bind
  face    ... alias of ble-face
  hook    ... alias of blehook
  sabbrev ... alias of ble-sabbrev
  palette ... alias of ble-color-show


[spiderunderurbed@daspidercave:~]$ 

Here is it found through the keybinds:

GNU bash, version 5.3.3(1)-release (x86_64-pc-linux-gnu) [NixOS 25.11 (Xantusia)]
ble.sh, version 0.4.0-devel3+1a5c451c (noarch) [git 2.39.0, GNU Make 4.3, GNU Awk 5.1.1, API: 3.1 (GNU MPFR 4.1.0-p13, GNU MP 6.2.1)]
bash-completion, version 2.16.0 (hash:480ffcc6a751e55621ec526eb5dea7a0d86d9e72, 17877 bytes) (noarch)
locale: LANG=en_NZ.UTF-8 LC_ADDRESS=en_NZ.UTF-8 LC_IDENTIFICATION=en_NZ.UTF-8 LC_MEASUREMENT=en_NZ.UTF-8 LC_MONETARY=en_NZ.UTF-8 LC_NAME=en_NZ.UTF-8 LC_NUMERIC=en_NZ.UTF-8 LC_PAPER=en_NZ.UTF-8 LC_TELEPHONE=en_NZ.UTF-8 LC_TIME=en_NZ.UTF-8
terminal: TERM=xterm-256color wcwidth=15.0-west/15.0-2+ri, konsole:220380 (1;115;0)

Actual issue (directly):

[spiderunderurbed@daspidercave:~]$ ls 'gawk: cmd. line:112: (FILENAME=- FNR=7) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale
'sch

Whenever I do ls I always get this issue with gawk

[akinomyoga]

Could you check the behavior with the latest master version of ble.sh?

[akinomyoga]

What is the result of the following command?

$ ble/bin/gawk -W version

[akinomyoga]

Does the issue stop when you set the following option?

$ bleopt complete_auto_complete=
$ ls <input something> # Does the issue reproduce here?

I also would like to know the result of the following command after the error message is shown:

$ ls 'gawk: cmd. line:112:... # Please cause the error message in the session.
$ complete -p ls         # After the error message is shown, what is the result of this command?

Also, does the issue only happen with a specific command ls? Or does it happen with any commands?

[SpiderUnderUrBed]

I updated again, this issue has persisted the last few updates of my distro (like the last few months), it doesn't seem to be here for once but it could still be present so ill test it more tommorow, maybe it was fixed in this version? ill test to make sure and if it is ill close the issue, previously it was happening with any command. Also i tried adding bleopt complete_auto_complete=, it didnt seem to contribute much, mostly on account for its hard for me to see that issue again and see if it made a diffrence.

[akinomyoga]

I updated again, this issue has persisted the last few updates of my distro (like the last few months),

Sorry, I didn't suggest updating the ble.sh package within your distro. I actually asked to check the behavior with the latest master branch of ble.sh, which can be installed using the instructions in the README in the repository. The version information you provided implies that you had been using ble.sh from two years ago, which implies that the ble.sh version distributed by your distro is too old. In fact, the package in nixpkgs hasn't been updated for two years, and it's useless to update ble.sh through nixpkgs.

maybe it was fixed in this version?

I don't have an idea, but as a first step, it is good to check if something has changed during these two years.

[HrishabhMittal]

I am facing this exact same issue. I installed ble.sh not from a package manager but from this github. I have also tried updating ble.sh using ble-update but the issue persists

I am only facing the issue while using yay. Here's how my terminal looks when i type yay -S some

$ yay -S sbash: warning: command substitution: ignored null byte in input
sombash: warning: command substitution: ignored null byte in input
somebash: warning: command substitution: ignored null byte in input
bash: warning: command substitution: ignored null byte in input
gawk: cmd. line:112: (FILENAME=- FNR=1) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale
some

the issue doesn't reproduce when $ bleopt complete_auto_complete= is done but happens when i try to tab complete.
also while tab completing, some text which looks like regex expressions gets printed while also containing non printable characters.

just after this error

$ complete -p ls
complete -F _comp_complete_longopt ls

this is the gawk version on my system

$ ble/bin/gawk -W version
GNU Awk 5.3.2, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.2, GNU MP 6.3.0)

[HrishabhMittal]

this is the magled output i get after bleopt complete_auto_complete= is run and i try to tab complete

[akinomyoga]

@HrishabhMittal Thanks, I could reproduce the problem in my environment. After investigation, it turned out that the yay completion generates invalid data, which causes the failure of ble.sh in processing the data using gawk.

I can reproduce the invalid output of the yay completion even in Bash sessions without ble.sh:

[murase@aventura 0 ble.sh]$ yay -S s[tab]bash: warning: command substitution: ignored null byte in input
[tab]bash: warning: command substitution: ignored null byte in input

Display all 640 possibilities? (y or n)y
s}^\�^F�
s$h�2�1�"^T^Y���^D�8�9�@�_hm�;���^W^^�/�+a^F^N��o�^Z]1��Cv^?;^^��='9�^o�^N�
s��$�ҠS^H;*_[^D�^F��<k�f��ݥ2�F��ұ^D����"�^H^NZ^W^N`�n��)��^R[,'�h즈9�]�^K
s1�w�^XW��ʃq�^OOr��^W^T���T^_�#Zm7R6����˄}W]^]V^Y���'�6Ik9O�0
s2n-tls
s^Y�=�[^N��&_��^N2��^N^^]QH^K�З8\
s3cmd
s3fs-fuse
s3rver
s4Ȟ�%�w�n��:����^Q@�c
s<^\�@�5^_^B�^G�^X(ߍ�rT`��4U���^\�]�^V�Y^W6&^\�^H�JL�^X
s6C�0�W'���]¤Y!`o��'��^R^F:V�^P�7�^[
saB^D�^PB>�^R�^\^N
sad
s�a#;�e�l�(Zk��7cޓ.������eti�^]6˽&��?��0J���e�j��K��G�|N�^LP��|�L-�Q;<+�>��!��ܧ^U%(T0]�c^A��^L^B�^U�^K�^N^N^P�0�]J"%��1E�^A4�z^V��Gv^]ai�l^]�^]���瘴�F/��$݌���mI�^\�"n^K��
sage-data-cunningham_tables
sage-data-elliptic_curves
sage-data-graphs
sage-data-polytopes_db
sagemath
sagemath-doc
--More--

Then, this is an issue with the yay completion and should be fixed there.

The original issue reported by @SpiderUnderUrBed seems to be caused in the completion for an arbitrary command (but not specifically for yay), so it isn't caused by the yay completion. However, considering the same error message by gawk in a similar situation, I suspect that the completion settings in @SpiderUnderUrBed's environment generate invalid data for a different reason. Then, this isn't a bug of ble.sh either.

By the way, I observed a ble.sh crash with the setting shopt -s failglob in my environment, which shouldn't happen even if the completion settings are broken. I'll also check it.

[akinomyoga]

By the way, I observed a ble.sh crash with the setting shopt -s failglob in my environment, which shouldn't happen even if the completion settings are broken. I'll also check it.

OK, I identified that failglob happens on this line (edit: no, this line) inside the shell function _yay_pkg provided by the yay completion. This is also a problem of the yay completion.

[akinomyoga]

The error message bash: warning: command substitution: ignored null byte in input is caused on this line.

It seems yay -Pc outputs binary data. yay -P --help says:

show specific options:
    -c --complete         Used for completions

man yay says:

HOW OPTIONS (APPLY TO -P AND --show)
       -c, --complete
              Print a list of all AUR and repo packages. This allows shell completion and is not intended to be used directly by the user.

so yay -Pc is an option specifically designed for completion. I'm not sure why it outputs broken data. yay -Pc seems to be implemented here, but this part only dumps an existing file specified to the argument completionPath. The argument is specified here as run.Cfg.CompletionPath. The default value comes from here, and completionFileName is defined to be completion.cache here. In fact, ~/.cache/yay/completion.cache seems to contain garbage.

[murase@aventura 1 ble.sh]$ head ~/.cache/yay/completion.cache
D�~5�;u�����Gu�Ǜ�su4÷����k!p�뫿���׷׏���cuU�àk$����y���z�Է�q����Nj��v�n�ܬTu�$"[w8�臗�߶V�^���`D�,�����V�|}u�:*{�Ç?[�8^#W5Ժ�t���@�ie¿*��>H.��_=��t�\��U�'�:�/�Sul2�n�p�K5����i�!a�PNHP:yB�^�+��v��R���췷�/u7.ͷ��ߕjz�2��ݎ�^�ř�8)ildG�V���[����mls�
 x�l'U�{�G��À���g���~���p#����*�KR�RN���G�VY��44�F
                                                  �Á��aC�Ў�ގ5}�Â�i���/ڞ��M��G_~X���k5/�&���agjx�L�[���`Ngm+���C��{���]-&�ط��~�{����4����Шn�ROs��s��O���P-��������ˀO����j���6�����bzu�P�Ӭ�������@u�7�p#��7^.�ꅺ<�\�L�bu���qq
...

[akinomyoga]

The cache content is created in this function. It downloads https://aur.archlinux.org/packages.gz and splits the contents into lines without extracting the .gz archive. Maybe the behavior of the module net/http was changed at some point.

[SpiderUnderUrBed]

I still get the gawk issue, I thought it was fixed upstream or here, it wasnt:

[spiderunderurbed@daspidercave:~]$ mkdir ai-tgawk: cmd. line:112: (FILENAME=- FNR=7) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale
ai-toolsgawk: cmd. line:112: (FILENAME=- FNR=7) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale
gawk: cmd. line:112: (FILENAME=- FNR=7) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale

If I do:

bleopt complete_auto_complete=

I do not get the error afterwards

Whats the issue with gawk and how do i debug it?

Also here are the complete commands:

[spiderunderurbed@daspidercave:~]$  complete -p ls  
complete -F _comp_complete_longopt ls

[spiderunderurbed@daspidercave:~]$  complete -p mkdir
complete -F _comp_complete_longopt mkdir

[spiderunderurbed@daspidercave:~]$ 

because I really like the autocomplete so i would like to fix this issue.

[akinomyoga]

@HrishabhMittal The issue with the yay completion seems to have been fixed in the upstream: Jguer/yay#2710. Now, you can update yay to the latest version 12.5.3 and run yay -Pcc to fix the issue.

[HrishabhMittal]

@akinomyoga thanks for the update!
I have updated my system and can confirm that I am no longer facing any problems with auto-completions