UnicodeDecodeError when copy android 14 system apk

[imianz]

Traceback (most recent call last):
  File "/home/user/.local/bin/adbsync", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/__init__.py", line 374, in main
    files_tree_source = fs_source.get_files_tree(path_source, follow_links = args.copy_links)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Base.py", line 45, in get_files_tree
    return self._get_files_tree(tree_path, statObject, follow_links = follow_links)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Base.py", line 33, in _get_files_tree
    tree[filename] = self._get_files_tree(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Base.py", line 33, in _get_files_tree
    tree[filename] = self._get_files_tree(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Base.py", line 30, in _get_files_tree
    for filename, stat_object_child, in self.lstat_in_dir(tree_path):
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Android.py", line 176, in lstat_in_dir
    for line in self.adb_shell(["ls", "-la", path]):
  File "/home/user/.local/share/pipx/venvs/betteradbsync/lib/python3.12/site-packages/BetterADBSync/FileSystems/Android.py", line 87, in adb_shell
    adb_line = adb_line.decode(self.adb_encoding).rstrip("\r\n")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 65: invalid start byte

Hi,
this error occurs during backup my data using "adbsync --show-progress pull".
The file responsible for the error is named "Connessione salute 14.apk" and is part of android system, the english version should be "Health Connect 14.apk".

Here the first 112 bytes of this file:

Connessione Salute 14.apk                                                      
00000 0000:  50 4B 03 04 0A 00 00 08  00 00 00 00 21 3A E9 9A  PK...... ....!:..
00000 0010:  A5 A9 0E 00 00 00 0E 00  00 00 2F 00 0B 00 4D 45  ........ ../...ME
00000 0020:  54 41 2D 49 4E 46 2F 61  6E 64 72 6F 69 64 78 2E  TA-INF/a ndroidx.
00000 0030:  61 63 74 69 76 69 74 79  5F 61 63 74 69 76 69 74  activity _activit
00000 0040:  79 2D 6B 74 78 2E 76 65  72 73 69 6F 6E FE CA 00  y-ktx.ve rsion...
00000 0050:  00 35 D9 03 00 04 00 00  31 2E 39 2E 30 2D 61 6C  .5...... 1.9.0-al
00000 0060:  70 68 61 30 33 0A 50 4B  03 04 0A 00 00 08 00 00  pha03.PK ........
                            ^^

[chrostino]

could you tell me how to check which file is responsible for the issue? I might be running into the same problem here

[GeorgiKarapetrov]

Same issue, please help:

adbsync --del push $HOME/$SOURCE_DIR/ $TARGET_DIR

Traceback (most recent call last):
  File "/usr/bin/adbsync", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/__init__.py", line 379, in main
    files_tree_destination = fs_destination.get_files_tree(path_destination, follow_links = args.copy_links)
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Base.py", line 45, in get_files_tree
    return self._get_files_tree(tree_path, statObject, follow_links = follow_links)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Base.py", line 33, in _get_files_tree
    tree[filename] = self._get_files_tree(
                     ~~~~~~~~~~~~~~~~~~~~^
        self.join(tree_path, filename),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        stat_object_child,
        ^^^^^^^^^^^^^^^^^^
        follow_links = follow_links)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Base.py", line 33, in _get_files_tree
    tree[filename] = self._get_files_tree(
                     ~~~~~~~~~~~~~~~~~~~~^
        self.join(tree_path, filename),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        stat_object_child,
        ^^^^^^^^^^^^^^^^^^
        follow_links = follow_links)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Base.py", line 30, in _get_files_tree
    for filename, stat_object_child, in self.lstat_in_dir(tree_path):
                                        ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Android.py", line 176, in lstat_in_dir
    for line in self.adb_shell(["ls", "-la", path]):
                ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/BetterADBSync/FileSystems/Android.py", line 87, in adb_shell
    adb_line = adb_line.decode(self.adb_encoding).rstrip("\r\n")
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 59: invalid start byte

I am trying with different encodings now but I can't even find the offending path.

[GeorgiKarapetrov]

Is maybe this relevant?
google#62

[jpstotz]

The file content is not relevant here as the error occurs when parsing the output of ls -la, thus a filename is the problem.

I am not sure if --show-progress will show you the causing file name as Python fails to decode the actual filename from the ls -la shell output. So if Python fails to decode the filename it can not print it.

I had similar problem with an mp3 file that had invalid unicode chars in it's file-name. I identified it by modifying the source code adb_line = adb_line.decode(self.adb_encoding).rstrip("\r\n") in Android.py to:

          try:
                adb_line = adb_line.decode(self.adb_encoding).rstrip("\r\n")
            except UnicodeDecodeError as e:
                errmsg = "Line to decode without the problematic characters: {}".format(adb_line.decode(self.adb_encoding,errors='ignore'))
                e.args += (errmsg,)
                raise e

This way the error includes the adb shell output line that fails to decode. The invalid character has been removed so we can print the output and see that file name causing the decode problem.

Edit: I have create a pull request containing the changes: #55

[GeorgiKarapetrov]

Sure, it's the filenames. I'll try with --show-progress and with the new error message and report back. However, I can only reproduce this at home for now and I rarely get computer time there.

Does maybe #45 deal with offending paths? It obviously does multiple things but maybe an atomic change can be extracted from it if it fixes the issue.

Is it possible to copy the files without changing the names like this? I mean the files already exist with these file names, why shouldn't they exist with the same names elsewhere?

[zorone]

Oh... I see. That's why I feel like I have been there before.

0a is newline character \n, I recommend you to rename them if plausible. Idk why some apps does contain newline in their name like that. It won't work well on Windows, which is what I mainly used.

But... I expect it should fail at file transfer operation, not here!

I have tackled this issue a bit in the past. I also get stuck at decode method as well. We might need to send this problem upstream to Python team, as we can't do anything except throwing for exception like @jpstotz did.

[imianz]

I believe the problem lies in the fact that the files are being treated as text instead of binary. Binary files can contain an infinite number of character combinations, some of which may be interpreted as invalid Unicode sequences. File copying should only be done in binary mode.

[jpstotz]

@imianz adb is not ftp, files are always pulled in "binary" mode. The problem is here the file name when listing it via ls -la. Not sure how it happens but some users have file(s) on their phone with invalid characters (when interpreted as utf-8 string) and this breaks listing them in better-adb-sync.

Unless these problems are fixed in the file-system better-adb-sync can not access those files. So the user needs to fix this which is of course pretty complicated if you don't know which file causes the problem.

[GeorgiKarapetrov]

I settled on renaming the offending files. Works fine now.