Skip to content

Encoding config support #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

iamgd67
Copy link

@iamgd67 iamgd67 commented Jan 22, 2020

add python script encoding config support, to handle some error like
UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 797: illegal multibyte sequence

…`UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 797: illegal multibyte sequence`
@adamtheturtle
Copy link
Owner

Thank you @iamgd67 . I know this is an old PR, but I would be happy to get a fix in for your issue.

Could you provide either/both of (ideally both) the following:

  • An issue which describes how to reproduce this problem.
  • A test case in this PR which fails without the fix.

@iamgd67
Copy link
Author

iamgd67 commented Jun 22, 2020

in my situation, it failes because when I process files with utf-8 encoding and contains chinese character on windows platform, looks like on windows open use gbk encoding default, and decode file failed.

@zhu
Copy link

zhu commented Jan 20, 2021

It seems that ast.parse() can handle the encoding, so we can use open(filename, mode='rb') to solve this problem.
The gbk encoding script file should add # coding: gbk line, see PEP263.

@adamtheturtle
Copy link
Owner

Thanks for the work @zhu . Please ping me when tests are passing or I can help!

@fabswt
Copy link

fabswt commented Feb 11, 2023

Getting the same kind of error:

(venv) root@web:/var/www/html/python-tests/pronunciation-demo# pip-extra-reqs .
Traceback (most recent call last):
  File "/var/www/html/python-tests/pronunciation-demo/venv/bin/pip-extra-reqs", line 8, in <module>
    sys.exit(main())
  File "/var/www/html/python-tests/pronunciation-demo/venv/lib/python3.9/site-packages/pip_check_reqs/find_extra_reqs.py", line 211, in main
    extras = find_extra_reqs(
  File "/var/www/html/python-tests/pronunciation-demo/venv/lib/python3.9/site-packages/pip_check_reqs/find_extra_reqs.py", line 35, in find_extra_reqs
    used_modules = common.find_imported_modules(
  File "/var/www/html/python-tests/pronunciation-demo/venv/lib/python3.9/site-packages/pip_check_reqs/common.py", line 151, in find_imported_modules
    content = file_obj.read()
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 64: invalid start byte

Maybe more annoying is that it won't tell me on which file scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants