Clarify error when ‘tokenize.detect_encoding’ receives text

[bignose]

BPO	23297
Nosy	@vstinner, @bitdancer, @berkerpeksag

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-01-22.04:40:26.874>
labels = ['3.7', '3.8', 'type-bug', 'library', 'docs']
title = 'Clarify error when \xe2\x80\x98tokenize.detect_encoding\xe2\x80\x99 receives text'
updated_at = <Date 2019-04-28.22:48:30.499>
user = 'https://bugs.python.org/bignose'

bugs.python.org fields:

activity = <Date 2019-04-28.22:48:30.499>
actor = 'bignose'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)']
creation = <Date 2015-01-22.04:40:26.874>
creator = 'bignose'
dependencies = []
files = []
hgrepos = []
issue_num = 23297
keywords = []
message_count = 9.0
messages = ['234471', '234472', '234474', '234481', '235831', '236338', '278536', '341028', '341038']
nosy_count = 6.0
nosy_names = ['vstinner', 'r.david.murray', 'docs@python', 'bignose', 'berker.peksag', 'Pod']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23297'
versions = ['Python 3.7', 'Python 3.8']

[bignose]

In tokenize.detect_encoding is the following code::

    first = read_or_stop()
    if first.startswith(BOM_UTF8):
        # …

The read_or_stop function is defined as::

    def read_or_stop():
        try:
            return readline()
        except StopIteration:
            return b''

So, on catching StopIteration, the return value will be a byte string. The detect_encoding code then immediately calls sartswith, which fails::

File "/usr/lib/python3.4/tokenize.py", line 409, in detect_encoding
  if first.startswith(BOM_UTF8):

TypeError: startswith first arg must be str or a tuple of str, not bytes

One or both of those locations in the code is wrong. Either read_or_stop should never return a byte string; or detect_encoding should not assume it can call startswith on the result.

[bignose]

Possibly related to bpo-9969.

[bitdancer]

bytes does support startswith:

>>> b'abc'.startswith(b'a')
True

[vstinner]

I don't understand why do you consider that this issue is a bug. Can you show an example where detect_encoding() raises an exception?

[Pod]

Not the OP, but I find this message a bug because it's confusing from the perspective of a user of the tokenize() function. If you give tokenize a readlines() that returns a str, you get this error message that confusingly states that something inside tokenize must be a string and NOT a bytes, even though the user gave readlines a string, not a bytes. It looks like an internal bug.

Turns out it's because the contact changed from python2 to 3.

Personally, I'd been accidentally reading the python2 page for the tokenize library instead of python3, and had been using tokenize.generate_tokens in my python 3 code which accepts a io.StringIO just fine. When I realising my mistake and switched to the python3 version of the page I noticed generate_tokens is no longer supported, even though the code I had was working, and I noticed that the definition of tokenize had changed to match the old generate_tokens (along with a subtle change in the definition of the acceptable readlines function).

So when I switched from tokenize.generate_tokens to tokenize.tokenize to try and use the library as intended, I get the same error as OP. Perhaps OP made a similar mistake?

To actually hit the error in question:

        $ cat -n temp.py
             1  import tokenize
             2  import io
             3
             4
             5  byte_reader = io.BytesIO(b"test bytes generate_tokens")
             6  tokens = tokenize.generate_tokens(byte_reader.readline)
             7
             8  byte_reader = io.BytesIO(b"test bytes tokenize")
             9  tokens = tokenize.tokenize(byte_reader.readline)
            10
            11  byte_reader = io.StringIO("test string generate")
            12  tokens = tokenize.generate_tokens(byte_reader.readline)
            13
            14  str_reader = io.StringIO("test string tokenize")
            15  tokens = tokenize.tokenize(str_reader.readline)
            16
            17
        
        $ python3 temp.py
        Traceback (most recent call last):
          File "temp.py", line 15, in <module>
            tokens = tokenize.tokenize(str_reader.readline)
          File "C:\work\env\python\Python34_64\Lib\tokenize.py", line 467, in tokenize
            encoding, consumed = detect_encoding(readline)
          File "C:\work\env\python\Python34_64\Lib\tokenize.py", line 409, in detect_encoding
            if first.startswith(BOM_UTF8):
        TypeError: startswith first arg must be str or a tuple of str, not bytes

[bitdancer]

The error message could indeed be made clearer by turning it into a message that tokenize itself requires bytes input. Or, more likely, the additional error handling needs to be in detect_encoding.

[berkerpeksag]

It looks like this can also be fixed by bpo-12486.

[berkerpeksag]

The original problem has already been solved by making tokenize.generate_tokens() public in bpo-12486.

However, the same exception can be raised when tokenize.open() is used with tokenize.tokenize(), because it returns a text stream:

https://github.com/python/cpython/blob/da63b321f63b697f75e7ab2f88f55d907f56c187/Lib/tokenize.py#L396

hello.py
--------

def say_hello():
    print("Hello, World!")

say_hello()

text.py
-------

import tokenize

with tokenize.open('hello.py') as f:
    token_gen = tokenize.tokenize(f.readline)
    for token in token_gen:
        print(token)

When we pass f.readline to tokenize.tokenize(), the second call to detect_encoding() fails, because f.readline() returns str.

In Lib/test/test_tokenize.py, it seems like tokenize.open() is only tested to open a file. Its output isn't passed to tokenize.tokenize(). Most of the tests either pass the readline() method of open(..., 'rb') or io.BytesIO() to tokenize.tokenize().

I will submit a documentation PR that suggests to use tokenize.generate_tokens() with tokenize.open().