File tree Expand file tree Collapse file tree 1 file changed +4
-10
lines changed Expand file tree Collapse file tree 1 file changed +4
-10
lines changed Original file line number Diff line number Diff line change @@ -318,16 +318,10 @@ def untokenize(iterable):
318318 with at least two elements, a token number and token value. If
319319 only two tokens are passed, the resulting output is poor.
320320
321- Round-trip invariant for full input:
322- Untokenized source will match input source exactly
323-
324- Round-trip invariant for limited input:
325- # Output bytes will tokenize back to the input
326- t1 = [tok[:2] for tok in tokenize(f.readline)]
327- newcode = untokenize(t1)
328- readline = BytesIO(newcode).readline
329- t2 = [tok[:2] for tok in tokenize(readline)]
330- assert t1 == t2
321+ The result is guaranteed to tokenize back to match the input so
322+ that the conversion is lossless and round-trips are assured.
323+ The guarantee applies only to the token type and token string as
324+ the spacing between tokens (column positions) may change.
331325 """
332326 ut = Untokenizer ()
333327 out = ut .untokenize (iterable )
You can’t perform that action at this time.
0 commit comments