argparse: action "extend" with 1 parameter splits strings into characters

[strjan]

BPO	40365
Nosy	@rhettinger, @berkerpeksag, @asottile, @isidentical, @jonathanhaigh
Files	argparse_bug.py: Minimal example

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-22.17:42:36.895>
labels = ['type-bug', '3.8', '3.9']
title = 'argparse: action "extend" with 1 parameter splits strings into characters'
updated_at = <Date 2020-06-09.13:20:45.393>
user = 'https://bugs.python.org/strjan'

bugs.python.org fields:

activity = <Date 2020-06-09.13:20:45.393>
actor = 'Jonathan Haigh'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2020-04-22.17:42:36.895>
creator = 'strjan'
dependencies = []
files = ['49086']
hgrepos = []
issue_num = 40365
keywords = []
message_count = 4.0
messages = ['367033', '367045', '370898', '371103']
nosy_count = 7.0
nosy_names = ['rhettinger', 'berker.peksag', 'paul.j3', 'Anthony Sottile', 'BTaskaya', 'strjan', 'Jonathan Haigh']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40365'
versions = ['Python 3.8', 'Python 3.9']

Linked PRs

• gh-84545: Clarify the 'extend' action documentation in argparse #125870
• [3.13] gh-84545: Clarify the 'extend' action documentation in argparse (GH-125870) #125964
• [3.12] gh-84545: Clarify the 'extend' action documentation in argparse (GH-125870) #125965

[strjan]

When positional argument with action='extend' is given to parse only one string argument, the string is splitted into individual characters.

The problem occures, when no "nargs" is given; it makes sense that nargs is requiered, though at least notion in documentation would be nice, as the issue does not occure with action='append'

Minimal example included.

[paulj3]

This is a consequence of Python's own definition of append vs extend

In [730]: alist = []
In [731]: alist.append('astring')
In [732]: alist
Out[732]: ['astring']
In [733]: alist.extend('astring')
In [734]: alist
Out[734]: ['astring', 'a', 's', 't', 'r', 'i', 'n', 'g']
In [735]: alist.extend(['astring'])
In [736]: alist
Out[736]: ['astring', 'a', 's', 't', 'r', 'i', 'n', 'g', 'astring']

Normally 'add_argument' doesn't check for valid parameters, but some Action subclasses do their own checking,

'extend' inherits the nargs==0 test from 'append'. (extend just modifies the __call__, not the __init__ method.

But I wonder, was this situation discussed in the original bug/issue?

[jonathanhaigh]

The situation for type=int and unspecified nargs or nargs="?" is also surprising:

Python 3.8.3 (default, May 21 2020, 12:19:36) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import argparse
>>> p = argparse.ArgumentParser()
>>> p.add_argument("--c", action="extend", type=int)
_ExtendAction(option_strings=['--c'], dest='c', nargs=None, const=None, default=None, type=<class 'int'>, choices=None, help=None, metavar=None)
>>> p.parse_args("--c 1".split())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 2006, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1946, in consume_optional
    take_action(action, args, option_string)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1171, in __call__
    items.extend(values)
TypeError: 'int' object is not iterable
>>> p = argparse.ArgumentParser()
>>> p.add_argument("--c", action="extend", type=int, nargs="?")
_ExtendAction(option_strings=['--c'], dest='c', nargs='?', const=None, default=None, type=<class 'int'>, choices=None, help=None, metavar=None)
>>> p.parse_args("--c 1".split())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 2006, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1946, in consume_optional
    take_action(action, args, option_string)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/jonathan/.pyenv/versions/3.8.3/lib/python3.8/argparse.py", line 1171, in __call__
    items.extend(values)
TypeError: 'int' object is not iterable
>>>

I suggest that the default nargs for extend should be "*" or "+" and an exception should be raised if nargs is given as "?". I don't see the current behaviour with unspecified nargs or nargs="?" being useful (and it certainly is surprising). In both cases, I think the least surprising behaviour would be for extend to act the same as append (or for an exception to be raised).

But I wonder, was this situation discussed in the original bug/issue?
Doesn't look like it:
https://bugs.python.org/issue23378
aa32a7e

[jonathanhaigh]

> But I wonder, was this situation discussed in the original bug/issue?
Doesn't look like it:

I was looking at the wrong PR link. This has more discussion: #13305.

nargs is discussed but I'm not sure it was realized that the nargs=None and nargs="?" cases would act in the way seen here rather than acting like append.

Having a default nargs of "+" was suggested but that suggestion was not addressed.

I suggest that the default nargs for extend should be "*" or "+" and an exception should be raised if nargs is given as "?".

I'm not convinced about that any more. Using append's behaviour is probably more reasonable for nargs=None and nargs="?".

[serhiy-storchaka]

I do not think that we should forbid nargs=None and nargs='?'. It may be surprising for some users when they experience this the first time, but it can be the needed behavior for others. And it can already be used in the user code. This is why it is even hard to change the default value of nargs.

But these peculiarities should be documented.