Skip to content

Bug Report: aws --output text with --query prints a spurious None line for the pagination resume-token hint #10449

Description

@Abuhaithem

Describe the bug

When --output text is combined with --query and a paginated response is
truncated (for example via --max-items), the AWS CLI appends a stray
None line to the output.

After the last page, the text formatter prints a NEXTTOKEN "resume token"
hint so users know how to continue paginating. That hint is pagination
metadata, but it is incorrectly run through the user's --query expression.
Evaluating a data expression such as Users[].UserName against the hint object
{"NextToken": {"NextToken": "<token>"}} returns None, and the text formatter
writes the literal string None.

In scripts that consume --output text --query line by line, the extra None
silently corrupts the data (an additional row, a wrong count, or a bogus field
value).

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The resume-token hint should render exactly as it does without --query,
because it is not part of the queried result set:

alice
NEXTTOKEN	<token>

Current Behavior

alice
None

The trailing None is the result of applying Users[].UserName to the
resume-token hint object instead of to the page data.

Reproduction Steps

aws iam list-users \
    --max-items 1 \
    --output text \
    --query 'Users[].UserName'

Any paginated operation reproduces this; --max-items is simply the easiest
way to force a truncated response (and therefore a resume token). The bug
appears whenever all of the following hold:

  1. output format is text,
  2. --query is supplied, and
  3. the response is truncated, so a resume token is generated.

Minimal, self-contained reproduction against mocked responses (no AWS account
or network required — mirrors the regression test):

from awscli.testutils import BaseAWSCommandParamsTest


class Repro(BaseAWSCommandParamsTest):
    def test_repro(self):
        self.parsed_responses = [
            {'Users': [{'UserName': 'alice', 'Path': '/',
                        'CreateDate': '2024-01-01T00:00:00Z',
                        'UserId': 'A1', 'Arn': 'arn:aws:iam::12345:user/alice'}],
             'IsTruncated': True, 'Marker': 'PAGE2'},
            {'Users': [{'UserName': 'bob', 'Path': '/',
                        'CreateDate': '2024-01-02T00:00:00Z',
                        'UserId': 'B1', 'Arn': 'arn:aws:iam::12345:user/bob'}],
             'IsTruncated': False},
        ]
        out = self.run_cmd(
            'iam list-users --max-items 1 --output text '
            '--query Users[].UserName', expected_rc=0)[0]
        # Actual:   'alice\nNone\n'
        # Expected: 'alice\nNEXTTOKEN\t<token>\n'
        print(repr(out))

Possible Solution

The bug is in awscli/formatter.py, TextFormatter.__call__:

if response.resume_token:
    # Tell the user about the next token so they can continue
    # if they want.
    self._format_response(
        {'NextToken': {'NextToken': response.resume_token}}, stream)

_format_response applies --query to whatever it is given:

def _format_response(self, response, stream):
    if self._args.query is not None:
        response = self._args.query.search(response)
    text.format_text(response, stream)

Render the resume-token hint directly with text.format_text(...), bypassing
--query, so the hint is preserved for all output cases:

if response.resume_token:
    text.format_text(
        {'NextToken': {'NextToken': response.resume_token}}, stream)

Only TextFormatter is affected. FullyBufferedFormatter (JSON and Table)
aggregates the full result with build_full_result() and applies --query
once to the entire response, so it has no separate hint emission and does not
exhibit this bug.

Additional Information/Context

  • Affects every paginated operation, not just iam list-users.
  • The hint is only emitted when a resume token exists, i.e. when the response
    is truncated (commonly --max-items), so non-truncated commands are
    unaffected.
  • Not a regression — the behavior has existed since the text formatter began
    streaming pages.

CLI version used

aws-cli/1.45.36 (reproduced on the develop branch). The same code path exists in the v2 line.

Environment details (OS name and version, etc.)

Platform independent (verified on Linux); no platform-specific behavior is involved.

Metadata

Metadata

Labels

bugThis issue is a bug.investigatingThis issue is being investigated and/or work is in progress to resolve the issue.p3This is a minor priority issue

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions