kinesis list-shards injecting NextToken

[jpassaro]

Describe the bug

when i call list-shards for an existing stream-name or stream-arn, without providing --next-token, awscli complains that i shouldn't provide stream ident and next token at once.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Expect a valid ListShards response. Note that this works in boto3:

Python 3.12.9 (main, Feb  4 2025, 14:38:38) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.40.0'
>>> str(
...   boto3.Session(profile_name="test", region_name="us-east-1")
...     .client("kinesis")
...     .list_shards(StreamName="geaccounting-datos-v2-data-qa")
... )[:50]
"{'Shards': [{'ShardId': 'shardId-000000002643', 'P"

Current Behavior

$ aws --version
aws-cli/2.28.0 Python/3.13.5 Darwin/24.2.0 source/arm64
$ aws --profile <PROFILE> --region <REGION> kinesis list-shards --stream-name <NAME>

An error occurred (InvalidArgumentException) when calling the ListShards operation: NextToken and StreamARN cannot be provided together.

Reproduction Steps

see "current behavior"

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.28.0 Python/3.13.5 Darwin/24.2.0 source/arm64

Environment details (OS name and version, etc.)

Darwin/24.2.0 source/arm64

[adev-code]

Hi @jpassaro, thanks for reaching out. I have tried: $ aws --profile <PROFILE> --region <REGION> kinesis list-shards --stream-name <NAME> with versions latest version: aws-cli/2.28.2 and aws-cli/2.28.0 and I did not get the exception: InvalidArgumentException. Could you provide the full --debug logs and redacting all security and sensitive information? Thank you.

[jpassaro]

thanks for the suggestion to use debug. this stream has a very high number of shards, and the debug log actually shows a valid response that includes a NextToken. I think when the CLI auto-paginates it is adding the StreamName parameter with NextToken.

Here's a more reproducible example:

$ aws kinesis create-stream --shard-count 10 --stream-name foo
$ aws kinesis list-shards --stream-name foo | head -n 5
{
    "Shards": [
        {
            "ShardId": "shardId-000000000000",
            "HashKeyRange": {
$ aws kinesis list-shards --page-size 1 --stream-name foo

An error occurred (InvalidArgumentException) when calling the ListShards operation: NextToken and StreamName cannot be provided together.

[jpassaro]

for what it's worth, i just tried a similar test for list-stream-consumers and there was no error. Debug log shows that ListStreamConsumers was called the second time with StreamARN and NextToken provided, and it completes successfully. I actually see the same in the list-shards debug log -- second call to ListShards has both StreamName and NextToken, and the API rejects it.

[adev-code]

Hi @jpassaro, thanks for the reply. I also see the issue too and the error:
An error occurred (InvalidArgumentException) when calling the ListShards operation: NextToken and StreamName cannot be provided together..
After checking the debug logs, this is coming from the Service Kinesis itself:

MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"InvalidArgumentException","message":"NextToken and StreamName cannot be provided together."}'

Also using Boto3, I get the similar Issue and the error is coming from the Service:

import boto3

boto3.set_stream_logger('')

client = boto3.client('kinesis')
paginator = client.get_paginator('list_shards')

for page in paginator.paginate(
    StreamName='foo',
    PaginationConfig={'PageSize': 1}
):
    for shard in page['Shards']:
        print(shard['ShardId'])

and Output:

botocore.parsers [DEBUG] Response body:
b'{"__type":"InvalidArgumentException","message":"NextToken and StreamName cannot be provided together."}'

I have reached out to the Kinesis Service Team in regards of the issue.

For Internal Tracking: P286120895