How to handle failure in putFrame 0x52000040 STATUS_CLIENT_AUTH_CALL_FAILED

[gustavofuhr]

Hello, I'm trying to capture when putFrame are failing with this specific status to restart the pipeline, but I'm unable to do that for now.

** Logging **
After some time, my cameras (Axis ACAPs) start to fail with the following:

2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [ERROR] [KVS] Put frame for B8A44FD014E5__distorted_tree failed with 0x52000040
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [ERROR] [KVS] Put frame for B8A44FD014E5__distorted_tree failed with 0x52000040
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [ERROR] [KVS] Put frame for B8A44FD014E5__distorted_tree failed with 0x52000040
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [ERROR] [KVS] Put frame for B8A44FD014E5__distorted_tree failed with 0x52000040
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] fromAuthClientState(): Invalid auth token as it is expiring in less than 30 seconds
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] fromAuthClientState(): Invalid auth token as it is expiring in less than 30 seconds
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] fromAuthClientState(): Invalid auth token as it is expiring in less than 30 seconds
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] putKinesisVideoFrame(): [B8A44FD014E5__distorted_tree] Failed to submit frame to Kinesis Video client. status: 0x52000040 decoding timestamp: 17527719123905702 presentation timestamp: 17527719123905702
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] putKinesisVideoFrame(): [B8A44FD014E5__distorted_tree] Failed to submit frame to Kinesis Video client. status: 0x52000040 decoding timestamp: 17527719124314757 presentation timestamp: 17527719124314757
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] putKinesisVideoFrame(): [B8A44FD014E5__distorted_tree] Failed to submit frame to Kinesis Video client. status: 0x52000040 decoding timestamp: 17527719124706314 presentation timestamp: 17527719124706314
2025-07-17T18:05:21.589+01:00 axis-b8a44fd014e5 [ INFO    ] iamotionstreamer[91409]: [WARN ] [KVS] putKinesisVideoFrame(): [B8A44FD014E5__distorted_tree] Failed to submit frame to Kinesis Video client. status: 0x52000040 decoding timestamp: 17527719125109437 presentation timestamp: 17527719125109437

** Any design considerations/constraints **
This an ACAP running the CPP KVS producer and sending frames, it works for some hours/days, but it starts to fail after a while, like an expired auth token or something. I'm using this credentials provider:

    credential_provider.reset(new IotCertCredentialProvider(kvs_opts.aws_iot_credentials_endpoint,
        "credentials/certificate.pem",
        "credentials/private.key",
        kvs_opts.aws_iot_role_alias,
        "credentials/AmazonRootCA1.pem",
        kvs_opts.aws_iot_thing_name));

[MushMal]

Suggest checking the clock drift on the device.

[gustavofuhr]

@MushMal in principle, the cameras sync their clocks via NTP each few minutes. Anyways, it would be nice to be able to handle the error so I can notify someone to check that, as of right now I'm unable to do it.

[MushMal]

I will let someone from AWS to comment but to experiment with it, you can override the auth callback provider and insert your code to check the time drift before delegating the call to the default iot callback provider implementation. I don't have the details here but surely someone can help I hope.

[sirknightj]

One suggestion might be to increase the credentials duration to 12 hours since it helps reduce the number of refreshes needed. You will need to increase it in two places (both in the IAM roles, and in the IoT role alias section in AWS console).

I'm wondering if the underlying issue is a network connectivity issue? Do you have earlier logs? It should have tried to refresh the credentials ~5 minutes before they actually expire so there may be some additional logs from earlier in the run.

The curl implementation of the callback is listed here:

• https://github.com/awslabs/amazon-kinesis-video-streams-producer-c/blob/7590b2317470041e139833bdc974e3619fad495a/src/source/Common/IotCredentialProvider.c#L218

To modify it you should be able to edit the file at the location: dependency/libkvscproducer/kvscproducer-src/src/source/Common/IotCredentialProvider.c to and run make again to rebuild.

I haven't explored this yet, but it may be worth looking at increasing the number of retries configured for the CLIENT_AUTH state, seeing as it's set to 5 retries (dependency/libkvscproducer/kvscproducer-src/dependency/libkvspic/kvspic-src/src/client/src/ClientState.c). That state is what calls the getSecurityTokenIoTFunc (definition) (additional refs: 1, 2, 3, 4). The state transition definition says that if the call fails, to retry the same state so I'm wondering if the retries got exhausted. Verbose logging may also help as it should print out the retry counts of what it's currently doing.

[gustavofuhr]

@sirknightj thanks for you answer, that was very helpful, I increase the credentials/policy durations, so I think I will have less errors now. I don't have verbose/debug logs before the event, but I'll try to drop internet connection in a local device to see what happens.

The camera devices might be loosing often the network, so when the credential error happens, it fails forever (each time I try to send another frame). My first idea, instead of building a custom version of the lib, was to capture the error somehow and try to wait until I have a better connection (more or less a backoff behavior). Is that something easy to do?

I see that putFrame returns a boolean if the frame was put in the queue, but am I able to capture errors when actually sending those? Maybe streamErrorReportHandler will capture that?

Thank you again!

[gustavofuhr]

Just checking if someone still have an idea on how to catch this error....