[BUG] Tags are not forwarded in call to createStreamSync when creating a new KVS stream

[ReeRichard]

Logging
Unable to create Kinesis Video stream. Error status: 0xf

Describe the bug
Whenever we enable SCP for KVS resources in our case it is a required tag such CostCentre, the creation of new streams no longer works and outputs the above error message even though we provide all the tags.

To Reproduce
Steps to reproduce the behavior:

1. Create a SCP for KVS streams that require the tag "CostCentre" be defined for each KVS stream
2. Create a StreamDefinition that includes the required tags ("CostCentre") for a non-existing KVS stream
3. Call com::amazonaws::kinesis::video::KinesisVideoProducer::createStreamSync
4. See error

When the SCP is not enabled the creation of the KVS stream works fine and the tags are also there as expected.

Expected behavior
Creating new KVS Streams with tags should comply with SCP as it does when using the aws-sdk-cpp or python equivalent.

SDK version number
v3.5.0

Platform (please complete the following information):

• OS: Linux
• Processor: x86
• Version 24.04

[sirknightj]

Hi @ReeRichard,

From my understanding of the issue description, you are using an IAM policy that looks something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": "kinesisvideo:CreateStream",
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/CostCentre": "true"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "kinesisvideo:*",
            "Resource": "*"
        }
    ]
}

Then you configure the stream like this:

map<string, string> tags;
char tag_name[MAX_TAG_NAME_LEN];
char tag_val[MAX_TAG_VALUE_LEN];
SNPRINTF(tag_name, MAX_TAG_NAME_LEN, "CostCentre");
SNPRINTF(tag_val, MAX_TAG_VALUE_LEN, "1234");
tags[tag_name] = tag_val;

...

unique_ptr<StreamDefinition> stream_definition(new StreamDefinition(
    ...
    &tags,
    ...

data->kinesis_video_stream = data->kinesis_video_producer->createStreamSync(std::move(stream_definition));

and the stream can't be created successfully (times out).

Root cause

Summary: The C++ KVS Producer SDK v3.5.0 currently doesn't support this.

The IAM policy above requires that the stream is created (CreateStream API call) with the tags added. CreateStreamSync method does not necessarily correspond to the Create Stream API call. CreateStreamSync handles PutMedia endpoint discovery and will create the stream for you if it doesn't exist. However, the stream creation in the Producer SDK is implemented in two steps when tags are configured:

1. CreateStream (with no tags)
2. TagStream (add the tags)

Therefore, there will be a short period where the stream does not have tags.

Resolution

We have two suggestions:

1. Create the stream outside of the Producer SDK. e.g.

aws kinesisvideo create-stream \
  --stream-name "demo-stream-with-tags" \
  --tags "CostCentre=1234"
./kvs_gstreamer_sample demo-stream-with-tags

2. Make a change to the SDK to add the parameter for the CreateStream API call. Here are diff which you can apply to dependency/libkvscproducer/kvscproducer-src. It builds the tags JSON string based on the streamInfo.tags.

diff --git a/src/source/CurlApiCallbacks.c b/src/source/CurlApiCallbacks.c
index 3d36c65..dfa6102 100644
--- a/src/source/CurlApiCallbacks.c
+++ b/src/source/CurlApiCallbacks.c
@@ -995,6 +995,12 @@ CleanUp:
     return retStatus;
 }
 
+#define CREATE_STREAM_PARAM_JSON_TEMPLATE_WITH_TAGS                                                                                                  \
+    "{\n\t\"DeviceName\": \"%s\",\n\t"                                                                                                               \
+    "\"StreamName\": \"%s\",\n\t\"MediaType\": \"%s\",\n\t"                                                                                          \
+    "%s"                                                                                                                                             \
+        "\"DataRetentionInHours\": %" PRIu64 "%s\n}"
+
 STATUS createStreamCurl(UINT64 customData, PCHAR deviceName, PCHAR streamName, PCHAR contentType, PCHAR kmsKeyId, UINT64 retentionPeriod,
                         PServiceCallContext pServiceCallContext)
 {
@@ -1003,6 +1009,7 @@ STATUS createStreamCurl(UINT64 customData, PCHAR deviceName, PCHAR streamName, P
     CHAR kmsKey[MAX_JSON_KMS_KEY_ID_STRING_LEN];
     CHAR paramsJson[MAX_JSON_PARAMETER_STRING_LEN];
     CHAR url[MAX_PATH_LEN + 1];
+    CHAR tagsJson[MAX_JSON_PARAMETER_STRING_LEN];
     UINT64 retentionInHours, currentTime;
     PAwsCredentials pCredentials = NULL;
     TID threadId = INVALID_TID_VALUE;
@@ -1011,11 +1018,13 @@ STATUS createStreamCurl(UINT64 customData, PCHAR deviceName, PCHAR streamName, P
     PCurlRequest pCurlRequest = NULL;
     BOOL startLocked = FALSE, requestLocked = FALSE, shutdownLocked = FALSE, streamShuttingDown = FALSE;
     STREAM_HANDLE streamHandle = INVALID_STREAM_HANDLE_VALUE;
+    PKinesisVideoStream pKinesisVideoStream;
 
     CHK(pCurlApiCallbacks != NULL && pCurlApiCallbacks->pCallbacksProvider != NULL && pServiceCallContext != NULL, STATUS_INVALID_ARG);
     pCallbacksProvider = pCurlApiCallbacks->pCallbacksProvider;
 
     streamHandle = (STREAM_HANDLE) pServiceCallContext->customData;
+    pKinesisVideoStream = FROM_STREAM_HANDLE(streamHandle);
 
     // Prepare the JSON string for the KMS param.
     // NOTE: the kms key id is optional
@@ -1026,9 +1035,22 @@ STATUS createStreamCurl(UINT64 customData, PCHAR deviceName, PCHAR streamName, P
         kmsKey[0] = '\0';
     }
 
+    // Build tags JSON
+    if (pKinesisVideoStream->streamInfo.tagCount > 0) {
+        STRCPY(tagsJson, ",\n\t\"Tags\": {");
+        for (UINT32 i = 0; i < pKinesisVideoStream->streamInfo.tagCount; i++) {
+            if (i > 0) STRCAT(tagsJson, ",");
+            sprintf(tagsJson + STRLEN(tagsJson), "\"%s\":\"%s\"", pKinesisVideoStream->streamInfo.tags[i].name, pKinesisVideoStream->streamInfo.tags[i].value);
+        }
+        STRCAT(tagsJson, "}");
+    } else {
+        STRCPY(tagsJson, "");
+    }
+
     // Expressed in hours
     retentionInHours = (UINT64) (retentionPeriod / HUNDREDS_OF_NANOS_IN_AN_HOUR);
-    SNPRINTF(paramsJson, ARRAY_SIZE(paramsJson), CREATE_STREAM_PARAM_JSON_TEMPLATE, deviceName, streamName, contentType, kmsKey, retentionInHours);
+    SNPRINTF(paramsJson, ARRAY_SIZE(paramsJson), CREATE_STREAM_PARAM_JSON_TEMPLATE_WITH_TAGS, deviceName, streamName, contentType, kmsKey, retentionInHours, tagsJson);
+
 
     // Validate the credentials
     CHK_STATUS(deserializeAwsCredentials(pServiceCallContext->pAuthInfo->data));
diff --git a/src/source/Include_i.h b/src/source/Include_i.h
index 2fa24df..6e75797 100644
--- a/src/source/Include_i.h
+++ b/src/source/Include_i.h
@@ -22,7 +22,8 @@ typedef StreamLatencyStateMachine* PStreamLatencyStateMachine;
 // Project include files
 ////////////////////////////////////////////////////
 #include <com/amazonaws/kinesis/video/cproducer/Include.h>
-#include <com/amazonaws/kinesis/video/common/Include.h>
+#include <com/amazonaws/kinesis/video/common/Include.h>
+#include "../../dependency/libkvspic/kvspic-src/src/client/src/Include_i.h"
 
 #include <curl/curl.h>
 

We'll be looking into supporting this out of the box in a future release. Please let us know if this resolves your issue and feel free to reach out to kinesis-video-support@amazon.com for additional support.

[ReeRichard]

We'll be looking into supporting this out of the box in a future release. Please let us know if this resolves your issue and feel free to reach out to kinesis-video-support@amazon.com for additional support.

Thank you for the elaborate response. We will try to hotfix this as you suggested in the meantime until the proper fix is available.