Skip to content

Commit 683eef0

Browse files
author
AWS
committed
Amazon Kinesis Firehose Update: Adds integration with Secrets Manager for Redshift, Splunk, HttpEndpoint, and Snowflake destinations
1 parent 6087ca6 commit 683eef0

File tree

2 files changed

+91
-17
lines changed

2 files changed

+91
-17
lines changed

‎.changes/next-release/feature-AmazonKinesisFirehose-c55852b.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon Kinesis Firehose",
4+
"contributor": "",
5+
"description": "Adds integration with Secrets Manager for Redshift, Splunk, HttpEndpoint, and Snowflake destinations"
6+
}

‎services/firehose/src/main/resources/codegen-resources/service-2.json

Lines changed: 85 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"endpointPrefix":"firehose",
66
"jsonVersion":"1.1",
77
"protocol":"json",
8+
"protocols":["json"],
89
"serviceAbbreviation":"Firehose",
910
"serviceFullName":"Amazon Kinesis Firehose",
1011
"serviceId":"Firehose",
@@ -130,7 +131,7 @@
130131
{"shape":"LimitExceededException"},
131132
{"shape":"InvalidKMSResourceException"}
132133
],
133-
"documentation":"<p>Enables server-side encryption (SSE) for the delivery stream. </p> <p>This operation is asynchronous. It returns immediately. When you invoke it, Firehose first sets the encryption status of the stream to <code>ENABLING</code>, and then to <code>ENABLED</code>. The encryption status of a delivery stream is the <code>Status</code> property in <a>DeliveryStreamEncryptionConfiguration</a>. If the operation fails, the encryption status changes to <code>ENABLING_FAILED</code>. You can continue to read and write data to your delivery stream while the encryption status is <code>ENABLING</code>, but the data is not encrypted. It can take up to 5 seconds after the encryption status changes to <code>ENABLED</code> before all records written to the delivery stream are encrypted. To find out whether a record or a batch of records was encrypted, check the response elements <a>PutRecordOutput$Encrypted</a> and <a>PutRecordBatchOutput$Encrypted</a>, respectively.</p> <p>To check the encryption status of a delivery stream, use <a>DescribeDeliveryStream</a>.</p> <p>Even if encryption is currently enabled for a delivery stream, you can still invoke this operation on it to change the ARN of the CMK or both its type and ARN. If you invoke this method to change the CMK, and the old CMK is of type <code>CUSTOMER_MANAGED_CMK</code>, Firehose schedules the grant it had on the old CMK for retirement. If the new CMK is of type <code>CUSTOMER_MANAGED_CMK</code>, Firehose creates a grant that enables it to use the new CMK to encrypt and decrypt data and to manage the grant.</p> <p>For the KMS grant creation to be successful, Firehose APIs <code>StartDeliveryStreamEncryption</code> and <code>CreateDeliveryStream</code> should not be called with session credentials that are more than 6 hours old.</p> <p>If a delivery stream already has encryption enabled and then you invoke this operation to change the ARN of the CMK or both its type and ARN and you get <code>ENABLING_FAILED</code>, this only means that the attempt to change the CMK failed. In this case, encryption remains enabled with the old CMK.</p> <p>If the encryption status of your delivery stream is <code>ENABLING_FAILED</code>, you can invoke this operation again with a valid CMK. The CMK must be enabled and the key policy mustn't explicitly deny the permission for Firehose to invoke KMS encrypt and decrypt operations.</p> <p>You can enable SSE for a delivery stream only if it's a delivery stream that uses <code>DirectPut</code> as its source. </p> <p>The <code>StartDeliveryStreamEncryption</code> and <code>StopDeliveryStreamEncryption</code> operations have a combined limit of 25 calls per delivery stream per 24 hours. For example, you reach the limit if you call <code>StartDeliveryStreamEncryption</code> 13 times and <code>StopDeliveryStreamEncryption</code> 12 times for the same delivery stream in a 24-hour period.</p>"
134+
"documentation":"<p>Enables server-side encryption (SSE) for the delivery stream. </p> <p>This operation is asynchronous. It returns immediately. When you invoke it, Firehose first sets the encryption status of the stream to <code>ENABLING</code>, and then to <code>ENABLED</code>. The encryption status of a delivery stream is the <code>Status</code> property in <a>DeliveryStreamEncryptionConfiguration</a>. If the operation fails, the encryption status changes to <code>ENABLING_FAILED</code>. You can continue to read and write data to your delivery stream while the encryption status is <code>ENABLING</code>, but the data is not encrypted. It can take up to 5 seconds after the encryption status changes to <code>ENABLED</code> before all records written to the delivery stream are encrypted. To find out whether a record or a batch of records was encrypted, check the response elements <a>PutRecordOutput$Encrypted</a> and <a>PutRecordBatchOutput$Encrypted</a>, respectively.</p> <p>To check the encryption status of a delivery stream, use <a>DescribeDeliveryStream</a>.</p> <p>Even if encryption is currently enabled for a delivery stream, you can still invoke this operation on it to change the ARN of the CMK or both its type and ARN. If you invoke this method to change the CMK, and the old CMK is of type <code>CUSTOMER_MANAGED_CMK</code>, Firehose schedules the grant it had on the old CMK for retirement. If the new CMK is of type <code>CUSTOMER_MANAGED_CMK</code>, Firehose creates a grant that enables it to use the new CMK to encrypt and decrypt data and to manage the grant.</p> <p>For the KMS grant creation to be successful, the Firehose API operations <code>StartDeliveryStreamEncryption</code> and <code>CreateDeliveryStream</code> should not be called with session credentials that are more than 6 hours old.</p> <p>If a delivery stream already has encryption enabled and then you invoke this operation to change the ARN of the CMK or both its type and ARN and you get <code>ENABLING_FAILED</code>, this only means that the attempt to change the CMK failed. In this case, encryption remains enabled with the old CMK.</p> <p>If the encryption status of your delivery stream is <code>ENABLING_FAILED</code>, you can invoke this operation again with a valid CMK. The CMK must be enabled and the key policy mustn't explicitly deny the permission for Firehose to invoke KMS encrypt and decrypt operations.</p> <p>You can enable SSE for a delivery stream only if it's a delivery stream that uses <code>DirectPut</code> as its source. </p> <p>The <code>StartDeliveryStreamEncryption</code> and <code>StopDeliveryStreamEncryption</code> operations have a combined limit of 25 calls per delivery stream per 24 hours. For example, you reach the limit if you call <code>StartDeliveryStreamEncryption</code> 13 times and <code>StopDeliveryStreamEncryption</code> 12 times for the same delivery stream in a 24-hour period.</p>"
134135
},
135136
"StopDeliveryStreamEncryption":{
136137
"name":"StopDeliveryStreamEncryption",
@@ -1842,7 +1843,7 @@
18421843
"CloudWatchLoggingOptions":{"shape":"CloudWatchLoggingOptions"},
18431844
"RequestConfiguration":{
18441845
"shape":"HttpEndpointRequestConfiguration",
1845-
"documentation":"<p>The configuration of the requeste sent to the HTTP endpoint specified as the destination.</p>"
1846+
"documentation":"<p>The configuration of the request sent to the HTTP endpoint that is specified as the destination.</p>"
18461847
},
18471848
"ProcessingConfiguration":{"shape":"ProcessingConfiguration"},
18481849
"RoleARN":{
@@ -1857,7 +1858,11 @@
18571858
"shape":"HttpEndpointS3BackupMode",
18581859
"documentation":"<p>Describes the S3 bucket backup options for the data that Firehose delivers to the HTTP endpoint destination. You can back up all documents (<code>AllData</code>) or only the documents that Firehose could not deliver to the specified HTTP endpoint destination (<code>FailedDataOnly</code>).</p>"
18591860
},
1860-
"S3Configuration":{"shape":"S3DestinationConfiguration"}
1861+
"S3Configuration":{"shape":"S3DestinationConfiguration"},
1862+
"SecretsManagerConfiguration":{
1863+
"shape":"SecretsManagerConfiguration",
1864+
"documentation":"<p> The configuration that defines how you access secrets for HTTP Endpoint destination. </p>"
1865+
}
18611866
},
18621867
"documentation":"<p>Describes the configuration of the HTTP endpoint destination.</p>"
18631868
},
@@ -1890,7 +1895,11 @@
18901895
"shape":"HttpEndpointS3BackupMode",
18911896
"documentation":"<p>Describes the S3 bucket backup options for the data that Kinesis Firehose delivers to the HTTP endpoint destination. You can back up all documents (<code>AllData</code>) or only the documents that Firehose could not deliver to the specified HTTP endpoint destination (<code>FailedDataOnly</code>).</p>"
18921897
},
1893-
"S3DestinationDescription":{"shape":"S3DestinationDescription"}
1898+
"S3DestinationDescription":{"shape":"S3DestinationDescription"},
1899+
"SecretsManagerConfiguration":{
1900+
"shape":"SecretsManagerConfiguration",
1901+
"documentation":"<p> The configuration that defines how you access secrets for HTTP Endpoint destination. </p>"
1902+
}
18941903
},
18951904
"documentation":"<p>Describes the HTTP endpoint destination.</p>"
18961905
},
@@ -1923,7 +1932,11 @@
19231932
"shape":"HttpEndpointS3BackupMode",
19241933
"documentation":"<p>Describes the S3 bucket backup options for the data that Kinesis Firehose delivers to the HTTP endpoint destination. You can back up all documents (<code>AllData</code>) or only the documents that Firehose could not deliver to the specified HTTP endpoint destination (<code>FailedDataOnly</code>).</p>"
19251934
},
1926-
"S3Update":{"shape":"S3DestinationUpdate"}
1935+
"S3Update":{"shape":"S3DestinationUpdate"},
1936+
"SecretsManagerConfiguration":{
1937+
"shape":"SecretsManagerConfiguration",
1938+
"documentation":"<p> The configuration that defines how you access secrets for HTTP Endpoint destination. </p>"
1939+
}
19271940
},
19281941
"documentation":"<p>Updates the specified HTTP endpoint destination.</p>"
19291942
},
@@ -2404,7 +2417,7 @@
24042417
"documentation":"<p>Indicates the version of row format to output. The possible values are <code>V1</code> and <code>V2</code>. The default is <code>V1</code>.</p>"
24052418
}
24062419
},
2407-
"documentation":"<p>A serializer to use for converting data to the Parquet format before storing it in Amazon S3. For more information, see <a href=\"https://parquet.apache.org/documentation/latest/\">Apache Parquet</a>.</p>"
2420+
"documentation":"<p>A serializer to use for converting data to the Parquet format before storing it in Amazon S3. For more information, see <a href=\"https://parquet.apache.org/docs/\">Apache Parquet</a>.</p>"
24082421
},
24092422
"ParquetWriterVersion":{
24102423
"type":"string",
@@ -2639,8 +2652,6 @@
26392652
"RoleARN",
26402653
"ClusterJDBCURL",
26412654
"CopyCommand",
2642-
"Username",
2643-
"Password",
26442655
"S3Configuration"
26452656
],
26462657
"members":{
@@ -2687,6 +2698,10 @@
26872698
"CloudWatchLoggingOptions":{
26882699
"shape":"CloudWatchLoggingOptions",
26892700
"documentation":"<p>The CloudWatch logging options for your delivery stream.</p>"
2701+
},
2702+
"SecretsManagerConfiguration":{
2703+
"shape":"SecretsManagerConfiguration",
2704+
"documentation":"<p> The configuration that defines how you access secrets for Amazon Redshift. </p>"
26902705
}
26912706
},
26922707
"documentation":"<p>Describes the configuration of a destination in Amazon Redshift.</p>"
@@ -2697,7 +2712,6 @@
26972712
"RoleARN",
26982713
"ClusterJDBCURL",
26992714
"CopyCommand",
2700-
"Username",
27012715
"S3DestinationDescription"
27022716
],
27032717
"members":{
@@ -2740,6 +2754,10 @@
27402754
"CloudWatchLoggingOptions":{
27412755
"shape":"CloudWatchLoggingOptions",
27422756
"documentation":"<p>The Amazon CloudWatch logging options for your delivery stream.</p>"
2757+
},
2758+
"SecretsManagerConfiguration":{
2759+
"shape":"SecretsManagerConfiguration",
2760+
"documentation":"<p> The configuration that defines how you access secrets for Amazon Redshift. </p>"
27432761
}
27442762
},
27452763
"documentation":"<p>Describes a destination in Amazon Redshift.</p>"
@@ -2790,6 +2808,10 @@
27902808
"CloudWatchLoggingOptions":{
27912809
"shape":"CloudWatchLoggingOptions",
27922810
"documentation":"<p>The Amazon CloudWatch logging options for your delivery stream.</p>"
2811+
},
2812+
"SecretsManagerConfiguration":{
2813+
"shape":"SecretsManagerConfiguration",
2814+
"documentation":"<p> The configuration that defines how you access secrets for Amazon Redshift. </p>"
27932815
}
27942816
},
27952817
"documentation":"<p>Describes an update for a destination in Amazon Redshift.</p>"
@@ -3021,6 +3043,31 @@
30213043
},
30223044
"documentation":"<p>Specifies the schema to which you want Firehose to configure your data before it writes it to Amazon S3. This parameter is required if <code>Enabled</code> is set to true.</p>"
30233045
},
3046+
"SecretARN":{
3047+
"type":"string",
3048+
"max":2048,
3049+
"min":1,
3050+
"pattern":"arn:.*"
3051+
},
3052+
"SecretsManagerConfiguration":{
3053+
"type":"structure",
3054+
"required":["Enabled"],
3055+
"members":{
3056+
"SecretARN":{
3057+
"shape":"SecretARN",
3058+
"documentation":"<p>The ARN of the secret that stores your credentials. It must be in the same region as the Firehose stream and the role. The secret ARN can reside in a different account than the delivery stream and role as Firehose supports cross-account secret access. This parameter is required when <b>Enabled</b> is set to <code>True</code>.</p>"
3059+
},
3060+
"RoleARN":{
3061+
"shape":"RoleARN",
3062+
"documentation":"<p> Specifies the role that Firehose assumes when calling the Secrets Manager API operation. When you provide the role, it overrides any destination specific role defined in the destination configuration. If you do not provide the then we use the destination specific role. This parameter is required for Splunk. </p>"
3063+
},
3064+
"Enabled":{
3065+
"shape":"BooleanObject",
3066+
"documentation":"<p>Specifies whether you want to use the the secrets manager feature. When set as <code>True</code> the secrets manager configuration overwrites the existing secrets in the destination configuration. When it's set to <code>False</code> Firehose falls back to the credentials in the destination configuration.</p>"
3067+
}
3068+
},
3069+
"documentation":"<p>The structure that defines how Firehose accesses the secret.</p>"
3070+
},
30243071
"SecurityGroupIdList":{
30253072
"type":"list",
30263073
"member":{"shape":"NonEmptyStringWithoutWhitespace"},
@@ -3089,8 +3136,6 @@
30893136
"type":"structure",
30903137
"required":[
30913138
"AccountUrl",
3092-
"PrivateKey",
3093-
"User",
30943139
"Database",
30953140
"Schema",
30963141
"Table",
@@ -3160,7 +3205,11 @@
31603205
"shape":"SnowflakeS3BackupMode",
31613206
"documentation":"<p>Choose an S3 backup mode</p>"
31623207
},
3163-
"S3Configuration":{"shape":"S3DestinationConfiguration"}
3208+
"S3Configuration":{"shape":"S3DestinationConfiguration"},
3209+
"SecretsManagerConfiguration":{
3210+
"shape":"SecretsManagerConfiguration",
3211+
"documentation":"<p> The configuration that defines how you access secrets for Snowflake. </p>"
3212+
}
31643213
},
31653214
"documentation":"<p>Configure Snowflake destination</p>"
31663215
},
@@ -3221,7 +3270,11 @@
32213270
"shape":"SnowflakeS3BackupMode",
32223271
"documentation":"<p>Choose an S3 backup mode</p>"
32233272
},
3224-
"S3DestinationDescription":{"shape":"S3DestinationDescription"}
3273+
"S3DestinationDescription":{"shape":"S3DestinationDescription"},
3274+
"SecretsManagerConfiguration":{
3275+
"shape":"SecretsManagerConfiguration",
3276+
"documentation":"<p> The configuration that defines how you access secrets for Snowflake. </p>"
3277+
}
32253278
},
32263279
"documentation":"<p>Optional Snowflake destination description</p>"
32273280
},
@@ -3286,7 +3339,11 @@
32863339
"shape":"SnowflakeS3BackupMode",
32873340
"documentation":"<p>Choose an S3 backup mode</p>"
32883341
},
3289-
"S3Update":{"shape":"S3DestinationUpdate"}
3342+
"S3Update":{"shape":"S3DestinationUpdate"},
3343+
"SecretsManagerConfiguration":{
3344+
"shape":"SecretsManagerConfiguration",
3345+
"documentation":"<p> Describes the Secrets Manager configuration in Snowflake. </p>"
3346+
}
32903347
},
32913348
"documentation":"<p>Update to configuration settings</p>"
32923349
},
@@ -3430,7 +3487,6 @@
34303487
"required":[
34313488
"HECEndpoint",
34323489
"HECEndpointType",
3433-
"HECToken",
34343490
"S3Configuration"
34353491
],
34363492
"members":{
@@ -3473,6 +3529,10 @@
34733529
"BufferingHints":{
34743530
"shape":"SplunkBufferingHints",
34753531
"documentation":"<p>The buffering options. If no value is specified, the default values for Splunk are used.</p>"
3532+
},
3533+
"SecretsManagerConfiguration":{
3534+
"shape":"SecretsManagerConfiguration",
3535+
"documentation":"<p> The configuration that defines how you access secrets for Splunk. </p>"
34763536
}
34773537
},
34783538
"documentation":"<p>Describes the configuration of a destination in Splunk.</p>"
@@ -3519,6 +3579,10 @@
35193579
"BufferingHints":{
35203580
"shape":"SplunkBufferingHints",
35213581
"documentation":"<p>The buffering options. If no value is specified, the default values for Splunk are used.</p>"
3582+
},
3583+
"SecretsManagerConfiguration":{
3584+
"shape":"SecretsManagerConfiguration",
3585+
"documentation":"<p> The configuration that defines how you access secrets for Splunk. </p>"
35223586
}
35233587
},
35243588
"documentation":"<p>Describes a destination in Splunk.</p>"
@@ -3565,6 +3629,10 @@
35653629
"BufferingHints":{
35663630
"shape":"SplunkBufferingHints",
35673631
"documentation":"<p>The buffering options. If no value is specified, the default values for Splunk are used.</p>"
3632+
},
3633+
"SecretsManagerConfiguration":{
3634+
"shape":"SecretsManagerConfiguration",
3635+
"documentation":"<p> The configuration that defines how you access secrets for Splunk. </p>"
35683636
}
35693637
},
35703638
"documentation":"<p>Describes an update for a destination in Splunk.</p>"
@@ -3776,7 +3844,7 @@
37763844
},
37773845
"SnowflakeDestinationUpdate":{
37783846
"shape":"SnowflakeDestinationUpdate",
3779-
"documentation":"<p>Update to the Snowflake destination condiguration settings</p>"
3847+
"documentation":"<p>Update to the Snowflake destination configuration settings.</p>"
37803848
}
37813849
}
37823850
},
@@ -3844,5 +3912,5 @@
38443912
"documentation":"<p>The details of the VPC of the Amazon ES destination.</p>"
38453913
}
38463914
},
3847-
"documentation":"<fullname>Amazon Data Firehose</fullname> <note> <p>Amazon Data Firehose was previously known as Amazon Kinesis Data Firehose.</p> </note> <p>Amazon Data Firehose is a fully managed service that delivers real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon OpenSearch Service, Amazon Redshift, Splunk, and various other supportd destinations.</p>"
3915+
"documentation":"<fullname>Amazon Data Firehose</fullname> <note> <p>Amazon Data Firehose was previously known as Amazon Kinesis Data Firehose.</p> </note> <p>Amazon Data Firehose is a fully managed service that delivers real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon OpenSearch Service, Amazon Redshift, Splunk, and various other supported destinations.</p>"
38483916
}

0 commit comments

Comments
 (0)