|
| 1 | +# Amazon Data Firehose Common Actions |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +This example shows how to use AWS SDKs to perform common actions with Amazon Data Firehose, such as putting individual records (`PutRecord`) and batches of records (`PutRecordBatch`) to a delivery stream. |
| 6 | + |
| 7 | +The Data Firehose API has a maximum limit of 500 records or 4MB per request for `PutRecordBatch`. This example demonstrates how to handle scenarios where the number of records exceeds the maximum limit by breaking down the requests into multiple batches. |
| 8 | + |
| 9 | +The following components are used in this example: |
| 10 | + |
| 11 | +- [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) is the service used to capture, transform, and load streaming data into data lakes, data stores, and analytics services. |
| 12 | +- [Amazon S3](https://aws.amazon.com/s3/) is used as the destination for the Data Firehose delivery stream, storing the ingested data. |
| 13 | +- [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) hosts the Firehose metrics used to monitor Firehose performance. |
| 14 | + |
| 15 | +For detailed information on this workflow, see the [firehose/README.md](../../../../../workflows/firehose/README.md). |
| 16 | + |
| 17 | +## ⚠ Important |
| 18 | + |
| 19 | +- Running this code might result in charges to your AWS account. |
| 20 | +- Running the tests might result in charges to your AWS account. |
| 21 | +- We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). |
| 22 | +- This code is not tested in every AWS Region. For more information, see [AWS Regional Services](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services). |
| 23 | + |
| 24 | +## Run this code |
| 25 | + |
| 26 | +### Prerequisites |
| 27 | + |
| 28 | +To run this example, you need to set up the necessary infrastructure and generate mock data. Follow the steps outlined in the [workflow README](../../../../../workflows/firehose/README.md#setup) to create a Data Firehose delivery stream and generate sample data. |
| 29 | + |
| 30 | +### Execution |
| 31 | + |
| 32 | +This Java example will perform the following actions: |
| 33 | + |
| 34 | +1. Initialize the AWS SDK for Java service clients. |
| 35 | +2. Define configuration parameters (delivery stream name, region, batch size, logging settings). |
| 36 | +3. Put individual records using the `PutRecord` API. |
| 37 | +4. Put batches of records using the `PutRecordBatch` API. |
| 38 | +5. Monitor `IncomingBytes` and `IncomingRecords` metrics to ensure there is incoming traffic, and `FailedPutCount` for batch operations. |
| 39 | + |
| 40 | +## Additional reading |
| 41 | + |
| 42 | +- [Data Firehose Developer Guide](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) |
| 43 | +- [Data Firehose API Reference](https://docs.aws.amazon.com/firehose/latest/APIReference/Welcome.html) |
| 44 | +- [SDK for Java Data Firehose reference](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/firehose/FirehoseClient.html) |
| 45 | + |
| 46 | +--- |
| 47 | + |
| 48 | +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: Apache-2.0 |
0 commit comments