You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/aws/services/firehose.md
+32-34Lines changed: 32 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,22 +1,20 @@
1
1
---
2
2
title: "Data Firehose"
3
-
linkTitle: "Data Firehose"
4
-
description: >
5
-
Get started with Data Firehose on LocalStack
3
+
description: Get started with Data Firehose on LocalStack
6
4
tags: ["Free"]
7
5
---
8
6
9
-
{{< callout >}}
7
+
:::note
10
8
This service was formerly called as 'Kinesis Data Firehose'.
11
-
{{< /callout >}}
9
+
:::
12
10
13
11
## Introduction
14
12
15
13
Data Firehose is a service provided by AWS that allows you to extract, transform and load streaming data into various destinations, such as Amazon S3, Amazon Redshift, and Elasticsearch.
16
14
With Data Firehose, you can ingest and deliver real-time data from different sources as it automates data delivery, handles buffering and compression, and scales according to the data volume.
17
15
18
16
LocalStack allows you to use the Data Firehose APIs in your local environment to load and transform real-time data.
19
-
The supported APIs are available on our [API coverage page]({{< ref "coverage_firehose" >}}), which provides information on the extent of Data Firehose's integration with LocalStack.
17
+
The supported APIs are available on our [API coverage page](), which provides information on the extent of Data Firehose's integration with LocalStack.
20
18
21
19
## Getting started
22
20
@@ -30,9 +28,9 @@ We will demonstrate how to use Firehose to load Kinesis data into Elasticsearch
30
28
You can create an Elasticsearch domain using the [`create-elasticsearch-domain`](https://docs.aws.amazon.com/cli/latest/reference/es/create-elasticsearch-domain.html) command.
31
29
Execute the following command to create a domain named `es-local`:
32
30
33
-
{{< command >}}
34
-
$ awslocal es create-elasticsearch-domain --domain-name es-local
35
-
{{< / command >}}
31
+
```bash
32
+
awslocal es create-elasticsearch-domain --domain-name es-local
33
+
```
36
34
37
35
Save the value of the `Endpoint` field from the response, as it will be required further down to confirm the setup.
38
36
@@ -43,17 +41,17 @@ Now let us create our target S3 bucket and our source Kinesis stream:
43
41
Before creating the stream, we need to create an S3 bucket to store our backup data.
44
42
You can do this using the [`mb`](https://docs.aws.amazon.com/cli/latest/reference/s3/mb.html) command:
You can now use the [`CreateStream`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_CreateStream.html) API to create a Kinesis stream named `kinesis-es-local-stream` with two shards:
51
49
52
-
{{< command >}}
53
-
$ awslocal kinesis create-stream \
50
+
```bash
51
+
awslocal kinesis create-stream \
54
52
--stream-name kinesis-es-local-stream \
55
53
--shard-count 2
56
-
{{< / command >}}
54
+
```
57
55
58
56
### Create a Firehouse delivery stream
59
57
@@ -64,20 +62,20 @@ Within the `kinesis-stream-source-configuration`, it is required to specify the
64
62
The `elasticsearch-destination-configuration` sets vital parameters, which includes the access role, `DomainARN` of the Elasticsearch domain where you wish to publish, and the settings including the `IndexName` and `TypeName` for the Elasticsearch setup.
65
63
Additionally to backup all documents to S3, the `S3BackupMode` parameter is set to `AllDocuments`, which is accompanied by `S3Configuration`.
66
64
67
-
{{< callout >}}
65
+
:::note
68
66
Within LocalStack's default configuration, IAM roles remain unverified and no strict validation is applied on ARNs.
69
67
However, when operating within the AWS environment, you need to check the access rights of the specified role for the task.
70
-
{{< /callout >}}
68
+
:::
71
69
72
70
You can use the [`CreateDeliveryStream`](https://docs.aws.amazon.com/firehose/latest/APIReference/API_CreateDeliveryStream.html) API to create a Firehose delivery stream named `activity-to-elasticsearch-local`:
On successful execution, the command will return the `DeliveryStreamARN` of the created delivery stream:
83
81
@@ -93,43 +91,43 @@ Before testing the integration, it's necessary to confirm if the local Elasticse
93
91
You can use the [`describe-elasticsearch-domain`](https://docs.aws.amazon.com/cli/latest/reference/es/describe-elasticsearch-domain.html) command to check the status of the Elasticsearch cluster.
Once the command returns `false`, you can move forward with data ingestion.
102
100
The data can be added to the source Kinesis stream or directly to the Firehose delivery stream.
103
101
104
102
You can add data to the Kinesis stream using the [`PutRecord`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html) API.
105
103
The following command adds a record to the stream:
106
104
107
-
{{< command >}}
108
-
$ awslocal kinesis put-record \
105
+
```bash
106
+
awslocal kinesis put-record \
109
107
--stream-name kinesis-es-local-stream \
110
108
--data '{ "target": "barry" }' \
111
109
--partition-key partition
112
-
{{< / command >}}
110
+
```
113
111
114
-
{{< callout "tip" >}}
112
+
:::note
115
113
For users using AWS CLI v2, consider adding `--cli-binary-format raw-in-base64-out` to the command mentioned above.
116
-
{{< /callout >}}
114
+
:::
117
115
118
116
You can use the [`PutRecord`](https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecord.html) API to add data to the Firehose delivery stream.
119
117
The following command adds a record to the stream:
Copy file name to clipboardExpand all lines: src/content/docs/aws/services/neptune.md
+2-4Lines changed: 2 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -142,7 +142,7 @@ if __name__ == '__main__':
142
142
143
143
Amazon Neptune resources with IAM DB authentication enabled require all requests to use AWS Signature Version 4.
144
144
145
-
When LocalStack starts with [IAM enforcement enabled]({{< ref "/user-guide/security-testing" >}}), the Neptune database checks user permissions before granting access.
145
+
When LocalStack starts with [IAM enforcement enabled](/aws/capabilities/security-testing/iam-policy-enforcement), the Neptune database checks user permissions before granting access.
146
146
The following Gremlin query actions are available for database engine versions `1.3.2.0` and higher:
The LocalStack Web Application provides a Resource Browser for managing Neptune databases and clusters.
238
238
You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the **Resources** section, and then clicking on **Neptune** under the **Database** section.
0 commit comments