Skip to content

Commit c779b9b

Browse files
[Logs] Add Kinesis as destination (#25325)
* Add Kinesis as destination * Apply suggestions from code review Co-authored-by: Maddy <[email protected]> --------- Co-authored-by: Maddy <[email protected]>
1 parent 049cbe9 commit c779b9b

File tree

1 file changed

+90
-0
lines changed
  • src/content/docs/logs/logpush/logpush-job/enable-destinations

1 file changed

+90
-0
lines changed
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
pcx_content_type: how-to
3+
title: Enable Amazon Kinesis
4+
sidebar:
5+
order: 98
6+
---
7+
8+
Logpush supports [Amazon Kinesis](https://aws.amazon.com/kinesis/) as a destination for all datasets. Each Kinesis record that Logpush sends will contain a batch of GZIP-compressed data in newline-delimited JSON format (by default), or in the format specified in the [`output_options`](/logs/logpush/logpush-job/log-output-options/) parameter when the job was created.
9+
10+
## Configure Kinesis using STS Assume Role (recommended)
11+
12+
1. Create an IAM Role for Cloudflare Logpush to Assume with the following trust relationship:
13+
14+
```java
15+
{
16+
"Version": "2012-10-17",
17+
"Statement": [
18+
{
19+
"Effect": "Allow",
20+
"Principal": {
21+
"AWS": [
22+
"arn:aws:iam::391854517948:user/cloudflare-logpush"
23+
]
24+
},
25+
"Action": "sts:AssumeRole"
26+
}
27+
]
28+
}
29+
```
30+
31+
2. Ensure that the IAM role has permissions to perform the `PutRecord` action on your Kinesis stream. Replace `<AWS_REGION>`, `<YOUR_AWS_ACCOUNT_ID>` and `<STREAM_NAME>` with your own values:
32+
33+
```java
34+
{
35+
"Version": "2012-10-17",
36+
"Statement": [
37+
{
38+
"Effect": "Allow",
39+
"Action": "kinesis:PutRecord",
40+
"Resource": "arn:aws:kinesis:<AWS_REGION>:<YOUR_AWS_ACCOUNT_ID>:stream/<STREAM_NAME>"
41+
}
42+
]
43+
}
44+
```
45+
46+
3. Create a Logpush job, using the following format for the `destination_conf` field:
47+
48+
```bash
49+
kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>
50+
```
51+
52+
4. (optional) When using STS Assume Role, you can include `sts-external-id` as a `destination_conf` parameter so it is included in your Logpush job's requests to Kinesis. Refer to [Securely Using External ID for Accessing AWS Accounts Owned by Others](https://aws.amazon.com/blogs/apn/securely-using-external-id-for-accessing-aws-accounts-owned-by-others/) for more information.
53+
54+
```bash
55+
kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>&sts-external-id=<EXTERNAL_ID>
56+
```
57+
58+
### STS Assume Role example
59+
60+
```bash
61+
$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
62+
-H 'Authorization: Bearer <API_TOKEN>' \
63+
-H 'Content-Type: application/json' -d '{
64+
"name": "kinesis",
65+
"destination_conf": "kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>",
66+
"dataset": "http_requests",
67+
"enabled": true
68+
}'
69+
```
70+
71+
## Configure Kinesis using IAM Access Keys
72+
73+
When configuring your Logpush job using IAM Access Keys, ensure that the IAM user has permission to perform the `PutRecord` action on your Kinesis stream:
74+
75+
```bash
76+
kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>
77+
```
78+
79+
### IAM Access Key example
80+
81+
```bash
82+
$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
83+
-H 'Authorization: Bearer <API_TOKEN>' \
84+
-H 'Content-Type: application/json' -d '{
85+
"name": "kinesis",
86+
"destination_conf": "kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>",
87+
"dataset": "http_requests",
88+
"enabled": true
89+
}'
90+
```

0 commit comments

Comments
 (0)