Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
---
pcx_content_type: how-to
title: Enable Amazon Kinesis
sidebar:
order: 98
---

Logpush supports [Amazon Kinesis](https://aws.amazon.com/kinesis/) as a destination for all datasets. Each Kinesis record that Logpush sends will contain a batch of GZIP-compressed data in newline-delimited JSON format (by default), or in the format specified in the [`output_options`](/logs/logpush/logpush-job/log-output-options/) parameter when the job was created.

## Configure Kinesis using STS Assume Role (recommended)

1. Create an IAM Role for Cloudflare Logpush to Assume with the following trust relationship:

```java
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::391854517948:user/cloudflare-logpush"
]
},
"Action": "sts:AssumeRole"
}
]
}
```

2. Ensure that the IAM role has permissions to perform the `PutRecord` action on your Kinesis stream. Replace `<AWS_REGION>`, `<YOUR_AWS_ACCOUNT_ID>` and `<STREAM_NAME>` with your own values:

```java
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "kinesis:PutRecord",
"Resource": "arn:aws:kinesis:<AWS_REGION>:<YOUR_AWS_ACCOUNT_ID>:stream/<STREAM_NAME>"
}
]
}
```

3. Create a Logpush job, using the following format for the `destination_conf` field:

```bash
kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>
```

4. (optional) When using STS Assume Role, you can include `sts-external-id` as a `destination_conf` parameter so it is included in your Logpush job's requests to Kinesis. Refer to [Securely Using External ID for Accessing AWS Accounts Owned by Others](https://aws.amazon.com/blogs/apn/securely-using-external-id-for-accessing-aws-accounts-owned-by-others/) for more information.

```bash
kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>&sts-external-id=<EXTERNAL_ID>
```

### STS Assume Role example

```bash
$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' -d '{
"name": "kinesis",
"destination_conf": "kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>",
"dataset": "http_requests",
"enabled": true
}'
```

## Configure Kinesis using IAM Access Keys

When configuring your Logpush job using IAM Access Keys, ensure that the IAM user has permission to perform the `PutRecord` action on your Kinesis stream:

```bash
kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>
```

### IAM Access Key example

```bash
$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' -d '{
"name": "kinesis",
"destination_conf": "kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>",
"dataset": "http_requests",
"enabled": true
}'
```
Loading