|
| 1 | +--- |
| 2 | +pcx_content_type: how-to |
| 3 | +title: Enable Amazon Kinesis |
| 4 | +sidebar: |
| 5 | + order: 98 |
| 6 | +--- |
| 7 | + |
| 8 | +Logpush supports [Amazon Kinesis](https://aws.amazon.com/kinesis/) as a destination for all datasets. Each Kinesis record that Logpush sends will contain a batch of GZIP-compressed data in newline-delimited JSON format (by default), or in the format specified in the [`output_options`](/logs/logpush/logpush-job/log-output-options/) parameter when the job was created. |
| 9 | + |
| 10 | +## Configure Kinesis using STS Assume Role (recommended) |
| 11 | + |
| 12 | +1. Create an IAM Role for Cloudflare Logpush to Assume with the following trust relationship: |
| 13 | + |
| 14 | +```java |
| 15 | +{ |
| 16 | + "Version": "2012-10-17", |
| 17 | + "Statement": [ |
| 18 | + { |
| 19 | + "Effect": "Allow", |
| 20 | + "Principal": { |
| 21 | + "AWS": [ |
| 22 | + "arn:aws:iam::391854517948:user/cloudflare-logpush" |
| 23 | + ] |
| 24 | + }, |
| 25 | + "Action": "sts:AssumeRole" |
| 26 | + } |
| 27 | + ] |
| 28 | +} |
| 29 | +``` |
| 30 | + |
| 31 | +2. Ensure that the IAM role has permissions to perform the `PutRecord` action on your Kinesis stream. Replace `<AWS_REGION>`, `<YOUR_AWS_ACCOUNT_ID>` and `<STREAM_NAME>` with your own values: |
| 32 | + |
| 33 | +```java |
| 34 | +{ |
| 35 | + "Version": "2012-10-17", |
| 36 | + "Statement": [ |
| 37 | + { |
| 38 | + "Effect": "Allow", |
| 39 | + "Action": "kinesis:PutRecord", |
| 40 | + "Resource": "arn:aws:kinesis:<AWS_REGION>:<YOUR_AWS_ACCOUNT_ID>:stream/<STREAM_NAME>" |
| 41 | + } |
| 42 | + ] |
| 43 | +} |
| 44 | +``` |
| 45 | + |
| 46 | +3. Create a Logpush job, using the following format for the `destination_conf` field: |
| 47 | + |
| 48 | +```bash |
| 49 | +kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME> |
| 50 | +``` |
| 51 | + |
| 52 | +4. (optional) When using STS Assume Role, you can include `sts-external-id` as a `destination_conf` parameter so it is included in your Logpush job's requests to Kinesis. Refer to [Securely Using External ID for Accessing AWS Accounts Owned by Others](https://aws.amazon.com/blogs/apn/securely-using-external-id-for-accessing-aws-accounts-owned-by-others/) for more information. |
| 53 | + |
| 54 | +```bash |
| 55 | +kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>&sts-external-id=<EXTERNAL_ID> |
| 56 | +``` |
| 57 | + |
| 58 | +### STS Assume Role example |
| 59 | + |
| 60 | +```bash |
| 61 | +$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \ |
| 62 | +-H 'Authorization: Bearer <API_TOKEN>' \ |
| 63 | +-H 'Content-Type: application/json' -d '{ |
| 64 | + "name": "kinesis", |
| 65 | + "destination_conf": "kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>", |
| 66 | + "dataset": "http_requests", |
| 67 | + "enabled": true |
| 68 | +}' |
| 69 | +``` |
| 70 | + |
| 71 | +## Configure Kinesis using IAM Access Keys |
| 72 | + |
| 73 | +When configuring your Logpush job using IAM Access Keys, ensure that the IAM user has permission to perform the `PutRecord` action on your Kinesis stream: |
| 74 | + |
| 75 | +```bash |
| 76 | +kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY> |
| 77 | +``` |
| 78 | + |
| 79 | +### IAM Access Key example |
| 80 | + |
| 81 | +```bash |
| 82 | +$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \ |
| 83 | +-H 'Authorization: Bearer <API_TOKEN>' \ |
| 84 | +-H 'Content-Type: application/json' -d '{ |
| 85 | + "name": "kinesis", |
| 86 | + "destination_conf": "kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>", |
| 87 | + "dataset": "http_requests", |
| 88 | + "enabled": true |
| 89 | +}' |
| 90 | +``` |
0 commit comments