Skip to content

Commit 941348f

Browse files
ShourieGShourieG
authored
Add GCS input to Cloudflare Logpush (elastic#4720)
* initial commit to add gcs input to cloudflare_logpush * updated changelog * updated manifest with version * updated docs * updated docs * updated docs * updated docs as per pr suggetions * updated docs * updated PR with config changes & suggetions
1 parent 028ad08 commit 941348f

File tree

18 files changed

+866
-3
lines changed

18 files changed

+866
-3
lines changed

‎packages/cloudflare_logpush/_dev/build/docs/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,16 @@ This module has been tested against **Cloudflare version v4**.
6565
- Credentials for the above AWS S3 and SQS input types should be configured using the [link](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config).
6666
- Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
6767

68+
### To collect data from GCS Buckets, follow the below steps:
69+
- Configure the [Data Forwarder](https://developers.cloudflare.com/logs/get-started/enable-destinations/google-cloud-storage/) to ingest data into a GCS bucket.
70+
- Configure the GCS bucket names and credentials along with the required configs under the "Collect Cloudflare Logpush logs via Google Cloud Storage" section.
71+
- Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
72+
73+
**Note**:
74+
- The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
75+
- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
76+
- The GCS input currently only supports json data.
77+
6878
### To collect data from the Cloudflare HTTP Endpoint, follow the below steps:
6979
- Reference link to [Enable HTTP destination](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) for Cloudflare Logpush.
7080
- Add same custom header along with its value on both the side for additional security.
@@ -88,7 +98,7 @@ curl --location --request POST 'https://api.cloudflare.com/client/v4/zones/<ZONE
8898
2. In the integrations search bar type **Cloudflare Logpush**.
8999
3. Click the **Cloudflare Logpush** integration from the search results.
90100
4. Click the **Add Cloudflare Logpush** button to add Cloudflare Logpush integration.
91-
5. Enable the Integration with the HTTP Endpoint or AWS S3 input.
101+
5. Enable the Integration with the HTTP Endpoint, AWS S3 input or GCS input.
92102
6. Under the AWS S3 input, there are two types of inputs: using AWS S3 Bucket or using SQS.
93103
7. Configure Cloudflare to send logs to the Elastic Agent.
94104

‎packages/cloudflare_logpush/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.4.0"
3+
changes:
4+
- description: Added support for GCS input.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/4720
27
- version: "0.3.0"
38
changes:
49
- description: Update package to ECS 8.5.0.

‎packages/cloudflare_logpush/data_stream/audit/agent/stream/gcs.yml.hbs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{{#if project_id}}
2+
project_id: {{project_id}}
3+
{{/if}}
4+
{{#if service_account_key}}
5+
auth.credentials_json.account_key: {{service_account_key}}
6+
{{/if}}
7+
{{#if service_account_file}}
8+
auth.credentials_file.path: {{service_account_file}}
9+
{{/if}}
10+
{{#if number_of_workers}}
11+
max_workers: {{number_of_workers}}
12+
{{/if}}
13+
{{#if polling}}
14+
poll: {{polling}}
15+
{{/if}}
16+
{{#if poll_interval}}
17+
poll_interval: {{poll_interval}}
18+
{{/if}}
19+
{{#if bucket_timeout}}
20+
bucket_timeout: {{bucket_timeout}}
21+
{{/if}}
22+
{{#if buckets}}
23+
buckets:
24+
{{buckets}}
25+
{{/if}}
26+
{{#if tags}}
27+
tags:
28+
{{#each tags as |tag|}}
29+
- {{tag}}
30+
{{/each}}
31+
{{/if}}
32+
{{#contains "forwarded" tags}}
33+
publisher_pipeline.disable_host: true
34+
{{/contains}}
35+
{{#if processors}}
36+
processors:
37+
{{processors}}
38+
{{/if}}

‎packages/cloudflare_logpush/data_stream/audit/manifest.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,3 +149,81 @@ streams:
149149
show_user: false
150150
description: >-
151151
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
152+
- input: gcs
153+
title: Audit logs via GCS
154+
description: Collect Audit logs from Cloudflare.
155+
template_path: gcs.yml.hbs
156+
vars:
157+
- name: max_workers
158+
type: integer
159+
title: 'Maximum number of workers'
160+
multi: false
161+
required: false
162+
show_user: true
163+
default: 3
164+
description: Determines how many workers are spawned per bucket.
165+
- name: poll
166+
type: bool
167+
title: 'Polling'
168+
multi: false
169+
required: false
170+
show_user: true
171+
default: true
172+
description: Determines if the bucket will be continuously polled for new documents.
173+
- name: poll_interval
174+
type: text
175+
title: 'Polling interval'
176+
multi: false
177+
required: false
178+
show_user: true
179+
default: 15s
180+
description: Determines the time interval between polling operations.
181+
- name: bucket_timeout
182+
type: text
183+
title: 'Bucket Timeout'
184+
multi: false
185+
required: false
186+
show_user: true
187+
default: 120s
188+
description: Defines the maximum time that the sdk will wait for a bucket api response before timing out.
189+
- name: buckets
190+
type: yaml
191+
title: Buckets
192+
description: "This attribute contains the details about a specific bucket like, name, max_workers, poll,
193+
poll_interval and bucket_timeout. The attribute 'name' is specific to a bucket as it describes the bucket name,
194+
while the fields max_workers, poll, poll_interval and bucket_timeout can exist both at the bucket level and at the global level.
195+
If you have already defined the attributes globally, then you can only specify the name in this yaml config. If you want to override any specific
196+
attribute for a specific bucket, then, you can define it here. Any attribute defined in the yaml will override the global definitions.
197+
Please see the relevant [Documentation](https://www.elastic.co/guide/en/beats/filebeat/8.5/filebeat-input-gcs.html#attrib-buckets) for further information.\n"
198+
required: true
199+
show_user: true
200+
default: |
201+
# You can define as many buckets as you want here.
202+
203+
- name: audit_logs
204+
- name: audit_logs_2
205+
206+
# The config below is an example of how to override the global config.
207+
208+
#- name: audit_logs_3
209+
# max_workers: 3
210+
# poll: true
211+
# poll_interval: 10s
212+
# bucket_timeout: 30s
213+
- name: processors
214+
type: yaml
215+
title: Processors
216+
multi: false
217+
required: false
218+
show_user: false
219+
description: |
220+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
221+
- name: tags
222+
type: text
223+
title: Tags
224+
multi: true
225+
required: true
226+
show_user: false
227+
default:
228+
- forwarded
229+
- cloudflare_logpush_audit

‎packages/cloudflare_logpush/data_stream/dns/agent/stream/gcs.yml.hbs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{{#if project_id}}
2+
project_id: {{project_id}}
3+
{{/if}}
4+
{{#if service_account_key}}
5+
auth.credentials_json.account_key: {{service_account_key}}
6+
{{/if}}
7+
{{#if service_account_file}}
8+
auth.credentials_file.path: {{service_account_file}}
9+
{{/if}}
10+
{{#if number_of_workers}}
11+
max_workers: {{number_of_workers}}
12+
{{/if}}
13+
{{#if polling}}
14+
poll: {{polling}}
15+
{{/if}}
16+
{{#if poll_interval}}
17+
poll_interval: {{poll_interval}}
18+
{{/if}}
19+
{{#if bucket_timeout}}
20+
bucket_timeout: {{bucket_timeout}}
21+
{{/if}}
22+
{{#if buckets}}
23+
buckets:
24+
{{buckets}}
25+
{{/if}}
26+
{{#if tags}}
27+
tags:
28+
{{#each tags as |tag|}}
29+
- {{tag}}
30+
{{/each}}
31+
{{/if}}
32+
{{#contains "forwarded" tags}}
33+
publisher_pipeline.disable_host: true
34+
{{/contains}}
35+
{{#if processors}}
36+
processors:
37+
{{processors}}
38+
{{/if}}

‎packages/cloudflare_logpush/data_stream/dns/manifest.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,3 +149,81 @@ streams:
149149
show_user: false
150150
description: >-
151151
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
152+
- input: gcs
153+
title: DNS logs via GCS
154+
description: Collect DNS logs from Cloudflare.
155+
template_path: gcs.yml.hbs
156+
vars:
157+
- name: max_workers
158+
type: integer
159+
title: 'Maximum number of workers'
160+
multi: false
161+
required: false
162+
show_user: true
163+
default: 3
164+
description: Determines how many workers are spawned per bucket.
165+
- name: poll
166+
type: bool
167+
title: 'Polling'
168+
multi: false
169+
required: false
170+
show_user: true
171+
default: true
172+
description: Determines if the bucket will be continuously polled for new documents.
173+
- name: poll_interval
174+
type: text
175+
title: 'Polling interval'
176+
multi: false
177+
required: false
178+
show_user: true
179+
default: 15s
180+
description: Determines the time interval between polling operations.
181+
- name: bucket_timeout
182+
type: text
183+
title: 'Bucket Timeout'
184+
multi: false
185+
required: false
186+
show_user: true
187+
default: 120s
188+
description: Defines the maximum time that the sdk will wait for a bucket api response before timing out.
189+
- name: buckets
190+
type: yaml
191+
title: Buckets
192+
description: "This attribute contains the details about a specific bucket like, name, max_workers, poll,
193+
poll_interval and bucket_timeout. The attribute 'name' is specific to a bucket as it describes the bucket name,
194+
while the fields max_workers, poll, poll_interval and bucket_timeout can exist both at the bucket level and at the global level.
195+
If you have already defined the attributes globally, then you can only specify the name in this yaml config. If you want to override any specific
196+
attribute for a specific bucket, then, you can define it here. Any attribute defined in the yaml will override the global definitions.
197+
Please see the relevant [Documentation](https://www.elastic.co/guide/en/beats/filebeat/8.5/filebeat-input-gcs.html#attrib-buckets) for further information.\n"
198+
required: true
199+
show_user: true
200+
default: |
201+
# You can define as many buckets as you want here.
202+
203+
- name: dns_logs
204+
- name: dns_logs_2
205+
206+
# The config below is an example of how to override the global config.
207+
208+
#- name: dns_logs_3
209+
# max_workers: 3
210+
# poll: true
211+
# poll_interval: 10s
212+
# bucket_timeout: 30s
213+
- name: processors
214+
type: yaml
215+
title: Processors
216+
multi: false
217+
required: false
218+
show_user: false
219+
description: |
220+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
221+
- name: tags
222+
type: text
223+
title: Tags
224+
multi: true
225+
required: true
226+
show_user: false
227+
default:
228+
- forwarded
229+
- cloudflare_logpush_audit

‎packages/cloudflare_logpush/data_stream/firewall_event/agent/stream/gcs.yml.hbs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{{#if project_id}}
2+
project_id: {{project_id}}
3+
{{/if}}
4+
{{#if service_account_key}}
5+
auth.credentials_json.account_key: {{service_account_key}}
6+
{{/if}}
7+
{{#if service_account_file}}
8+
auth.credentials_file.path: {{service_account_file}}
9+
{{/if}}
10+
{{#if number_of_workers}}
11+
max_workers: {{number_of_workers}}
12+
{{/if}}
13+
{{#if polling}}
14+
poll: {{polling}}
15+
{{/if}}
16+
{{#if poll_interval}}
17+
poll_interval: {{poll_interval}}
18+
{{/if}}
19+
{{#if bucket_timeout}}
20+
bucket_timeout: {{bucket_timeout}}
21+
{{/if}}
22+
{{#if buckets}}
23+
buckets:
24+
{{buckets}}
25+
{{/if}}
26+
{{#if tags}}
27+
tags:
28+
{{#each tags as |tag|}}
29+
- {{tag}}
30+
{{/each}}
31+
{{/if}}
32+
{{#contains "forwarded" tags}}
33+
publisher_pipeline.disable_host: true
34+
{{/contains}}
35+
{{#if processors}}
36+
processors:
37+
{{processors}}
38+
{{/if}}

0 commit comments

Comments
 (0)