Skip to content

Commit 266f8ef

Browse files
ShourieGShourieG
authored
Add GCS input to Akamai (elastic#4728)
* initial commit for gcs input in akamai package * updated manifest * updated changelog * updated config * updated config * updated yml.hbs * updated pr removed unused if condition in gcs.yml.hbs
1 parent 941348f commit 266f8ef

File tree

6 files changed

+169
-4
lines changed

6 files changed

+169
-4
lines changed

‎packages/akamai/_dev/build/docs/README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,17 @@ Use this API to get security event data generated on the ​Akamai​ platform a
1212

1313
See [Akamai API get started](https://techdocs.akamai.com/siem-integration/reference/api-get-started) to set up your Akamai account and get your credentials.
1414

15+
### To collect data from GCS Bucket [Beta], follow the below steps:
16+
- Configure the [Data Forwarder](https://techdocs.akamai.com/datastream2/docs/stream-google-cloud/) to ingest data into a GCS bucket.
17+
- Configure the GCS bucket names and credentials along with the required configs under the "Collect Akamai SIEM logs via Google Cloud Storage" section.
18+
- Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
19+
20+
**Note**:
21+
- The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
22+
- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
23+
- The GCS input currently only supports JSON data.
24+
- This input is still in beta.
25+
1526
{{fields "siem"}}
1627

1728
{{event "siem"}}

‎packages/akamai/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "2.3.0"
3+
changes:
4+
- description: Added support for GCS input.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/4728
27
- version: "2.2.0"
38
changes:
49
- description: Update package to ECS 8.5.0.

‎packages/akamai/data_stream/siem/agent/stream/gcs.yml.hbs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{{#if project_id}}
2+
project_id: {{project_id}}
3+
{{/if}}
4+
{{#if service_account_key}}
5+
auth.credentials_json.account_key: {{service_account_key}}
6+
{{/if}}
7+
{{#if service_account_file}}
8+
auth.credentials_file.path: {{service_account_file}}
9+
{{/if}}
10+
{{#if number_of_workers}}
11+
max_workers: {{number_of_workers}}
12+
{{/if}}
13+
{{#if polling}}
14+
poll: {{polling}}
15+
{{/if}}
16+
{{#if poll_interval}}
17+
poll_interval: {{poll_interval}}
18+
{{/if}}
19+
{{#if bucket_timeout}}
20+
bucket_timeout: {{bucket_timeout}}
21+
{{/if}}
22+
{{#if buckets}}
23+
buckets:
24+
{{buckets}}
25+
{{/if}}
26+
{{#if tags}}
27+
tags:
28+
{{#each tags as |tag|}}
29+
- {{tag}}
30+
{{/each}}
31+
{{/if}}
32+
{{#contains "forwarded" tags}}
33+
publisher_pipeline.disable_host: true
34+
{{/contains}}
35+
{{#if processors}}
36+
processors:
37+
{{processors}}
38+
{{/if}}

‎packages/akamai/data_stream/siem/manifest.yml‎

Lines changed: 100 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@ title: Akamai SIEM Logs
33
release: experimental
44
streams:
55
- input: httpjson
6+
template_path: httpjson.yml.hbs
7+
title: Akamai SIEM logs
8+
description: Collect Akamai logs via the SIEM API
69
vars:
710
- name: api_host
811
type: text
@@ -100,6 +103,100 @@ streams:
100103
required: false
101104
show_user: false
102105
description: "Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. \nThis executes in the agent before the logs are parsed. \nSee [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.\n"
103-
template_path: httpjson.yml.hbs
104-
title: Akamai SIEM logs
105-
description: Collect Akamai logs via the SIEM API
106+
- input: gcs
107+
title: Collect Akamai SIEM logs via Google Cloud Storage [Beta]
108+
description: Collecting SIEM logs from Akamai via Google Cloud Storage.
109+
template_path: gcs.yml.hbs
110+
vars:
111+
- name: project_id
112+
type: text
113+
title: "Project Id"
114+
description: It is a required parameter to collect logs via GCS.
115+
multi: false
116+
required: true
117+
show_user: true
118+
default: my-project-id
119+
- name: service_account_key
120+
type: password
121+
title: "Credentials json key"
122+
description: It is an optional parameter for authentication.
123+
multi: false
124+
required: false
125+
show_user: true
126+
- name: service_account_file
127+
type: text
128+
title: "Credentials file path"
129+
description: It is an optional parameter for authentication.
130+
multi: false
131+
required: false
132+
show_user: false
133+
- name: max_workers
134+
type: integer
135+
title: 'Maximum number of workers'
136+
multi: false
137+
required: false
138+
show_user: true
139+
default: 3
140+
description: Determines how many workers are spawned per bucket.
141+
- name: poll
142+
type: bool
143+
title: 'Polling'
144+
multi: false
145+
required: false
146+
show_user: true
147+
default: true
148+
description: Determines if the bucket will be continuously polled for new documents.
149+
- name: poll_interval
150+
type: text
151+
title: 'Polling interval'
152+
multi: false
153+
required: false
154+
show_user: true
155+
default: 15s
156+
description: Determines the time interval between polling operations.
157+
- name: bucket_timeout
158+
type: text
159+
title: 'Bucket Timeout'
160+
multi: false
161+
required: false
162+
show_user: true
163+
default: 120s
164+
description: Defines the maximum time that the sdk will wait for a bucket api response before timing out.
165+
- name: buckets
166+
type: yaml
167+
title: Buckets
168+
description: "This attribute contains the details about a specific bucket like, name, max_workers, poll,
169+
poll_interval and bucket_timeout. The attribute 'name' is specific to a bucket as it describes the bucket name,
170+
while the fields max_workers, poll, poll_interval and bucket_timeout can exist both at the bucket level and at the global level.
171+
If you have already defined the attributes globally, then you can only specify the name in this yaml config. If you want to override any specific
172+
attribute for a specific bucket, then, you can define it here. Any attribute defined in the yaml will override the global definitions.
173+
Please see the relevant [Documentation](https://www.elastic.co/guide/en/beats/filebeat/8.5/filebeat-input-gcs.html#attrib-buckets) for further information.\n"
174+
required: true
175+
show_user: true
176+
default: |
177+
# You can define as many buckets as you want here.
178+
- name: siem_gcs_bucket_1
179+
- name: siem_gcs_bucket_2
180+
# The config below is an example of how to override the global config.
181+
#- name: siem_gcs_bucket_3
182+
# max_workers: 3
183+
# poll: true
184+
# poll_interval: 10s
185+
# bucket_timeout: 30s
186+
- name: processors
187+
type: yaml
188+
title: Processors
189+
multi: false
190+
required: false
191+
show_user: false
192+
description: |
193+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
194+
- name: tags
195+
type: text
196+
title: Tags
197+
multi: true
198+
required: true
199+
show_user: false
200+
default:
201+
- forwarded
202+
- akamai-siem

‎packages/akamai/docs/README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,17 @@ Use this API to get security event data generated on the ​Akamai​ platform a
1212

1313
See [Akamai API get started](https://techdocs.akamai.com/siem-integration/reference/api-get-started) to set up your Akamai account and get your credentials.
1414

15+
### To collect data from GCS Bucket [Beta], follow the below steps:
16+
- Configure the [Data Forwarder](https://techdocs.akamai.com/datastream2/docs/stream-google-cloud/) to ingest data into a GCS bucket.
17+
- Configure the GCS bucket names and credentials along with the required configs under the "Collect Akamai SIEM logs via Google Cloud Storage" section.
18+
- Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
19+
20+
**Note**:
21+
- The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
22+
- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
23+
- The GCS input currently only supports JSON data.
24+
- This input is still in beta.
25+
1526
**Exported fields**
1627

1728
| Field | Description | Type |

‎packages/akamai/manifest.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: akamai
22
title: Akamai
3-
version: "2.2.0"
3+
version: "2.3.0"
44
release: ga
55
description: Collect logs from Akamai with Elastic Agent.
66
type: integration
@@ -22,5 +22,8 @@ policy_templates:
2222
- type: httpjson
2323
title: "Collect Akamai SIEM logs via API"
2424
description: "Collecting SIEM logs from Akamai via API"
25+
- type: gcs
26+
title: "Collect Akamai SIEM logs via Google Cloud Storage"
27+
description: "Collecting SIEM logs from Akamai via Google Cloud Storage"
2528
owner:
2629
github: elastic/security-external-integrations

0 commit comments

Comments
 (0)