Skip to content

Commit eae0c45

Browse files
gi-erregi-erre
authored
Add GCP Ingestion onboarding module (#4)
1 parent 7cfbba3 commit eae0c45

File tree

6 files changed

+274
-0
lines changed

6 files changed

+274
-0
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,7 @@ Provides unified threat-detection, compliance, forensics and analysis through th
66

77
* **[CSPM](https://docs.sysdig.com/en/docs/sysdig-secure/benchmarks/)**: It evaluates periodically your cloud configuration, using Cloud Custodian, against some benchmarks and returns the results and remediation you need to fix. Managed through `trust-relationship` module. <br/>
88

9+
* **CDR (Cloud Detection and Response)**: It send periodically the Audit Logs collected from a GCP project to Sysdig's systems, this by collecting them in a PubSub topic through a Sink and then sending them through a `PUSH` integration. Managed through `webhook-datasource` module. <br/>
10+
911
For other Cloud providers check: [AWS](https://github.com/draios/terraform-aws-secure-for-cloud)
1012

‎modules/services/webhook-datasource/README.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# GCP Webhook Datasource Module
2+
3+
This Module creates the resources required to send Project-specific AuditLogs to Sysdig by creating a dedicated Push Subscription tied to a ingestion PubSub topic.
4+
5+
6+
The following resources will be created in each instrumented account:
7+
- A Sink to direct the AuditLogs towards a dedicated PubSub topic
8+
- A PubSub ingestion topic that will hold all the AuditLogs coming from the specified project
9+
- A Push Subscription that will POST the AuditLogs collected from the project towards Sysdig's backend
10+
- All the necessary Service Accounts and Policies to enable the AuditLogs publishing operation
11+
12+
## Requirements
13+
14+
| Name | Version |
15+
|------|---------|
16+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
17+
| <a name="requirement_google"></a> [google](#requirement\_google) | >= 4.21.0 |
18+
19+
## Providers
20+
21+
| Name | Version |
22+
|------|---------|
23+
| <a name="provider_google"></a> [google](#provider\_google) | >= 4.21.0 |
24+
25+
## Modules
26+
27+
No modules.
28+
29+
## Resources
30+
31+
| Name | Type |
32+
|------|------|
33+
| [google_logging_project_sink.ingestion_sink](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink) | resource |
34+
| [google_project_iam_audit_config.audit_config](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_audit_config) | resource |
35+
| [google_pubsub_subscription.ingestion_topic_push_subscription](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription) | resource |
36+
| [google_pubsub_topic.deadletter_topic](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic) | resource |
37+
| [google_pubsub_topic.ingestion_topic](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic) | resource |
38+
| [google_pubsub_topic_iam_member.publisher_iam_member](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam_member) | resource |
39+
| [google_service_account.push_auth](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
40+
| [google_service_account_iam_binding.push_auth_binding](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_iam_binding) | resource |
41+
42+
## Inputs
43+
44+
| Name | Description | Type | Default | Required |
45+
|------|-------------|------|---------|:--------:|
46+
| <a name="input_ack_deadline_seconds"></a> [ack\_deadline\_seconds](#input\_ack\_deadline\_seconds) | (Optional) Maximum time in seconds after Sysdig's subscriber receives a message before the subscriber should acknowledge the message | `number` | `60` | no |
47+
| <a name="input_labels"></a> [labels](#input\_labels) | (Optional) Labels to be associated with Sysdig-originated resources | `map(string)` | <pre>{<br> "originator": "Sysdig"<br>}</pre> | no |
48+
| <a name="input_max_delivery_attempts"></a> [max\_delivery\_attempts](#input\_max\_delivery\_attempts) | (Optional) Number of attempts redelivering missed messages from the deadletter topic to the main one | `number` | `5` | no |
49+
| <a name="input_maximum_backoff"></a> [maximum\_backoff](#input\_maximum\_backoff) | (Optional) Maximum backoff time for exponential backoff of the push subscription retry policy | `string` | `"600s"` | no |
50+
| <a name="input_message_retention_duration"></a> [message\_retention\_duration](#input\_message\_retention\_duration) | (Optional) How long unacknowledged messages are retained in Sysdig's subscription backlog, from the moment a message is published | `string` | `"604800s"` | no |
51+
| <a name="input_minimum_backoff"></a> [minimum\_backoff](#input\_minimum\_backoff) | (Optional) Minimum backoff time for exponential backoff of the push subscription retry policy | `string` | `"10s"` | no |
52+
| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | (Required) Target Project identifier provided by the customer | `string` | n/a | yes |
53+
| <a name="input_push_endpoint"></a> [push\_endpoint](#input\_push\_endpoint) | (Required) Final endpoint towards which audit logs POST calls will be directed | `string` | n/a | yes |
54+
55+
## Outputs
56+
57+
| Name | Description |
58+
|------|-------------|
59+
| <a name="output_project_id"></a> [project\_id](#output\_project\_id) | GCP Project Identifier |
60+
| <a name="output_push_endpoint"></a> [push\_endpoint](#output\_push\_endpoint) | Push endpoint towards which the POST request will be directed |
61+
| <a name="output_push_subscription_service_account"></a> [push\_subscription\_service\_account](#output\_push\_subscription\_service\_account) | Service Account used to send POST messages, a KMS key needs to be manually added in order to properly authenticate the requests at Sysdig's side |

‎modules/services/webhook-datasource/main.tf‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
########################################################################################
2+
# The webhook-datasource module takes care of provisioning the necessary resources to make Sysdig's
3+
# backend able to ingest data from a single GCP project.
4+
#
5+
# Before applying the changes defined in this module, the following operations need to
6+
# be performed on the target GCP environment:
7+
# - Creating a GCP project
8+
# - Enabling the following APIs in the target environment (https://support.google.com/googleapi/answer/6158841?hl=en)
9+
# - Cloud Pub/Sub API
10+
# - Identity and Access Management (IAM) API
11+
# - IAM Service Account Credentials API
12+
# - Cloud Resource Manager API
13+
#
14+
# Given that, this module will take care of enabling the AuditLogs for the selected
15+
# project, direct them to a dedicated PubSub through a Sink and finally creating a Push
16+
# Subscription that will send the data to Sysdig's backend. This module takes also care
17+
# of creating the necessary service accounts along with the necessary policies to enable
18+
# pushing logs to Sysdig's system.
19+
########################################################################################
20+
21+
#-------------#
22+
# GCP Project #
23+
#-------------#
24+
25+
data "google_project" "target_project" {
26+
project_id = var.project_id
27+
}
28+
29+
#------------#
30+
# Audit Logs #
31+
#------------#
32+
33+
resource "google_project_iam_audit_config" "audit_config" {
34+
project = var.project_id
35+
service = "allServices"
36+
37+
audit_log_config {
38+
log_type = "ADMIN_READ"
39+
}
40+
41+
audit_log_config {
42+
log_type = "DATA_READ"
43+
}
44+
45+
audit_log_config {
46+
log_type = "DATA_WRITE"
47+
}
48+
}
49+
50+
#------#
51+
# Sink #
52+
#------#
53+
54+
resource "google_logging_project_sink" "ingestion_sink" {
55+
name = "${google_pubsub_topic.ingestion_topic.name}_sink"
56+
description = "Sysdig sink to direct the AuditLogs to the PubSub topic used for data gathering"
57+
58+
# NOTE: The target destination is a PubSub topic
59+
destination = "pubsub.googleapis.com/projects/${var.project_id}/topics/${google_pubsub_topic.ingestion_topic.name}"
60+
61+
filter = "protoPayload.@type = \"type.googleapis.com/google.cloud.audit.AuditLog\""
62+
63+
# NOTE: Used to create a dedicated writer identity and not using the default one
64+
unique_writer_identity = true
65+
}
66+
67+
resource "google_pubsub_topic_iam_member" "publisher_iam_member" {
68+
project = google_pubsub_topic.ingestion_topic.project
69+
topic = google_pubsub_topic.ingestion_topic.name
70+
role = "roles/pubsub.publisher"
71+
member = google_logging_project_sink.ingestion_sink.writer_identity
72+
}
73+
74+
#-----------------#
75+
# Ingestion topic #
76+
#-----------------#
77+
78+
resource "google_pubsub_topic" "ingestion_topic" {
79+
name = "ingestion_topic"
80+
labels = var.labels
81+
project = var.project_id
82+
message_retention_duration = var.message_retention_duration
83+
}
84+
85+
resource "google_pubsub_topic" "deadletter_topic" {
86+
name = "dl-${google_pubsub_topic.ingestion_topic.name}"
87+
88+
project = var.project_id
89+
message_retention_duration = var.message_retention_duration
90+
}
91+
92+
#-------------------#
93+
# Push Subscription #
94+
#-------------------#
95+
96+
resource "google_service_account" "push_auth" {
97+
account_id = "ingestion-topic-push-auth"
98+
display_name = "Push Auth Service Account"
99+
}
100+
101+
resource "google_service_account_iam_binding" "push_auth_binding" {
102+
service_account_id = google_service_account.push_auth.name
103+
role = "roles/iam.serviceAccountTokenCreator"
104+
105+
members = [
106+
"serviceAccount:${google_service_account.push_auth.email}",
107+
]
108+
}
109+
110+
resource "google_pubsub_subscription" "ingestion_topic_push_subscription" {
111+
name = "${google_pubsub_topic.ingestion_topic.name}_push_subscription"
112+
topic = google_pubsub_topic.ingestion_topic.name
113+
labels = var.labels
114+
ack_deadline_seconds = var.ack_deadline_seconds
115+
message_retention_duration = var.message_retention_duration
116+
117+
push_config {
118+
push_endpoint = var.push_endpoint
119+
120+
attributes = {
121+
x-goog-version = "v1"
122+
}
123+
124+
oidc_token {
125+
service_account_email = google_service_account.push_auth.email
126+
audience = "sysdig_secure"
127+
}
128+
}
129+
130+
retry_policy {
131+
minimum_backoff = var.minimum_backoff
132+
maximum_backoff = var.maximum_backoff
133+
}
134+
135+
dead_letter_policy {
136+
dead_letter_topic = google_pubsub_topic.deadletter_topic.id
137+
max_delivery_attempts = var.max_delivery_attempts
138+
}
139+
}
140+

‎modules/services/webhook-datasource/outputs.tf‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
output "project_id" {
2+
value = data.google_project.target_project.id
3+
description = "GCP Project Identifier"
4+
}
5+
6+
output "push_subscription_service_account" {
7+
value = google_service_account.push_auth.name
8+
description = "Service Account used to send POST messages, a KMS key needs to be manually added in order to properly authenticate the requests at Sysdig's side"
9+
}
10+
11+
output "push_endpoint" {
12+
value = google_pubsub_subscription.ingestion_topic_push_subscription.push_config[0].push_endpoint
13+
description = "Push endpoint towards which the POST request will be directed"
14+
}

‎modules/services/webhook-datasource/variables.tf‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
variable "project_id" {
2+
type = string
3+
description = "(Required) Target Project identifier provided by the customer"
4+
}
5+
6+
variable "push_endpoint" {
7+
type = string
8+
description = "(Required) Final endpoint towards which audit logs POST calls will be directed"
9+
}
10+
11+
variable "labels" {
12+
type = map(string)
13+
description = "(Optional) Labels to be associated with Sysdig-originated resources"
14+
default = {
15+
originator = "sysdig"
16+
}
17+
}
18+
19+
variable "ack_deadline_seconds" {
20+
type = number
21+
description = "(Optional) Maximum time in seconds after Sysdig's subscriber receives a message before the subscriber should acknowledge the message"
22+
default = 60
23+
}
24+
25+
variable "message_retention_duration" {
26+
type = string
27+
description = "(Optional) How long unacknowledged messages are retained in Sysdig's subscription backlog, from the moment a message is published"
28+
default = "604800s"
29+
}
30+
31+
variable "max_delivery_attempts" {
32+
type = number
33+
description = "(Optional) Number of attempts redelivering missed messages from the deadletter topic to the main one"
34+
default = 5
35+
}
36+
37+
variable "minimum_backoff" {
38+
type = string
39+
description = "(Optional) Minimum backoff time for exponential backoff of the push subscription retry policy"
40+
default = "10s"
41+
}
42+
43+
variable "maximum_backoff" {
44+
type = string
45+
description = "(Optional) Maximum backoff time for exponential backoff of the push subscription retry policy"
46+
default = "600s"
47+
}

‎modules/services/webhook-datasource/versions.tf‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
terraform {
2+
required_version = ">= 1.0.0"
3+
4+
required_providers {
5+
google = {
6+
source = "hashicorp/google"
7+
version = ">= 4.21.0"
8+
}
9+
}
10+
}

0 commit comments

Comments
 (0)