Skip to content

Commit 8c6ff9a

Browse files
ksobrenat32ksobrenat32
authored
Add a security GCP log analyzer (#35922)
* Add the base log_analyzer * Add github action for security logging * Enhance LogAnalyzer to filter logs by time range and include file names in event summary * Add dry-run option for weekly email report generation in LogAnalyzer * Better error handling for timezones and missing details * Refactor LogAnalyzer to use SinkCls for type consistency and enhance bucket permission management for log sinks
1 parent 4549214 commit 8c6ff9a

File tree

5 files changed

+556
-0
lines changed

5 files changed

+556
-0
lines changed

‎.github/workflows/beam_Infrastructure_SecurityLogging.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# This workflow works with the GCP security log analyzer to
19+
# generate weekly security reports and initialize log sinks
20+
21+
name: GCP Security Log Analyzer
22+
23+
on:
24+
workflow_dispatch:
25+
schedule:
26+
# Once a week at 9:00 AM on Monday
27+
- cron: '0 9 * * 1'
28+
push:
29+
paths:
30+
- 'infra/security/config.yml'
31+
32+
# This allows a subsequently queued workflow run to interrupt previous runs
33+
concurrency:
34+
group: '${{ github.workflow }} @ ${{ github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}'
35+
cancel-in-progress: true
36+
37+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
38+
permissions:
39+
contents: read
40+
41+
jobs:
42+
beam_GCP_Security_LogAnalyzer:
43+
name: GCP Security Log Analysis
44+
runs-on: [self-hosted, ubuntu-20.04, main]
45+
timeout-minutes: 30
46+
steps:
47+
- uses: actions/checkout@v4
48+
49+
- name: Setup Python
50+
uses: actions/setup-python@v4
51+
with:
52+
python-version: '3.13'
53+
54+
- name: Install Python dependencies
55+
working-directory: ./infra/security
56+
run: |
57+
python -m pip install --upgrade pip
58+
pip install -r requirements.txt
59+
60+
- name: Setup gcloud
61+
uses: google-github-actions/setup-gcloud@v2
62+
63+
- name: Initialize Log Sinks
64+
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
65+
working-directory: ./infra/security
66+
run: python log_analyzer.py --config config.yml initialize
67+
68+
- name: Generate Weekly Security Report
69+
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
70+
working-directory: ./infra/security
71+
env:
72+
SMTP_SERVER: smtp.gmail.com
73+
SMTP_PORT: 465
74+
EMAIL_ADDRESS: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_ADDRESS }}
75+
EMAIL_PASSWORD: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_PASSWORD }}
76+
EMAIL_RECIPIENT: "dev@beam.apache.org"
77+
run: python log_analyzer.py --config config.yml generate-report --dry-run

‎infra/security/README.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# GCP Security Analyzer
21+
22+
This document describes the implementation of a security analyzer for Google Cloud Platform (GCP) resources. The analyzer is designed to enhance security monitoring within our GCP environment by capturing critical events and generating alerts for specific security-sensitive actions.
23+
24+
## How It Works
25+
26+
1. **Log Sinks**: The system uses [GCP Log Sinks](https://cloud.google.com/logging/docs/export/configure_export_v2) to capture specific security-related log entries. These sinks are configured to filter for events like IAM policy changes or service account key creation.
27+
2. **Log Storage**: The filtered logs are routed to a dedicated Google Cloud Storage (GCS) bucket for persistence and analysis.
28+
3. **Report Generation**: A scheduled job runs weekly, executing the `log_analyzer.py` script.
29+
4. **Email Alerts**: The script analyzes the logs from the past week, compiles a summary of security events, and sends a report to a configured email address.
30+
31+
## Configuration
32+
33+
The behavior of the log analyzer is controlled by a `config.yml` file. Here’s an overview of the configuration options:
34+
35+
- `project_id`: The GCP project ID where the resources are located.
36+
- `bucket_name`: The name of the GCS bucket where logs will be stored.
37+
- `logging`: Configures the logging level and format for the script.
38+
- `sinks`: A list of log sinks to be created. Each sink has the following properties:
39+
- `name`: A unique name for the sink.
40+
- `description`: A brief description of what the sink monitors.
41+
- `filter_methods`: A list of GCP API methods to include in the filter (e.g., `SetIamPolicy`).
42+
- `excluded_principals`: A list of service accounts or user emails to exclude from monitoring, such as CI/CD service accounts.
43+
44+
### Example Configuration (`config.yml`)
45+
46+
```yaml
47+
project_id: your-gcp-project-id
48+
bucket_name: your-log-storage-bucket
49+
50+
sinks:
51+
- name: iam-policy-changes
52+
description: Monitors changes to IAM policies.
53+
filter_methods:
54+
- "SetIamPolicy"
55+
excluded_principals:
56+
- "ci-cd-account@your-project.iam.gserviceaccount.com"
57+
```
58+
59+
## Usage
60+
61+
The `log_analyzer.py` script provides two main commands for managing the security analyzer.
62+
63+
### Initializing Sinks
64+
65+
To create or update the log sinks in GCP based on your `config.yml` file, run the following command:
66+
67+
```bash
68+
python log_analyzer.py --config config.yml initialize
69+
```
70+
71+
This command ensures that the log sinks are correctly configured to capture the desired security events.
72+
73+
### Generating Weekly Reports
74+
75+
To generate and send the weekly security report, run this command:
76+
77+
```bash
78+
python log_analyzer.py --config config.yml generate-report
79+
```
80+
81+
This is typically run as a scheduled job (GitHub Action) to automate the delivery of weekly security reports.
82+
83+
84+

‎infra/security/config.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
project_id: testing-me-460223
17+
18+
# Logging
19+
logging:
20+
level: DEBUG
21+
format: "[%(asctime)s] %(levelname)s: %(message)s"
22+
23+
# gcloud storage bucket
24+
bucket_name: "testing-me-460223-tfstate"
25+
26+
# GCP Log sinks
27+
sinks:
28+
- name: iam-policy-changes
29+
description: Monitors changes to IAM policies, excluding approved CI/CD service accounts.
30+
filter_methods:
31+
- "SetIamPolicy"
32+
excluded_principals:
33+
- beam-github-actions@apache-beam-testing.iam.gserviceaccount.com
34+
- github-self-hosted-runners@apache-beam-testing.iam.gserviceaccount.com
35+
36+
- name: sa-key-management
37+
description: Monitors creation and deletion of service account keys.
38+
filter_methods:
39+
- "google.iam.admin.v1.IAM.CreateServiceAccountKey"
40+
- "google.iam.admin.v1.IAM.DeleteServiceAccountKey"
41+
excluded_principals:
42+
- beam-github-actions@apache-beam-testing.iam.gserviceaccount.com
43+
- github-self-hosted-runners@apache-beam-testing.iam.gserviceaccount.com

0 commit comments

Comments
 (0)