Skip to content

Commit c75f3cd

Browse files
pyth0n1cpyth0n1c
authored
Merge pull request #334 from splunk/contentctl_5
contentctl 5 - Step 4 - ESCU 5.0 This PR has not been fully reviewed and we anticipate some issues. However, in order to enable testing in a number of different workflows, we are released it as an alpha.
2 parents c26e492 + 248e436 commit c75f3cd

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+1785
-1505
lines changed

‎.github/dependabot.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,8 @@ updates:
1212
schedule:
1313
interval: "daily"
1414
open-pull-requests-limit: 6
15+
- package-ecosystem: "github-actions"
16+
directory: "/"
17+
schedule:
18+
# Check for updates to GitHub Actions every week
19+
interval: "weekly"

‎.github/workflows/ruff.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: lint & format
2+
on:
3+
pull_request:
4+
types: [opened, reopened, synchronize]
5+
6+
jobs:
7+
lint:
8+
runs-on: ubuntu-latest
9+
steps:
10+
- uses: actions/checkout@v4
11+
- name: Install Python
12+
uses: actions/setup-python@v5
13+
with:
14+
python-version: "3.11"
15+
- name: Install ruff
16+
run: |
17+
python -m pip install --upgrade pip
18+
pip install ruff
19+
- name: Run lint
20+
run: ruff check --output-format=github contentctl/
21+
- name: Run Formatter
22+
run: ruff format --check contentctl/

‎.github/workflows/testEndToEnd.yml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
name: testEndToEnd
22
on:
3-
push:
43
pull_request:
5-
types: [opened, reopened]
4+
types: [opened, reopened, synchronize]
65
schedule:
76
- cron: "44 4 * * *"
87

@@ -11,8 +10,8 @@ jobs:
1110
strategy:
1211
fail-fast: false
1312
matrix:
14-
python_version: ["3.11", "3.12"]
15-
operating_system: ["ubuntu-20.04", "ubuntu-22.04", "macos-latest", "macos-14", "windows-2022"]
13+
python_version: ["3.11", "3.12", "3.13"]
14+
operating_system: ["ubuntu-24.04", "macos-15", "windows-2022"]
1615
#operating_system: ["ubuntu-20.04", "ubuntu-22.04", "macos-latest"]
1716

1817

‎.github/workflows/test_against_escu.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,8 @@
66
# note yet been fixed in security_content, we may see this workflow fail.
77
name: test_against_escu
88
on:
9-
push:
109
pull_request:
11-
types: [opened, reopened]
10+
types: [opened, reopened, synchronize]
1211
schedule:
1312
- cron: "44 4 * * *"
1413

@@ -17,9 +16,9 @@ jobs:
1716
strategy:
1817
fail-fast: false
1918
matrix:
20-
python_version: ["3.11", "3.12"]
19+
python_version: ["3.11", "3.12", "3.13"]
2120

22-
operating_system: ["ubuntu-20.04", "ubuntu-22.04", "macos-latest", "macos-14"]
21+
operating_system: ["ubuntu-24.04", "macos-15"]
2322
# Do not test against ESCU until known character encoding issue is resolved
2423
# operating_system: ["ubuntu-20.04", "ubuntu-22.04", "macos-latest", "macos-14", "windows-2022"]
2524

@@ -36,6 +35,7 @@ jobs:
3635
with:
3736
path: security_content
3837
repository: splunk/security_content
38+
ref: rba_migration
3939

4040
#Install the given version of Python we will test against
4141
- name: Install Required Python Version

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ poetry.lock
33
# usual mac files
44
.DS_Store
55
*/.DS_Store
6+
.ruff_cache
67

78
# custom
89
dist/*

‎.pre-commit-config.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
repos:
2+
- repo: https://github.com/pre-commit/pre-commit-hooks
3+
rev: v5.0.0 # Use the ref you want to point at
4+
hooks:
5+
- id: check-json
6+
- id: check-symlinks
7+
- id: check-yaml
8+
- id: detect-aws-credentials
9+
- id: detect-private-key
10+
- id: forbid-submodules
11+
- repo: https://github.com/astral-sh/ruff-pre-commit
12+
rev: v0.9.2
13+
hooks:
14+
- id: ruff
15+
args: [ --fix ]
16+
- id: ruff-format

‎.vscode/extensions.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"recommendations": [
3+
"charliermarsh.ruff"
4+
]
5+
}

‎.vscode/settings.json‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,15 @@
44
"python.testing.cwd": "${workspaceFolder}",
55
"python.languageServer": "Pylance",
66
"python.analysis.typeCheckingMode": "strict",
7-
"editor.defaultFormatter": "ms-python.black-formatter"
7+
"[python]": {
8+
"editor.formatOnSave": true,
9+
"editor.codeActionsOnSave": {
10+
"source.fixAll": "explicit",
11+
"source.organizeImports": "explicit"
12+
},
13+
"editor.defaultFormatter": "charliermarsh.ruff",
14+
},
15+
"ruff.nativeServer": "on"
816

917

1018
}

‎contentctl/actions/build.py‎

Lines changed: 39 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,17 @@
44

55
from dataclasses import dataclass
66

7-
from contentctl.objects.enums import SecurityContentProduct, SecurityContentType
7+
from contentctl.objects.enums import SecurityContentType
88
from contentctl.input.director import Director, DirectorOutputDto
99
from contentctl.output.conf_output import ConfOutput
1010
from contentctl.output.conf_writer import ConfWriter
1111
from contentctl.output.api_json_output import ApiJsonOutput
1212
from contentctl.output.data_source_writer import DataSourceWriter
13-
from contentctl.objects.lookup import Lookup
13+
from contentctl.objects.lookup import CSVLookup, Lookup_Type
1414
import pathlib
1515
import json
1616
import datetime
17-
from typing import Union
17+
import uuid
1818

1919
from contentctl.objects.config import build
2020

@@ -34,27 +34,41 @@ def execute(self, input_dto: BuildInputDto) -> DirectorOutputDto:
3434
updated_conf_files:set[pathlib.Path] = set()
3535
conf_output = ConfOutput(input_dto.config)
3636

37+
38+
# Construct a path to a YML that does not actually exist.
39+
# We mock this "fake" path since the YML does not exist.
40+
# This ensures the checking for the existence of the CSV is correct
41+
data_sources_fake_yml_path = input_dto.config.getPackageDirectoryPath() / "lookups" / "data_sources.yml"
42+
3743
# Construct a special lookup whose CSV is created at runtime and
38-
# written directly into the output folder. It is created with model_construct,
39-
# not model_validate, because the CSV does not exist yet.
44+
# written directly into the lookups folder. We will delete this after a build,
45+
# assuming that it is successful.
4046
data_sources_lookup_csv_path = input_dto.config.getPackageDirectoryPath() / "lookups" / "data_sources.csv"
41-
DataSourceWriter.writeDataSourceCsv(input_dto.director_output_dto.data_sources, data_sources_lookup_csv_path)
42-
input_dto.director_output_dto.addContentToDictMappings(Lookup.model_construct(description= "A lookup file that will contain the data source objects for detections.",
43-
filename=data_sources_lookup_csv_path,
44-
name="data_sources"))
4547

48+
49+
50+
DataSourceWriter.writeDataSourceCsv(input_dto.director_output_dto.data_sources, data_sources_lookup_csv_path)
51+
input_dto.director_output_dto.addContentToDictMappings(CSVLookup.model_construct(name="data_sources",
52+
id=uuid.UUID("b45c1403-6e09-47b0-824f-cf6e44f15ac8"),
53+
version=1,
54+
author=input_dto.config.app.author_name,
55+
date = datetime.date.today(),
56+
description= "A lookup file that will contain the data source objects for detections.",
57+
lookup_type=Lookup_Type.csv,
58+
file_path=data_sources_fake_yml_path))
4659
updated_conf_files.update(conf_output.writeHeaders())
47-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.detections, SecurityContentType.detections))
48-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.stories, SecurityContentType.stories))
49-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.baselines, SecurityContentType.baselines))
50-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.investigations, SecurityContentType.investigations))
51-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.lookups, SecurityContentType.lookups))
52-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.macros, SecurityContentType.macros))
53-
updated_conf_files.update(conf_output.writeObjects(input_dto.director_output_dto.dashboards, SecurityContentType.dashboards))
60+
updated_conf_files.update(conf_output.writeLookups(input_dto.director_output_dto.lookups))
61+
updated_conf_files.update(conf_output.writeDetections(input_dto.director_output_dto.detections))
62+
updated_conf_files.update(conf_output.writeStories(input_dto.director_output_dto.stories))
63+
updated_conf_files.update(conf_output.writeBaselines(input_dto.director_output_dto.baselines))
64+
updated_conf_files.update(conf_output.writeInvestigations(input_dto.director_output_dto.investigations))
65+
updated_conf_files.update(conf_output.writeMacros(input_dto.director_output_dto.macros))
66+
updated_conf_files.update(conf_output.writeDashboards(input_dto.director_output_dto.dashboards))
5467
updated_conf_files.update(conf_output.writeMiscellaneousAppFiles())
5568

5669

5770

71+
5872
#Ensure that the conf file we just generated/update is syntactically valid
5973
for conf_file in updated_conf_files:
6074
ConfWriter.validateConfFile(conf_file)
@@ -67,17 +81,15 @@ def execute(self, input_dto: BuildInputDto) -> DirectorOutputDto:
6781
if input_dto.config.build_api:
6882
shutil.rmtree(input_dto.config.getAPIPath(), ignore_errors=True)
6983
input_dto.config.getAPIPath().mkdir(parents=True)
70-
api_json_output = ApiJsonOutput()
71-
for output_objects, output_type in [(input_dto.director_output_dto.detections, SecurityContentType.detections),
72-
(input_dto.director_output_dto.stories, SecurityContentType.stories),
73-
(input_dto.director_output_dto.baselines, SecurityContentType.baselines),
74-
(input_dto.director_output_dto.investigations, SecurityContentType.investigations),
75-
(input_dto.director_output_dto.lookups, SecurityContentType.lookups),
76-
(input_dto.director_output_dto.macros, SecurityContentType.macros),
77-
(input_dto.director_output_dto.deployments, SecurityContentType.deployments)]:
78-
api_json_output.writeObjects(output_objects, input_dto.config.getAPIPath(), input_dto.config.app.label, output_type )
79-
80-
84+
api_json_output = ApiJsonOutput(input_dto.config.getAPIPath(), input_dto.config.app.label)
85+
api_json_output.writeDetections(input_dto.director_output_dto.detections)
86+
api_json_output.writeStories(input_dto.director_output_dto.stories)
87+
api_json_output.writeBaselines(input_dto.director_output_dto.baselines)
88+
api_json_output.writeInvestigations(input_dto.director_output_dto.investigations)
89+
api_json_output.writeLookups(input_dto.director_output_dto.lookups)
90+
api_json_output.writeMacros(input_dto.director_output_dto.macros)
91+
api_json_output.writeDeployments(input_dto.director_output_dto.deployments)
92+
8193

8294
#create version file for sse api
8395
version_file = input_dto.config.getAPIPath()/"version.json"

‎contentctl/actions/detection_testing/DetectionTestingManager.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
from contentctl.actions.detection_testing.infrastructures.DetectionTestingInfrastructureServer import DetectionTestingInfrastructureServer
66
from urllib.parse import urlparse
77
from copy import deepcopy
8-
from contentctl.objects.enums import DetectionTestingTargetInfrastructure
98
import signal
109
import datetime
1110
# from queue import Queue

0 commit comments

Comments
 (0)