Skip to content

Commit 9c0f638

Browse files
Merge pull request #360 from MateSaary/main
OSD-28248: Investigation bootstrap make target
2 parents 241ed3b + f714a82 commit 9c0f638

File tree

3 files changed

+133
-9
lines changed

3 files changed

+133
-9
lines changed

Makefile

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,11 @@ test-interceptor-e2e: check-go121-install check-jq-install check-vault-install b
8383
@echo "Running e2e tests for interceptor..."
8484
cd interceptor && ./test/e2e.sh
8585

86+
##@ Boilerplate:
87+
.PHONY: boilerplate
88+
bootstrap-investigation: ## Bootstrap a new boilerplate investigation
89+
@cd hack && ./bootstrap-investigation.sh
90+
8691
##@ Template-updater:
8792
.PHONY: template-updater
8893
template-updater: build-template-updater lint-template-updater ## Run all targets for template-updater

README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -43,30 +43,31 @@ CAD consists of:
4343

4444
1) [PagerDuty Webhooks](https://support.pagerduty.com/docs/webhooks) are used to trigger Configuration-Anomaly-Detection when a [PagerDuty incident](https://support.pagerduty.com/docs/incidents) is created
4545
2) The webhook routes to a [Tekton EventListener](https://tekton.dev/docs/triggers/eventlisteners/)
46-
3) Received webhooks are filtered by a [Tekton Interceptor](https://tekton.dev/docs/triggers/interceptors/) that uses the payload to evaluate whether the alert has an implemented handler function in `cadctl` or not. If there is no handler implemented, the alert is directly forwarded to a human SRE.
46+
3) Received webhooks are filtered by a [Tekton Interceptor](https://tekton.dev/docs/triggers/interceptors/) that uses the payload to evaluate whether the alert has an implemented handler function in `cadctl` or not. If there is no handler implemented, the alert is directly forwarded to a human SRE.
4747
4) If `cadctl` implements a handler for the received payload/alert, a [Tekton PipelineRun](https://tekton.dev/docs/pipelines/pipelineruns/) is started.
48-
5) The pipeline runs `cadctl` which determines the handler function by itself based on the payload.
48+
5) The pipeline runs `cadctl` which determines the handler function by itself based on the payload.
4949

5050
![CAD Overview](./images/cad_overview/cad_architecture_dark.png#gh-dark-mode-only)
5151
![CAD Overview](./images/cad_overview/cad_architecture_light.png#gh-light-mode-only)
5252

53-
## Contributing
53+
## Contributing
5454

5555
### Building
5656

57-
For build targets, see `make help`.
57+
For build targets, see `make help`.
5858

5959
### Adding a new investigation
6060

6161
CAD investigations are triggered by PagerDuty webhooks. Currently, CAD supports the following two formats of webhooks:
62-
- WebhookV3
62+
- WebhookV3
6363
- EventOrchestrationWebhook
6464

65-
The required investigation is identified by CAD based on the incident and its payload.
65+
The required investigation is identified by CAD based on the incident and its payload.
6666
As PagerDuty itself does not provide finer granularity for webhooks than service-based, CAD filters out the alerts it should investigate. For more information, please refer to https://support.pagerduty.com/docs/webhooks.
6767

6868
To add a new alert investigation:
69-
- Create a mapping for the alert in `registry.go` and write a corresponding CAD investigation (e.g. `Investigate()` in `chgm.go`).
69+
70+
- run `make bootstrap-investigation` to generate boilerplate code in `pkg/investigations` (This creates the corresponding folder & .go file, and also appends the investigation to the `availableInvestigations` interface in `registry.go`.).
7071
- investigation.Resources contain initialized clients for the clusters aws environment, ocm and more. See [Integrations](#integrations)
7172

7273
### Integrations
@@ -86,14 +87,13 @@ They are initialized for you and passed to the investigation via investigation.R
8687
* [k8sclient](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client) -- Interact with clusters kube-api
8788
- Requires RBAC definitions for your investigation to be added to `metadata.yaml`
8889

89-
9090
## Testing locally
9191

9292
### Pre-requirements
9393
- an existing cluster
9494
- an existing PagerDuty incident for the cluster and alert type that is being tested
9595

96-
To quickly create an incident for a cluster_id, you can run `./test/generate_incident.sh <alertname> <clusterid>`.
96+
To quickly create an incident for a cluster_id, you can run `./test/generate_incident.sh <alertname> <clusterid>`.
9797
Example usage:`./test/generate_incident.sh ClusterHasGoneMissing 2b94brrrrrrrrrrrrrrrrrrhkaj`.
9898

9999
### Running cadctl for an incident ID

hack/bootstrap-investigation.sh

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
read -p "Enter the new investigation (package) name: " INVESTIGATION_NAME
6+
if [[ "${INVESTIGATION_NAME}" == "" ]] ; then
7+
echo "Investigation name cannot be empty."
8+
exit 1
9+
elif [[ "${INVESTIGATION_NAME}" =~ [^a-zA-Z0-9_] ]] ; then
10+
echo "Investigation name must be alphanumeric."
11+
exit 1
12+
fi
13+
14+
read -p "Enter new investigation description: " INVESTIGATION_DESCRIPTION
15+
if [[ "${INVESTIGATION_DESCRIPTION}" == "" ]] ; then
16+
INVESTIGATION_DESCRIPTION="TODO"
17+
fi
18+
19+
read -p "Should Investigate Alert (y/n): " INVESTIGATE_ALERT_BOOL
20+
if [[ "${INVESTIGATE_ALERT_BOOL}" == "y" ]] ; then
21+
read -p "Investigation alert string: " INVESTIGATION_ALERT_STRING
22+
INVESTIGATION_ALERT="strings.Contains(alert, \"${INVESTIGATION_ALERT_STRING}\")"
23+
elif [[ "${INVESTIGATE_ALERT_BOOL}" == "n" ]] ; then
24+
INVESTIGATION_ALERT="false"
25+
else
26+
echo "Invalid input. Please enter 'y' or 'n'."
27+
exit 1
28+
fi
29+
30+
INVESTIGATION_NAME=$(echo "${INVESTIGATION_NAME}" | tr '[:upper:]' '[:lower:]')
31+
32+
INVESTIGATION_DIR="../pkg/investigations/${INVESTIGATION_NAME}"
33+
34+
if [ -d "${INVESTIGATION_DIR}" ]; then
35+
echo "Investigation of name ${INVESTIGATION_NAME} already exists."
36+
exit 1
37+
fi
38+
39+
mkdir -p "${INVESTIGATION_DIR}"
40+
ls "${INVESTIGATION_DIR}"
41+
42+
touch "${INVESTIGATION_DIR}/${INVESTIGATION_NAME}.go"
43+
touch "${INVESTIGATION_DIR}/metadata.yaml"
44+
touch "${INVESTIGATION_DIR}/README.md"
45+
46+
# Create README.md file
47+
cat <<EOF > "${INVESTIGATION_DIR}/README.md"
48+
# ${INVESTIGATION_NAME} Investigation
49+
50+
${INVESTIGATION_DESCRIPTION}
51+
52+
EOF
53+
54+
# Create metadata.yaml file
55+
cat <<EOF > "${INVESTIGATION_DIR}/metadata.yaml"
56+
name: ${INVESTIGATION_NAME}
57+
rbac:
58+
roles: []
59+
clusterRoleRules: []
60+
customerDataAccess: false
61+
62+
EOF
63+
64+
# Create boilerplate investigation file
65+
cat <<EOF > "${INVESTIGATION_DIR}/${INVESTIGATION_NAME}.go"
66+
// Package ${INVESTIGATION_NAME} contains...TODO
67+
package ${INVESTIGATION_NAME}
68+
69+
import (
70+
"strings"
71+
72+
"github.com/openshift/configuration-anomaly-detection/pkg/investigations/investigation"
73+
"github.com/openshift/configuration-anomaly-detection/pkg/logging"
74+
"github.com/openshift/configuration-anomaly-detection/pkg/notewriter"
75+
)
76+
77+
type Investigation struct{}
78+
79+
func (c *Investigation) Run(r *investigation.Resources) (investigation.InvestigationResult, error) {
80+
result := investigation.InvestigationResult{}
81+
82+
// Initialize PagerDuty note writer
83+
notes := notewriter.New(r.Name, logging.RawLogger)
84+
85+
// TODO: Implement investigation logic here
86+
87+
return result, r.PdClient.EscalateIncidentWithNote(notes.String())
88+
}
89+
90+
func (c *Investigation) Name() string {
91+
return "${INVESTIGATION_NAME}"
92+
}
93+
94+
func (c *Investigation) Description() string {
95+
return "${INVESTIGATION_DESCRIPTION}"
96+
}
97+
98+
func (c *Investigation) ShouldInvestigateAlert(alert string) bool {
99+
return ${INVESTIGATION_ALERT}
100+
}
101+
102+
func (c *Investigation) IsExperimental() bool {
103+
// TODO: Update to false when graduating to production.
104+
return true
105+
}
106+
107+
EOF
108+
109+
echo "${INVESTIGATION_NAME} created in ${INVESTIGATION_DIR}"
110+
echo "metadata.yaml file created in ${INVESTIGATION_DIR}"
111+
112+
# Update registry.go to contain new investigation
113+
if ! grep -q "${INVESTIGATION_NAME}" ../pkg/investigations/registry.go && ! grep -q "${INVESTIGATION_NAME}" ../pkg/investigations/registry.go; then
114+
sed -i "/import (/a \\\t\"github.com/openshift/configuration-anomaly-detection/pkg/investigations/${INVESTIGATION_NAME}\"" ../pkg/investigations/registry.go
115+
sed -i "/var availableInvestigations = \[/a \\\t&${INVESTIGATION_NAME}.Investigation{}," ../pkg/investigations/registry.go
116+
echo "${INVESTIGATION_NAME} added to registry.go"
117+
else
118+
echo "${INVESTIGATION_NAME} already exists in registry.go"
119+
fi

0 commit comments

Comments
 (0)