Skip to content

Commit 88c2ebc

Browse files
ZIJZIJ
authored
Docs/drift section (#2341)
* Plan for drift docs * Drift docs - firt pass * Update drift section docs * Fix slack notifications page slug * Title Slack Notifications
1 parent e271ff0 commit 88c2ebc

14 files changed

+396
-166
lines changed

‎agent-tasks/drift-docs-section-plan.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Drift Documentation: New Section Plan
2+
3+
## Goals
4+
5+
- Create a dedicated Drift section (parallel to State Management / Self-host Digger) that clearly explains concepts, setup, operations, and troubleshooting.
6+
- Reduce fragmentation by moving/adapting current content under a coherent information architecture (IA).
7+
- Fill current gaps: org/project settings, scheduling strategies, remediation, and self-host specifics.
8+
9+
## Current Inventory (what exists today)
10+
11+
- Features → Drift Detection: `docs/ce/features/drift-detection.mdx`
12+
- Covers: GH Actions scheduled workflow for drift; Slack and GitHub Issues examples; note about 403/no-backend.
13+
- How To → Scope Drift Detection: `docs/ce/howto/scope-drift-detection.mdx`
14+
- Covers: Using `digger-filename` to limit drift to certain projects.
15+
- Reference → Action inputs: `docs/ce/reference/action-inputs.mdx`
16+
- Mentions: `mode: drift-detection`, `drift-detection-slack-notification-url`, `digger-filename`, `no-backend`.
17+
- Codebase capabilities (underdocumented in docs):
18+
- Org-level drift settings: enable/disable, cron string, Slack webhook URL (`/api/orgs/settings/`).
19+
- Project-level drift enable flag (`/api/projects/:project_id`).
20+
- Internal drift service endpoints and scheduler hooks (for backend-driven scans and Slack rollups).
21+
- Plan snapshots and counts stored per project; status surfaced in UI/API.
22+
23+
## Proposed Section Name and Nav
24+
25+
- Group: Drift.
26+
- Placement: Top-level group in `docs/mint.json` after “Getting Started” and “State Management”.
27+
- Remove “Drift Detection” from Features; move/retitle into this new section.
28+
29+
## Information Architecture (pages and purpose)
30+
31+
1) `ce/drift/overview.mdx`
32+
- What is drift, why it matters, how Digger detects it (scheduled plans on default branch per project).
33+
- Two operating modes: GitHub Actions-only vs backend-orchestrated scans.
34+
- Optional high-level mention of drift states (no dedicated page).
35+
36+
2) `ce/drift/how-it-works.mdx`
37+
- Architecture: drift as a separate service with a cron-driven scheduler; internal endpoints and DB updates.
38+
- Backendless mode: running entirely via GitHub Actions with `mode: drift-detection`.
39+
- When to choose each approach.
40+
41+
3) `ce/drift/quickstart-github-actions.mdx`
42+
- Minimal GH Actions workflow using `mode: drift-detection` and a cron.
43+
- Includes a note that it is "backendless mode"
44+
- Tips: install Terraform (`setup-terraform: true`), store webhook secret, use `no-backend: true` when applicable.
45+
46+
4) `ce/drift/scoping-projects.mdx`
47+
- Consolidate/replace How To → Scope Drift Detection.
48+
- Techniques: dedicated `digger.yml` via `digger-filename`, project `drift_detection: false` in digger.yml, branch filters, dependency considerations.
49+
50+
5) `ce/drift/notifications.mdx`
51+
- Slack rollup notifications: format, frequency, org-level webhook, test endpoint, common errors.
52+
- Mentions GitHub Issues exist for drift, with a link to the dedicated page below.
53+
54+
6) `ce/drift/github-issues.mdx`
55+
- Dedicated setup and permissions for creating GitHub Issues during drift.
56+
- What’s in the issue, labels, ownership, and housekeeping guidance.
57+
58+
7) `ce/drift/remediation.mdx`
59+
- How to remediate drift using GitHub Issues integration.
60+
- Describe remediation by commenting `digger apply` in the GitHub issue; include a clear note that this flow is only relevant for the GitHub Issues integration.
61+
62+
8) `ce/drift/scheduling.mdx`
63+
- GH Actions cron examples (simple mode).
64+
- Backend scheduling (self-host/Cloud): org-level `drift_cron_tab` and how scans are batched; internal endpoints overview.
65+
66+
9) `ce/drift/self-host.mdx`
67+
- Running the drift service (`Dockerfile_drift`), required env vars: `DIGGER_HOSTNAME`, `DIGGER_WEBHOOK_SECRET`, `DIGGER_APP_URL`, `DIGGER_DRIFT_REPORTER_HOSTNAME`.
68+
- Database/scheduler notes (pg_cron/pg_net SQL jobs in `drift/scripts/cron/*.sql`).
69+
- Securing internal endpoints, topology, scaling.
70+
71+
10) `ce/drift/troubleshooting.mdx`
72+
- 403/permissions during drift reporting → often missing `no-backend: true` in Actions.
73+
- Slack webhook failures and how to test.
74+
- Issues creation failures (token scopes), large plan outputs, runner timeouts.
75+
76+
## Content Outlines (key sections per page)
77+
78+
- Overview
79+
- Concept, detection flow description, supported IaC; optional brief mention of states.
80+
81+
- How It Works
82+
- Separate service architecture and cron; internal endpoints; backendless mode explained.
83+
84+
- Quickstart (GH Actions)
85+
- YAML snippet; secrets; validation; verification steps; example outputs.
86+
87+
- Scoping Projects
88+
- `digger-filename` technique; per-project `drift_detection` flag; branch selection considerations.
89+
90+
- Notifications
91+
- Slack rollup design (aggregated per org), color legend, test endpoint; direct link to GitHub Issues page.
92+
93+
- GitHub Issues
94+
- Enablement, required env vars/permissions, example outputs, labels/assignment.
95+
96+
- Remediation
97+
- Comment `digger apply` in the GitHub issue to remediate drift (only relevant for Issues integration).
98+
99+
- Scheduling
100+
- GH Actions cron; backend scheduler overview (SQL jobs in `drift/scripts/cron/`), `drift_cron_tab` semantics; examples.
101+
102+
- Self-host
103+
- Deploying the drift service; required components; env vars; security; health checks; scaling.
104+
105+
- Troubleshooting
106+
- Symptoms, likely causes, resolutions; logs to check; sample curl commands for internal endpoints (self-host operators).
107+
108+
## Migration and Link Strategy
109+
110+
- Move: `ce/features/drift-detection` → `ce/drift/quickstart-github-actions` (rename and retitle).
111+
- Move: `ce/howto/scope-drift-detection` → `ce/drift/scoping-projects` (update links site-wide).
112+
- Update: cross-links in `action-inputs.mdx` to reference new “Quickstart”, “Notifications”, and “Scheduling” pages.
113+
114+
## Navigation Changes (mint.json)
115+
116+
- Add a top-level group:
117+
- Drift → pages:
118+
- `ce/drift/overview`
119+
- `ce/drift/how-it-works`
120+
- `ce/drift/quickstart-github-actions`
121+
- `ce/drift/scoping-projects`
122+
- `ce/drift/notifications`
123+
- `ce/drift/github-issues`
124+
- `ce/drift/remediation`
125+
- `ce/drift/scheduling`
126+
- `ce/drift/self-host`
127+
- `ce/drift/troubleshooting`
128+
- Remove `ce/features/drift-detection` from “Features”.
129+
- Remove `ce/howto/scope-drift-detection` from “How To”.
130+
131+
## Acceptance Criteria
132+
133+
- New Drift group appears as a dedicated section with the listed pages.
134+
- Quickstart reproducible end-to-end (Slack and Issues options tested).
135+
- Remediation page accurately reflects the supported comment flow for Issues integration.
136+
- Self-host and How It Works pages list required env vars and components and reference the SQL scheduler snippets.
137+
138+
## Implementation Plan (steps)
139+
140+
1. Create new directory and placeholder pages under `docs/ce/drift/` with titles and short summaries.
141+
2. Move/adapt content from `features/drift-detection` and `howto/scope-drift-detection` into the new pages.
142+
3. Update `docs/mint.json` to add the Drift group; remove moved pages from Features/How To.
143+
4. Update cross-links in the docs (search for references to old paths and fix).
144+
5. QA pass: local build of docs, link checker, skim for tone and consistency.
145+
146+
## References (code pointers for authoring)
147+
148+
- Action inputs mentioning drift: `docs/ce/reference/action-inputs.mdx`.
149+
- Current feature page: `docs/ce/features/drift-detection.mdx`.
150+
- How-to: `docs/ce/howto/scope-drift-detection.mdx`.
151+
- Backend endpoints and models: `backend/controllers/orgs.go`, `backend/controllers/projects.go`, `backend/bootstrap/main.go`, `backend/models/orgs.go`.
152+
- Issue comment handling: `backend/controllers/github_comment.go`.
153+
- Drift service and scheduler: `drift/controllers/*.go`, `drift/scripts/cron/*.sql`.

‎docs/ce/drift/github-issues.mdx‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
title: "GitHub Issues"
3+
---
4+
5+
You can create a GitHub Issue whenever drift is detected, to route remediation through your normal triage process. Digger supports sending a summary of Drift notifications via Slack. Another way to get notified is via [Slack](/ce/drift/slack-notifications)
6+
7+
8+
## Enable in GitHub Actions
9+
10+
Add the following to your drift workflow:
11+
12+
```
13+
env:
14+
GITHUB_CONTEXT: ${{ toJson(github) }}
15+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
16+
INPUT_DRIFT_GITHUB_ISSUES: 'true'
17+
# Optional: DIGGER_GITHUB_TOKEN or PAT with appropriate scopes
18+
```
19+
20+
## Notes
21+
22+
- Ensure the token has permission to create issues in the repository.
23+
- The issue records the project(s) with drift and links back to context for quick follow-up.
24+
25+
## Next step
26+
27+
- See [Remediation](/ce/drift/remediation) for closing the loop by applying from the issue.
28+

‎docs/ce/drift/remediation.mdx‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: "Remediation"
3+
---
4+
5+
If you use the GitHub Issues integration for drift, you can remediate directly from the issue by commenting a Digger command.
6+
7+
## Apply changes from the issue
8+
9+
- Comment `digger apply` in the GitHub issue created for the drift event.
10+
- Digger runs the apply for the impacted project(s) and reports status back.
11+
12+
## Notes
13+
14+
- This flow is specific to the GitHub Issues integration. If you are not using Issues, remediate by opening a PR with fixes and using your normal PR flow.
15+
16+
## Related
17+
18+
- GitHub Issues: see [/ce/drift/github-issues](/ce/drift/github-issues)

‎docs/ce/drift/scoping-projects.mdx‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
title: "Scope Drift to Specific Projects"
3+
---
4+
5+
Use separate Digger config files and workflows to scope drift detection to only the projects you want. The drift config file is a regular `digger.yml` listing a subset of projects (for example, only `dev` or only `prod`).
6+
7+
## Approach
8+
9+
- Create a dedicated `digger.yml` that lists only the projects or blocks you want scanned.
10+
- Point your drift workflow to that file using the `digger-filename` input.
11+
- Repeat per environment if needed.
12+
13+
## Examples
14+
15+
### Explicit projects
16+
17+
```yml
18+
# digger-drift-dev.yml
19+
projects:
20+
- name: app-dev-a
21+
dir: terraform/aws_devel/app-a
22+
workflow: default
23+
- name: app-dev-b
24+
dir: terraform/aws_devel/app-b
25+
workflow: default
26+
```
27+
28+
### Terragrunt-generated blocks
29+
30+
```yml
31+
# digger-drift-dev.yml
32+
generate_projects:
33+
blocks:
34+
- block_name: aws_devel
35+
terragrunt: true
36+
root_dir: terraform/aws_devel/
37+
workflow: default
38+
```
39+
40+
## Referencing the file in your workflow
41+
42+
```yml
43+
name: Drift (dev)
44+
45+
on:
46+
workflow_dispatch:
47+
48+
jobs:
49+
detect-drift:
50+
runs-on: ubuntu-latest
51+
steps:
52+
- uses: diggerhq/digger@vLatest
53+
with:
54+
mode: drift-detection
55+
digger-filename: digger-drift-dev.yml
56+
```
57+
58+
## Notes
59+
60+
- There is no per-project drift filter in the action; scoping via a dedicated config file is the recommended approach.
61+
- You can also mark projects with `drift_detection: false` in your main config to disable drift checks for them.
62+
63+
## Related
64+
65+
- Quickstart: see [/ce/drift/quickstart-github-actions](/ce/drift/quickstart-github-actions)
66+
- Scheduling: see [/ce/drift/scheduling](/ce/drift/scheduling)

‎docs/ce/drift/self-host.mdx‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
title: "Self-hosting"
3+
---
4+
5+
Drift checks are triggered by a separate service, with its own Dockerfile
6+
7+
## Requirements
8+
9+
- Build/run the drift service (see `Dockerfile_drift`).
10+
- Backend database accessible to the service.
11+
- Webhook secret configured and used to protect internal endpoints.
12+
13+
## Key environment variables
14+
15+
- `DIGGER_HOSTNAME`: Base URL of your backend, used to call internal endpoints.
16+
- `DIGGER_WEBHOOK_SECRET`: Shared secret to authenticate internal requests.
17+
- `DIGGER_APP_URL`: Base URL for links in notifications.
18+
- `DIGGER_DRIFT_REPORTER_HOSTNAME`: Hostname for the reporter in CI job specs.
19+
20+
## Scheduling and notifications
21+
22+
- Set the org-level `drift_cron_tab` for when to scan.
23+
- Slack rollups use an org-level webhook URL when configured.
24+
- SQL helper snippets for periodic invocation live in `drift/scripts/cron/`.
25+
26+
## Security
27+
28+
- Expose internal endpoints only behind your network boundary and verify the webhook secret on requests.
29+

‎docs/ce/drift/set-up-in-ui.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
title: "Set up in UI"
3+
---
4+
5+
Configuring Drift Detection in [Digger UI](https://ui.digger.dev).
6+
7+
# Schedule
8+
9+
You can run drift detection on pre-defined intervals (e.g. hourly or daily) - or use a custom crontab:
10+
11+
![Drift UI Schedule](/images/drift-ui-schedule.png)
12+
13+
# Slack notifications
14+
15+
You can also configure Slack notifications via webhooks:
16+
17+
![Drift UI Slack](/images/drift-ui-slack.png)
18+
19+

‎docs/ce/drift/slack-notifications.mdx‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
title: "Slack Notifications"
3+
---
4+
5+
Digger supports sending a summary of Drift notifications via Slack. Another way to get notified is [GitHub Issues](/ce/drift/github-issues)
6+
7+
## Configuration in UI
8+
9+
You can cofigure and test Slack webhooks in the Drift section of the dashboard:
10+
11+
![Drift UI Slack](/images/drift-ui-slack.png)
12+
13+
## Backendless mode
14+
15+
Drift detection can also run in Backendless mode. In this case, it is a regular GitHub Action envoked on schedule - you just need to pass `drift-detection-slack-notification-url` action input:
16+
17+
```
18+
name: Drift Detection
19+
20+
on:
21+
workflow_dispatch:
22+
#schedule:
23+
# - cron: '0 0 * * *' # 12am daily
24+
25+
jobs:
26+
detect-drift:
27+
runs-on: ubuntu-latest
28+
steps:
29+
- name: digger drift detection
30+
uses: diggerhq/digger@743844a930fd404882869ee036213cd1b24d20dd
31+
with:
32+
mode: drift-detection
33+
drift-detection-slack-notification-url: SLACK_WEBHOOK_URL_HERE
34+
setup-terraform: true
35+
setup-aws: true
36+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
37+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
38+
no-backend: true
39+
env:
40+
GITHUB_CONTEXT: ${{ toJson(github) }}
41+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
42+
DIGGER_GITHUB_TOKEN: ${{ secrets.DIGGER_GITHUB_TOKEN }}
43+
INPUT_DRIFT_GITHUB_ISSUES: 'true'
44+
```

‎docs/ce/drift/troubleshooting.mdx‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
title: "Troubleshooting"
3+
---
4+
5+
## Common issues
6+
7+
- 403 or permission errors when reporting results
8+
- Ensure `no-backend: true` is set in backendless workflows.
9+
- Confirm tokens used by the workflow have the required scopes.
10+
11+
- Slack messages not received
12+
- Verify the webhook URL and that it’s configured in the correct place (action input or org settings).
13+
- Check the configured schedule (Actions cron or org-level crontab).
14+
15+
- Issue creation failures
16+
- Confirm the token has permission to create issues in the repository.
17+
18+
## Operator tips (self-hosting)
19+
20+
- Check service logs around the scheduled run time.
21+
- Use the SQL helpers in `drift/scripts/cron/` to invoke internal endpoints on a predictable cadence.

0 commit comments

Comments
 (0)