Skip to content

Commit b3a970d

Browse files
concentratorconcentrator
authored
Merge pull request #93 from wallarm/doc/adopt-hits-refs
Adopt hits references to doc framework
2 parents 7aac50d + 7e61da9 commit b3a970d

2 files changed

Lines changed: 352 additions & 23 deletions

File tree

‎references/hits-to-rules.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# Hits-to-Rules (false-positive mitigation)
2+
3+
Reference for the provider feature that turns Wallarm **hit** data into
4+
false-positive **suppression rules** managed as Terraform state. Describes how
5+
the code behaves; for the operator procedure see the registry guide
6+
`docs/guides/hits_to_rules.md`.
7+
8+
## 1. Overview
9+
10+
Wallarm detects threats and records them as **hits**. Some hits are false
11+
positives - legitimate traffic flagged as an attack. The fix is a suppression
12+
rule that tells Wallarm to stop flagging that signature or attack type at that
13+
request point and scope.
14+
15+
Hits are **ephemeral** (they age out of the API), so the suppression config
16+
cannot be derived from live hits on every plan - it must be captured into
17+
Terraform state once and then persist independently. This feature does that: a
18+
data source reads hits and transforms them into a scope (`action`) plus grouped
19+
signatures; a gating resource ensures each request is fetched only once; and two
20+
rule resources create the actual suppressions. The pieces are wired together in
21+
the reference module `examples/hits-to-rules/`.
22+
23+
Scope of this doc: `data.wallarm_hits`, `wallarm_hits_index`,
24+
`wallarm_rule_disable_stamp`, `wallarm_rule_disable_attack_type`, and how they
25+
compose. The HCL generator's role in the flow is summarized here; its full
26+
behavior is in `hcl-generator.md`.
27+
28+
## 2. Model
29+
30+
A hit carries a detection **point** (where in the request the signature
31+
matched), one or more **stamps** (numeric signature IDs), an **attack type**
32+
(`sqli`, `xss`, ...), and request metadata (`domain`, `path`, `poolid`,
33+
`attack_id`, `request_id`). Hits of the same HTTP request share `request_id`;
34+
hits of the same campaign share `attack_id`.
35+
36+
Suppression is expressed against an **action** (the match scope: host + URL path
37+
+ optionally the application instance) and a **point**. Two rule shapes exist:
38+
39+
- **disable_stamp** - allow one specific signature (`stamp`) at a point.
40+
- **disable_attack_type** - allow a whole attack type at a point (the only
41+
option for stampless types, see § 6).
42+
43+
```mermaid
44+
flowchart LR
45+
H[Hits in API<br/>ephemeral] -->|request_id| DS[data.wallarm_hits]
46+
IDX[wallarm_hits_index<br/>ready + cached_request_ids] -->|gates new ids only| DS
47+
DS -->|aggregated JSON| CACHE[terraform_data.cache<br/>persists in state]
48+
CACHE -->|dedup by action_hash in HCL locals| EXP{{expand per stamp / type}}
49+
EXP --> RS[wallarm_rule_disable_stamp]
50+
EXP --> RA[wallarm_rule_disable_attack_type]
51+
DS -.->|source=rules| GEN[wallarm_rule_generator<br/>standalone .tf + moved]
52+
```
53+
54+
The action scope produced by the data source uses the **exact same schema** as
55+
every `wallarm_rule_*` resource (`resourcerule.ScopeActionSchema()`), so
56+
`data.wallarm_hits.<x>.action` can be passed straight into a rule's `action`
57+
argument.
58+
59+
## 3. Elements
60+
61+
| Element | Kind | Responsibility |
62+
|---|---|---|
63+
| `data.wallarm_hits` | data source | Fetch hits for one `request_id`, validate their action is consistent, optionally expand by attack campaign, compute the action scope + hash, group signatures per point, emit the compact `aggregated` payload. |
64+
| `wallarm_hits_index` | resource | Persistent per-client index of already-fetched request IDs. Exposes `ready` and `cached_request_ids` so HCL can gate the data source to new IDs only. Holds no API state (`Delete` is state-only). |
65+
| `wallarm_rule_disable_stamp` | resource | Create/read/update/delete a `disable_stamp` rule (allow one `stamp` at a `point`/`action`). |
66+
| `wallarm_rule_disable_attack_type` | resource | Same lifecycle for a `disable_attack_type` rule (allow one `attack_type`). |
67+
| `terraform_data.cache` | Terraform built-in (module) | Persists each request's `aggregated` output in state with `ignore_changes`, so rules survive after hits expire. Not a provider resource. |
68+
| `wallarm_rule_generator` (`source = "rules"`) | resource | Optional: writes standalone `.tf` files plus `moved` blocks from the cached rules, for migration off `for_each`. Detailed in `hcl-generator.md`. |
69+
70+
## 4. Behavior
71+
72+
### 4.1 Data source read pipeline
73+
74+
`dataSourceWallarmHitsRead` runs in phases:
75+
76+
1. **Fetch direct hits** (`fetchDirectHits`) - `HitRead` filtered by
77+
`client_id` + `request_id`, time range, and noise filters (see § 6.3).
78+
Empty result -> `setEmptyHitsState` (empty `action`, `aggregated` with empty
79+
arrays, `hits_count = 0`) and return; this is what makes a re-fetch of
80+
expired hits destroy rules, and is why fetching is gated (§ 4.2).
81+
2. **Action-consistency check** - all direct hits must share `domain`, `path`,
82+
and `poolid`; otherwise the read fails with an `inconsistent hit data` error.
83+
The first hit is the reference (`refDomain`/`refPath`/`refPoolID`).
84+
3. **Attack expansion** (mode `attack` only) -
85+
`fetchRelatedHitsByAttackIDs` collects unique `attack_id`s, pages `HitRead`
86+
(batch `HitFetchBatchSize`) filtered to `attack_types`, keeps only hits whose
87+
action matches the reference, then `mergeHits` dedupes by hit ID. When
88+
`refPath` is `[multiple]`, matching is on `domain` + `poolid` only.
89+
4. **Build action + hashes** - `buildActionFromHit` turns
90+
domain/path/poolid into action conditions; `ConditionsHash` -> `action_hash`,
91+
`ActionDirName` -> `action_dir_name`.
92+
5. **API validation** - if the hit ID has >=2 elements, `ActionReadByHitID`
93+
fetches the API's own conditions and compares hashes. A fetch error is a
94+
`[WARN]` (read proceeds); a **hash mismatch is a hard error** with a
95+
full condition-by-condition diff.
96+
6. **Group + aggregate** - `groupHitsForRules` groups by `point_hash` +
97+
attack type, unions stamps, drops hits whose type is not in `attack_types`.
98+
`buildAggregatedJSON` filters by `rule_types`, truncates hashes to 16 chars,
99+
and marshals `{action_hash, action, groups[]}`.
100+
101+
### 4.2 Gating and persistence
102+
103+
`wallarm_hits_index.ready` is `false` on create and `true` afterwards, made
104+
known at plan time by `hitsIndexCustomizeDiff` (`SetNew`). `cached_request_ids`
105+
is empty on create and, on update, the diff **preserves the old state value** so
106+
newly added IDs read as uncached during plan - that is the signal the module
107+
uses to fetch only new IDs (`ready ? new_ids : all_ids`). `Create`/`Read`/
108+
`Update` then sync `cached_request_ids` to the configured `request_ids`.
109+
110+
On first apply with request IDs, `ready = false` fetches everything, caches it,
111+
and creates rules in a single apply. Subsequent applies fetch only new IDs.
112+
Deduplication by `action_hash` happens in the module's HCL locals (stamps
113+
unioned via `distinct(flatten(...))`), so identical rules from different request
114+
IDs collapse to one resource - this prevents drift loops where the API would
115+
merge duplicate rules.
116+
117+
### 4.3 Rule create semantics
118+
119+
Both rule resources build a `wallarm.ActionCreate` and call `HintCreate`:
120+
121+
- `Type` = `"disable_stamp"` / `"disable_attack_type"`.
122+
- `VariativityDisabled` = **`true`** (hardcoded) - these rules are exact, not
123+
variative.
124+
- `Validated` = `false`.
125+
- `action` (scope) is expanded from the `action {}` blocks
126+
(`ExpandSetToActionDetailsList`); `point` from the `point` attribute
127+
(`ExpandPointsToTwoDimensionalArray`).
128+
- Resource ID is `clientID/actionID/ruleID`; `rule_id`, `action_id`,
129+
`rule_type` are set from the response.
130+
131+
Update routes through `resourcerule.Update(apiClient, WithStamp |
132+
WithAttackType)`, Delete through `resourcerule.Delete`, Import through
133+
`resourcerule.Import("<type>")`, and both use
134+
`resourcerule.ActionScopeCustomizeDiff`.
135+
136+
### 4.4 Action-condition construction
137+
138+
`buildActionFromHit` + `locationToConditions` port the Ruby
139+
`LocationToConditions`:
140+
141+
- **instance** condition emitted when `include_instance` is true and
142+
`poolid != 0` (`{instance: <poolid>}`, `equal`, empty value).
143+
- **HOST** header always `iequal` to `domain`.
144+
- **path** split on `/` into `equal` segment conditions, terminated by an
145+
`absent` condition one index past the last segment (fixes chain length).
146+
- final path segment splits into `action_name` + `action_ext` on the **first**
147+
dot, matching the API (`archive.tar.gz` -> name `archive`, ext `tar.gz`); no
148+
dot -> `action_name` = segment and `action_ext` `absent`. **Known bug (R-002):**
149+
the code currently splits on the *last* dot (`actionNameExtConditions` here and
150+
`parseLastSegment` on the `action_path` side); the fix is `strings.Index` at
151+
both sites. See `action.md §4.3`.
152+
- root path `/` -> `action_name` empty + `path[0]` absent.
153+
- `path == "[multiple]"` -> host-only wildcard scope (no path/action_name/
154+
action_ext conditions).
155+
156+
### 4.5 Failure and edge modes
157+
158+
| Situation | Handling |
159+
|---|---|
160+
| Direct hits disagree on domain/path/poolid | Hard error (`inconsistent hit data`). |
161+
| Provider action hash != API hash | Hard error with per-condition diff. |
162+
| `ActionReadByHitID` call fails | `[WARN]`, read proceeds unvalidated. |
163+
| No hits (or all expired) | Empty state; downstream rules destroyed if not cached. |
164+
| Stampless attack type (`xxe`, `invalid_xml`) | No stamps; only `disable_attack_type` rules. With `rule_types=["disable_stamp"]` they yield nothing. |
165+
| `nil` stamps slice | Coerced to `[]` before marshal (JSON `null` breaks HCL). |
166+
167+
## 5. Parameters
168+
169+
### 5.1 `data.wallarm_hits`
170+
171+
| input | type | req? | default | notes |
172+
|---|---|---|---|---|
173+
| `client_id` | int | optional | provider default | tenant scope; resolved via `retrieveClientID`. |
174+
| `request_id` | string | **required** | - | the request whose hits to fetch. |
175+
| `mode` | string | optional | `request` | `request` \| `attack` (validated). |
176+
| `attack_types` | list(string) | optional | 16 default types (§ 6.1) | filter; in `attack` mode also limits what is fetched. |
177+
| `rule_types` | list(string) | optional | both | `disable_stamp` \| `disable_attack_type` (validated). |
178+
| `include_instance` | bool | optional | `true` | include `instance`/poolid in action scope. |
179+
| `time` | list(int), max 2 | optional | [6 months ago, now] | `[from, to]` unix timestamps. |
180+
| `action` | set(block) | optional/computed | computed from hits | rule-compatible action scope. |
181+
182+
Computed outputs: `action_hash` (16-char-truncated in keys/aggregated, full
183+
SHA256 in the `action_hash` attribute), `action_dir_name`, `action_conditions`
184+
(type/point/value list), `aggregated` (JSON, see § 6.4), `hits_count`, and
185+
`hits` (per-hit detail: `id`, `type`, `ip`, `statuscode`, `time`, `value`,
186+
`stamps`, `stamps_hash`, `point`, `point_wrapped`, `point_hash`, `poolid`,
187+
`attack_id`, `block_status`, `request_id`, `domain`, `path`, `protocol`,
188+
`known_attack`, `node_uuid`).
189+
190+
### 5.2 `wallarm_hits_index`
191+
192+
| attribute | type | req? | notes |
193+
|---|---|---|---|
194+
| `client_id` | int | optional | tenant scope. |
195+
| `request_ids` | set(string) | **required** | IDs to track. |
196+
| `ready` | bool | computed | `false` on create, `true` after. |
197+
| `cached_request_ids` | set(string) | computed | mirrors `request_ids` after apply; old value preserved during plan. |
198+
199+
ID: `hits_index_<client_id>`. `Delete` clears the ID only (no API call).
200+
201+
### 5.3 `wallarm_rule_disable_stamp` / `wallarm_rule_disable_attack_type`
202+
203+
| input | type | req? | notes |
204+
|---|---|---|---|
205+
| `stamp` (stamp rule) | int | **required** | `>= 1`. |
206+
| `attack_type` (attack-type rule) | string | **required** | one of § 6.2 (includes `any`). |
207+
| `action` | set(block) | optional+computed | rule scope (`ScopeActionSchema`). |
208+
| `point` | list(list(string)) | **required**, ForceNew | detection point (`defaultPointSchema`). |
209+
| `comment`, `active`, `set`, `title` | common | optional | `commonResourceRuleFields`; `comment` defaults to `"Managed by Terraform"`. |
210+
| `variativity_disabled` | bool | - | forced `true` at create. |
211+
212+
Computed: `rule_id`, `action_id`, `rule_type`. Import ID and lifecycle helpers
213+
per § 4.3.
214+
215+
## 6. Reference data
216+
217+
### 6.1 Data-source default attack-type filter (16)
218+
219+
`xss`, `sqli`, `rce`, `ptrav`, `crlf`, `redir`, `nosqli`, `ldapi`, `scanner`,
220+
`mass_assignment`, `ssrf`, `ssi`, `mail_injection`, `ssti`, `xxe`, `invalid_xml`
221+
(`defaultAllowedAttackTypes`).
222+
223+
### 6.2 `disable_attack_type.attack_type` allowed values (17)
224+
225+
The 16 above plus `any` (the resource's `StringInSlice`; note `any` is not in
226+
the data-source default filter).
227+
228+
### 6.3 Hit fetch filters
229+
230+
| filter | direct | attack-related |
231+
|---|---|---|
232+
| `NotType` | `warn`, `infoleak` | - |
233+
| `Type` (allowlist) | - | `attack_types` |
234+
| `NotState` | `falsepositive` | `falsepositive` |
235+
| `NotExperimental` / `NotAasmEvent` | yes | yes |
236+
| `NotWallarmScanner` | - | yes |
237+
| batch / order | `HitFetchBatchSize`, `time` desc | same, paged by offset |
238+
239+
`HitFetchBatchSize = 500` (`constants.go`).
240+
241+
### 6.4 `aggregated` JSON shape
242+
243+
```json
244+
{
245+
"action_hash": "<16 hex>",
246+
"action": [ { "type": "...", "value": "...", "point": { "...": "..." } } ],
247+
"groups": [
248+
{ "key": "<point_hash16>_<attack_type>", "point": [["header","HOST"]],
249+
"stamps": [6961], "attack_type": "sqli", "disable_attack_type": true }
250+
]
251+
}
252+
```
253+
254+
`groups` are filtered by `rule_types`: `stamps` populated only when
255+
`disable_stamp` is requested; `disable_attack_type` true only when that type is
256+
requested and an attack type is present. A group with neither is dropped.
257+
258+
### 6.5 Hashes
259+
260+
`action_hash` = Ruby-compatible `resourcerule.ConditionsHash` (SHA256 of sorted
261+
conditions); `point_hash` = `resourcerule.PointHash`. Both are truncated to 16
262+
hex chars where used as `for_each`/group keys.
263+
264+
## 7. References
265+
266+
- `docs/guides/hits_to_rules.md` - operator how-to (the procedure).
267+
- `examples/hits-to-rules/` - the reference module wiring these together.
268+
- `hits.md` - domain notes on hits and the FP workflow.
269+
- `rules-core.md`, `action.md`, `point.md` -
270+
Action/Condition/Hint model, action path-expansion, point chaining.
271+
- `hcl-generator.md` - full `wallarm_rule_generator` behavior.

‎references/hits.md‎

Lines changed: 81 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,93 @@
1-
# Hits, Attacks & False Positive Suppression
1+
# Hits and attacks
22

3-
## Domain model
3+
Reference for the Wallarm hits/attacks domain model and the false-positive
4+
suppression concept. The provider flow that turns hits into rules is
5+
`hits-to-rules.md`; this doc is the domain model those pieces operate on.
46

5-
A **hit** represents a single detected threat within an HTTP request. Hits sharing the same HTTP request are linked by `request_id`. Hits from the same attack campaign share an `attack_id`.
7+
## 1. Overview
68

7-
**IMPORTANT: Hits are ephemeral** — they have a retention period and can be dropped from the API at any time.
9+
A **hit** is a single detected threat within an HTTP request. Hits are
10+
**ephemeral** - they have a retention period and can be dropped from the API at
11+
any time. Some hits are false positives (legitimate traffic flagged as an
12+
attack); the fix is a suppression rule scoped to a request point. This doc
13+
defines the entities; `hits-to-rules.md` is the implementation that captures
14+
them into Terraform state.
815

9-
## False positive workflow
16+
## 2. Model
1017

11-
1. **Fetch**: `wallarm_hits` data source retrieves hits for given `request_id`(s)
12-
2. **Group by Action**: Hits grouped by Host header + URI path (the Action scope)
13-
3. **Group by Point**: Within each action, grouped by detection point
14-
4. **Generate Rules**: Two rule types for FP suppression:
15-
- **`disable_stamp`** — allows specific attack signatures (stamps) at a given point
16-
- **`disable_attack_type`** — allows specific attack types at a given point
17-
5. **One resource per rule**: Each stamp and each attack_type is a separate Terraform resource, matching the API 1:1. The `for_each` key is `{action_hash}_{point_hash}_{attack_type}_{stamp}` for stamp rules or `{action_hash}_{point_hash}_{attack_type}` for attack_type rules. Hash prefixes are 16 hex chars.
18+
```mermaid
19+
erDiagram
20+
REQUEST ||--o{ HIT : "request_id"
21+
ATTACK ||--o{ HIT : "attack_id"
22+
HIT {
23+
string attack_type "sqli / xss / ..."
24+
array stamps "numeric signature IDs"
25+
array point "detection point"
26+
string domain
27+
string path
28+
int poolid
29+
}
30+
```
1831

19-
**Stampless attack types:** `xxe` and `invalid_xml` do not produce stamps. Hits of these types can only be suppressed via `disable_attack_type` rules.
32+
A hit carries a detection **point** (where in the request the signature
33+
matched), one or more **stamps** (numeric signature IDs), an **attack type**,
34+
and request metadata (`domain`, `path`, `poolid`, `request_id`, `attack_id`).
35+
Hits of the same HTTP request share `request_id`; hits of the same campaign
36+
share `attack_id`. Suppression is expressed against an **action** (the match
37+
scope: host + URL path + optionally the application instance) and a **point**.
2038

21-
## Data source: `wallarm_hits`
39+
## 3. Elements
2240

23-
**Input**: `request_id` (single string) + `mode` variable (`"request"` or `"attack"`). Called per-request_id via `for_each` in HCL.
41+
| Entity | Meaning |
42+
|---|---|
43+
| hit | one detected threat in a request |
44+
| stamp | a numeric signature ID (a specific attack fingerprint) |
45+
| attack type | the attack category (`sqli`, `xss`, ...) |
46+
| action | the match scope (host + path + optional instance) a rule targets |
47+
| point | the detection point within the request |
48+
| `data.wallarm_hits` | the data source that fetches and transforms hits (see `hits-to-rules.md`) |
2449

25-
**Hit filtering — allowed attack types:**
26-
`xss`, `sqli`, `rce`, `ptrav`, `crlf`, `redir`, `nosqli`, `ldapi`, `scanner`, `mass_assignment`, `ssrf`, `ssi`, `mail_injection`, `ssti`, `xxe`, `invalid_xml`
50+
## 4. Behavior
2751

28-
**Key computed outputs:**
29-
- `aggregated` — compact JSON with `action_hash` (16 chars), `action` conditions, and `groups` (each keyed by `point_hash_16 + "_" + attack_type`, containing `stamps`, `attack_type`, and `disable_attack_type` bool controlled by `rule_types` filter)
30-
- `action_hash` — Ruby-compatible `ConditionsHash`
31-
- Action validation via `ActionReadByHitID` hash comparison
52+
The false-positive suppression workflow:
3253

33-
## Hits-to-rules flow
54+
1. **Fetch** - `data.wallarm_hits` retrieves hits for the given `request_id`(s).
55+
2. **Group by action** - hits are grouped by their Host + URL-path scope.
56+
3. **Group by point** - within each action, by detection point.
57+
4. **Generate rules** - two suppression shapes:
58+
- `disable_stamp` - allow one specific signature (stamp) at a point;
59+
- `disable_attack_type` - allow a whole attack type at a point.
60+
5. **One resource per rule** - each stamp and each attack type is a separate
61+
Terraform resource, matching the API 1:1.
3462

35-
Three components: `wallarm_hits_index` (gating), `data.wallarm_hits` (fetching), `terraform_data.cache` (persistence). Deduplication by action_hash in HCL locals. See `docs/guides/hits_to_rules.md`.
63+
Because hits age out of the API, the suppression config is captured into state
64+
once and then persists independently of live hits (`hits-to-rules.md §4.2`).
65+
**Stampless attack types** (`xxe`, `invalid_xml`) produce no stamps, so they can
66+
be suppressed only via `disable_attack_type`.
67+
68+
## 5. Parameters
69+
70+
The `data.wallarm_hits` inputs/outputs and the `disable_stamp` /
71+
`disable_attack_type` resource fields are in `hits-to-rules.md §5`; this doc adds
72+
no parameters of its own.
73+
74+
## 6. Reference data
75+
76+
- **Default attack-type filter (16)** (`data.wallarm_hits`): `xss`, `sqli`,
77+
`rce`, `ptrav`, `crlf`, `redir`, `nosqli`, `ldapi`, `scanner`,
78+
`mass_assignment`, `ssrf`, `ssi`, `mail_injection`, `ssti`, `xxe`,
79+
`invalid_xml`.
80+
- **Stampless types**: `xxe`, `invalid_xml` (no stamps; `disable_attack_type`
81+
only).
82+
- **`for_each` key formats**: `{action_hash}_{point_hash}_{attack_type}_{stamp}`
83+
(stamp rules) or `{action_hash}_{point_hash}_{attack_type}` (attack-type
84+
rules); hash prefixes are 16 hex chars (`ConditionsHash` / `PointHash`
85+
truncated, `hits-to-rules.md §6.5`).
86+
87+
## 7. References
88+
89+
- `hits-to-rules.md` - the provider flow, data source, and rule behavior.
90+
- `rules-core.md` - Action/Condition/Hint model; the `disable_stamp` /
91+
`disable_attack_type` rules.
92+
- `action.md` - action-scope construction from a hit.
93+
- `docs/guides/hits_to_rules.md` - operator how-to.

0 commit comments

Comments
 (0)