Skip to content

Commit f7641f4

Browse files
committed
docs/reference/filebeat: document run_as_cel and cel.* options
Add reference documentation for the run_as_cel option and the cel.program, cel.state, cel.max_executions, cel.regexp, cel.xsd, and cel.redact configuration fields. Includes a field mapping table, a note on cursor state migration, and a worked example.
1 parent 74424e0 commit f7641f4

File tree

1 file changed

+119
-2
lines changed

1 file changed

+119
-2
lines changed

docs/reference/filebeat/filebeat-input-httpjson.md

Lines changed: 119 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ applies_to:
1212

1313
Use the `httpjson` input to read messages from an HTTP API with JSON payloads.
1414

15-
If you are starting development of a new custom HTTP API input, we recommend that you use the [Common Expression Language input](/reference/filebeat/filebeat-input-cel.md) which provides greater flexibility and an improved developer experience.
15+
If you are starting development of a new custom HTTP API input, we recommend that you use the [Common Expression Language input](/reference/filebeat/filebeat-input-cel.md) which provides greater flexibility and an improved developer experience. Existing `httpjson` inputs can be migrated to CEL using the [`run_as_cel`](#run-as-cel) option.
1616

1717
This input supports:
1818

@@ -1499,7 +1499,7 @@ Example:
14991499
}
15001500
```
15011501

1502-
This behaviour of targeted fixed pattern replacement in the url helps solve various use cases.
1502+
This behavior of targeted fixed pattern replacement in the url helps solve various use cases.
15031503

15041504

15051505
**Some useful points to remember:**
@@ -1729,6 +1729,123 @@ filebeat.inputs:
17291729
```
17301730

17311731

1732+
### `run_as_cel` [run-as-cel]
1733+
1734+
```{applies_to}
1735+
stack: ga 9.4.0
1736+
```
1737+
1738+
When set to `true`, the input is transparently redirected to the [Common Expression Language input](/reference/filebeat/filebeat-input-cel.md). This allows an existing `httpjson` configuration to run under the CEL engine without changing the input `type`. A `cel.program` must be provided when this option is enabled.
1739+
1740+
Shared configuration fields are automatically translated to their CEL equivalents:
1741+
1742+
| httpjson field | CEL field |
1743+
| --- | --- |
1744+
| `interval` | `interval` |
1745+
| `id` | `id` |
1746+
| `request.url` | `resource.url` |
1747+
| `request.timeout` | `resource.timeout` |
1748+
| `request.ssl` | `resource.ssl` |
1749+
| `request.proxy_url` | `resource.proxy_url` |
1750+
| `request.proxy_headers` | `resource.proxy_headers` |
1751+
| `request.proxy_disable` | `resource.proxy_disable` |
1752+
| `request.idle_connection_timeout` | `resource.idle_connection_timeout` |
1753+
| `request.keep_alive` | `resource.keep_alive` |
1754+
| `request.retry` | `resource.retry` |
1755+
| `request.redirect` | `resource.redirect` |
1756+
| `request.tracer` | `resource.tracer` |
1757+
| `auth` | `auth` |
1758+
1759+
Fields that are specific to httpjson (such as `request.transforms`, `response.transforms`, `response.split`, `response.pagination`, and `chain`) are not transferred and have no effect when `run_as_cel` is enabled. The CEL program is responsible for equivalent logic.
1760+
1761+
If the httpjson input has existing cursor state, it is automatically carried over to the CEL input on the first run. After that first run the CEL input writes its own cursor, and all subsequent runs, whether the next interval or after a restart, read from the CEL cursor. The original httpjson cursor is not modified, so removing `run_as_cel` restores the original httpjson behavior with its last cursor intact.
1762+
1763+
Default: `false`.
1764+
1765+
1766+
### `cel.program` [cel-program]
1767+
1768+
```{applies_to}
1769+
stack: ga 9.4.0
1770+
```
1771+
1772+
The CEL program to execute when [`run_as_cel`](#run-as-cel) is enabled. This is the same program format used by the [CEL input's `program` field](/reference/filebeat/filebeat-input-cel.md). Required when `run_as_cel` is `true`.
1773+
1774+
1775+
### `cel.state` [cel-state]
1776+
1777+
```{applies_to}
1778+
stack: ga 9.4.0
1779+
```
1780+
1781+
Initial state for the CEL program, equivalent to the [CEL input's `state` field](/reference/filebeat/filebeat-input-cel.md). May include an initial `cursor` object that is used as the bootstrap value on the first execution when no stored cursor exists.
1782+
1783+
1784+
### `cel.max_executions` [cel-max-executions]
1785+
1786+
```{applies_to}
1787+
stack: ga 9.4.0
1788+
```
1789+
1790+
The maximum number of CEL program executions per interval. Equivalent to the [CEL input's `max_executions` field](/reference/filebeat/filebeat-input-cel.md). Default: `1000`.
1791+
1792+
1793+
### `cel.regexp` [cel-regexp]
1794+
1795+
```{applies_to}
1796+
stack: ga 9.4.0
1797+
```
1798+
1799+
A map of named regular expression patterns available to the CEL program. Equivalent to the [CEL input's `regexp` field](/reference/filebeat/filebeat-input-cel.md).
1800+
1801+
1802+
### `cel.xsd` [cel-xsd]
1803+
1804+
```{applies_to}
1805+
stack: ga 9.4.0
1806+
```
1807+
1808+
A map of named XSD schemas for XML decoding in the CEL program. Equivalent to the [CEL input's `xsd` field](/reference/filebeat/filebeat-input-cel.md).
1809+
1810+
1811+
### `cel.redact` [cel-redact]
1812+
1813+
```{applies_to}
1814+
stack: ga 9.4.0
1815+
```
1816+
1817+
Redaction configuration for the CEL program. Equivalent to the [CEL input's `redact` field](/reference/filebeat/filebeat-input-cel.md).
1818+
1819+
::::{admonition} Example: migrating an httpjson input to CEL
1820+
```yaml
1821+
filebeat.inputs:
1822+
- type: httpjson
1823+
id: my-api-input
1824+
interval: 60s
1825+
run_as_cel: true
1826+
request.url: https://api.example.com/events
1827+
auth.oauth2:
1828+
client.id: my-client-id
1829+
client.secret: my-client-secret
1830+
token_url: https://auth.example.com/oauth2/token
1831+
request.ssl.verification_mode: full
1832+
request.timeout: 30s
1833+
cel.program: |
1834+
state.url.with({
1835+
"Header": {"Accept": ["application/json"]},
1836+
}).as(req, request("GET", req).as(resp,
1837+
bytes(resp.Body).decode_json().as(body, {
1838+
"events": body.items.map(e, {"message": e.encode_json()}),
1839+
"cursor": {"since": body.items[body.items.size()-1].updated_at},
1840+
})
1841+
))
1842+
cel.state:
1843+
cursor:
1844+
since: "2024-01-01T00:00:00Z"
1845+
```
1846+
::::
1847+
1848+
17321849
## Request life cycle [_request_life_cycle]
17331850

17341851
![Request lifecycle](images/input-httpjson-lifecycle.png "")

0 commit comments

Comments
 (0)