Skip to content

Pipeline: filters: nightfall: style updates #1694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 29, 2025
Merged

Pipeline: filters: nightfall: style updates #1694

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d06f490
Pipeline: filters: nightfall: style updates
esmerel May 29, 2025
5c50069
Apply suggestions from code review
esmerel May 29, 2025
ae9d226
Apply suggestions from code review
esmerel May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 36 additions & 28 deletions pipeline/filters/nightfall.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,27 @@
# Nightfall

The Nightfall filter scans logs for sensitive data and redacts the sensitive portions. This filter supports scanning for
various sensitive information, ranging from API keys and personally identifiable information(PII) to custom regexes you
The _Nightfall_ filter scans logs for sensitive data and redacts the sensitive portions. This filter supports scanning for
various sensitive information, ranging from API keys and Personally Identifiable Information (PII) to custom regular expressions you
define. You can configure what to scan for in the [Nightfall Dashboard](https://app.nightfall.ai).

> This filter is not enabled by default in 1.9.0 due to a typo. It must be enabled by setting flag ```-DFLB_FILTER_NIGHTFALL=ON``` when building. In 1.9.1 and above this is fixed.
## Configuration Parameters
This filter isn't enabled by default in 1.9.0 due to a typo. To enable it, set the flag ```-DFLB_FILTER_NIGHTFALL=ON``` when building. In 1.9.1 and above this is fixed.

## Configuration parameters

The plugin supports the following configuration parameters:

| Key | Description | Default |
| :--- | :--- | :--- |
| nightfall\_api_key | The Nightfall API key to scan your logs with, obtainable from the [Nightfall Dashboard](https://app.nightfall.ai) | |
| policy\_id | The Nightfall dev platform policy to scan your logs with, configurable in the [Nightfall Dashboard](https://app.nightfall.ai/developer-platform/policies). | |
| sampling\_rate | The rate controlling how much of your logs you wish to be scanned, must be a float between (0,1]. 1 means all logs will be scanned. Useful for avoiding rate limits in conjunction with Fluent Bit's match rule.| 1 |
| tls.debug | Debug level between 0 (nothing) and 4 (every detail). | 0 |
| tls.verify | When enabled, turns on certificate validation when connecting to the Nightfall API. | true |
| tls.ca_path | Absolute path to root certificates, required if tls.verify is true. | |
| `nightfall_api_key` | The Nightfall API key to scan your logs with, obtainable from the [Nightfall Dashboard](https://app.nightfall.ai) | _none_ |
| `policy_id` | The Nightfall developer platform policy to scan your logs with, configurable in the [Nightfall Dashboard](https://app.nightfall.ai/developer-platform/policies). | _none_ |
| `sampling_rate` | The rate controlling how much of your logs you wish to be scanned. Must be a float between `(0,1]`. `1` means all logs will be scanned. Use this setting to avoid rate limits in conjunction with Fluent Bit match rules.| `1` |
| `tls.debug` | Debug level between `0` (nothing) and `4` (every detail). | `0` |
| `tls.verify` | When enabled, turns on certificate validation when connecting to the Nightfall API. | `true` |
| `tls.ca_path` | Absolute path to root certificates, required if `tls.verify` is true. | _none_ |

### Command Line
### Configuration file

```text
$ bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf

[2022/02/09 19:46:22] [ info] [engine] started (pid=53844)
[2022/02/09 19:46:22] [ info] [storage] version=1.1.5, initializing...
[2022/02/09 19:46:22] [ info] [storage] in-memory
[2022/02/09 19:46:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/02/09 19:46:22] [ info] [cmetrics] version=0.2.2
[2022/02/09 19:46:22] [ info] [input:http:http.0] listening on 0.0.0.0:8000
[2022/02/09 19:46:22] [ info] [sp] stream processor started
[2022/02/09 19:46:30] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
[0] app.log: [1644464790.280412000, {"A"=>"there is nothing sensitive here", "B"=>[{"A"=>"my credit card number is *******************"}, {"A"=>"*********** is my social security."}], "C"=>false, "D"=>"key ********************"}]
[2022/02/09 19:47:25] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
[0] app.log: [1644464845.675431000, {"A"=>"a very safe string"}]
```

### Configuration File
The following is an example of a configuration file for the Nightfall filter.

```text
[INPUT]
Expand All @@ -56,3 +41,26 @@ $ bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
Name stdout
```

### Command line

Once configured, you can use the filter from the command line by running a command like:

```shell
bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
```

Which results in output like:

```text
[2022/02/09 19:46:22] [ info] [engine] started (pid=53844)
[2022/02/09 19:46:22] [ info] [storage] version=1.1.5, initializing...
[2022/02/09 19:46:22] [ info] [storage] in-memory
[2022/02/09 19:46:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/02/09 19:46:22] [ info] [cmetrics] version=0.2.2
[2022/02/09 19:46:22] [ info] [input:http:http.0] listening on 0.0.0.0:8000
[2022/02/09 19:46:22] [ info] [sp] stream processor started
[2022/02/09 19:46:30] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
[0] app.log: [1644464790.280412000, {"A"=>"there is nothing sensitive here", "B"=>[{"A"=>"my credit card number is *******************"}, {"A"=>"*********** is my social security."}], "C"=>false, "D"=>"key ********************"}]
[2022/02/09 19:47:25] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
[0] app.log: [1644464845.675431000, {"A"=>"a very safe string"}]
```