Skip to content

Commit 5d81c13

Browse files
esmerelesmerel
authored
Merge pull request #1977 from fluent/lynettemiles/sc-136268/update-fluent-bit-docs-pipeline-outputs-stackdriver
2 parents d8fda03 + 4efc94c commit 5d81c13

File tree

3 files changed

+101
-92
lines changed

3 files changed

+101
-92
lines changed

‎pipeline/outputs/stackdriver_special_fields.md

Lines changed: 97 additions & 92 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,35 @@
1-
# Stackdriver Special fields
1+
# Stackdriver special fields
22

33
When the [google-logging-agent](https://cloud.google.com/logging/docs/agent) receives a structured log record, it treats [some fields](https://cloud.google.com/logging/docs/agent/configuration#special-fields) specially, allowing users to set specific fields in the LogEntry object that get written to the Logging API.
44

5-
## Log Entry Fields
5+
## LogEntry fields
66

7-
Currently, we support some special fields in fluent-bit for setting fields on the LogEntry object:
8-
| JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | type | Description |
7+
Fluent Bit support some special fields for setting fields on the LogEntry object:
8+
9+
| JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | Type | Description |
910
| :--- | :--- | :--- | :--- |
1011
| `logging.googleapis.com/logName` | `logName` | `string` | The log name to write this log to. |
1112
| `logging.googleapis.com/labels` | `labels` | `object<string, string>` | The labels for this log. |
12-
| `logging.googleapis.com/severity` | `severity` | [`LogSeverity` enum](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity) | The severity of this log. |
13+
| `logging.googleapis.com/severity` | `severity` | [`LogSeverity` Enum](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity) | The severity of this log. |
1314
| `logging.googleapis.com/monitored_resource` | `resource` | [`MonitoredResource`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) (without `type`) | Resource labels for this log. |
1415
| `logging.googleapis.com/operation` | `operation` | [`LogEntryOperation`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntryOperation) | Additional information about a potentially long-running operation. |
15-
| `logging.googleapis.com/insertId` | `insertId` | `string` | A unique identifier for the log entry. It is used to order logEntries. |
16+
| `logging.googleapis.com/insertId` | `insertId` | `string` | A unique identifier for the log entry. It's used to order `logEntries`. |
1617
| `logging.googleapis.com/sourceLocation` | `sourceLocation` | [`LogEntrySourceLocation`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntrySourceLocation) | Additional information about the source code location that produced the log entry. |
1718
| `logging.googleapis.com/http_request` | `httpRequest` | [`HttpRequest`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) | A common proto for logging HTTP requests. |
1819
| `logging.googleapis.com/trace` | `trace` | `string` | Resource name of the trace associated with the log entry. |
1920
| `logging.googleapis.com/traceSampled` | `traceSampled` | boolean | The sampling decision associated with this log entry. |
2021
| `logging.googleapis.com/spanId` | `spanId` | `string` | The ID of the trace span associated with this log entry. |
21-
| `timestamp` | `timestamp` | `object` ([protobuf `Timestamp` object format](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp)) | An object including the seconds and nanos fields that represents the time. |
22-
| `timestampSecond` & `timestampNanos` | `timestamp` | `int` | The seconds and nanos that represents the time. |
22+
| `timestamp` | `timestamp` | `object` ([protobuf `Timestamp` object format](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp)) | An object including the seconds and nanoseconds fields that represent the time. |
23+
| `timestampSecond` and `timestampNanos` | `timestamp` | `int` | The seconds and nanoseconds that represent the time. |
2324

24-
## Other Special Fields
25+
## Other special fields
2526

26-
| JSON log field | Description |
27-
|:-------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------|
28-
| `logging.googleapis.com/projectId` | Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project. |
29-
| `logging.googleapis.com/local_resource_id` | Overrides the [configured `local_resource_id`](./stackdriver.md#resource-labels). |
27+
| JSON log field | Description |
28+
|:---------------|:------------|
29+
| `logging.googleapis.com/projectId` | Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project. |
30+
| `logging.googleapis.com/local_resource_id` | Overrides the [configured `local_resource_id`](./stackdriver.md#resource-labels). |
3031

31-
## Using Special Fields
32+
## Use special fields
3233

3334
To use a special field, you must add a field with the right name and value to your log. Given an example structured log (internally in MessagePack but shown in JSON for demonstration):
3435

@@ -38,7 +39,7 @@ To use a special field, you must add a field with the right name and value to yo
3839
}
3940
```
4041

41-
To use the `logging.googleapis.com/logName` special field, you would add it to your structured log as follows:
42+
To use the `logging.googleapis.com/logName` special field, add it to your structured log as follows:
4243

4344
```json
4445
{
@@ -47,7 +48,7 @@ To use the `logging.googleapis.com/logName` special field, you would add it to y
4748
}
4849
```
4950

50-
For the special fields that map to `LogEntry` prototypes, you will need to add them as objects with field names that match the proto. For example, to use the `logging.googleapis.com/operation`:
51+
For the special fields that map to `LogEntry` prototypes, add them as objects with field names that match the proto. For example, to use the `logging.googleapis.com/operation`:
5152

5253
```json
5354
{
@@ -63,62 +64,62 @@ For the special fields that map to `LogEntry` prototypes, you will need to add t
6364

6465
Adding special fields to logs is best done through the [`modify` filter](https://docs.fluentbit.io/manual/pipeline/filters/modify) for simple fields, or [a Lua script using the `lua` filter](https://docs.fluentbit.io/manual/pipeline/filters/lua) for more complex fields.
6566

66-
## Simple Type Special Fields
67+
## Simple type special fields
6768

68-
For special fields with simple types (except for the [`logging.googleapis.com/insertId` field](#insert-id)), they will follow this pattern (demonstrated with the `logging.googleapis.com/logName` field):
69+
Special fields with simple types (except for the [`logging.googleapis.com/insertId` field](#insert-id)) will follow this pattern (demonstrated with the `logging.googleapis.com/logName` field):
6970

7071
1. If the special field matches the type, it will be moved to the corresponding LogEntry field. For example:
7172

72-
```text
73-
{
74-
...
75-
"logging.googleapis.com/logName": "my_log"
76-
...
77-
}
78-
```
79-
80-
the logEntry will be:
81-
82-
```text
83-
{
84-
"jsonPayload": {
85-
...
86-
}
87-
"logName": "my_log"
88-
...
89-
}
90-
```
91-
92-
2. If the field is non-empty but an invalid, it will be left in the jsonPayload. For example:
93-
94-
```text
95-
{
96-
...
97-
"logging.googleapis.com/logName": 12345
98-
...
99-
}
100-
```
101-
102-
the logEntry will be:
103-
104-
```text
105-
{
106-
"jsonPayload": {
107-
"logging.googleapis.com/logName": 12345
108-
...
109-
}
110-
}
111-
```
73+
```text
74+
{
75+
...
76+
"logging.googleapis.com/logName": "my_log"
77+
...
78+
}
79+
```
80+
81+
the `logEntry` will be:
82+
83+
```text
84+
{
85+
"jsonPayload": {
86+
...
87+
}
88+
"logName": "my_log"
89+
...
90+
}
91+
```
92+
93+
1. If the field is non-empty but an invalid, it will be left in the `jsonPayload`. For example:
94+
95+
```text
96+
{
97+
...
98+
"logging.googleapis.com/logName": 12345
99+
...
100+
}
101+
```
102+
103+
the `logEntry` will be:
104+
105+
```text
106+
{
107+
"jsonPayload": {
108+
"logging.googleapis.com/logName": 12345
109+
...
110+
}
111+
}
112+
```
112113

113114
### Exceptions
114115

115116
#### Insert ID
116117

117118
If the `logging.googleapis.com/insertId` field has an invalid type, the log will be rejected by the plugin and not sent to Cloud Logging.
118119

119-
#### Trace Sampled
120+
#### Trace sampled
120121

121-
If the [`autoformat_stackdriver_trace` plugin configuration option]() is set to `true`, the value provided in the `trace` field will be formatted into the format that Cloud Logging expects along with the detected Project ID (from the Google Metadata server, configured in the plugin, or provided via special field).
122+
If the`autoformat_stackdriver_trace` is set to `true`, the value provided in the `trace` field will be formatted into the format that Cloud Logging expects along with the detected Project ID (from the Google Metadata server, configured in the plugin, or provided using a special field).
122123

123124
For example, if `autoformat_stackdriver_trace` is enabled, this:
124125

@@ -148,11 +149,11 @@ Will become this:
148149

149150
The `timestampSecond` and `timestampNano` fields don't map directly to the `timestamp` field in `LogEntry` so the parsing behaviour deviates from other special fields. Read more in the [Timestamp section](#timestamp).
150151

151-
## Proto Special Fields
152+
## Proto special fields
152153

153154
For special fields that expect the format of a prototype from the `LogEntry` (except for the `logging.googleapis.com/monitored_resource` field) will follow this pattern (demonstrated with the `logging.googleapis.com/operation` field):
154155

155-
If any subfields of the proto are empty or in incorrect type, the plugin will set these subfields empty. For example:
156+
If any sub-fields of the proto are empty or in incorrect type, the plugin will set these sub-fields empty. For example:
156157

157158
```text
158159
{
@@ -166,7 +167,7 @@ If any subfields of the proto are empty or in incorrect type, the plugin will se
166167
}
167168
```
168169

169-
the logEntry will be:
170+
the `logEntry` will be:
170171

171172
```text
172173
{
@@ -181,7 +182,7 @@ the logEntry will be:
181182
}
182183
```
183184

184-
If the field itself is not a map, the plugin will leave this field untouched. For example:
185+
If the field itself isn't a map, the plugin will leave this field untouched. For example:
185186

186187
```text
187188
{
@@ -191,7 +192,7 @@ If the field itself is not a map, the plugin will leave this field untouched. Fo
191192
}
192193
```
193194

194-
the logEntry will be:
195+
the `logEntry` will be:
195196

196197
```text
197198
{
@@ -203,7 +204,7 @@ the logEntry will be:
203204
}
204205
```
205206

206-
If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload. For example:
207+
If there are extra sub-fields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra sub-fields in `jsonPayload`. For example:
207208

208209
```text
209210
{
@@ -221,7 +222,7 @@ If there are extra subfields, the plugin will add the recognized fields to the c
221222
}
222223
```
223224

224-
the logEntry will be:
225+
the `logEntry will be:
225226

226227
```text
227228
{
@@ -249,45 +250,48 @@ the logEntry will be:
249250

250251
The `logging.googleapis.com/monitored_resource` field is parsed in a special way, meaning it has some important exceptions:
251252

252-
The `type` field from the [`MonitoredResource` proto]() is not parsed out of the special field. It is read from the [`resource` plugin configuration option](https://docs.fluentbit.io/manual/pipeline/outputs/stackdriver#configuration-parameters). If it is supplied in the `logging.googleapis.com/monitored_resource` special field, it will not be recognized.
253+
The `type` field from the `MonitoredResource` proto isn't parsed out of the special field. It's read from the [`resource` plugin configuration option](https://docs.fluentbit.io/manual/pipeline/outputs/stackdriver#configuration-parameters). If it's supplied in the `logging.googleapis.com/monitored_resource` special field, it won't be recognized.
253254

254-
The `labels` field is expected to be an `object<string, string>`. If any fields have a value that is not a string, the value is ignored and not preserved. The plugin logs an error and drops the field.
255+
The `labels` field is expected to be an `object<string, string>`. If any fields have a value that's not a string, the value is ignored and not preserved. The plugin logs an error and drops the field.
255256

256257
If no valid `labels` field is found, or if all entries in the `labels` object provided are invalid, the `logging.googleapis.com/monitored_resource` field is dropped in favour of automatically setting resource labels using other available information based on the configured `resource` type.
257258

258259
## Timestamp
259260

260-
We support two formats of time-related fields:
261+
Fluent Bit supports the following formats of time-related fields:
261262

262-
Format 1 - timestamp:
263-
Log body contains a `timestamp` field that includes the seconds and nanos fields.
263+
- `timestamp`
264264

265-
```text
266-
{
267-
"timestamp": {
268-
"seconds": CURRENT_SECONDS,
269-
"nanos": CURRENT_NANOS
265+
Log body contains a `timestamp` field that includes the seconds and nanoseconds fields.
266+
267+
```text
268+
{
269+
"timestamp": {
270+
"seconds": CURRENT_SECONDS,
271+
"nanos": CURRENT_NANOS
272+
}
270273
}
271-
}
272-
```
274+
```
273275

274-
Format 2 - timestampSeconds/timestampNanos:
275-
Log body contains both the `timestampSeconds` and `timestampNanos` fields.
276+
- `timestampSeconds`/`timestampNanos`
276277

277-
```text
278-
{
279-
"timestampSeconds": CURRENT_SECONDS,
280-
"timestampNanos": CURRENT_NANOS
281-
}
282-
```
278+
Log body contains both the `timestampSeconds` and `timestampNanos` fields.
279+
280+
```text
281+
{
282+
"timestampSeconds": CURRENT_SECONDS,
283+
"timestampNanos": CURRENT_NANOS
284+
}
285+
```
283286

284287
If one of the following JSON timestamp representations is present in a structured record, the plugin collapses them into a single representation in the timestamp field in the `LogEntry` object.
285288

286289
Without time-related fields, the plugin will set the current time as timestamp.
287290

288291
### Format 1
289292

290-
Set the input log as followed:
293+
Set the input log as follows:
294+
291295
```text
292296
{
293297
"timestamp": {
@@ -298,7 +302,7 @@ Set the input log as followed:
298302
}
299303
```
300304

301-
the logEntry will be:
305+
the `logEntry` will be:
302306

303307
```text
304308
{
@@ -312,7 +316,8 @@ the logEntry will be:
312316

313317
### Format 2
314318

315-
Set the input log as followed:
319+
Set the input log as follows:
320+
316321
```text
317322
{
318323
"timestampSeconds":1596149787,
@@ -321,7 +326,7 @@ Set the input log as followed:
321326
}
322327
```
323328

324-
the logEntry will be:
329+
the `logEntry` will be:
325330

326331
```text
327332
{
@@ -333,4 +338,4 @@ the logEntry will be:
333338
}
334339
```
335340

336-
If the `timestamp` object or the `timestampSeconds` and `timestampNanos` fields end up being invalid, they will remain in the `jsonPayload` untouched.
341+
If the `timestamp` object or the `timestampSeconds` and `timestampNanos` fields end up being invalid, they will remain in the `jsonPayload` untouched.

‎vale-styles/FluentBit/Headings.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ exceptions:
7171
- Libpq
7272
- Linux
7373
- LogDNA
74+
- LogEntry
7475
- LTSV
7576
- macOS
7677
- Marketplace

‎vale-styles/FluentBit/Spelling-exceptions.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ endcode
6767
endhint
6868
endtab
6969
endtabs
70+
Enum
7071
Eventhouse
7172
Exabeam
7273
Fargate
@@ -146,6 +147,8 @@ preprocessor
146147
Profiler
147148
Prometheus
148149
PromQL
150+
proto
151+
protobuf
149152
Protobuf
150153
proxying
151154
Pulumi

0 commit comments

Comments
 (0)