diff --git a/pipeline/outputs/stackdriver.md b/pipeline/outputs/stackdriver.md index f8161d934..5b66e9c34 100644 --- a/pipeline/outputs/stackdriver.md +++ b/pipeline/outputs/stackdriver.md @@ -55,7 +55,7 @@ If you are using a _Google Cloud Credentials File_, the following configuration Example configuration file for k8s resource type: -local_resource_id is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log. +`local_resource_id` is used by the Stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the `local_resource_id` field in the log entry. If there is no field `logging.googleapis.com/local_resource_id` in the log, the plugin will then construct it by using the tag value of the log. The local_resource_id should be in format: @@ -157,6 +157,15 @@ For instance, for a K8s resource type, `resource_labels` can be used in tandem w ``` `resource_labels` also supports validation for required labels based on the input resource type. This allows fluent-bit to check if all specified labels are present for a given configuration before runtime. If validation is not currently supported for a resource type that you would like to use this API with, we encourage you to open a pull request for it. Adding validation for a new resource type is simple - all that is needed is to specify the resources associated with the type alongside the required labels [here](https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver_resource_types.c#L27). + +## Log Name + +By default, the plugin will write to the following log name: +``` +/projects//logs/ +``` +You may be in a scenario where being more specific about the log name is important (for example [integration with Log Router rules](https://cloud.google.com/logging/docs/routing/overview) or [controlling cardinality of log based metrics]((https://cloud.google.com/logging/docs/logs-based-metrics/troubleshooting#too-many-time-series))). You can control the log name directly on a per-log basis by using the [`logging.googleapis.com/logName` special field][StackdriverSpecialFields]. You can configure a `log_name_key` if you'd like to use something different than `logging.googleapis.com/logName`, i.e. if the `log_name_key` is set to `mylognamefield` will extract the log name from `mylognamefield` in the log. + ## Troubleshooting Notes ### Upstream connection error diff --git a/pipeline/outputs/stackdriver_special_fields.md b/pipeline/outputs/stackdriver_special_fields.md index b329eaecb..3bb2b9cd3 100644 --- a/pipeline/outputs/stackdriver_special_fields.md +++ b/pipeline/outputs/stackdriver_special_fields.md @@ -5,96 +5,66 @@ When the [google-logging-agent](https://cloud.google.com/logging/docs/agent) rec ## Log Entry Fields Currently, we support some special fields in fluent-bit for setting fields on the LogEntry object: -| JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | Description | -| :--- | :--- | :--- | -| logging.googleapis.com/operation | operation | Additional information about a potentially long-running operation | -| logging.googleapis.com/labels | labels | The value of this field should be a structured record | -| logging.googleapis.com/insertId | insertId | A unique identifier for the log entry. It is used to order logEntries | -| logging.googleapis.com/sourceLocation | sourceLocation | Additional information about the source code location that produced the log entry. | -| logging.googleapis.com/http_request | httpRequest | A common proto for logging HTTP requests. | -| logging.googleapis.com/trace | trace | Resource name of the trace associated with the log entry | -| logging.googleapis.com/traceSampled | traceSampled | The sampling decision associated with this log entry. | -| logging.googleapis.com/spanId | spanId | The ID of the trace span associated with this log entry. | -| timestamp | timestamp | An object including the seconds and nanos fields that represents the time | -| timestampSecond & timestampNanos | timestamp | The seconds and nanos that represents the time | +| JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | type | Description | +| :--- | :--- | :--- | :--- | +| `logging.googleapis.com/logName` | `logName` | `string` | The log name to write this log to. | +| `logging.googleapis.com/labels` | `labels` | `object` | The labels for this log. | +| `logging.googleapis.com/severity` | `severity` | [`LogSeverity` enum](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity) | The severity of this log. | +| `logging.googleapis.com/monitored_resource` | `resource` | [`MonitoredResource`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) (without `type`) | Resource labels for this log. See [caveats](#monitored-resource). | +| `logging.googleapis.com/operation` | `operation` | [`LogEntryOperation`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntryOperation) | Additional information about a potentially long-running operation. | +| `logging.googleapis.com/insertId` | `insertId` | `string` | A unique identifier for the log entry. It is used to order logEntries. | +| `logging.googleapis.com/sourceLocation` | `sourceLocation` | [`LogEntrySourceLocation`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntrySourceLocation) | Additional information about the source code location that produced the log entry. | +| `logging.googleapis.com/http_request` | `httpRequest` | [`HttpRequest`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) | A common proto for logging HTTP requests. | +| `logging.googleapis.com/trace` | `trace` | `string` | Resource name of the trace associated with the log entry. | +| `logging.googleapis.com/traceSampled` | `traceSampled` | boolean | The sampling decision associated with this log entry. | +| `logging.googleapis.com/spanId` | `spanId` | `string` | The ID of the trace span associated with this log entry. | +| `timestamp` | `timestamp` | `object` ([protobuf `Timestamp` object format](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp)) | An object including the seconds and nanos fields that represents the time. | +| `timestampSecond` & `timestampNanos` | `timestamp` | `int` | The seconds and nanos that represents the time. | ## Other Special Fields | JSON log field | Description | | :--- | :--- | -| logging.googleapis.com/projectId | Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project. | +| `logging.googleapis.com/projectId` | Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project. | +| `logging.googleapis.com/local_resource_id` | Overrides the [configured `local_resource_id`](./stackdriver.md#resource-labels). | -## Operation -Operation field contains additional information about a potentially long-running operation with which a log entry is associated. +## Using Special Fields -The JSON representation is as followed: -```text +To use a special field, you must add a field with the right name and value to your log. Given an example structured log (internally in MessagePack but shown in JSON for demonstration): +```json { - "id": string, - "producer": string, - "first": boolean, - "last": boolean + "log": "Hello world!" } ``` - -For example, when the jsonPayload contains the subfield `logging.googleapis.com/operation`: -```text -jsonPayload { - "logging.googleapis.com/operation": { - "id": "test_id", - "producer": "test_producer", - "first": true, - "last": true - } - ... +To use the `logging.googleapis.com/logName` special field, you would add it to your structured log as follows: +```json +{ + "log": "Hello world!", + "logging.googleapis.com/logName": "my_log" } ``` -the stackdriver output plugin will extract the operation field and remove it from jsonPayload. LogEntry will be: -```text +For the special fields that map to `LogEntry` protos, you will need to add them as objects with field names that match the proto. For example, to use the `logging.googleapis.com/operation`: +```json { - "jsonPayload": { - ... - } - "operation": { + "log": "Hello world!", + "logging.googleapis.com/operation": { "id": "test_id", "producer": "test_producer", "first": true, "last": true } - ... } ``` +Adding special fields to logs is best done through the [`modify` filter](https://docs.fluentbit.io/manual/pipeline/filters/modify) for simple fields, or [a Lua script using the `lua` filter](https://docs.fluentbit.io/manual/pipeline/filters/lua) for more complex fields. -### Use Cases -**1. If the subfields are empty or in incorrect type, stackdriver output plugin will set these subfields empty.** For example: -```text -jsonPayload { - "logging.googleapis.com/operation": { - "id": 123, #incorrect type - # no producer here - "first": true, - "last": true - } - ... -} -``` -the logEntry will be: +## Simple Type Special Fields + +For special fields with simple types (with the exception of the [`logging.googleapis.com/insertId` field](#insert-id)), they will follow this pattern (demonstrated with the `logging.googleapis.com/logName` field): + +1. If the special field matches the type, it will be moved to the corresponding LogEntry field. For example: ```text { - "jsonPayload": { - ... - } - "operation": { - "first": true, - "last": true - } - ... -} -``` -**2. If the `logging.googleapis.com/operation` itself is not a map, stackdriver output plugin will leave this field untouched.** For example: -```text -jsonPayload { - "logging.googleapis.com/operation": "some string", + "logging.googleapis.com/logName": "my_log" ... } ``` @@ -102,25 +72,17 @@ the logEntry will be: ```text { "jsonPayload": { - "logging.googleapis.com/operation": "some string", ... } + "logName": "my_log" ... } ``` -**3. If there are extra subfields, stackdriver output plugin will add operation field to logEntry and preserve the extra subfields in jsonPayload.** For example: -```text -jsonPayload { - "logging.googleapis.com/operation": { - "id": "test_id", - "producer": "test_producer", - "first": true, - "last": true, - "extra1": "some string", - "extra2": 123, - "extra3": true - } +2. If the field is non-empty but an invalid, it will be left in the jsonPayload. For example: +```text +{ + "logging.googleapis.com/logName": 12345 ... } ``` @@ -128,64 +90,57 @@ the logEntry will be: ```text { "jsonPayload": { - "logging.googleapis.com/operation": { - "extra1": "some string", - "extra2": 123, - "extra3": true - } + "logging.googleapis.com/logName": 12345 ... } - "operation": { - "id": "test_id", - "producer": "test_producer", - "first": true, - "last": true - } - ... } ``` -## Labels -labels field contains specific labels in a structured entry that will be added to LogEntry labels. -For example, when the jsonPayload contains the subfield `logging.googleapis.com/labels`: +### Exceptions + +#### Insert ID + +If the `logging.googleapis.com/insertId` field has an invalid type, the log will be rejected by the plugin and not sent to Cloud Logging. + +#### Trace Sampled + +If the [`autoformat_stackdriver_trace` plugin configuration option]() is set to `true`, the value provided in the `trace` field will be formatted into the format that Cloud Logging expects along with the detected Project ID (from the Google Metadata server, configured in the plugin, or provided via special field). + +For example, if `autoformat_stackdriver_trace` is enabled, this: ```text -jsonPayload { - "logging.googleapis.com/labels": { - "A": "valA", - "B": "valB", - "C": "valC" - } - ... +{ + "logging.googleapis.com/projectId": "my-project-id", + "logging.googleapis.com/trace": "12345" } ``` -the stackdriver output plugin will extract labels from the subfield `logging.googleapis.com/labels` and move it up from jsonPayload to LogEntry Labels. LogEntry will be: +Will become this: ```text { "jsonPayload": { ... - } - "labels": { - "A": "valA", - "B": "valB", - "C": "valC" - } - ... + }, + "projectId": "my-project-id", + "trace": "/projects/my-project-id/traces/12345" } ``` -## insertId -InsertId is a unique identifier for the log entry. It is used to order logEntries. +#### `timestampSecond` and `timestampNano` -The JSON representation is as followed: -```text -"insertId": string -``` +The `timestampSecond` and `timestampNano` fields don't map directly to the `timestamp` field in `LogEntry` so the parsing behaviour deviates from other special fields. Read more in the [Timestamp section](#timestamp). + +## Proto Special Fields + +For special fields that expect the format of a proto type from the `LogEntry` (with the exception of the `logging.googleapis.com/monitored_resource` field) will follow this pattern (demonstrated with the `logging.googleapis.com/operation` field): -### Use Cases -**1. If the insertId is a non-empty string.** For example: +If any subfields of the proto are empty or in incorrect type, the plugin will set these subfields empty. For example: ```text -jsonPayload { - "logging.googleapis.com/insertId": "test_id" +{ + "logging.googleapis.com/operation": { + "id": 123, #incorrect type + # no producer here + "first": true, + "last": true + } ... } ``` @@ -195,41 +150,18 @@ the logEntry will be: "jsonPayload": { ... } - "insertId": "test_id" - ... -} -``` - -**2. If the insertId is invalid (not non-empty string).** For example: -```text -jsonPayload { - "logging.googleapis.com/insertId": 12345 + "operation": { + "first": true, + "last": true + } ... } ``` -The logging agent will log an error and reject the entry. - -## SourceLocation -SourceLocation field contains additional information about the source code location that produced the log entry. The format. -The JSON representation is as followed: +If the field itself is not a map, the plugin will leave this field untouched. For example: ```text { - "file": string, - "line": string, - "function": string -} -``` - -### Use Cases -Set the input log as followed: -```text -jsonPayload { - "logging.googleapis.com/sourceLocation": { - "file": "my_file", - "line": 123, - "function": "foo()" - } + "logging.googleapis.com/operation": "some string", ... } ``` @@ -237,62 +169,25 @@ the logEntry will be: ```text { "jsonPayload": { + "logging.googleapis.com/operation": "some string", ... } - "sourceLocation": { - "file": "my_file", - "line": 123, - "function": "foo()" - } ... } ``` -## httpRequest -HttpRequest field is a common proto for logging HTTP requests. - -The JSON representation is as followed: +If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload. For example: ```text { - "requestMethod": string, - "requestUrl": string, - "requestSize": string, - "status": integer, - "responseSize": string, - "userAgent": string, - "remoteIp": string, - "serverIp": string, - "referer": string, - "latency": string, - "cacheLookup": boolean, - "cacheHit": boolean, - "cacheValidatedWithOriginServer": boolean, - "cacheFillBytes": string, - "protocol": string -} - -``` + "logging.googleapis.com/operation": { + "id": "test_id", + "producer": "test_producer", + "first": true, + "last": true, -### Use Cases -Set the input log as followed: -```text -jsonPayload { - "logging.googleapis.com/http_request": { - "requestMethod":"GET", - "requestUrl":"logging.googleapis.com", - "requestSize":"12", - "status":200, - "responseSize":"12", - "userAgent":"Mozilla", - "remoteIp":"255.0.0.1", - "serverIp":"255.0.0.1", - "referer":"referer", - "latency":"1s", - "cacheLookup":true, - "cacheHit":true, - "cacheValidatedWithOriginServer":true, - "cacheFillBytes":"12", - "protocol":"HTTP/1.2" + "extra1": "some string", + "extra2": 123, + "extra3": true } ... } @@ -301,75 +196,41 @@ the logEntry will be: ```text { "jsonPayload": { + "logging.googleapis.com/operation": { + "extra1": "some string", + "extra2": 123, + "extra3": true + } ... } - "httpRequest": { - "requestMethod":"GET", - "requestUrl":"logging.googleapis.com", - "requestSize":"12", - "status":200, - "responseSize":"12", - "userAgent":"Mozilla", - "remoteIp":"255.0.0.1", - "serverIp":"255.0.0.1", - "referer":"referer", - "latency":"1s", - "cacheLookup":true, - "cacheHit":true, - "cacheValidatedWithOriginServer":true, - "cacheFillBytes":"12", - "protocol":"HTTP/1.2" + "operation": { + "id": "test_id", + "producer": "test_producer", + "first": true, + "last": true } ... } ``` -## Trace +### Exceptions -TraceId is resource name of the trace associated with the log entry. -If enable autoformat_stackdriver_trace flag in config the entry will automatically get the projectID from the Google Metadata server and add it. +#### Monitored Resource ID -The JSON representation is as followed: -```text -"trace": string -``` +The `logging.googleapis.com/monitored_resource` field is parsed in a special way, meaning it has some important exceptions: -## Span ID +The `type` field from the [`MonitoredResource` proto]() is not parsed out of the special field. It is read from the [`resource` plugin configuration option](https://docs.fluentbit.io/manual/pipeline/outputs/stackdriver#configuration-parameters). If it is supplied in the `logging.googleapis.com/monitored_resource` special field, it will not be recognized. -The span ID within the trace associated with the log entry. +The `labels` field is expected to be an `object`. If any fields have a value that is not a string, the value is ignored and not preserved. The plugin logs an error and drops the field. -The JSON representation is as followed: -```text -"spanId": string -``` - -### Use Cases -Set the input log as followed: -```text -jsonPayload { - "logging.googleapis.com/trace": "0123456789abcdef0123456789abcdef" - "logging.googleapis.com/spanId": "1234567890abcdef" - ... -} -``` -the logEntry will be: -```text -{ - "jsonPayload": { - ... - } - "trace": "projects/your-project-name/traces/0123456789abcdef0123456789abcdef" - "spanId": "1234567890abcdef" - ... -} -``` +If no valid `labels` field is found, or if all of entries in the `labels` object provided are invalid, the `logging.googleapis.com/monitored_resource` field is dropped in favour of automatically setting resource labels using other available information based on the configured `resource` type. ## Timestamp We support two formats of time-related fields: Format 1 - timestamp: -JsonPayload contains a timestamp field that includes the seconds and nanos fields. +Log body contains a `timestamp` field that includes the seconds and nanos fields. ```text { "timestamp": { @@ -379,7 +240,7 @@ JsonPayload contains a timestamp field that includes the seconds and nanos field } ``` Format 2 - timestampSeconds/timestampNanos: -JsonPayload contains both the timestampSeconds and timestampNanos fields. +Log body contains both the `timestampSeconds` and `timestampNanos` fields. ```text { "timestampSeconds": CURRENT_SECONDS, @@ -388,16 +249,15 @@ JsonPayload contains both the timestampSeconds and timestampNanos fields. ``` -If one of the following JSON timestamp representations is present in a structured record, the Logging agent collapses them into a single representation in the timestamp field in the LogEntry object. +If one of the following JSON timestamp representations is present in a structured record, the plugin collapses them into a single representation in the timestamp field in the `LogEntry` object. -Without time-related fields, the logging agent will set the current time as timestamp. Supporting time-related fields enables users to get more information about the logEntry. +Without time-related fields, the plugin will set the current time as timestamp. +### Format 1 -### Use Cases -**Format 1** Set the input log as followed: ```text -jsonPayload { +{ "timestamp": { "seconds": 1596149787, "nanos": 12345 @@ -416,10 +276,11 @@ the logEntry will be: } ``` -**Format 2** +### Format 2 + Set the input log as followed: ```text -jsonPayload { +{ "timestampSeconds":1596149787, "timestampNanos": 12345 ... @@ -435,3 +296,5 @@ the logEntry will be: ... } ``` + +If the `timestamp` object or the `timestampSeconds` and `timestampNanos` fields end up being invalid, they will remain in the `jsonPayload` untouched. \ No newline at end of file