Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/reuse/apps/opentelemetry/logs-advance-option-otel.md

This file was deleted.

1 change: 1 addition & 0 deletions docs/reuse/apps/opentelemetry/timestamp-parsing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**Timestamp Parsing**. You can configure timestamp parsing for logs ingested using this source template. For more information, see [Timestamps, Time Zones, and Date Formats for the OpenTelemetry collector](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference).
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ In this step, you will configure the yaml required for Apache Collection. Below
- **Error file log path**. Enter the path to the error log file for your Apache instance.
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, sumo tags `_sourceCategory` with the value otel/apache user needs to provide the value for `webengine.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ In this step, you will configure the yaml required for Docker Collection. Below
- **Excluded Image List**. A list of strings, [regexes](https://golang.org/pkg/regexp/), or [globs](https://github.com/gobwas/glob) whose referent container image names will not be among the queried containers for scrapping metrics. Learn more about [*excluded_images*](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/dockerstatsreceiver/README.md#configuration).
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/docker.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,16 @@ import CollectorInstallation from '../../../../../reuse/apps/opentelemetry/colle
In this step, you will configure the yaml required for Elasticsearch collection. Below are the inputs required for configuration:

- **Name**. Name of the source template.
- **Description**. Description for the source template.
- **Description**. Description for the source template.
- **Log Filepath**. Location where the Elasticsearch logs are logged. Please refer to your elasticsearch.conf file.
- **Endpoint**. Enter the url of the server you need to monitor. (default: `localhost:9200`).
- **Username**. Enter the Elasticsearch username.
- **Password Environment Variable Name**. Enter the Elasticsearch password environment variable name.
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/elasticsearch user needs to provide the value for `db.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand All @@ -91,4 +92,4 @@ import DataConfiguration from '../../../../../reuse/apps/opentelemetry/data-conf

:::info
Refer to the [changelog](changelog.md) for information on periodic updates to this source template.
:::
:::
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,9 @@ In this step, you will configure the yaml required for Kafka collection. Below a
- **Endpoint**. The URL of the broker endpoint (default: `localhost:9092`).
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/kafka user needs to provide the value for `webengine.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@ In this step, you will configure the yaml required for Local File collection. Be
- **Collection should begin from**. Defines where will the collection of the logs start from. Possible values are "End of File" and "Beginning of File".
- **Detect messages spanning multiple lines**. You can enable this option when dealing with logs which span over multiple lines. On enabling this option you will need to specify **Boundary regex location** where you can specify if the expression defines end or start of the log line and **Expression to match message boundary** where you will define the expression.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add processing rules for logs collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,16 +80,17 @@ import CollectorInstallation from '../../../../../reuse/apps/opentelemetry/colle
In this step, you will configure the yaml required for MySQL collection. Below are the inputs required for configuration:

- **Name**. Name of the source template.
- **Description**. Description for the source template.
- **Description**. Description for the source template.
- **Error log path**. Location where the SQL Errors are logged. Please refer to your my.cnf file.
- **Slow Transaction log file path (optional)**. Location where the Slow SQL transactions are logged. Please refer to your my.cnf file.
- **Endpoint**. The URL of the MySQL endpoint (default: `localhost:3306`).
- **Username**. Enter the MySQL username.
- **Password Environment Variable Name**. Enter the MySQL password environment variable name.
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/mysql user needs to provide the value for `db.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand All @@ -101,4 +102,4 @@ import DataConfiguration from '../../../../../reuse/apps/opentelemetry/data-conf

:::info
Refer to the [changelog](changelog.md) for information on periodic updates to this source template.
:::
:::
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,9 @@ In this step, you will configure the yaml required for Nginx collection. Below a
- **Path to Nginx error Log file**. Enter the path to the error log file for your Nginx instance.
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default sumo tags `_sourceCategory` with the value otel/nginx user needs to provide the value for `webengine.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
@@ -1,27 +1,29 @@
---
id: otrm-time-reference
title: Timestamps, Time Zones, and Date Formats
title: Timestamps, Time Zones, Time Ranges, and Date Formats for OTRM
description: Learn how Sumo Logic manages timestamps, time zones, and dates, and the configuration options that are available with OTRM source templates.
keywords:
- time
- time reference
- time zone
- timezone
- time zone
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import useBaseUrl from '@docusaurus/useBaseUrl';

We support several options for timestamps, time zones, time ranges, and dates. When collecting log data, the timestamp attached to messages is vital, both for the integrity of the data in your account, and for accurate query results.
We support several options for handling timestamps, time zones, and date formats in logs ingested through our OpenTelemetry Remote Management (OTRM) source templates.

This guide covers timestamp parsing behavior, configuration, and troubleshooting specific to OTRM. If you're using traditional Sumo Logic sources (not OTRM), refer to the general [Time Reference documentation](/docs/send-data/reference-information/time-reference/).

Because of the importance of timestamps, Sumo Logic indexes the timestamp of each message, making sure that data relevant to a query’s time range is returned properly in search results, which allows you to reconstruct a correct event timeline.
When collecting log data, the timestamp attached to messages is critical for data integrity and accurate search results. Sumo Logic indexes the timestamp of each message to ensure results fall within the query’s time range, allowing you to reconstruct event timelines reliably.

## Timestamps

Timestamp is the part of a log message that marks the time that an event occurred. During ingestion, we can detect the message timestamp, convert it to Unix epoch time (the number of milliseconds since midnight, January 1, 1970 UTC), and index it. The timestamp is parsed either using the default timestamp parsing settings, or a custom format that you specify, including the time zone.

When configuring a source template, specify a custom format to parse timestamps in your log messages. 
When configuring a source template, you can specify a custom format to parse timestamps in your log messages. 

:::note
Currently, only `strptime` timestamps are supported in the source templates.
Expand All @@ -42,7 +44,7 @@ OpenTelemetry Collectors can automatically parse most timestamps without any is

1. Perform one of the following steps:
* If you're configuring a new Source template, proceed to Step 2. Or,
* To edit the timestamp settings for an existing Source template, navigate to the source template. Then click on **Edit**, to the right of the Source name and go to Step 2.<br/><img src={useBaseUrl('img/send-data/source-template-edit.png')} alt="Screenshot showing the editing interface for a source template in Sumo Logic, highlighting the section for editing advanced options including timestamp settings" style={{border: '1px solid gray'}} width="400"/>
* To edit the timestamp settings for an existing Source template, navigate to the source template. Then click **Edit** to the right of the Source name and go to Step 2.<br/><img src={useBaseUrl('img/send-data/source-template-edit.png')} alt="Screenshot showing the editing interface for a source template in Sumo Logic, highlighting the section for editing advanced options including timestamp settings" style={{border: '1px solid gray'}} width="400"/>
1. Navigate to the **Timestamp Parsing** section and select **Specify the format** and enter the below details:
1. **Select Timezone**. Define the geographic location (time zone) to use while parsing a timestamp that does not include a time zone. The available locations depend on the local IANA Time Zone database. For example, `America/New_York`. For more examples, refer to the [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones).
1. **Format**. Specify the exact layout of the timestamp to be parsed. For example, `- %Y-%m-%dT%H:%M:%S.%LZ`. To learn more about the formatting rules, refer to [this guide](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/internal/coreinternal/timeutils/internal/ctimefmt/ctimefmt.go#L68).
Expand Down Expand Up @@ -142,7 +144,7 @@ The following conventions are some examples of the supported formats for `strpti

The following considerations apply to time zones:

We highly recommend that the time zone be set explicitly on any source template where the logs does not have a time zone available. Sumo Logic always attempts to determine the time zone for the Source. However, if that is not possible, the time zone will revert to UTC. In these cases, the time zone will be incorrect, and that could significantly affect forensic analysis and reporting.
We highly recommend that the time zone be set explicitly on any source template where the logs do not have a time zone available. Sumo Logic always attempts to determine the time zone for the Source. However, if that is not possible, the time zone will revert to UTC. In these cases, the time zone will be incorrect, and that could significantly affect forensic analysis and reporting.

### Default time zone

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,9 @@ In this step, you will configure the yaml required for PostgreSQL collection. Be
- **Password Environment Variable Name**. Enter the PostgreSQL password environment variable name.
- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/postgresql user needs to provide the value for `db.cluster.name`.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand All @@ -111,4 +113,4 @@ import DataConfiguration from '../../../../../reuse/apps/opentelemetry/data-conf

:::info
Refer to the [changelog](changelog.md) for information on periodic updates to this source template.
:::
:::
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,9 @@ In this step, you will configure the yaml required for Local File Collection. Be
- **Username**. Required. Enter the RabbitMQ username.
- **Password Environment Variable Name**. Required. Enter the RabbitMQ password environment variable name.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add processing rules for logs collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,9 @@ separated by a colon.
- **Username** (Optional). Enter the Redis username in case you are using a specific user for monitoring.
- **Password Environment Variable Name** (Required). Enter the Redis password environment variable name.

**Timestamp Parsing**. You can do timestamp parsing for log ingested using this source template. For more details, refer to [Timestamps, Time Zones, and Date Formats](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) for OpenTelemetry collector.
import TimestampParsing from '../../../../../reuse/apps/opentelemetry/timestamp-parsing.md';

<TimestampParsing/>

**Processing Rules**. You can add processing rules for logs collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).

Expand Down
4 changes: 4 additions & 0 deletions docs/send-data/reference-information/time-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ The timestamp is the part of a log message that marks the time that an event occ

When configuring a Source, you can choose to use the default timestamp parsing settings, or you can specify a custom format for us to parse timestamps in your log messages. The **Enable Timestamp Parsing** option is selected by default. If it's deselected, no timestamp information is parsed at all. Instead, we stamp logs with the time at which the messages are processed.

:::note Using OpenTelemetry Remote Management (OTRM)?
This page covers timestamp parsing for standard Sumo Logic sources. If you're using OTRM source templates, refer to [Timestamps, time zones, time ranges, and date formats for OTRM](/docs/send-data/opentelemetry-collector/remote-management/source-templates/otrm-time-reference) instead.
:::

### Timestamp considerations

By default, we can automatically detect timestamps in your log messages. Automatic detection identifies timestamps in common formats and prefers timestamps that appear early in the message.
Expand Down
Binary file modified static/img/send-data/source-template-edit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.