Skip to content

Commit 3184e23

Browse files
JV0812JV0812
committed
OTel Timestamps Date Formats docs added
1 parent ccf56c4 commit 3184e23

File tree

3 files changed

+99
-0
lines changed

3 files changed

+99
-0
lines changed

‎docs/send-data/opentelemetry-collector/index.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,15 @@ In this section, we'll introduce the following concepts:
5555
</a>
5656
<p>Perform string hashing and masking operations using the Transform Processor and OTTL in OpenTelemetry.</p>
5757
</div>
58+
</div>
59+
<div className="box smallbox card">
60+
<div className="container">
61+
<a href="/docs/send-data/opentelemetry-collector/time-reference">
62+
<img src={useBaseUrl('img/send-data/otel-color.svg')} alt="OTel thumbnail icon" width="25" />
63+
<h4>OTel Timestamps Date Formats</h4>
64+
</a>
65+
<p>Learn how Sumo Logic manages OTel timestamps and the configuration options that are available.</p>
66+
</div>
5867
</div>
5968
<div className="box smallbox card">
6069
<div className="container">

‎docs/send-data/opentelemetry-collector/time-reference.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
id: time-reference
3+
title: OTel Timestamps Date Formats
4+
description: Learn how Sumo Logic manages OTel timestamps and the configuration options that are available.
5+
keywords:
6+
- time-stamp
7+
- otel-collector
8+
---
9+
10+
import Tabs from '@theme/Tabs';
11+
import TabItem from '@theme/TabItem';
12+
import useBaseUrl from '@docusaurus/useBaseUrl';
13+
14+
We support several options for timestamps. When collecting log data, the timestamp attached to messages is vital, both for the integrity of the data in your account, and for accurate query results.
15+
16+
Because of the importance of timestamps, Sumo Logic indexes the timestamp of each message, making sure that data relevant to a query’s time range is returned properly in search results, which allows you to reconstruct a correct event timeline.
17+
18+
## Timestamps
19+
20+
The timestamp is the part of a log message that marks the time that an event occurred. During ingestion, we can detect the message timestamp, convert it to Unix epoch time (the number of milliseconds since midnight, January 1, 1970 UTC), and index it. The timestamp is parsed either using the default timestamp parsing settings, or a custom format that you specify.
21+
22+
### Timestamp considerations
23+
24+
By default, we can automatically detect timestamps in your log messages. Automatic detection identifies timestamps in common formats and prefers timestamps that appear early in the message.
25+
26+
If your log messages from a Source contain multiple timestamps, timestamps in unusual formats, or a mix of distinct timestamp formats, you have two options:
27+
* Configure a Source for each log format
28+
* Configure a custom timestamp format for your Source
29+
30+
The OTel collector assumes that all log messages coming from a particular Source will have timestamps that are close together. If a message comes through that appears to be more than one day earlier or later than recent messages from that Source it will be auto-corrected to match the current time. You can stop this auto-correction by explicitly configuring a custom timestamp format on your Source.
31+
32+
The OTel collector also assumes that all log messages coming from a particular Source will have timestamps that are within a window of -1 year through +2 days compared to the current time. Any log message with a parsed timestamp outside of that window is automatically re-stamped with the current time. You must contact [Sumo Logic Support](https://support.sumologic.com/) to adjust this auto-correction behavior. See [How to ingest old or historical data](/docs/send-data/collector-faq#how-to-ingest-old-or-historical-data) for further details.
33+
34+
### OTel automated timestamp parsing
35+
36+
OTel collectors can automatically parse any of the following timestamp formats. If more than one valid timestamp is detected in a log message, the OTel collector will select the timestamp that appears "furthest left" in the message. 
37+
38+
:::note
39+
The Java SimpleDateFormat library is used for timestamp parsing. [Learn more](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html).
40+
:::
41+
42+
| OTel Timestamp Format | Example |
43+
|:---|:---|
44+
| `%Y-%m-%d'T'%H:%M:%S*%f%z` | 2023-08-20'T'13:20:10*633+0000 |
45+
| `%Y %b %d %H:%M:%S.%f %Z` | 2023 Mar 03 05:12:41.211 PDT |
46+
| `%b %d %H:%M:%S %z %Y` | Jan 21 18:20:11 +0000 2023 |
47+
| `%d/%b/%Y:%H:%M:%S %z` | 19/Apr/2023:06:36:15 -0700 |
48+
| `%b %d, %Y %l:%M:%S %p` | Dec 2, 2023 2:39:58 AM |
49+
| `%b %d %Y %H:%M:%S` | Jun 09 2023 15:28:14 |
50+
| `%b %d %H:%M:%S %Y` | Apr 20 00:00:35 2010 |
51+
| `%b %d %H:%M:%S %z` | Sep 28 19:00:00 +0000 |
52+
| `%b %d %H:%M:%S` | Mar 16 08:12:04 |
53+
| `%Y-%m-%dT%H:%M:%S%z` | 2023-10-14T22:11:20+0000 |
54+
| `%Y-%m-%d %H:%M:%S %z` | 2023-08-19 12:17:55 -0400 |
55+
| `%Y-%m-%d %H:%M:%S%z` | 2023-08-19 12:17:55-0400 |
56+
| `%Y %b %d %H:%M:%S.%f*%Z` | 2023 Apr 13 22:08:13.211*PDT |
57+
| `%Y %b %d %l:%M:%S` | 2023 Mar 10 1:44:20 |
58+
| `%Y-%m-%d %H:%M:%S,%f%z` | 2023-03-10 14:30:12,655+0000 |
59+
| `%Y-%m-%d %H:%M:%S` | 2023-02-27 15:35:20 |
60+
| `%Y-%m-%d %H:%M:%S.%f%z` | 2023-03-12 13:11:34.222-0700 |
61+
| `%Y-%m-%d'T'%H:%M:%S.%f` | 2023-07-22'T'16:28:55.444 |
62+
| `%Y-%m-%d'T'%H:%M:%S` | 2023-09-08'T'03:13:10 |
63+
| `%Y-%m-%d'T'%H:%M:%S'%z` | 2023-03-12'T'17:56:22'-0700' |
64+
| `%Y-%m-%dT%H:%M:%S.%f%z` | 2023-11-22'T'10:10:15.455 |
65+
| `%Y-%m-%d'T'%H:%M:%S` | 2023-02-11'T'18:31:44 |
66+
| `%Y-%m-%d*%H:%M:%S:%f` | 2023-10-30*02:47:33:899 |
67+
| `%Y-%m-%d*%H:%M:%S` | 2023-07-04*13:23:55 |
68+
| `%y-%m-%d %H:%M:%S,%f %z` | 23-02-11 16:47:35,985 +0000 |
69+
| `%y-%m-%d %H:%M:%S,%f` | 23-06-26 02:31:29,573 |
70+
| `%y-%m-%d %H:%M:%S` | 23-04-19 12:00:17 |
71+
| `%m/%d/%y %l:%M:%S` | 06/01/23 4:11:05 |
72+
| `%m%d%y %H:%M:%S` | 220423 11:42:35 |
73+
| `%Y%m%d %H:%M:%S.%f` | 20230423 11:42:35.173 |
74+
| `%m/%d/%y*%H:%M:%S` | 08/10/23*13:33:56 |
75+
| `%m/%d/%Y*%H:%M:%S` | 11/23/2023*05:13:11 |
76+
| `%m/%d/%y %H:%M:%S %z` | 04/23/23 04:34:22 +0000 |
77+
| `%m/%d/%Y %H:%M:%S %z` | 10/03/2023 07:29:46 -0700 |
78+
| `%H:%M:%S` | 11:42:35 |
79+
| `%H:%M:%S,%f` | 11:42:35,173 |
80+
| `%d/%b/%Y:%H:%M:%S` | 23/Apr/2023:11:42:35 |
81+
| `%d/%b%Y %H:%M:%S` | 23/Apr/2023 11:42:35 |
82+
| `%d-%b-%Y %H:%M:%S` | 23-Apr-2023 11:42:35 |
83+
| `%d-%b-%Y %H:%M:%S` | 23-Apr-2023 11:42:36 |
84+
| `%d %b %Y %H:%M:%S` | 23 Apr 2023 11:42:35 |
85+
| `%d %b %Y %H:%M:%S*%f` | 23 Apr 2023 10:32:35*311 |
86+
| `%m%d_%H:%M:%S` | 0423_11:42:35 |
87+
| `%m%d_%H:%M:%S.%f` | 0423_11:42:35.883 |
88+
| `%q/%g/%Y %l:%M:%S %p:%f` | 8/5/2023 3:31:18 AM:234 |
89+
| `%q/%d/%Y %I:%M:%S %p` | 9/28/2023 2:23:15 PM |

‎sidebars.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,6 +261,7 @@ module.exports = {
261261
'send-data/opentelemetry-collector/auto-discovery',
262262
'send-data/opentelemetry-collector/performance-benchmarks',
263263
'send-data/opentelemetry-collector/data-transformations',
264+
'send-data/opentelemetry-collector/time-reference',
264265
'send-data/opentelemetry-collector/troubleshooting',
265266
'send-data/opentelemetry-collector/sumo-logic-opentelemetry-vs-opentelemetry-upstream-relationship',
266267
'send-data/opentelemetry-collector/faq',

0 commit comments

Comments
 (0)