Skip to content

Commit d0512d6

Browse files
paliwalparitoshpaliwalparitosh
authored
timezone override support for logs collected via fluetnd agent (#93)
1 parent 7206227 commit d0512d6

File tree

8 files changed

+148
-6
lines changed

8 files changed

+148
-6
lines changed

‎charts/logan/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ apiVersion: v2
55
name: oci-onm-logan
66
description: Charts for sending Kubernetes platform logs, compute logs, and Kubernetes Objects information to OCI Logging Analytics.
77
type: application
8-
version: 3.5.0
8+
version: 3.5.1
99
appVersion: "3.0.0"
1010

1111
dependencies:

‎charts/logan/templates/discovery-cronjob.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@ spec:
2222
spec:
2323
restartPolicy: {{ .Values.k8sDiscovery.objects.restartPolicy }}
2424
serviceAccountName: {{ $serviceAccount }}
25+
{{- if .Values.image.imagePullSecrets }}
2526
imagePullSecrets:
2627
- name: {{ .Values.image.imagePullSecrets }}
28+
{{- end }}
2729
containers:
2830
- name: k8-discovery-job
2931
image: {{ .Values.image.url }}

‎charts/logan/templates/ekscp-logs-configmap.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,13 @@ data:
166166
{{- else }}
167167
oci_la_log_set "{{ $.Values.fluentd.eksControlPlane.ociLALogSet | default $.Values.ociLALogSet }}"
168168
{{- end }}
169+
{{- if $logDefinition.timezone }}
170+
oci_la_timezone "{{ $logDefinition.timezone }}"
171+
{{- else if $.Values.fluentd.eksControlPlane.timezone }}
172+
oci_la_timezone "{{ $.Values.fluentd.eksControlPlane.timezone }}"
173+
{{- else if $.Values.fluentd.timezone }}
174+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
175+
{{- end }}
169176
message "${record['message']}"
170177
tag ${tag}
171178
</record>
@@ -241,6 +248,13 @@ data:
241248
oci_la_log_set "{{ $.Values.fluentd.eksControlPlane.ociLALogSet | default $.Values.ociLALogSet }}"
242249
{{- end }}
243250
message "${record['message']}"
251+
{{- if $logDefinition.timezone }}
252+
oci_la_timezone "{{ $logDefinition.timezone }}"
253+
{{- else if $.Values.fluentd.eksControlPlane.timezone }}
254+
oci_la_timezone "{{ $.Values.fluentd.eksControlPlane.timezone }}"
255+
{{- else if $.Values.fluentd.timezone }}
256+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
257+
{{- end }}
244258
tag ${tag}
245259
</record>
246260
</filter>
@@ -270,6 +284,7 @@ data:
270284
@type oci-logging-analytics
271285
enable_ruby true
272286
namespace "{{ $.Values.ociLANamespace }}"
287+
collection_source "kubernetes_solution"
273288
endpoint "{{ $.Values.fluentd.ociLoggingAnalyticsOutputPlugin.endpoint }}"
274289
{{- if eq $authtype "config" }}
275290
config_file_location {{ $.Values.oci.path }}/{{ $.Values.oci.file }}

‎charts/logan/templates/logs-configmap.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,13 @@ data:
110110
{{- if eq $runtime "docker" }}
111111
message "${record['log']}"
112112
{{- end }}
113+
{{- if $logDefinition.timezone }}
114+
oci_la_timezone "{{ $logDefinition.timezone }}"
115+
{{- else if $.Values.fluentd.kubernetesSystem.timezone }}
116+
oci_la_timezone "{{ $.Values.fluentd.kubernetesSystem.timezone }}"
117+
{{- else if $.Values.fluentd.timezone }}
118+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
119+
{{- end }}
113120
tag ${tag}
114121
</record>
115122
</filter>
@@ -208,6 +215,13 @@ data:
208215
{{- else }}
209216
oci_la_log_set "{{ $.Values.fluentd.linuxSystem.ociLALogSet | default $.Values.ociLALogSet }}"
210217
{{- end }}
218+
{{- if $logDefinition.timezone }}
219+
oci_la_timezone "{{ $logDefinition.timezone }}"
220+
{{- else if $.Values.fluentd.linuxSystem.timezone }}
221+
oci_la_timezone "{{ $.Values.fluentd.linuxSystem.timezone }}"
222+
{{- else if $.Values.fluentd.timezone }}
223+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
224+
{{- end }}
211225
tag ${tag}
212226
</record>
213227
</filter>
@@ -292,6 +306,13 @@ data:
292306
{{- else }}
293307
oci_la_log_set "{{ $.Values.fluentd.linuxSystem.ociLALogSet | default $.Values.ociLALogSet }}"
294308
{{- end }}
309+
{{- if $.Values.fluentd.linuxSystem.logs.kubeletlog.timezone }}
310+
oci_la_timezone "{{ $.Values.fluentd.linuxSystem.logs.kubeletlog.timezone }}"
311+
{{- else if $.Values.fluentd.linuxSystem.timezone }}
312+
oci_la_timezone "{{ $.Values.fluentd.linuxSystem.timezone }}"
313+
{{- else if $.Values.fluentd.timezone }}
314+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
315+
{{- end }}
295316
tag ${tag}
296317
</record>
297318
</filter>
@@ -322,6 +343,13 @@ data:
322343
{{- else }}
323344
oci_la_log_set "{{ $.Values.fluentd.linuxSystem.ociLALogSet | default $.Values.ociLALogSet }}"
324345
{{- end }}
346+
{{- if $.Values.fluentd.linuxSystem.logs.syslog.timezone }}
347+
oci_la_timezone "{{ $.Values.fluentd.linuxSystem.logs.syslog }}"
348+
{{- else if $.Values.fluentd.linuxSystem.timezone }}
349+
oci_la_timezone "{{ $.Values.fluentd.linuxSystem.timezone }}"
350+
{{- else if $.Values.fluentd.timezone }}
351+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
352+
{{- end }}
325353
tag ${tag}
326354
</record>
327355
</filter>
@@ -395,6 +423,11 @@ data:
395423
{{- if and (ne "false" ($logDefinition.isContainerLog | toString)) (eq $runtime "docker") }}
396424
message "${record['log']}"
397425
{{- end }}
426+
{{- if $logDefinition.timezone }}
427+
oci_la_timezone "{{ $logDefinition.timezone }}"
428+
{{- else if $.Values.fluentd.timezone }}
429+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
430+
{{- end }}
398431
tag ${tag}
399432
</record>
400433
</filter>
@@ -502,6 +535,11 @@ data:
502535
{{- if eq $runtime "docker" }}
503536
message "${record['log']}"
504537
{{- end }}
538+
{{- if $.Values.fluentd.genericContainerLogs.timezone }}
539+
oci_la_timezone "{{ $.Values.fluentd.genericContainerLogs.timezone }}"
540+
{{- else if $.Values.fluentd.timezone }}
541+
oci_la_timezone "{{ $.Values.fluentd.timezone }}"
542+
{{- end }}
505543
tag ${tag}
506544
</record>
507545
</filter>
@@ -566,6 +604,7 @@ data:
566604
enable_ruby true
567605
namespace "{{ $.Values.ociLANamespace }}"
568606
endpoint "{{ $.Values.fluentd.ociLoggingAnalyticsOutputPlugin.endpoint }}"
607+
collection_source "kubernetes_solution"
569608
{{- if eq $authtype "config" }}
570609
config_file_location {{ $.Values.oci.path }}/{{ $.Values.oci.file }}
571610
profile_name "{{ $.Values.fluentd.ociLoggingAnalyticsOutputPlugin.profile_name }}"

‎charts/logan/values.yaml‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ image:
4848
# Image pull secrets for. Secret must be in the namespace defined by namespace
4949
imagePullSecrets:
5050
# -- Replace this value with actual docker image url
51-
url: container-registry.oracle.com/oci_observability_management/oci-la-fluentd-collector:1.5.0
51+
url: container-registry.oracle.com/oci_observability_management/oci-la-fluentd-collector:1.5.1
5252
# -- Image pull policy
5353
imagePullPolicy: Always
5454

@@ -164,6 +164,8 @@ fluentd:
164164
# -- To enable MultiProcessWorkers, set this to (> 0) the number of required workers. Defaults to 0.
165165
# -- Set corresponding workerId using worker field against each source/log. When enabled MultiProcessWorkers, the default worker would be 0 until unless set for each source/log.
166166
multiProcessWorkers: 0
167+
# -- To set timezone override for all logs collected using this solution (applies only to log records without explicit timezone identifier in the record itself)
168+
timezone:
167169

168170
# Configuration for oci-logging-analytics fluentd output plugin
169171
ociLoggingAnalyticsOutputPlugin:
@@ -244,6 +246,8 @@ fluentd:
244246
#encoding:
245247
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
246248
#worker:
249+
# -- To set timezone override for all kubernetesSystem logs (applies only to log records without explicit timezone identifier in the record itself)
250+
#timezone:
247251

248252
logs:
249253
# -- Kube Proxy logs collection configuration
@@ -261,7 +265,9 @@ fluentd:
261265
#ociLALogGroupID:
262266
#encoding:
263267
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
264-
#worker:
268+
#worker:
269+
# -- To set timezone override for kube-proxy logs (applies only to log records without explicit timezone identifier in the record itself)
270+
#timezone:
265271

266272
# -- Kube Flannel logs collection configuration
267273
kube-flannel:
@@ -371,6 +377,9 @@ fluentd:
371377
#encoding:
372378
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
373379
#worker:
380+
# -- To set timezone override for all linuxSystem logs (applies only to log records without explicit timezone identifier in the record itself)
381+
#timezone:
382+
374383
logs:
375384
# -- Linux CRON logs collection configuration
376385
cronlog:
@@ -380,6 +389,8 @@ fluentd:
380389
ociLALogSourceName: "Linux Cron Logs"
381390
# The regular expression pattern for the starting line in case of multi-line logs.
382391
multilineStartRegExp: /^(?:(?:\d+\s+)?<([^>]*)>(?:\d+\s+)?)?\S+\s+\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2}\s+/
392+
# -- To set timezone override for cronlog (applies only to log records without explicit timezone identifier in the record itself)
393+
#timezone:
383394

384395
# -- Linux CRON logs collection configuration
385396
securelog:
@@ -403,6 +414,7 @@ fluentd:
403414
ociLALogSourceName: "Linux Syslog Logs"
404415
# The regular expression pattern for the starting line in case of multi-line logs.
405416
multilineStartRegExp: /^(?:(?:\d+\s+)?<([^>]*)>(?:\d+\s+)?)?\S+\s+\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2}\s+/
417+
406418
# -- Linux maillog collection configuration
407419
maillog:
408420
# maillog file path
@@ -462,6 +474,9 @@ fluentd:
462474
#"Third Key": "Third Value"
463475
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
464476
#worker:
477+
# -- To set timezone override for all eksControlPlane logs (applies only to log records without explicit timezone identifier in the record itself)
478+
#timezone:
479+
465480
logs:
466481
# If using cloudwatch collection mechanism, apiserver and audit logs need to be part of the same worker as they share the same log stream name prefix.
467482
# Thus "worker" variable is only picked up from "apiserver" section.
@@ -482,6 +497,9 @@ fluentd:
482497
#ociLALogGroupID:
483498
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
484499
#worker:
500+
# -- To set timezone override for apiserver logs (applies only to log records without explicit timezone identifier in the record itself)
501+
#timezone:
502+
485503
audit:
486504
sqsQueue: "audit"
487505
# S3 object key
@@ -494,6 +512,7 @@ fluentd:
494512
#"Third Key": "Third Value"
495513
#ociLALogGroupID:
496514
#worker:
515+
497516
authenticator:
498517
cwLogStreamName: "authenticator"
499518
sqsQueue: "authenticator"
@@ -508,6 +527,7 @@ fluentd:
508527
#"Third Key": "Third Value"
509528
#ociLALogGroupID:
510529
#worker:
530+
511531
kubecontrollermanager:
512532
cwLogStreamName: "kube-controller-manager"
513533
sqsQueue: "kube-controller-manager"
@@ -522,6 +542,7 @@ fluentd:
522542
#"Third Key": "Third Value"
523543
#ociLALogGroupID:
524544
#worker:
545+
525546
cloudcontrollermanager:
526547
cwLogStreamName: "cloud-controller-manager"
527548
sqsQueue: "cloud-controller-manager"
@@ -536,6 +557,7 @@ fluentd:
536557
#"Third Key": "Third Value"
537558
#ociLALogGroupID:
538559
#worker:
560+
539561
scheduler:
540562
cwLogStreamName: "kube-scheduler"
541563
sqsQueue: "scheduler"
@@ -551,6 +573,7 @@ fluentd:
551573
#ociLALogGroupID:
552574
#worker:
553575

576+
554577
# Generic configuration for all container/pod logs
555578
genericContainerLogs:
556579
# -- Default Logging Analytics log source to use for parsing and processing the logs: Kubernetes Container Generic Logs.
@@ -574,6 +597,8 @@ fluentd:
574597
- '"/var/log/containers/kube-scheduler-*.log"'
575598
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
576599
#worker: 1
600+
# -- To set timezone override for genericContainerLogs (applies only to log records without explicit timezone identifier in the record itself)
601+
#timezone:
577602

578603
# -- Configuration for any custom logs which are not part of the default configuration defined in this file.
579604
# All the pod/container logs will be collected as per "genericContainerLogs" section.
@@ -598,6 +623,8 @@ fluentd:
598623
#encoding:
599624
# Worker number in case of multi process workers enabled. If not set when multi process workers enabled, then it defaults to 0.
600625
#worker:
626+
# -- To set timezone override for "custom-id1" (applies only to log records without explicit timezone identifier in the record itself)
627+
#timezone:
601628
#custom-id2:
602629
#path: /var/log/custom/*.log
603630
# Logging Analytics log source to use for parsing and processing the logs:

‎charts/oci-onm/Chart.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ type: application
1818
# This is the chart version. This version number should be incremented each time you make changes
1919
# to the chart and its templates, including the app version.
2020
# Versions are expected to follow Semantic Versioning (https://semver.org/)
21-
version: 3.5.0
21+
version: 3.5.1
2222

2323
# This is the version number of the application being deployed. This version number should be
2424
# incremented each time you make changes to the application. Versions are not expected to
@@ -32,7 +32,7 @@ dependencies:
3232
repository: "file://../common"
3333
condition: oci-onm-common.enabled
3434
- name: oci-onm-logan
35-
version: "3.5.0"
35+
version: "3.5.1"
3636
repository: "file://../logan"
3737
condition: oci-onm-logan.enabled
3838
- name: oci-onm-mgmt-agent

‎charts/oci-onm/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ oci-onm-logan:
3131
kubernetesClusterID: "{{ .Values.global.kubernetesClusterID }}"
3232
kubernetesClusterName: "{{ .Values.global.kubernetesClusterName }}"
3333
image:
34-
url: container-registry.oracle.com/oci_observability_management/oci-la-fluentd-collector:1.5.0
34+
url: container-registry.oracle.com/oci_observability_management/oci-la-fluentd-collector:1.5.1
3535
# Go to OCI Logging Analytics Administration, click Service Details, and note the namespace value.
3636
ociLANamespace:
3737
# OCI Logging Analytics Default Log Group OCID

‎docs/FAQ.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,65 @@ oci-onm-logan:
154154
encoding: <ENCODING-VALUE>
155155
```
156156

157+
### How to set timezone override ?
158+
159+
If a log record contains a timezone identifier, the **Logging Analytics service** will use that timezone. However, if there is no timezone information, the service defaults to **UTC**.
160+
161+
To override this default, use the `timezone` parameter in your `values.yaml` file. This parameter can be configured at different levels.
162+
163+
#### timezone override
164+
165+
**Note:** If a log record already has a timezone identifier, this setting may not be applicable.
166+
167+
* Setting `oci-onm-logan.fluentd.timezone` to **PST** applies PST as the default timezone for all logs collected via the Fluentd agent.
168+
* Setting `oci-onm-logan.fluentd.genericContainerLogs.timezone` to **IST** applies IST as the default timezone specifically for generic container logs.
169+
170+
171+
```
172+
..
173+
..
174+
oci-onm-logan:
175+
fluentd:
176+
timezone: <Set default timezone for all logs collected via fluentd agent>
177+
...
178+
...
179+
180+
kubernetesSystem:
181+
timezone: <Set default timezone for all Kubernetes System logs>
182+
logs:
183+
kube-proxy:
184+
timezone: <Set default timezone for kube-proxy logs>
185+
...
186+
...
187+
188+
linuxSystem:
189+
logs:
190+
cronlog:
191+
timezone: <Set default timezone for cron logs>
192+
...
193+
...
194+
195+
eksControlPlane:
196+
logs:
197+
apiserver:
198+
timezone: <Set default timezone for EKS API server logs>
199+
...
200+
...
201+
202+
203+
genericContainerLogs:
204+
timezone: <Set default timezone for generic container logs>
205+
...
206+
...
207+
208+
customLogs:
209+
custom-log-1:
210+
timezone: <Set default timezone for custom logs>
211+
...
212+
...
213+
```
214+
215+
#### Specific log level
157216
### How to use Configfile based AuthZ (User Principal) instead of default AuthZ (Instance Principal) ?
158217

159218
**Note**: This is supported only through the helm chart based deployment.

0 commit comments

Comments
 (0)