Skip to content

Commit 746aa72

Browse files
santhoshkvudaSanthosh Kumar Vuda
santhoshkvuda
and
Santhosh Kumar Vuda
authored
Annotations support (#4)
* Pod Annotations based customiation of configuration paremetersfor logs collected through "Kubernetes Container Generic Logs" Co-authored-by: Santhosh Kumar Vuda <[email protected]>
1 parent 2bca3e7 commit 746aa72

File tree

5 files changed

+278
-91
lines changed

5 files changed

+278
-91
lines changed

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Change Log
22

3+
## 2022-04-20
4+
### Added
5+
- Pod Annotations based customiation of configuration paremeters (oci_la_log_source_name, oci_la_log_group_id, oci_la_entity_id) for logs collected through "Kubernetes Container Generic Logs".
6+
- README update for custom configuration documentation.
7+
- Flush interval and timeout label configuration for Concat plugin section.
8+
39
## 2022-02-24
410
### Added
511
- Oracle Linux 8 based Docker Image support.

‎README.md‎

Lines changed: 67 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,10 +125,11 @@ These yaml files needs to be applied using kubectl to create the necessary resou
125125
- Use configmap-docker.yaml for Kubernetes clusters based off Docker runtime (e.g., OKE < 1.20) and configmap-cri.yaml for Kubernetes clusters based off CRI-O.
126126
- Inline comments are available in the file for each of the source/filter/match blocks for easy reference for making any changes to the configuration.
127127
- Refer [this](https://docs.oracle.com/en/learn/oci_logging_analytics_fluentd/) to learn about each of the Logging Analytics Fluentd Output plugin configuration parameters.
128-
- *Note*: A generic source with time only parser is defined/configured for collecting all application pod logs from /var/log/containers/ out of the box.
128+
- **Note**: A generic source with time only parser is defined/configured for collecting all application pod logs from /var/log/containers/ out of the box.
129129
It is recommended to define and use a LogSource/LogParser at Logging Analytics for a given log type and then modify the configuration accordingly.
130130
When adding a configuration (Source, Filter section) for any new container log, also exclude the log path from generic log collection,
131131
by adding the log path to *exclude_path* field in *in_tail_containerlogs* source block. This is to avoid the duplicate collection of logs through generic log collection.
132+
Refer [this](#custom-configuration) section for further details.
132133
133134
##### fluentd-daemonset.yaml
134135
@@ -249,3 +250,68 @@ Use the following `helm uninstall` command to delete the chart. Provide the rele
249250
helm uninstall <release-name>
250251
```
251252
253+
## Custom Configuration
254+
255+
### How to use custom logSource (oci_la_log_source_name) and/or other custom configuration for Pod/Container Logs collected through "Kubernetes Container Generic Logs" logSource ?
256+
257+
A generic source with time only parser is defined/configured for collecting all application pod logs from /var/log/containers/ out of the box.
258+
This is to ensure that all the logs generated by all pods are collected and pushed to Logging Analytics.
259+
Often you may need to configure a custom logSource for a particular pod log, either by using one of the existing OOB logSources at Logging Analytics or by defining one custom logSource matching to the requirements.
260+
Once you have defined/identified a logSource for a particular pod log, the following are couple of ways to get those pod logs associated to the logSource.
261+
262+
#### Through Pod Annotations
263+
264+
In this approach, all that you need to do is add the following annotation, "oracle.com/oci_la_log_source_name" (with logSourceName as value) to all the pods of choice.
265+
This approach works for all the use-cases, except for multi-line plain text formatted logs.
266+
267+
- Refer [this doc](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) to find how to add the annotation through Pod's metadata section. This is the recommended approach as it provides the persistent behavior.
268+
- Refer [this doc](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#annotate) to find how to add annotation through 'kubectl annotate' command. You may use this approach for quick testing.
269+
270+
**Note** The following configuration parameters are supported for customisation through Pod Annotations in addition to logSource,
271+
- oracle.com/oci_la_log_group_id => to use custom logGroupId (oci_la_log_group_id)
272+
- oracle.com/oci_la_entity_id => to use custom entityId (oci_la_entity_id)
273+
274+
#### Through Custom Fluentd conf
275+
276+
In this approach, a new set of Source, Filter sections have to be created in the fluentd config.
277+
The following example demonstrates a custom fluentd config to tag /var/log/containers/frontend*.log with logSource "Guestbook Frontend Logs"
278+
(*to be added to helm-chart values.yaml, under fluentd:configMapLogsFiles:kubernetes.conf value if using helm chart OR to either of configmap-cri.yaml / configmap-docker.yaml if using kubectl approach).
279+
280+
```
281+
<source>
282+
@type tail
283+
@id in_tail_frontend
284+
path_key tailed_path
285+
path /var/log/containers/frontend-*.log
286+
pos_file /var/log/oci_la_fluentd_outplugin/pos/frontend.logs.pos
287+
tag oci.oke.frontend.*
288+
read_from_head "#{ENV['FLUENT_OCI_READ_FROM_HEAD'] || true}"
289+
<parse>
290+
{{- if eq $runtime "docker" }}
291+
@type json
292+
{{- else}}
293+
@type cri
294+
{{- end }}
295+
</parse>
296+
</source>
297+
298+
# Record transformer filter to apply Logging Analytics configuration to each record.
299+
<filter oci.oke.frontend.**>
300+
@type record_transformer
301+
enable_ruby true
302+
<record>
303+
oci_la_metadata ${{"{{"}}"Kubernetes Cluster Name": "#{ENV['FLUENT_OCI_KUBERNETES_CLUSTER_NAME'] || 'UNDEFINED'}", "Kubernetes Cluster ID": "#{ENV['FLUENT_OCI_KUBERNETES_CLUSTER_ID'] || 'UNDEFINED'}"{{"}}"}}
304+
oci_la_log_group_id "#{ENV['FLUENT_OCI_KUBERNETES_LOGGROUP_ID'] || ENV['FLUENT_OCI_DEFAULT_LOGGROUP_ID']}"
305+
oci_la_log_path "${record['tailed_path']}"
306+
oci_la_log_source_name "Guestbook Frontend Logs"
307+
{{- if eq $runtime "docker" }}
308+
message "${record['log']}"
309+
{{- end }}
310+
tag ${tag}
311+
</record>
312+
</filter>
313+
```
314+
**Note**: The log path */var/log/containers/frontend-*.log* has to be excluded from the generic container logs to avoid duplicate log collection. Add the log path to *exclude_path* value under *in_tail_containerlogs* source section.
315+
316+
In addition to the above, you may need to modify the source section to add *multiline* parser, if the logs are of plain text multi-line format (OR) add a concat plugin filter if the logs are of say multi-line but wrapped in json.
317+
Refer OOB fluentd config in the helm-cahrt values.yaml for examples.

‎logan/helm-chart/values.yaml‎

Lines changed: 69 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -98,39 +98,49 @@ fluentd:
9898
skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
9999
skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
100100
watch "#{ENV['FLUENT_KUBERNETES_WATCH'] || 'true'}"
101+
de_dot false
102+
annotation_match [ ".*" ]
101103
</filter>
102104
103-
# Match block to set info required for oci-logging-analytics fluentd output plugin
105+
# Match block to ensure all the logs including concat plugin timeout logs will have same label
104106
<match oci.**>
105-
@type oci-logging-analytics
106-
enable_ruby true
107-
namespace "#{ENV['FLUENT_OCI_NAMESPACE']}"
108-
endpoint "#{ENV['FLUENT_OCI_ENDPOINT']}"
109-
{{- if eq $authtype "config" }}
110-
config_file_location "#{ENV['FLUENT_OCI_CONFIG_LOCATION'] || '/var/opt/.oci/config'}"
111-
profile_name "#{ENV['FLUENT_OCI_PROFILE'] || 'DEFAULT'}"
112-
{{- end }}
113-
plugin_log_location "#{ENV['FLUENT_OCI_LOG_LOCATION'] || '/var/log/'}"
114-
plugin_log_level "#{ENV['FLUENT_OCI_LOG_LEVEL'] || 'info'}"
115-
plugin_log_file_size "#{ENV['FLUENT_OCI_LOG_FILE_SIZE'] || '10MB'}"
116-
plugin_log_file_count "#{ENV['FLUENT_OCI_LOG_FILE_COUNT'] || 10}"
117-
kubernetes_metadata_keys_mapping {"container_name":"Container","namespace_name":"Namespace","pod_name":"Pod","container_image":"Container Image Name","host":"Node"}
118-
<buffer>
119-
@type file
120-
path "#{ENV['FLUENT_OCI_BUFFER_PATH'] || '/var/log/oci_la_fluentd_outplugin/logs/buffer/'}"
121-
flush_thread_count "#{ENV['FLUENT_OCI_FLUSH_THREAD_COUNT'] || 1}"
122-
chunk_limit_size "#{ENV['FLUENT_OCI_CHUNK_LIMIT_SIZE'] || '2m'}" # 2MB
123-
total_limit_size "#{ENV['FLUENT_OCI_TOTAL_LIMIT_SIZE'] || 5368709120}" # 5GB
124-
flush_interval "#{ENV['FLUENT_OCI_FLUSH_INTERVAL'] || 30}" # seconds
125-
flush_thread_interval "#{ENV['FLUENT_OCI_FLUSH_THREAD_INTERVAL'] || 0.5}"
126-
flush_thread_burst_interval "#{ENV['FLUENT_OCI_FLUSH_THREAD_BURST_INTERVAL'] || 0.05}"
127-
retry_wait "#{ENV['FLUENT_OCI_RETRY_WAIT'] || 2}" # seconds
128-
retry_max_times "#{ENV['FLUENT_OCI_RETRY_MAX_TIMES'] || 17}"
129-
retry_exponential_backoff_base "#{ENV['FLUENT_OCI_RETRY_EXPONENTIAL_BACKOFF_BASE'] || 2}"
130-
retry_forever "#{ENV['FLUENT_OCI_RETRY_FOREVER'] || true}"
131-
disable_chunk_backup true
132-
</buffer>
107+
@type relabel
108+
@label @NORMAL
133109
</match>
110+
111+
# Match block to set info required for oci-logging-analytics fluentd output plugin
112+
<label @NORMAL>
113+
<match oci.**>
114+
@type oci-logging-analytics
115+
enable_ruby true
116+
namespace "#{ENV['FLUENT_OCI_NAMESPACE']}"
117+
endpoint "#{ENV['FLUENT_OCI_ENDPOINT']}"
118+
{{- if eq $authtype "config" }}
119+
config_file_location "#{ENV['FLUENT_OCI_CONFIG_LOCATION'] || '/var/opt/.oci/config'}"
120+
profile_name "#{ENV['FLUENT_OCI_PROFILE'] || 'DEFAULT'}"
121+
{{- end }}
122+
plugin_log_location "#{ENV['FLUENT_OCI_LOG_LOCATION'] || '/var/log/'}"
123+
plugin_log_level "#{ENV['FLUENT_OCI_LOG_LEVEL'] || 'info'}"
124+
plugin_log_file_size "#{ENV['FLUENT_OCI_LOG_FILE_SIZE'] || '10MB'}"
125+
plugin_log_file_count "#{ENV['FLUENT_OCI_LOG_FILE_COUNT'] || 10}"
126+
kubernetes_metadata_keys_mapping {"container_name":"Container","namespace_name":"Namespace","pod_name":"Pod","container_image":"Container Image Name","host":"Node"}
127+
<buffer>
128+
@type file
129+
path "#{ENV['FLUENT_OCI_BUFFER_PATH'] || '/var/log/oci_la_fluentd_outplugin/logs/buffer/'}"
130+
flush_thread_count "#{ENV['FLUENT_OCI_FLUSH_THREAD_COUNT'] || 1}"
131+
chunk_limit_size "#{ENV['FLUENT_OCI_CHUNK_LIMIT_SIZE'] || '2m'}" # 2MB
132+
total_limit_size "#{ENV['FLUENT_OCI_TOTAL_LIMIT_SIZE'] || 5368709120}" # 5GB
133+
flush_interval "#{ENV['FLUENT_OCI_FLUSH_INTERVAL'] || 30}" # seconds
134+
flush_thread_interval "#{ENV['FLUENT_OCI_FLUSH_THREAD_INTERVAL'] || 0.5}"
135+
flush_thread_burst_interval "#{ENV['FLUENT_OCI_FLUSH_THREAD_BURST_INTERVAL'] || 0.05}"
136+
retry_wait "#{ENV['FLUENT_OCI_RETRY_WAIT'] || 2}" # seconds
137+
retry_max_times "#{ENV['FLUENT_OCI_RETRY_MAX_TIMES'] || 17}"
138+
retry_exponential_backoff_base "#{ENV['FLUENT_OCI_RETRY_EXPONENTIAL_BACKOFF_BASE'] || 2}"
139+
retry_forever "#{ENV['FLUENT_OCI_RETRY_FOREVER'] || true}"
140+
disable_chunk_backup true
141+
</buffer>
142+
</match>
143+
</label>
134144
# kubernetes config file data which is included in main fluentd config file.
135145
kubernetes.conf: |
136146
{{- $runtime := .Values.runtime | lower }}
@@ -181,6 +191,8 @@ fluentd:
181191
@type concat
182192
key message
183193
stream_identity_key stream
194+
flush_interval "#{ENV['FLUENT_CONCAT_FLUSH_INTERVAL'] || 60}" # seconds
195+
timeout_label "@NORMAL"
184196
multiline_start_regexp /^\S\d{2}\d{2}\s+[^\:]+:[^\:]+:[^\.]+\.\d{0,3}/
185197
</filter>
186198
@@ -224,6 +236,8 @@ fluentd:
224236
@type concat
225237
key message
226238
stream_identity_key stream
239+
flush_interval "#{ENV['FLUENT_CONCAT_FLUSH_INTERVAL'] || 60}" # seconds
240+
timeout_label "@NORMAL"
227241
multiline_start_regexp /^\S\d{2}\d{2}\s+[^\:]+:[^\:]+:[^\.]+\.\d{0,3}/
228242
</filter>
229243
@@ -267,6 +281,8 @@ fluentd:
267281
@type concat
268282
key message
269283
stream_identity_key stream
284+
flush_interval "#{ENV['FLUENT_CONCAT_FLUSH_INTERVAL'] || 60}" # seconds
285+
timeout_label "@NORMAL"
270286
multiline_start_regexp /^\S\d{2}\d{2}\s+[^\:]+:[^\:]+:[^\.]+\.\d{0,3}/
271287
</filter>
272288
@@ -310,6 +326,8 @@ fluentd:
310326
@type concat
311327
key message
312328
stream_identity_key stream
329+
flush_interval "#{ENV['FLUENT_CONCAT_FLUSH_INTERVAL'] || 60}" # seconds
330+
timeout_label "@NORMAL"
313331
multiline_start_regexp /^\[[^\]]+\]\s+/
314332
</filter>
315333
@@ -423,6 +441,8 @@ fluentd:
423441
@type concat
424442
key message
425443
stream_identity_key stream
444+
flush_interval "#{ENV['FLUENT_CONCAT_FLUSH_INTERVAL'] || 60}" # seconds
445+
timeout_label "@NORMAL"
426446
multiline_start_regexp /^\S\d{2}\d{2}\s+[^\:]+:[^\:]+:[^\.]+\.\d{0,3}/
427447
</filter>
428448
@@ -681,15 +701,33 @@ fluentd:
681701
</parse>
682702
</source>
683703
704+
# Filter to add kubernetes metadata
705+
<filter oci.oke.containerlogs.**>
706+
@type kubernetes_metadata
707+
@id filter_kube_metadata_containerlogs
708+
kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
709+
verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
710+
ca_file "#{ENV['KUBERNETES_CA_FILE']}"
711+
skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
712+
skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
713+
skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
714+
skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
715+
watch "#{ENV['FLUENT_KUBERNETES_WATCH'] || 'true'}"
716+
de_dot false
717+
annotation_match [ ".*" ]
718+
</filter>
719+
684720
# Record transformer filter to apply Logging Analytics configuration to each record.
685721
<filter oci.oke.containerlogs.**>
686722
@type record_transformer
687723
enable_ruby true
688724
<record>
689725
oci_la_metadata ${{"{{"}}"Kubernetes Cluster Name": "#{ENV['FLUENT_OCI_KUBERNETES_CLUSTER_NAME'] || 'UNDEFINED'}", "Kubernetes Cluster ID": "#{ENV['FLUENT_OCI_KUBERNETES_CLUSTER_ID'] || 'UNDEFINED'}"{{"}}"}}
690-
oci_la_log_group_id "#{ENV['FLUENT_OCI_KUBERNETES_LOGGROUP_ID'] || ENV['FLUENT_OCI_DEFAULT_LOGGROUP_ID']}"
726+
oci_la_log_group_id ${record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_group_id") ? record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_group_id") : "#{ENV['FLUENT_OCI_KUBERNETES_LOGGROUP_ID'] || ENV['FLUENT_OCI_DEFAULT_LOGGROUP_ID']}"}
691727
oci_la_log_path "${record['tailed_path']}"
692-
oci_la_log_source_name "Kubernetes Container Generic Logs"
728+
oci_la_log_source_name ${record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_source_name") ? record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_source_name") : "Kubernetes Container Generic Logs"}
729+
oci_la_entity_id ${record.dig("kubernetes", "annotations", "oracle.com/oci_la_entity_id") ? record.dig("kubernetes", "annotations", "oracle.com/oci_la_entity_id") : ""}
730+
oci_la_log_set ${record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_set") ? record.dig("kubernetes", "annotations", "oracle.com/oci_la_log_set") : ""}
693731
{{- if eq $runtime "docker" }}
694732
message "${record['log']}"
695733
{{- end }}

0 commit comments

Comments
 (0)