Skip to content

Commit 33b1b6a

Browse files
wanlonghenrypfrcksAmol Agrawal
authored
Containerlogv2 Metadata and annotations based filtering (#1148)
* add fluent-bit kubernetes filter (#1115) * add fluent-bit kubernetes filter --------- Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * add metadata feilds including labels, annotations, uid and image (#1120) * add metadata feilds including labels, annotations, uid and image * set the include_fields if customer set it otherwise go with default if enabled * fix minor bugs * change placeholder * add exclude specific kubernetes config * update tls verify for linux * make log filtering opt in explicit (#1140) Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * Longw/metadata containerlogv2 kubernetes (#1139) * add metadata feilds including labels, annotations, uid and image * rename to podUid and add nil check * set the include_fields if customer set it otherwise go with default if enabled * add workload to testing clusters * containerlogv2 metadata backend change with feature flag on * adjust the marshal and log the output * address comments add check and scenario for empty list * remove ADX support * go fmt for better format * fix space in tomlparser.rb * update trivy ignore * fix Kube_Tag_Prefix for windows * update trivy * add ttl for metadata cache * update containerlogv2 test workloads * change typo and update tests * Longw/metadata containerlogv2 address feedback (#1166) * add argument for sendMetric * update SendMetric * update SendMetric * Longw/metadata containerlogv2 address feedback2 (#1186) * extend to support more configs on metadata * add warning message if feilds not match * add plugin for geneva path * add kubernetesMetadataCollection for geneva path * Longw/metadata containerlogv2 address feedback2 (#1188) * remove send metric * update trivy * Longw/metadata containerlogv2 address feedback3 (#1195) * update doc for containerlogv2 linux * add kube_meta_cache_ttl to agent settings * update logic to adjust colonLocation as the end of the list if it is not found * address comments and feedback * update conf file for geneva * adjust geneva and add new metrics * address feedback for geneva path * add telemetry in go * address comments for geneva conf filter config --------- Co-authored-by: Amol Agrawal <pfrcks@gmail.com> Co-authored-by: Amol Agrawal <amagraw@microsoft.com>
1 parent 83946e7 commit 33b1b6a

24 files changed

+552
-32
lines changed

‎.trivyignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ CVE-2024-28110
1111
CVE-2024-27289
1212
CVE-2024-27304
1313
GHSA-7jwh-3vrq-q3m8
14-
CVE-2024-27304
1514
CVE-2024-24786
1615

1716
#telegraf HIGH
@@ -29,3 +28,4 @@ CVE-2023-5678
2928

3029
#golang MEDIUM
3130
CVE-2023-48795
31+
CVE-2024-24786

‎Documentation/Internal/ContainerLogV2-Linux.xml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<?xml version="1.0" encoding="utf-8"?>
2-
<MonitoringManagement version="1.0" namespace="<NamespaceForLinuxContainers>" eventVersion="1" timestamp="2016-01-20T00:00:00.000">
2+
<MonitoringManagement version="1.1" namespace="<NamespaceForLinuxContainers>" eventVersion="1" timestamp="2016-01-20T00:00:00.000">
33
<Accounts>
44
<Account moniker="<GenevaLogsAccountMoniker>" isDefault="true" />
55
</Accounts>
@@ -25,6 +25,7 @@
2525
<Column name="Computer" type="str" mdstype="mt:wstr" />
2626
<Column name="ContainerName" type="str" mdstype="mt:wstr" />
2727
<Column name="AzureResourceId" type="str" mdstype="mt:wstr" />
28+
<Column name="KubernetesMetadata" type="str" mdstype="mt:wstr" />
2829
</Schema>
2930
</Schemas>
3031
<Sources>

‎build/common/installer/scripts/fluent-bit-conf-customizer.rb‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,8 @@ def substituteFluentBitPlaceHolders
7777
stacktraceLanguages = ENV["AZMON_MULTILINE_LANGUAGES"]
7878
resourceOptimizationEnabled = ENV["AZMON_RESOURCE_OPTIMIZATION_ENABLED"]
7979
windowsFluentBitDisabled = ENV["AZMON_WINDOWS_FLUENT_BIT_DISABLED"]
80+
kubernetesMetadataCollection = ENV["AZMON_KUBERNETES_METADATA_ENABLED"]
81+
annotationBasedLogFiltering = ENV["AZMON_ANNOTATION_BASED_LOG_FILTERING"]
8082

8183
serviceInterval = (!interval.nil? && is_number?(interval) && interval.to_i > 0) ? interval : @default_service_interval
8284
serviceIntervalSetting = "Flush " + serviceInterval
@@ -113,6 +115,16 @@ def substituteFluentBitPlaceHolders
113115
new_contents = new_contents.gsub("\n ${TAIL_IGNORE_OLDER}\n", "\n")
114116
end
115117

118+
if !kubernetesMetadataCollection.nil? && kubernetesMetadataCollection.to_s.downcase == "true"
119+
new_contents = new_contents.gsub("#${KubernetesFilterEnabled}", "")
120+
end
121+
122+
if !annotationBasedLogFiltering.nil? && annotationBasedLogFiltering.to_s.downcase == "true"
123+
# enabled kubernetes filter plugin if not already enabled
124+
new_contents = new_contents.gsub("#${KubernetesFilterEnabled}", "")
125+
new_contents = new_contents.gsub("#${AnnotationBasedLogFilteringEnabled}", "")
126+
end
127+
116128
new_contents = substituteMultiline(multilineLogging, stacktraceLanguages, new_contents)
117129

118130
if !@isWindows || (@isWindows && (!windowsFluentBitDisabled.nil? && windowsFluentBitDisabled.to_s.downcase == "false"))

‎build/common/installer/scripts/fluent-bit-geneva-conf-customizer.rb‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,19 @@
44
LINUX_CONFIG_PATHS = {
55
"common" => "/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva.conf",
66
"infra" => "/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_infra.conf",
7+
"infra_filter" => "/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_infra_filter.conf",
78
"tenant" => "/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_tenant.conf",
9+
"tenant_filter" => "/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_tenant_filter.conf",
810
}
911

1012
WINDOWS_CONFIG_PATHS = {
1113
"common" => "/etc/fluent-bit/fluent-bit-geneva.conf",
1214
"infra" => "/etc/fluent-bit/fluent-bit-geneva-logs_infra.conf",
15+
"infra_filter" => "/etc/fluent-bit/fluent-bit-geneva-logs_infra_filter.conf",
1316
"tenant" => "/etc/fluent-bit/fluent-bit-geneva-logs_tenant.conf",
17+
"tenant_filter" => "/etc/fluent-bit/fluent-bit-geneva-logs_tenant_filter.conf",
1418
}
15-
SUPPORTED_CONFIG_TYPES = ["common", "infra", "tenant"]
19+
SUPPORTED_CONFIG_TYPES = ["common", "infra", "tenant", "infra_filter", "tenant_filter"]
1620

1721
@default_service_interval = "15"
1822
@default_mem_buf_limit = "10"
@@ -40,6 +44,8 @@ def substituteFluentBitPlaceHolders(configFilePath)
4044
multilineLogging = ENV["AZMON_MULTILINE_ENABLED"]
4145
stacktraceLanguages = ENV["AZMON_MULTILINE_LANGUAGES"]
4246
enableFluentBitThreading = ENV["ENABLE_FBIT_THREADING"]
47+
kubernetesMetadataCollection = ENV["AZMON_KUBERNETES_METADATA_ENABLED"]
48+
annotationBasedLogFiltering = ENV["AZMON_ANNOTATION_BASED_LOG_FILTERING"]
4349

4450
serviceInterval = is_valid_number?(interval) ? interval : @default_service_interval
4551
serviceIntervalSetting = "Flush " + serviceInterval
@@ -82,6 +88,16 @@ def substituteFluentBitPlaceHolders(configFilePath)
8288
new_contents = new_contents.gsub("\n ${TAIL_THREADED}\n", "\n")
8389
end
8490

91+
if !kubernetesMetadataCollection.nil? && kubernetesMetadataCollection.to_s.downcase == "true"
92+
new_contents = new_contents.gsub("#${KubernetesFilterEnabled}", "")
93+
end
94+
95+
if !annotationBasedLogFiltering.nil? && annotationBasedLogFiltering.to_s.downcase == "true"
96+
# enabled kubernetes filter plugin if not already enabled
97+
new_contents = new_contents.gsub("#${KubernetesFilterEnabled}", "")
98+
new_contents = new_contents.gsub("#${AnnotationBasedLogFilteringEnabled}", "")
99+
end
100+
85101
if !multilineLogging.nil? && multilineLogging.to_s.downcase == "true"
86102
if !stacktraceLanguages.nil? && !stacktraceLanguages.empty?
87103
new_contents = new_contents.gsub("#${MultilineEnabled}", "")

‎build/common/installer/scripts/tomlparser-common-agent-config.rb‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
@configSchemaVersion = ""
1010

1111
@disableTelemetry = false
12+
@logEnableKubernetesMetadataCacheTTLSeconds = 60
1213

1314
def is_windows?
1415
return !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0
@@ -57,6 +58,20 @@ def populateSettingValuesFromConfigMap(parsedConfig)
5758
puts "Using config map value: disable_telemetry = #{@disableTelemetry}"
5859
end
5960
end
61+
62+
# agent config for kube_meta_cache_ttl_secs which used in containerlogv2 kubernetes metadata and annotation based filtering
63+
if !parsedConfig.nil? && !parsedConfig[:agent_settings].nil?
64+
k8s_metadata_config = parsedConfig[:agent_settings][:k8s_metadata_config]
65+
if !k8s_metadata_config.nil? && !k8s_metadata_config[:kube_meta_cache_ttl_secs].nil?
66+
ttl_value = k8s_metadata_config[:kube_meta_cache_ttl_secs]
67+
if ttl_value.is_a?(Integer) && ttl_value >= 0
68+
@logEnableKubernetesMetadataCacheTTLSeconds = ttl_value
69+
puts "config::INFO: Using config map value: kube_meta_cache_ttl_secs = #{@logEnableKubernetesMetadataCacheTTLSeconds}"
70+
else
71+
puts "config::WARN: Using the default value for kube_meta_cache_ttl_secs since provided config value is invalid"
72+
end
73+
end
74+
end
6075
rescue => errorStr
6176
puts "config::error:Exception while reading config settings for agent configuration setting - #{errorStr}, using defaults"
6277
end
@@ -87,6 +102,8 @@ def get_command_windows(env_variable_name, env_variable_value)
87102
commands = get_command_windows("DISABLE_TELEMETRY", @disableTelemetry)
88103
file.write(commands)
89104
end
105+
commands = get_command_windows("AZMON_KUBERNETES_METADATA_CACHE_TTL_SECONDS", @logEnableKubernetesMetadataCacheTTLSeconds)
106+
file.write(commands)
90107
# Close file after writing all environment variables
91108
file.close
92109
puts "****************End Config Processing********************"
@@ -101,7 +118,7 @@ def get_command_windows(env_variable_name, env_variable_value)
101118
if @disableTelemetry
102119
file.write("export DISABLE_TELEMETRY=#{@disableTelemetry}\n")
103120
end
104-
121+
file.write("export AZMON_KUBERNETES_METADATA_CACHE_TTL_SECONDS=#{@logEnableKubernetesMetadataCacheTTLSeconds}\n")
105122
# Close file after writing all environment variables
106123
file.close
107124
else

‎build/common/installer/scripts/tomlparser.rb‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@
2525
@adxDatabaseName = "containerinsights" # default for all configurations
2626
@logEnableMultiline = "false"
2727
@stacktraceLanguages = "go,java,python" #supported languages for multiline logs. java is also used for dotnet stacktraces
28+
@logEnableKubernetesMetadata = false
29+
@logKubernetesMetadataIncludeFields = "podlabels,podannotations,poduid,image,imageid,imagerepo,imagetag"
30+
@annotationBasedLogFiltering = false
2831
if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0
2932
@containerLogsRoute = "v1" # default is v1 for windows until windows agent integrates windows ama
3033
# This path format is necessary for fluent-bit in windows
@@ -228,6 +231,44 @@ def populateSettingValuesFromConfigMap(parsedConfig)
228231
rescue => errorStr
229232
ConfigParseErrorLogger.logError("Exception while reading config map settings for adx database name - #{errorStr}, using default #{@adxDatabaseName}, please check config map for errors")
230233
end
234+
235+
#Get Kubernetes Metadata setting
236+
begin
237+
if !parsedConfig[:log_collection_settings][:metadata_collection].nil? && !parsedConfig[:log_collection_settings][:metadata_collection][:enabled].nil?
238+
puts "config::INFO: Using config map setting for kubernetes metadata"
239+
@logEnableKubernetesMetadata = parsedConfig[:log_collection_settings][:metadata_collection][:enabled]
240+
if !parsedConfig[:log_collection_settings][:metadata_collection][:include_fields].nil?
241+
puts "config::INFO: Using config map setting for kubernetes metadata include fields"
242+
include_fields = parsedConfig[:log_collection_settings][:metadata_collection][:include_fields]
243+
if include_fields.empty?
244+
puts "config::WARN: Include fields specified for Kubernetes metadata is empty, disabling Kubernetes metadata"
245+
@logEnableKubernetesMetadata = false
246+
elsif include_fields.kind_of?(Array)
247+
include_fields.map!(&:downcase)
248+
predefined_fields = @logKubernetesMetadataIncludeFields.downcase.split(',')
249+
any_field_match = include_fields.any? { |field| predefined_fields.include?(field) }
250+
if any_field_match
251+
@logKubernetesMetadataIncludeFields = include_fields.join(",")
252+
else
253+
puts "config:: WARN: Include fields specified for Kubernetes metadata does not match any predefined fields, disabling Kubernetes metadata"
254+
@logEnableKubernetesMetadata = false
255+
end
256+
end
257+
end
258+
end
259+
rescue => errorStr
260+
ConfigParseErrorLogger.logError("config::error: Exception while reading config map settings for kubernetes metadata - #{errorStr}, please check config map for errors")
261+
end
262+
263+
#Get annotation based log filtering setting
264+
begin
265+
if !parsedConfig[:log_collection_settings][:filter_using_annotations].nil? && !parsedConfig[:log_collection_settings][:filter_using_annotations][:enabled].nil?
266+
puts "config::INFO: Using config map setting for annotation based log filtering"
267+
@annotationBasedLogFiltering = parsedConfig[:log_collection_settings][:filter_using_annotations][:enabled]
268+
end
269+
rescue => errorStr
270+
ConfigParseErrorLogger.logError("config::error: Exception while reading config map settings for annotation based log filtering - #{errorStr}, please check config map for errors")
271+
end
231272
end
232273
end
233274

@@ -276,6 +317,9 @@ def populateSettingValuesFromConfigMap(parsedConfig)
276317
file.write("export AZMON_ADX_DATABASE_NAME=#{@adxDatabaseName}\n")
277318
file.write("export AZMON_MULTILINE_ENABLED=#{@logEnableMultiline}\n")
278319
file.write("export AZMON_MULTILINE_LANGUAGES=#{@stacktraceLanguages}\n")
320+
file.write("export AZMON_KUBERNETES_METADATA_ENABLED=#{@logEnableKubernetesMetadata}\n")
321+
file.write("export AZMON_KUBERNETES_METADATA_INCLUDES_FIELDS=#{@logKubernetesMetadataIncludeFields}\n")
322+
file.write("export AZMON_ANNOTATION_BASED_LOG_FILTERING=#{@annotationBasedLogFiltering}\n")
279323
# Close file after writing all environment variables
280324
file.close
281325
puts "Both stdout & stderr log collection are turned off for namespaces: '#{@excludePath}' "
@@ -342,6 +386,12 @@ def get_command_windows(env_variable_name, env_variable_value)
342386
file.write(commands)
343387
commands = get_command_windows("AZMON_MULTILINE_LANGUAGES", @stacktraceLanguages)
344388
file.write(commands)
389+
commands = get_command_windows("AZMON_KUBERNETES_METADATA_ENABLED", @logEnableKubernetesMetadata)
390+
file.write(commands)
391+
commands = get_command_windows("AZMON_KUBERNETES_METADATA_INCLUDES_FIELDS", @logKubernetesMetadataIncludeFields)
392+
file.write(commands)
393+
commands = get_command_windows("AZMON_ANNOTATION_BASED_LOG_FILTERING", @annotationBasedLogFiltering)
394+
file.write(commands)
345395
# Close file after writing all environment variables
346396
file.close
347397
puts "Both stdout & stderr log collection are turned off for namespaces: '#{@excludePath}' "

‎build/linux/installer/conf/fluent-bit-geneva-logs_infra_filter.conf‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#${KubernetesFilterEnabled}[FILTER]
2+
#${KubernetesFilterEnabled} Name kubernetes
3+
#${KubernetesFilterEnabled} Alias geneva_kubernetes_filter
4+
#${KubernetesFilterEnabled} Match geneva.container.log.infra.*
5+
#${KubernetesFilterEnabled} Merge_Log Off
6+
#${KubernetesFilterEnabled} Buffer_Size 0
7+
#${KubernetesFilterEnabled} Use_Kubelet true
8+
#${KubernetesFilterEnabled} Kubelet_Port 10250
9+
#${KubernetesFilterEnabled} Kubelet_Host ${NODE_IP}
10+
#${KubernetesFilterEnabled} tls.verify Off
11+
#${KubernetesFilterEnabled}#${AnnotationBasedLogFilteringEnabled} K8S-Logging.Exclude On
12+
#${KubernetesFilterEnabled} Kube_Meta_Cache_TTL ${AZMON_KUBERNETES_METADATA_CACHE_TTL_SECONDS}
13+
#${KubernetesFilterEnabled} Kube_Tag_Prefix geneva.container.log.infra.var.log.containers.

‎build/linux/installer/conf/fluent-bit-geneva-logs_tenant_filter.conf‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#${KubernetesFilterEnabled}[FILTER]
2+
#${KubernetesFilterEnabled} Name kubernetes
3+
#${KubernetesFilterEnabled} Alias geneva_kubernetes_filter
4+
#${KubernetesFilterEnabled} Match geneva.container.log.tenant.*
5+
#${KubernetesFilterEnabled} Merge_Log Off
6+
#${KubernetesFilterEnabled} Buffer_Size 0
7+
#${KubernetesFilterEnabled} Use_Kubelet true
8+
#${KubernetesFilterEnabled} Kubelet_Port 10250
9+
#${KubernetesFilterEnabled} Kubelet_Host ${NODE_IP}
10+
#${KubernetesFilterEnabled} tls.verify Off
11+
#${KubernetesFilterEnabled}#${AnnotationBasedLogFilteringEnabled} K8S-Logging.Exclude On
12+
#${KubernetesFilterEnabled} Kube_Meta_Cache_TTL ${AZMON_KUBERNETES_METADATA_CACHE_TTL_SECONDS}
13+
#${KubernetesFilterEnabled} Kube_Tag_Prefix geneva.container.log.tenant.var.log.containers.

‎build/linux/installer/conf/fluent-bit.conf‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,5 +47,19 @@
4747
Match oms.container.log.la.*
4848
Exclude stream ${AZMON_LOG_EXCLUSION_REGEX_PATTERN}
4949

50+
#${KubernetesFilterEnabled}[FILTER]
51+
#${KubernetesFilterEnabled} Name kubernetes
52+
#${KubernetesFilterEnabled} Alias oms_kubernetes_filter
53+
#${KubernetesFilterEnabled} Match oms.container.log.la.*
54+
#${KubernetesFilterEnabled} Merge_Log Off
55+
#${KubernetesFilterEnabled} Buffer_Size 0
56+
#${KubernetesFilterEnabled} Use_Kubelet true
57+
#${KubernetesFilterEnabled} Kubelet_Port 10250
58+
#${KubernetesFilterEnabled} Kubelet_Host ${NODE_IP}
59+
#${KubernetesFilterEnabled} tls.verify Off
60+
#${KubernetesFilterEnabled}#${AnnotationBasedLogFilteringEnabled} K8S-Logging.Exclude On
61+
#${KubernetesFilterEnabled} Kube_Meta_Cache_TTL ${AZMON_KUBERNETES_METADATA_CACHE_TTL_SECONDS}
62+
#${KubernetesFilterEnabled} Kube_Tag_Prefix oms.container.log.la.var.log.containers.
63+
5064
@INCLUDE fluent-bit-common.conf
5165
@INCLUDE fluent-bit-internal-metrics.conf

‎build/linux/installer/datafiles/base_container.data‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,9 @@ MAINTAINER: 'Microsoft Corporation'
5050
/etc/opt/microsoft/docker-cimprov/fluent-bit-common.conf; build/linux/installer/conf/fluent-bit-common.conf; 644; root; root
5151
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva.conf; build/linux/installer/conf/fluent-bit-geneva.conf; 644; root; root
5252
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_infra.conf; build/linux/installer/conf/fluent-bit-geneva-logs_infra.conf; 644; root; root
53+
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_infra_filter.conf; build/linux/installer/conf/fluent-bit-geneva-logs_infra_filter.conf; 644; root; root
5354
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_tenant.conf; build/linux/installer/conf/fluent-bit-geneva-logs_tenant.conf; 644; root; root
55+
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-logs_tenant_filter.conf; build/linux/installer/conf/fluent-bit-geneva-logs_tenant_filter.conf; 644; root; root
5456
/etc/opt/microsoft/docker-cimprov/fluent-bit-geneva-telemetry-svc.conf; build/linux/installer/conf/fluent-bit-geneva-telemetry-svc.conf; 644; root; root
5557
/etc/opt/microsoft/docker-cimprov/fluent-bit-internal-metrics.conf; build/linux/installer/conf/fluent-bit-internal-metrics.conf; 644; root; root
5658

0 commit comments

Comments
 (0)