Skip to content

Commit ee6d26e

Browse files
FabianMeiswinkelFabianMeiswinkel
authored
Removing dependencies on Log4j (and as a necessary side effect stopping to export Log4J logs to Azuremonitor) (#46608)
* Removing Export of Log4J logs to AzureMonitor * Fixing build errors * Update external_dependencies.txt * Removed exclusions to log4j
1 parent 839381f commit ee6d26e

File tree

10 files changed

+2
-178
lines changed

10 files changed

+2
-178
lines changed

‎eng/versioning/external_dependencies.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -268,7 +268,6 @@ cosmos_com.microsoft.azure:applicationinsights-core;2.6.4
268268
cosmos_io.micrometer:micrometer-core;1.15.1
269269
cosmos_io.micrometer:micrometer-registry-azure-monitor;1.15.1
270270
cosmos_io.micrometer:micrometer-registry-graphite;1.15.1
271-
cosmos_io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17;2.15.0-alpha
272271

273272
# Cosmos Spark connector tests only
274273
cosmos_org.scalatest:scalatest_2.12;3.2.2

‎sdk/cosmos/azure-cosmos-spark_3-3_2-12/pom.xml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -164,10 +164,6 @@
164164
<groupId>org.slf4j</groupId>
165165
<artifactId>*</artifactId>
166166
</exclusion>
167-
<exclusion>
168-
<groupId>org.apache.logging.log4j</groupId>
169-
<artifactId>*</artifactId>
170-
</exclusion>
171167
</exclusions>
172168
<scope>provided</scope>
173169
</dependency>
@@ -184,10 +180,6 @@
184180
<groupId>org.slf4j</groupId>
185181
<artifactId>*</artifactId>
186182
</exclusion>
187-
<exclusion>
188-
<groupId>org.apache.logging.log4j</groupId>
189-
<artifactId>*</artifactId>
190-
</exclusion>
191183
</exclusions>
192184
<scope>test</scope>
193185
</dependency>

‎sdk/cosmos/azure-cosmos-spark_3-4_2-12/pom.xml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -164,10 +164,6 @@
164164
<groupId>org.slf4j</groupId>
165165
<artifactId>*</artifactId>
166166
</exclusion>
167-
<exclusion>
168-
<groupId>org.apache.logging.log4j</groupId>
169-
<artifactId>*</artifactId>
170-
</exclusion>
171167
</exclusions>
172168
<scope>provided</scope>
173169
</dependency>
@@ -184,10 +180,6 @@
184180
<groupId>org.slf4j</groupId>
185181
<artifactId>*</artifactId>
186182
</exclusion>
187-
<exclusion>
188-
<groupId>org.apache.logging.log4j</groupId>
189-
<artifactId>*</artifactId>
190-
</exclusion>
191183
</exclusions>
192184
<scope>test</scope>
193185
</dependency>

‎sdk/cosmos/azure-cosmos-spark_3-5_2-12/pom.xml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -164,10 +164,6 @@
164164
<groupId>org.slf4j</groupId>
165165
<artifactId>*</artifactId>
166166
</exclusion>
167-
<exclusion>
168-
<groupId>org.apache.logging.log4j</groupId>
169-
<artifactId>*</artifactId>
170-
</exclusion>
171167
</exclusions>
172168
<scope>provided</scope>
173169
</dependency>
@@ -184,10 +180,6 @@
184180
<groupId>org.slf4j</groupId>
185181
<artifactId>*</artifactId>
186182
</exclusion>
187-
<exclusion>
188-
<groupId>org.apache.logging.log4j</groupId>
189-
<artifactId>*</artifactId>
190-
</exclusion>
191183
</exclusions>
192184
<scope>test</scope>
193185
</dependency>

‎sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,6 @@
4141
| `spark.cosmos.diagnostics.azureMonitor.sampling.maxCount` | `10000` | The max. number of dependency events sampled-in every interval. |
4242
| `spark.cosmos.diagnostics.azureMonitor.sampling.intervalInSeconds` | `60` | The interval after which the max. number of events allowed to be sampled-in is reset. |
4343
| `spark.cosmos.diagnostics.azureMonitor.metrics.intervalInSeconds` | `60` | The interval (in seconds) defining how often Metrics are captured and emitted to Azure Monitor. A negative value results in disabling metric collection. |
44-
| `spark.cosmos.diagnostics.azureMonitor.log.level` | `INFO` | Determines which logs should be emitted to Azure Monitor. With the default `INFO` effectively all logs emitted to the built-in Spark logs are also emitted to Azure Monitor (unless the limit defined below is hit). |
45-
| `spark.cosmos.diagnostics.azureMonitor.log.sampling.maxCount` | `None` | The max. number of log records emitted to Azure Monitor every interval. |
46-
| `spark.cosmos.diagnostics.azureMonitor.log.sampling.intervalInSeconds` | `None` | The interval (in seconds) after which the max. number of log records allowed to be emitted to Azure Monitor is getting reset. |
4744

4845
### Additional Tuning
4946
| Config Property Name | Default | Description |

‎sdk/cosmos/azure-cosmos-spark_3_2-12/pom.xml‎

Lines changed: 0 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -133,26 +133,11 @@
133133
<artifactId>jackson-module-afterburner</artifactId>
134134
<version>2.18.4</version> <!-- {x-version-update;com.fasterxml.jackson.module:jackson-module-afterburner;external_dependency} -->
135135
</dependency>
136-
<dependency>
137-
<groupId>org.apache.logging.log4j</groupId>
138-
<artifactId>log4j-core</artifactId>
139-
<version>2.17.2</version> <!-- {x-version-update;org.apache.logging.log4j:log4j-core;external_dependency} -->
140-
</dependency>
141136
<dependency>
142137
<groupId>io.opentelemetry</groupId>
143138
<artifactId>opentelemetry-api</artifactId>
144139
<version>1.49.0</version> <!-- {x-version-update;io.opentelemetry:opentelemetry-api;external_dependency} -->
145140
</dependency>
146-
<dependency>
147-
<groupId>io.opentelemetry</groupId>
148-
<artifactId>opentelemetry-exporter-logging</artifactId>
149-
<version>1.49.0</version> <!-- {x-version-update;io.opentelemetry:opentelemetry-exporter-logging;external_dependency} -->
150-
</dependency>
151-
<dependency>
152-
<groupId>io.opentelemetry.instrumentation</groupId>
153-
<artifactId>opentelemetry-log4j-appender-2.17</artifactId>
154-
<version>2.15.0-alpha</version> <!-- {x-version-update;cosmos_io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17;external_dependency} -->
155-
</dependency>
156141
<dependency>
157142
<groupId>com.azure</groupId>
158143
<artifactId>azure-core-tracing-opentelemetry</artifactId>
@@ -238,18 +223,6 @@
238223
<version>1.7.36</version> <!-- {x-version-update;org.slf4j:slf4j-api;external_dependency} -->
239224
<scope>compile</scope>
240225
</dependency>
241-
<dependency>
242-
<groupId>org.apache.logging.log4j</groupId>
243-
<artifactId>log4j-slf4j-impl</artifactId>
244-
<version>2.17.2</version> <!-- {x-version-update;org.apache.logging.log4j:log4j-slf4j-impl;external_dependency} -->
245-
<scope>compile</scope>
246-
<exclusions>
247-
<exclusion>
248-
<groupId>org.slf4j</groupId>
249-
<artifactId>*</artifactId>
250-
</exclusion>
251-
</exclusions>
252-
</dependency>
253226
<dependency>
254227
<groupId>io.micrometer</groupId>
255228
<artifactId>micrometer-registry-azure-monitor</artifactId>
@@ -332,10 +305,6 @@
332305
<include>com.microsoft.azure:applicationinsights-core:[2.6.4]</include> <!-- {x-include-update;cosmos_com.microsoft.azure:applicationinsights-core;external_dependency} -->
333306
<include>com.globalmentor:hadoop-bare-naked-local-fs:[0.1.0]</include> <!-- {x-include-update;cosmos_com.globalmentor:hadoop-bare-naked-local-fs;external_dependency} -->
334307
<include>io.opentelemetry:opentelemetry-api:[1.49.0]</include> <!-- {x-include-update;io.opentelemetry:opentelemetry-api;external_dependency} -->
335-
<include>io.opentelemetry:opentelemetry-exporter-logging:[1.49.0]</include> <!-- {x-include-update;io.opentelemetry:opentelemetry-exporter-logging;external_dependency} -->
336-
<include>org.apache.logging.log4j:log4j-core:[2.17.2]</include> <!-- {x-include-update;org.apache.logging.log4j:log4j-core;external_dependency} -->
337-
<include>io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:[2.15.0-alpha]</include> <!-- {x-include-update;cosmos_io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17;external_dependency} -->
338-
<include>org.apache.logging.log4j:log4j-slf4j-impl:[2.17.2]</include> <!-- {x-include-update;org.apache.logging.log4j:log4j-slf4j-impl;external_dependency} -->
339308
</includes>
340309
</bannedDependencies>
341310
</rules>

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/AzureMonitorConfig.scala‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,6 @@
22
// Licensed under the MIT License.
33
package com.azure.cosmos.spark
44

5-
import com.azure.cosmos.{CosmosAsyncClient, CosmosClientBuilder, ReadConsistencyStrategy}
6-
import org.apache.logging.log4j.Level
7-
85
import java.util.concurrent.atomic.AtomicReference
96

107
private[spark] case class AzureMonitorConfig
@@ -17,10 +14,7 @@ private[spark] case class AzureMonitorConfig
1714
samplingRate: Float,
1815
samplingRateMaxCount: Int,
1916
samplingRateIntervalInSeconds: Int,
20-
metricCollectionIntervalInSeconds: Int,
21-
logLevel: Level,
22-
logSamplingMaxCount: Int,
23-
logSamplingIntervalInSeconds: Int
17+
metricCollectionIntervalInSeconds: Int
2418
)
2519

2620
private[spark] object AzureMonitorConfig {

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosClientCache.scala‎

Lines changed: 0 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,10 @@ import io.micrometer.core.instrument.composite.CompositeMeterRegistry
2424
import io.netty.util.ResourceLeakDetector
2525
import io.opentelemetry.api.common.AttributeKey
2626
import io.opentelemetry.api.logs.Severity
27-
import io.opentelemetry.instrumentation.log4j.appender.v2_17.OpenTelemetryAppender
2827
import io.opentelemetry.sdk.OpenTelemetrySdk
2928
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk
3029
import io.opentelemetry.sdk.metrics.SdkMeterProvider
3130
import io.opentelemetry.sdk.trace.samplers.Sampler
32-
import org.apache.logging.log4j.core.Filter.Result
33-
import org.apache.logging.log4j.{Level, LogManager}
34-
import org.apache.logging.log4j.core.filter.ThresholdFilter
35-
import org.apache.logging.log4j.core.{Filter, LoggerContext}
3631
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd}
3732
import org.apache.spark.sql.SparkSession
3833
import org.apache.spark.{SparkContext, TaskContext}
@@ -337,8 +332,6 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
337332
"none")
338333
additionalSystemPropertyOverrides.put(
339334
"otel.traces.exporter", "azure_monitor")
340-
additionalSystemPropertyOverrides.put(
341-
"otel.logs.exporter", "azure_monitor")
342335
additionalSystemPropertyOverrides.put(
343336
"applicationinsights.live.metrics.enabled",
344337
azMonConfig.liveMetricsEnabled.toString)
@@ -408,16 +401,6 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
408401
)
409402
)
410403
})
411-
.addLogRecordProcessorCustomizer((processor, _) => {
412-
logInfo(s"$processor")
413-
processor
414-
})
415-
.addLogRecordExporterCustomizer((exporter, _) => new CosmosMaxCountPerIntervalLogExporter(
416-
exporter,
417-
Severity.WARN,
418-
azMonConfig.logSamplingMaxCount,
419-
azMonConfig.logSamplingIntervalInSeconds
420-
))
421404
.build
422405
.getOpenTelemetrySdk
423406

@@ -436,60 +419,6 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
436419
.build()
437420
}
438421

439-
private[this] def configureLoggingForAzureMonitor
440-
(
441-
cosmosClientConfiguration: CosmosClientConfiguration,
442-
azMonConfig: AzureMonitorConfig,
443-
openTelemetry: OpenTelemetrySdk
444-
): Unit = {
445-
// 1. Get the context and config
446-
val ctx = LogManager.getContext(false).asInstanceOf[LoggerContext]
447-
val config = ctx.getConfiguration
448-
449-
// 2. Create the appender and filter
450-
val filter: Filter = ThresholdFilter.createFilter(
451-
azMonConfig.logLevel, // Minimum log level to emit (e.g., WARN or ERROR)
452-
Result.ACCEPT, // If above threshold
453-
Result.DENY // If below threshold
454-
)
455-
456-
val sampledDiagnosticsLoggerFilter: Filter = ThresholdFilter.createFilter(
457-
Level.INFO, // Minimum log level to emit (e.g., WARN or ERROR)
458-
Result.ACCEPT, // If above threshold
459-
Result.DENY // If below threshold
460-
)
461-
462-
val otelAppenderBuilder: OpenTelemetryAppender.Builder[OpenTelemetryAppender.Builder[_]] =
463-
OpenTelemetryAppender.builder()
464-
465-
otelAppenderBuilder.setName("otel-appender")
466-
otelAppenderBuilder.setFilter(filter)
467-
otelAppenderBuilder.setOpenTelemetry(openTelemetry)
468-
val otelAppender = otelAppenderBuilder.build()
469-
470-
otelAppender.start
471-
472-
// 3. Add to config
473-
config.addAppender(otelAppender)
474-
475-
// 4. Attach to **all** existing logger configs
476-
config.getLoggers.values().forEach { loggerConfig =>
477-
if ((azMonConfig.logLevel == Level.WARN || azMonConfig.logLevel == Level.ERROR)
478-
&& loggerConfig.getName.contains("CosmosSamplingDiagnosticsLogger")) {
479-
480-
loggerConfig.addAppender(otelAppender, Level.INFO, sampledDiagnosticsLoggerFilter)
481-
} else {
482-
loggerConfig.addAppender(otelAppender, azMonConfig.logLevel, filter)
483-
}
484-
}
485-
486-
// 5. Also attach to root logger
487-
val rootLoggerConfig = config.getLoggerConfig(LogManager.ROOT_LOGGER_NAME)
488-
rootLoggerConfig.addAppender(otelAppender, azMonConfig.logLevel, filter)
489-
490-
ctx.updateLoggers // Apply changes
491-
}
492-
493422
private[this] def configureAzureMonitorDiagnostics
494423
(
495424
cosmosClientConfiguration: CosmosClientConfiguration,
@@ -499,10 +428,6 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
499428
val azMonConfig = cosmosClientConfiguration.azureMonitorConfig.get
500429
val openTelemetry = configureOpenTelemetrySdk(cosmosClientConfiguration, azMonConfig)
501430

502-
if (azMonConfig.enabled && azMonConfig.logLevel != Level.OFF) {
503-
configureLoggingForAzureMonitor(cosmosClientConfiguration, azMonConfig, openTelemetry)
504-
}
505-
506431
// Pass OpenTelemetry container to TracingOptions.
507432
val tracingOptions = new OpenTelemetryTracingOptions()
508433
.setOpenTelemetry(openTelemetry)

0 commit comments

Comments
 (0)