Skip to content

Commit d0fbb15

Browse files
dongjoon-hyunsarutak
authored andcommitted
[SPARK-54794][CORE] Suppress verbose FsHistoryProvider.checkForLogs scanning logs
### What changes were proposed in this pull request? This PR aims to suppress verbose `FsHistoryProvider.checkForLogs` scanning logs. ### Why are the changes needed? This is a kind of regression at Apache Spark 4.1.0 switched from `logDebug` to `logInfo` accidentally. We had better recover them back to `DEBUG` level because it's excessively verbose currently. - #52382 ``` 25/12/21 22:27:53 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356063285 25/12/21 22:28:03 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356073312 25/12/21 22:28:13 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356083338 25/12/21 22:28:23 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356093347 25/12/21 22:28:33 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356103369 25/12/21 22:28:43 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356113391 25/12/21 22:28:53 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356123402 25/12/21 22:29:03 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356133422 25/12/21 22:29:13 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356143443 25/12/21 22:29:23 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356153458 25/12/21 22:29:33 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356163471 25/12/21 22:29:43 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356173491 25/12/21 22:29:53 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356183502 25/12/21 22:30:03 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356193517 25/12/21 22:30:13 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356203529 25/12/21 22:30:23 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356213555 25/12/21 22:30:33 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356223567 25/12/21 22:30:43 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356233585 25/12/21 22:30:53 INFO FsHistoryProvider: Scanning s3a://spark-events with lastScanTime=1766356243599 ``` ### Does this PR introduce _any_ user-facing change? No behavior change except log message level. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53555 from dongjoon-hyun/SPARK-54794. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Kousuke Saruta <[email protected]>
1 parent d5b51fe commit d0fbb15

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -502,7 +502,7 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
502502
var count: Int = 0
503503
try {
504504
val newLastScanTime = clock.getTimeMillis()
505-
logInfo(log"Scanning ${MDC(HISTORY_DIR, logDir)} with " +
505+
logDebug(log"Scanning ${MDC(HISTORY_DIR, logDir)} with " +
506506
log"lastScanTime=${MDC(LAST_SCAN_TIME, lastScanTime)}")
507507

508508
// Mark entries that are processing as not stale. Such entries do not have a chance to be

0 commit comments

Comments
 (0)