Skip to content

KAFKA-9366: Upgrade log4j to log4j2 #17373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 77 commits into from
Dec 13, 2024
Merged

KAFKA-9366: Upgrade log4j to log4j2 #17373

Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
4cc09d9
WIP
frankvicky Oct 2, 2024
d71bbdf
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 4, 2024
347388a
KAFKA-9366: Upgrade log4j to log4j2
frankvicky Oct 4, 2024
9fc1203
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 10, 2024
d12f311
KAFKA-9366: Cleanup
frankvicky Oct 10, 2024
7200b43
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 11, 2024
7419f5f
revert unnecessary change
frankvicky Oct 11, 2024
6ce4e5f
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 11, 2024
d69df89
revert unnecessary change
frankvicky Oct 11, 2024
98157cb
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 11, 2024
8ec412d
KAFKA-9366: update readme
frankvicky Oct 11, 2024
84c467c
KAFKA-9366: Address comments for core module
frankvicky Oct 23, 2024
32ae43c
KAFKA-9366: Address comments for tools module
frankvicky Oct 23, 2024
ae77d17
KAFKA-9366: upgrade trogdor to log4j2
frankvicky Oct 23, 2024
dfdcb1c
KAFKA-9366: update build.gradle for lib reference
frankvicky Oct 24, 2024
ae45089
KAFKA-9366: Remove unnecessary semicolon
frankvicky Oct 24, 2024
d1ed1d2
KAFKA-9366: Revert the line break
frankvicky Oct 24, 2024
403002c
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 24, 2024
bafb177
KAFKA-9366: Remove reload4j and sl4j-reload4j dependencies
frankvicky Oct 24, 2024
6dc14fd
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 24, 2024
aa06946
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 29, 2024
2b224ea
KAFKA-9366: Address comments
frankvicky Oct 29, 2024
4ac974e
Merge branch 'trunk' into KAFKA-9366
frankvicky Oct 31, 2024
125f6f1
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 3, 2024
b301c92
WIP
frankvicky Nov 3, 2024
c00618a
Update LogEvent to immutable
frankvicky Nov 8, 2024
07376e3
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 8, 2024
686b56b
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 8, 2024
5fc37b4
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 10, 2024
7408589
KAFKA-9366: Resolve conflicts
frankvicky Nov 10, 2024
1340837
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 10, 2024
f10e2d2
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 13, 2024
c959967
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 14, 2024
97509f4
Reslove Conflicts
frankvicky Nov 14, 2024
5f5f0ae
KAFKA-9366: Address comments
frankvicky Nov 15, 2024
20088a3
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 15, 2024
c30b309
KAFKA-9366: Update README
frankvicky Nov 16, 2024
2b6be7e
Update dependencies.gradle
frankvicky Nov 16, 2024
7032ffd
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 16, 2024
168557b
KAFKA-9366: update Loggers.java
frankvicky Nov 17, 2024
89047c0
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 18, 2024
efaab51
KAFKA-9366: Remove log4j properties in config directory
frankvicky Nov 18, 2024
dfdf33b
KAFKA-9366: Address comments
frankvicky Nov 18, 2024
0aff2c7
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 19, 2024
4aee10f
KAFKA-9366: Address comments
frankvicky Nov 19, 2024
5e1eaf5
KAFKA-9366: Address comments
frankvicky Nov 19, 2024
a1c2294
KAFKA-9366: migration to yaml
frankvicky Nov 19, 2024
7672fa5
KAFKA-9366: Update script and readme
frankvicky Nov 20, 2024
f3a68e1
KAFKA-9366: update e2e
frankvicky Nov 20, 2024
9343fc3
KAFKA-9366: fix yaml
frankvicky Nov 20, 2024
5761828
KAFKA-9366: fix typo and compact the single policy
frankvicky Nov 20, 2024
723e87c
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 21, 2024
40a80f4
KAFKA-9366: update e2e
frankvicky Nov 21, 2024
a6fa0ac
KAFKA-9366: update script
frankvicky Nov 21, 2024
17cc7fc
KAFKA-9366: Fix typo
frankvicky Nov 21, 2024
8fa5a73
KAFKA-9366: update util.py
frankvicky Nov 21, 2024
1d06e84
KAFKA-9366: Fix typo
frankvicky Nov 21, 2024
e488b0a
KAFKA-9366: update e2e log4j2.yaml
frankvicky Nov 22, 2024
89b2168
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 22, 2024
8113166
KAFKA-9366: Fix import-control
frankvicky Nov 22, 2024
2b749e8
KAFKA-9366: update e2e log4j2.yaml
frankvicky Nov 22, 2024
07d1bb1
KAFKA-9366: revert uncessary changes
frankvicky Nov 24, 2024
87fc720
KAFKA-9366: Update README and e2e log4j2.yaml
frankvicky Nov 25, 2024
56b306c
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 27, 2024
6d41bcd
KAFKA-9366: Fix the bugs of Loggers#currentLoggers
frankvicky Nov 27, 2024
064fe52
KAFKA-9366: Update log4j2.yaml of e2e
frankvicky Nov 29, 2024
762a54d
KAFKA-9366: replace tools and trogdor log4j.properties with log4j2.yaml
frankvicky Nov 30, 2024
3eb5522
KAFKA-9366: render log4j config according to node version
frankvicky Nov 30, 2024
fb61428
Merge branch 'trunk' into KAFKA-9366
frankvicky Nov 30, 2024
99cc467
KAFKA-9366: Update script file and e2e test
frankvicky Dec 1, 2024
440463a
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 2, 2024
fc4e351
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 4, 2024
9bf59d9
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 5, 2024
23e391b
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 8, 2024
38f6155
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 10, 2024
eab8334
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 11, 2024
fd72409
Merge branch 'trunk' into KAFKA-9366
frankvicky Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ jackson-annotations-2.16.2
jackson-core-2.16.2
jackson-databind-2.16.2
jackson-dataformat-csv-2.16.2
jackson-dataformat-yaml-2.16.2
jackson-datatype-jdk8-2.16.2
jackson-jaxrs-base-2.16.2
jackson-jaxrs-json-provider-2.16.2
Expand All @@ -239,6 +240,11 @@ jetty-servlets-9.4.56.v20240826
jetty-util-9.4.56.v20240826
jetty-util-ajax-9.4.56.v20240826
jose4j-0.9.4
log4j-api-2.24.1
log4j-core-2.24.1
log4j-core-test-2.24.1
log4j-slf4j-impl-2.24.1
log4j-1.2-api-2.24.1
lz4-java-1.8.0
maven-artifact-3.9.6
metrics-core-4.1.12.1
Expand All @@ -254,7 +260,6 @@ netty-transport-native-epoll-4.1.115.Final
netty-transport-native-unix-common-4.1.115.Final
opentelemetry-proto-1.0.0-alpha
plexus-utils-3.5.1
reload4j-1.2.25
rocksdbjni-7.9.2
scala-library-2.13.15
scala-logging_2.13-3.9.5
Expand Down Expand Up @@ -313,7 +318,6 @@ argparse4j-0.7.0, see: licenses/argparse-MIT
classgraph-4.8.173, see: licenses/classgraph-MIT
jopt-simple-5.0.4, see: licenses/jopt-simple-MIT
slf4j-api-1.7.36, see: licenses/slf4j-MIT
slf4j-reload4j-1.7.36, see: licenses/slf4j-MIT
pcollections-4.0.1, see: licenses/pcollections-MIT

---------------------------------------
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,10 @@ Follow instructions in https://kafka.apache.org/quickstart
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testTimeToNextUpdate

### Running a particular unit/integration test with log4j output ###
By default, there will be only small number of logs output while testing. You can adjust it by changing the `log4j.properties` file in the module's `src/test/resources` directory.
By default, there will be only small number of logs output while testing. You can adjust it by changing the `log4j2.yml` file in the module's `src/test/resources` directory.

For example, if you want to see more logs for clients project tests, you can modify [the line](https://github.com/apache/kafka/blob/trunk/clients/src/test/resources/log4j.properties#L21) in `clients/src/test/resources/log4j.properties`
to `log4j.logger.org.apache.kafka=INFO` and then run:
For example, if you want to see more logs for clients project tests, you can modify [the line](https://github.com/apache/kafka/blob/trunk/clients/src/test/resources/log4j2.yml#L35) in `clients/src/test/resources/log4j2.yml`
to `level: INFO` and then run:

./gradlew cleanTest clients:test --tests NetworkClientTest

Expand Down
9 changes: 8 additions & 1 deletion bin/connect-distributed.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,15 @@ fi

base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
if [ -f "$base_dir/../config/connect-log4j.properties" ]; then
echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/connect-log4j.properties >&2
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2
echo You can also use the \$KAFKA_HOME/config/connect-log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration. >&2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we do this in many files instead of kafka-run-class?

export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/connect-log4j.properties"
elif [ -f "$base_dir/../config/connect-log4j2.yaml" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we override the config path? it breaks all e2e since the custom log4j config can't be used.

export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.yaml"
elif [ -f "$base_dir/../config/connect-log4j2.xml" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.xml"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
Expand Down
9 changes: 8 additions & 1 deletion bin/connect-mirror-maker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,15 @@ fi

base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
if [ -f "$base_dir/../config/connect-log4j.properties" ]; then
echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/connect-log4j.properties >&2
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2
echo You can also use the \$KAFKA_HOME/config/connect-log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration. >&2
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/connect-log4j.properties"
elif [ -f "$base_dir/../config/connect-log4j2.yaml" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.yaml"
elif [ -f "$base_dir/../config/connect-log4j2.xml" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.xml"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
Expand Down
9 changes: 8 additions & 1 deletion bin/connect-standalone.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,15 @@ fi

base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
if [ -f "$base_dir/../config/connect-log4j.properties" ]; then
echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/connect-log4j.properties >&2
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2
echo You can also use the \$KAFKA_HOME/config/connect-log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration. >&2
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/connect-log4j.properties"
elif [ -f "$base_dir/../config/connect-log4j2.yaml" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.yaml"
elif [ -f "$base_dir/../config/connect-log4j2.xml" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/connect-log4j2.xml"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
Expand Down
9 changes: 8 additions & 1 deletion bin/kafka-server-start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,15 @@ then
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
if [ -f "$base_dir/../config/log4j.properties" ]; then
echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/log4j.properties >&2
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2
echo You can also use the \$KAFKA_HOME/config/log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration. >&2
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could break some existing Kafka installations. If users are extracting in place or copying previous config files to a new installation directory, they will be expecting the log4j.properties to still work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The switch from the legacy to the new configuration format can be based on the presence of specific files:

if [ -f "$base_dir/../config/log4j.properties" ]; then
    echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/log4j.properties >&2
    echo To use a Log4j 2.x configuration, create a \$KAFKA_HOME/config/log4j2.xml file and remove the Log4j 1.x configration. >&2
    echo See https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
elif [ -f "$base_dir/../config/log4j2.xml" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/log4j2.xml"
fi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure why we override the KAFKA_LOG4J_OPTS here. We typically allow users to define custom KAFKA_LOG4J_OPTS. Moreover, overriding KAFKA_LOG4J_OPTS can break many end-to-end tests, as they often create log4j configurations dynamically and pass them through KAFKA_LOG4J_OPTS

Noted that we do not require users to strictly use the path $base_dir/../config/log4j2.xml.

elif [ -f "$base_dir/../config/log4j2.yaml" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/log4j2.yaml"
elif [ -f "$base_dir/../config/log4j2.xml" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/log4j2.xml"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
Expand Down
11 changes: 9 additions & 2 deletions bin/windows/connect-distributed.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,15 @@ set BASE_DIR=%CD%
popd

rem Log4j settings
IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%BASE_DIR%/config/connect-log4j.properties
IF EXIST "%BASE_DIR%/config/connect-log4j.properties" (
echo DEPRECATED: Using Log4j 1.x configuration file %BASE_DIR%/config/connect-log4j.properties
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration.
echo You can also use the %BASE_DIR%/config/connect-log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration.
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%BASE_DIR%/config/connect-log4j.properties
) ELSE IF EXIST "%BASE_DIR%/config/connect-log4j2.yaml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%BASE_DIR%/config/connect-log4j2.yaml
) ELSE IF EXIST "%BASE_DIR%/config/connect-log4j2.xml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%BASE_DIR%/config/connect-log4j2.xml
)

"%~dp0kafka-run-class.bat" org.apache.kafka.connect.cli.ConnectDistributed %*
Expand Down
11 changes: 9 additions & 2 deletions bin/windows/connect-standalone.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,15 @@ set BASE_DIR=%CD%
popd

rem Log4j settings
IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%BASE_DIR%/config/connect-log4j.properties
IF EXIST "%BASE_DIR%/config/connect-log4j.properties" (
echo DEPRECATED: Using Log4j 1.x configuration file %BASE_DIR%/config/connect-log4j.properties
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration.
echo You can also use the %BASE_DIR%/config/connect-log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration.
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%BASE_DIR%/config/connect-log4j.properties
) ELSE IF EXIST "%BASE_DIR%/config/connect-log4j2.yaml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%BASE_DIR%/config/connect-log4j2.yaml
) ELSE IF EXIST "%BASE_DIR%/config/connect-log4j2.xml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%BASE_DIR%/config/connect-log4j2.xml
)

"%~dp0kafka-run-class.bat" org.apache.kafka.connect.cli.ConnectStandalone %*
Expand Down
9 changes: 8 additions & 1 deletion bin/windows/kafka-server-start.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,15 @@ IF [%1] EQU [] (
)

SetLocal
IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
IF EXIST "%~dp0../../config/log4j.properties" (
echo DEPRECATED: Using Log4j 1.x configuration file %~dp0../../config/log4j.properties
echo To use a Log4j 2.x configuration, please see https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration.
echo You can also use the %~dp0../../config/log4j2.yaml file as a starting point. Make sure to remove the Log4j 1.x configuration after completing the migration.
set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%~dp0../../config/log4j.properties
) ELSE IF EXIST "%~dp0../../config/log4j2.yaml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%~dp0../../config/log4j2.yaml
) ELSE IF EXIST "%~dp0../../config/log4j2.xml" (
set KAFKA_LOG4J_OPTS=-Dlog4j2.configurationFile=%~dp0../../config/log4j2.xml
)
IF ["%KAFKA_HEAP_OPTS%"] EQU [""] (
rem detect OS architecture
Expand Down
Loading