log4j library

[ptabandzelic]

Hey, i have been going through the log4j2 library. There is so limited resources other than Java applications. But a simple question:
We get our log4j logger with this getLogger method and we provide name which is a module name like this:

 log4j_logger = self.spark._jvm.org.apache.log4j  # noqa
 log4j_logger.LogManager.getLogger(self.__class__.__name__)

Which logger is actually called like this? Since we dont have loggers that are named as our modules inside log4j configuration.
In default xml configuration all loggers starts with com.databricks.logging.
I have been trying to configure my own log4j2 configuration like this:

appenders = FileLogger
appender.FileLogger.type = File
appender.FileLogger.name = FileLogger
appender.FileLogger.layout.type = PatternLayout
appender.FileLogger.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
appender.FileLogger.fileName = logs/stdout.production.log

loggers = production
logger.production=DEBUG, FileLogger
logger.production.name = com.production
logger.production.additivity = true

And then call it like this:

 log4j_logger = self.spark._jvm.org.apache.log4j  # noqa
 log4j_logger.LogManager.getLogger("com.production")

[renardeinside]

hi @ptabandzelic ,

We get our log4j logger with this getLogger method and we provide name which is a module name like this:

 log4j_logger.LogManager.getLogger(self.__class__.__name__)

Using it inside the class, for instance:

class SomeTask:
    def __init__(self):
        self.logger =  log4j_logger.LogManager.getLogger(self.__class__.__name__)

Will lead to logger name:

SampleETLTask

If you would like to make your logger static, you do it correctly:

 log4j_logger = self.spark._jvm.org.apache.log4j  # noqa
 log4j_logger.LogManager.getLogger("com.production")

In your log4j.properties you'll need to configure it as follows:

log4j.rootCategory=INFO, console
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=INFO

# configure the format and output for the console appender
log4j.appender.publicFile.layout.ConversionPattern=[spark][%p][%d{yy/MM/dd HH:mm:ss}][%c][%m]%n


log4j.logger.com.production=DEBUG

[ptabandzelic]

Yeah, i understand all of this. My question is when u get logger by module name, which logger is using in background, rootLogger?

[renardeinside]

My question is when u get logger by module name, which logger is using in background, rootLogger?

I would assume it's not the rootLogger, but the py4j logger since its used as a the bridge to Java. Although it's only an assumption

[ptabandzelic]

And if i want to modify py4j logging inside log4j2.properties, how can i reference it?

logger.py4j = DEBUG, appender1

So that i want to use:

logger =  log4j_logger.LogManager.getLogger(__class__.__name__)

[renardeinside]

Please read this and this on how to call the py4j logger directly.