Skip to content

Update the logging page for 2025.01 #2063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 21, 2025
Merged

Update the logging page for 2025.01 #2063

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
03a7aff
Update the logging page for 2025.01
renetapopova Jan 17, 2025
b654b94
reword the notes
renetapopova Jan 18, 2025
c00f70a
Update modules/ROOT/pages/monitoring/logging.adoc
renetapopova Jan 20, 2025
f6cbd0d
Update modules/ROOT/pages/monitoring/logging.adoc
MishaDemianenko Jan 21, 2025
fb4d654
Update modules/ROOT/pages/monitoring/logging.adoc
MishaDemianenko Jan 21, 2025
7b83f6b
Update modules/ROOT/pages/monitoring/logging.adoc
MishaDemianenko Jan 21, 2025
85ef8d2
Update modules/ROOT/pages/monitoring/logging.adoc
MishaDemianenko Jan 21, 2025
d2ad65c
Update modules/ROOT/pages/monitoring/logging.adoc
MishaDemianenko Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 81 additions & 58 deletions modules/ROOT/pages/monitoring/logging.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ For Debian and RPM packages run `journalctl --unit=neo4j`.

| _debug.log_
| Log information required by Neo4j Customer Support to investigate problems.
It is highly recommended to keep it enabled and not to alter the _debug.log_ format.
It is highly recommended to keep it enabled and not to alter the _debug.log_ format to ensure the best support.
If you require the _debug.log_ messages in a different format, create an additional Appender.
Starting from Neo4j 2025.01, the default format is JSON.
| xref:configuration/configuration-settings.adoc#config_server.logs.debug.enabled[`server.logs.debug.enabled`]
| `true`

Expand Down Expand Up @@ -80,15 +81,19 @@ and xref:configuration/configuration-settings.adoc#config_db.logs.query.threshol
Neo4j uses https://logging.apache.org/log4j/2.x/[Log4j 2] for logging.
The logging configuration is located in the _conf_ directory and consists of two files:

* _user-log.xml_ -- provides configuration for _neo4j.log_.
* _user-logs.xml_ -- provides configuration for _neo4j.log_.
* _server-logs.xml_ -- provides configuration for _debug.log_, _http.log_, _query.log_, and _security.log_.

The _gc.log_ is handled by the Java Virtual Machine(JVM) and is configured using the JVM parameters.
For details, see <<garbage-collection-log-file-configuration>>.

[TIP]
[json-format]
[NOTE]
====
If you want to use a custom directory for your Log4j configuration files, set the new paths to your XML files in the _neo4j.conf_ file by using the parameters xref:configuration/configuration-settings.adoc#config_server.logs.user.config[`server.logs.user.config`] and xref:configuration/configuration-settings.adoc#config_server.logs.config[`server.logs.config`].
Starting from Noe4j 2025.01, the default configuration files are located in the _server.directories.configuration=conf_ directory.
Therefore, the default paths to the configuration files are _conf/user-logs.xml_ and _conf/server-logs.xml_.
And <conf> by default is <home>/conf.
If you use a custom non-absolute paths set in the _neo4j.conf_ file for either of your Log4j configuration files, by using the parameters xref:configuration/configuration-settings.adoc#config_server.logs.user.config[`server.logs.user.config`] and xref:configuration/configuration-settings.adoc#config_server.logs.config[`server.logs.config`], then these values will likely be wrong, as these settings now depend on <conf> rather than <home>.
====

Each configuration file comprises two main elements: _Appenders_ and _Loggers_:
Expand All @@ -113,14 +118,14 @@ For more details, see https://logging.apache.org/log4j/2.x/manual/configuration.

The following example shows the default configuration of the _user-logs.xml_ file.

.Default _user-log.xml_ configuration
[source%linenums,xml,options="nowrap", role="nocopy"]
.Default _user-logs.xml_ configuration
[source,xml,options="nowrap", role="nocopy"]
----
<?xml version="1.0" encoding="UTF-8"?>
<!--

Copyright (c) "Neo4j"
Neo4j Sweden AB [http://neo4j.com]
Neo4j Sweden AB [https://neo4j.com]
This file is a commercial add-on to Neo4j Enterprise Edition.

-->
Expand All @@ -138,11 +143,11 @@ The following example shows the default configuration of the _user-logs.xml_ fil
<Appenders> \ #<2>
<RollingRandomAccessFile name="Neo4jLog" fileName="${config:server.directories.logs}/neo4j.log" #<3>
filePattern="$${config:server.directories.logs}/neo4j.log.%02i"> #<4>
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p %m%n"/> #<5>
<Policies> \ #<6>
<SizeBasedTriggeringPolicy size="20 MB"/> \ #<7>
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p %m%n"/> #<6>
<Policies> \ #<7>
<SizeBasedTriggeringPolicy size="20 MB"/> \ #<8>
</Policies>
<DefaultRolloverStrategy fileIndex="min" max="7"/> \ #<8>
<DefaultRolloverStrategy fileIndex="min" max="7"/> \ #<9>
</RollingRandomAccessFile>

<!-- Only used by "neo4j console", will be ignored otherwise -->
Expand All @@ -151,9 +156,9 @@ The following example shows the default configuration of the _user-logs.xml_ fil
</Console>
</Appenders>

<Loggers> \ #<9>
<Loggers> \ #<10>
<!-- Log level for the neo4j log. One of DEBUG, INFO, WARN, ERROR or OFF -->
<Root level="INFO"> \ #<10>
<Root level="INFO"> \ #<11>
<AppenderRef ref="Neo4jLog"/>
<AppenderRef ref="ConsoleAppender"/>
</Root>
Expand All @@ -168,41 +173,47 @@ The following example shows the default configuration of the _user-logs.xml_ fil
The following example shows the default configuration of the _server-logs.xml_ file.

.Default _server-logs.xml_ configuration
[source%linenums,xml,options="nowrap", role="nocopy"]
[source,xml,options="nowrap", role="nocopy"]
----
<?xml version="1.0" encoding="UTF-8"?>
<!--

Copyright (c) "Neo4j"
Neo4j Sweden AB [http://neo4j.com]
Neo4j Sweden AB [https://neo4j.com]
This file is a commercial add-on to Neo4j Enterprise Edition.

-->
<!--
This is a log4j 2 configuration file.

Please do not alter the original debug.log.
Changing the debug.log format can interfere with Neo4j's ability to offer customer support and invalidate your support warranty.
Please keep the original "debug.log" as is, to make sure enough data is captured in of case errors
in a format that neo4j support engineers and developers can work with. Changing the "debug.log" format
can interfere with neo4j's ability to offer customer support and invalidate your support warranty.

If you require the debug.log messages in a different format, you can copy the Appender and change the filename.
See Neo4j documentation for details.
If you require the "debug.log" messages in a different format you can copy the Appender and change the
filename. See Neo4j documentation.

All configuration values can be queried with the lookup prefix "config:". You can, for example, resolve
All configuration values can be queried with the lookup prefix "config:". You can for example, resolve
the path to your neo4j home directory with ${config:dbms.directories.neo4j_home}.

Please consult https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions and
available configuration options.
-->
<Configuration status="ERROR" monitorInterval="30" packages="org.neo4j.logging.log4j"> \ #<1>
<Appenders> \ #<2>
<!-- Neo4j debug.log,do not change. Required by Neo4j customer support. -->
<!-- Neo4j debug.log, do not change. Required by Neo4j customer support. -->
<RollingRandomAccessFile name="DebugLog" fileName="${config:server.directories.logs}/debug.log" \ #<3>
filePattern="$${config:server.directories.logs}/debug.log.%02i"> \ #<4>
<Neo4jDebugLogLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p [%c{1.}] %m%n"/> \ #<5>
<Policies> \ #<6>
<SizeBasedTriggeringPolicy size="20 MB"/> \ #<7>
<!-- JSON format -->
<JsonTemplateLayout eventTemplateUri="classpath:org/neo4j/logging/StructuredLayoutWithMessage.json"/> \ #<5>

<!-- Text format -->
<!-- <Neo4jDebugLogLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p [%c{1.}] %m%n"/> --> \ #<6>

<Policies> \ #<7>
<SizeBasedTriggeringPolicy size="20 MB"/> \ #<8>
</Policies>
<DefaultRolloverStrategy fileIndex="min" max="7"/> \ #<8>
<DefaultRolloverStrategy fileIndex="min" max="7"/> \ #<9>
</RollingRandomAccessFile>

<RollingRandomAccessFile name="HttpLog" fileName="${config:server.directories.logs}/http.log"
Expand Down Expand Up @@ -233,16 +244,16 @@ The following example shows the default configuration of the _server-logs.xml_ f
</RollingRandomAccessFile>
</Appenders>

<Loggers> \ #<9>
<Loggers> \ #10>
<!-- Log levels. One of DEBUG, INFO, WARN, ERROR or OFF -->

<!-- The debug log is used as the root logger to catch everything -->
<Root level="INFO"> \ #<10>
<Root level="INFO"> \ #<11>
<AppenderRef ref="DebugLog"/> <!-- Keep this -->
</Root>

<!-- The query log, must be named "QueryLogger" -->
<Logger name="QueryLogger" level="INFO" additivity="false"> \ #<11>
<Logger name="QueryLogger" level="INFO" additivity="false"> \ #<12>
<AppenderRef ref="QueryLog"/>
</Logger>

Expand Down Expand Up @@ -277,30 +288,34 @@ See <<general-logging-appenders>> for more information.
<4> `filePattern` specifies a file pattern to be used when the file is rolled.
The pattern renames the files to _debug.log.01_ and _http.log.01_ when they reach the defined trigger.

<5> `PatternLayout` defines the layout for the appender, in this case with the `GMT+2` timezone.
<5> The `JsonTemplateLayout` layout is used to format the log messages in JSON format.
Starting from Neo4j 2025.01, the default _debug.log_ format is JSON, and it is recommended to keep it enabled to ensure the best support.
If you require the _debug.log_ messages in a different format, create an additional Appender.

<6> `PatternLayout` defines the layout for the appender, in this case with the `GMT+2` timezone.
See <<general-logging-log-layout>> for more information.

<6> The `Policies` element defines when the files are rolled and how many files to keep as history before they are deleted.
<7> The `Policies` element defines when the files are rolled and how many files to keep as history before they are deleted.

<7> The `SizeBasedTriggeringPolicy` defines when the files are rolled.
<8> The `SizeBasedTriggeringPolicy` defines when the files are rolled.
In this case, when the size of the files reaches 20 MB, the files are renamed according to the `filePattern`, and the log files start over.
In Neo4j 4.0, this was configured with the parameter `dbms.logs.user.rotation.size`.

<8> The `DefaultRolloverStrategy` defines how many files to keep as history. +
<9> The `DefaultRolloverStrategy` defines how many files to keep as history. +
The `fileIndex=min` implies that the minimum/the lowest number is the most recent one. +
The `max` attribute defines the number of files to keep as history before they are deleted, in this case, 7 files.
In Neo4j 4.0, this was configured with the parameter `dbms.logs.user.rotation.keep_number`.

<9> Loggers are used to define the log level and which appender to use for the log messages.
<10> Loggers are used to define the log level and which appender to use for the log messages.
The loggers are referenced by the `name` attribute.
See <<general-logging-loggers>> for more information.

<10> The root logger is a "catch-all" logger that catches everything that is not caught by the other loggers and sends it to the appender(s) specified in the `AppenderRef` element(s).
<11> The root logger is a "catch-all" logger that catches everything that is not caught by the other loggers and sends it to the appender(s) specified in the `AppenderRef` element(s).
The root logger is referenced by the `Root` element.
It can be set to `DEBUG`, `INFO`, `WARN`, `ERROR`, or `OFF`.
The default log level is `INFO`.

<11> You can also define custom loggers to catch specific log events and send them to the appender(s) specified in the `AppenderRef` element(s).
<12> You can also define custom loggers to catch specific log events and send them to the appender(s) specified in the `AppenderRef` element(s).
For example, the `QueryLogger` logger (configured in _server-logs.xml_) is used to catch log events with a log level of `INFO` or above and send them to the `QueryLog` appender. +
The `additivity="false"` is set to fully consume the log event and not send it to the root logger. +
If `additivity="true"` is set, which is the default, the log event is also sent to the root logger.
Expand Down Expand Up @@ -428,23 +443,16 @@ a| `%n`

For all available converters, consult the https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout[Log4j 2 Pattern Layout documentation].

[[general-logging-debug-log-pattern]]
==== `<Neo4jDebugLogLayout>`

The `<Neo4jDebugLogLayout>` layout is essentially the same as the xref:monitoring/logging.adoc#general-logging-pattern-layout[`PatternLayout`].
The main difference is that a header is injected at the start of the log file with diagnostic information useful for Neo4j developers.
This layout should typically only be used for the _debug.log_ file.

.An example usage of the Neo4j debug log layout
[source,xml]
----
<Neo4jDebugLogLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p [%c{1.}] %m%n"/>
----

[[general-logging-json-template-layout]]
==== `<JsonTemplateLayout>`
The `<JsonTemplateLayout>` is equivalent to the pattern layout.
For more information, see the https://logging.apache.org/log4j/2.x/manual/json-template-layout.html[Log4j official documentation].
The `<JsonTemplateLayout>` is equivalent to the pattern layout and is used to format log messages in JSON format.

[NOTE]
====
Starting from Neo4j 2025.01, JSON is the default format for the _debug.log_.
It is highly recommended not to alter the _debug.log_ format as it is required by Neo4j Customer Support to investigate problems.
If you require the _debug.log_ messages in a different format, create an additional Appender.
====

There are two ways of configuring the JSON template layout.

Expand All @@ -458,7 +466,7 @@ The JSON template file can be on the file system.

* The JSON event template file can be embedded in the XML configuration:
+
[source%linenums,xml,highlight=4..12]
[source,xml,highlight=4..12]
----
<JsonTemplateLayout>
<eventTemplate>
Expand Down Expand Up @@ -514,6 +522,21 @@ There are also a couple of built-in templates available from the classpath, for
| Same layout as the less flexible `<JsonLayout>`.
|===

For more information, see the https://logging.apache.org/log4j/2.x/manual/json-template-layout.html[Log4j official documentation].

[[general-logging-debug-log-pattern]]
==== `<Neo4jDebugLogLayout>`

The `<Neo4jDebugLogLayout>` layout is essentially the same as the xref:monitoring/logging.adoc#general-logging-pattern-layout[`PatternLayout`].
The main difference is that a header is injected at the start of the log file with diagnostic information useful for Neo4j developers.
This layout should typically only be used for the _debug.log_ file.

.An example usage of the Neo4j debug log layout
[source,xml]
----
<Neo4jDebugLogLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p [%c{1.}] %m%n"/>
----

=== Filters

You can also configure filters to determine if and what log events are published and how.
Expand Down Expand Up @@ -575,7 +598,7 @@ The root logger is a special logger that matches everything, so if another logge
Therefore, it is best practice always to include a root logger so that no log events are missed.

.Configuration of loggers
[source%linenums,xml,options="nowrap"]
[source,xml,options="nowrap"]
----
<Configuration>
<!-- Appenders -->
Expand Down Expand Up @@ -684,7 +707,7 @@ dbms.security.log_successful_authentication=false
The security log can use a JSON layout.
To change the format, the layout for the `SecurityLogger` must be changed from using the `PatternLayout`:

[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
<RollingRandomAccessFile name="SecurityLog" fileName="${config:server.directories.logs}/security.log"
filePattern="$${config:server.directories.logs}/security.log.%02i">
Expand All @@ -698,7 +721,7 @@ To change the format, the layout for the `SecurityLogger` must be changed from u

to using the `JsonTemplateLayout`:

[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
<RollingRandomAccessFile name="SecurityLog" fileName="${config:server.directories.logs}/security.log"
filePattern="$${config:server.directories.logs}/security.log.%02i">
Expand Down Expand Up @@ -1131,7 +1154,7 @@ In Neo4j Browser and Bloom, the user-provided metadata is always replaced by the
The query log can use a JSON layout.
To change the format, the layout for the `QueryLogger` must be changed from using the `PatternLayout`:

[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
<RollingRandomAccessFile name="QueryLog" fileName="${config:server.directories.logs}/query.log"
filePattern="${config:server.directories.logs}/query.log.%02i">
Expand All @@ -1145,7 +1168,7 @@ To change the format, the layout for the `QueryLogger` must be changed from usin

to using the `JsonTemplateLayout`:

[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
<RollingRandomAccessFile name="QueryLog" fileName="${config:server.directories.logs}/query.log"
filePattern="${config:server.directories.logs}/query.log.%02i">
Expand Down Expand Up @@ -1310,7 +1333,7 @@ Additionally, GQL codes for external procedures are not yet stable.
The following are examples of the `errorInfo` JSON object:

.`errorInfo` JSON object of a database error
[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
...
"errorInfo": {
Expand All @@ -1327,7 +1350,7 @@ The following are examples of the `errorInfo` JSON object:
----

.`errorInfo` JSON object of a client error
[source%linenums,xml,options="nowrap",highlight=4]
[source,xml,options="nowrap",highlight=4]
----
...
"errorInfo": {
Expand Down
Loading