Skip to content

Commit e6ceec4

Browse files
authored
Merge pull request ceph#60813 from anthonyeleven/improve-subsystem
doc/rados/troubleshooting: Improve log-and-debug.rst Reviewed-by: Zac Dover <[email protected]>
2 parents 87089a0 + 44c9519 commit e6ceec4

File tree

1 file changed

+17
-15
lines changed

1 file changed

+17
-15
lines changed

doc/rados/troubleshooting/log-and-debug.rst

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,23 +6,24 @@ Ceph component debug log levels can be adjusted at runtime, while services are
66
running. In some circumstances you might want to adjust debug log levels in
77
``ceph.conf`` or in the central config store. Increased debug logging can be
88
useful if you are encountering issues when operating your cluster. By default,
9-
Ceph log files are in ``/var/log/ceph``.
9+
Ceph log files are in ``/var/log/ceph``; containerized deployments often log
10+
elsewhere under ``/var/log``.
1011

1112
.. tip:: Remember that debug output can slow down your system, and that this
1213
latency sometimes hides race conditions.
1314

1415
Debug logging is resource intensive. If you encounter a problem in a specific
1516
component of your cluster, begin troubleshooting by enabling logging for only
16-
that component of the cluster. For example, if your OSDs are running without
17-
errors, but your metadata servers are not, enable logging for any specific
18-
metadata server instances that are having problems. Continue by enabling
17+
that component. For example, if your OSDs are running without
18+
errors, but your CephFS metadata servers (MDS) are not, enable logging for specific
19+
instances that are having problems. Continue by enabling
1920
logging for each subsystem only as needed.
2021

2122
.. important:: Verbose logging sometimes generates over 1 GB of data per hour.
2223
If the disk that your operating system runs on (your "OS disk") reaches its
2324
capacity, the node associated with that disk will stop working.
2425

25-
Whenever you enable or increase the rate of debug logging, make sure that you
26+
Whenever you enable or increase the level of debug logging, ensure that you
2627
have ample capacity for log files, as this may dramatically increase their
2728
size. For details on rotating log files, see `Accelerating Log Rotation`_.
2829
When your system is running well again, remove unnecessary debugging settings
@@ -34,7 +35,7 @@ For details on available settings, see `Subsystem, Log and Debug Settings`_.
3435
Runtime
3536
=======
3637

37-
To see the configuration settings at runtime, log in to a host that has a
38+
To see configuration settings at runtime, log in to a host that has a
3839
running daemon and run a command of the following form:
3940

4041
.. prompt:: bash $
@@ -57,7 +58,7 @@ tell`` command of the following form:
5758
5859
Here ``{daemon-type}`` is ``osd``, ``mon``, or ``mds``. Apply the runtime
5960
setting either to a specific daemon (by specifying its ID) or to all daemons of
60-
a particular type (by using the ``*`` operator). For example, to increase
61+
a particular type (by using the ``*`` wildcard as the ID). For example, to increase
6162
debug logging for a specific ``ceph-osd`` daemon named ``osd.0``, run the
6263
following command:
6364

@@ -81,7 +82,8 @@ Boot Time
8182
=========
8283

8384
To activate Ceph's debugging output (that is, the ``dout()`` logging function)
84-
at boot time, you must add settings to your Ceph configuration file.
85+
at boot time, you must add settings to your Ceph configuration file (or
86+
set corresponding values in the central config store).
8587
Subsystems that are common to all daemons are set under ``[global]`` in the
8688
configuration file. Subsystems for a specific daemon are set under the relevant
8789
daemon section in the configuration file (for example, ``[mon]``, ``[osd]``,
@@ -115,7 +117,7 @@ For details, see `Subsystem, Log and Debug Settings`_.
115117
Accelerating Log Rotation
116118
=========================
117119

118-
If your log filesystem is nearly full, you can accelerate log rotation by
120+
If a host's log filesystem is nearly full, you can accelerate log rotation by
119121
modifying the Ceph log rotation file at ``/etc/logrotate.d/ceph``. To increase
120122
the frequency of log rotation (which will guard against a filesystem reaching
121123
capacity), add a ``size`` directive after the ``weekly`` frequency directive.
@@ -149,8 +151,8 @@ setting is shown immediately below.
149151

150152
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
151153

152-
In this example, the ``etc/logrotate.d/ceph`` file will be checked every 30
153-
minutes.
154+
In this example, the ``etc/logrotate.d/ceph`` file will be checked and possibly
155+
rotated every 30 minutes.
154156

155157
Valgrind
156158
========
@@ -175,7 +177,7 @@ For each subsystem, there is a logging level for its output logs (a so-called
175177
"log level") and a logging level for its in-memory logs (a so-called "memory
176178
level"). Different values may be set for these two logging levels in each
177179
subsystem. Ceph's logging levels operate on a scale of ``1`` to ``20``, where
178-
``1`` is terse and ``20`` is verbose. In certain rare cases, there are logging
180+
``1`` is terse and ``20`` is verbose. In a certain few cases, there are logging
179181
levels that can take a value greater than 20. The resulting logs are extremely
180182
verbose.
181183

@@ -184,7 +186,7 @@ following conditions are true:
184186

185187
- a fatal signal has been raised or
186188
- an assertion within Ceph code has been triggered or
187-
- the sending of in-memory logs to the output log has been manually triggered.
189+
- sending in-memory logs to the output log has been manually triggered.
188190
Consult `the portion of the "Ceph Administration Tool documentation
189191
that provides an example of how to submit admin socket commands
190192
<http://docs.ceph.com/en/latest/man/8/ceph/#daemon>`_ for more detail.
@@ -206,8 +208,8 @@ following:
206208
debug mds balancer = 1/20
207209
208210
The following table provides a list of Ceph subsystems and their default log and
209-
memory levels. Once you complete your logging efforts, restore the subsystems
210-
to their default level or to a level suitable for normal operations.
211+
memory levels. Once you complete your logging efforts, restore each subsystem's
212+
values to their defaults or to a level suitable for normal operations.
211213

212214
+--------------------------+-----------+--------------+
213215
| Subsystem | Log Level | Memory Level |

0 commit comments

Comments
 (0)