Skip to content

Commit d988a8f

Browse files
anthonyelevenanthonyeleven
authored
Merge pull request ceph#65501 from anthonyeleven/ceph-osd-df
doc: Enhance the ceph df and ceph osd commands
2 parents 9b3bfb6 + 12bbd8b commit d988a8f

File tree

3 files changed

+100
-16
lines changed

3 files changed

+100
-16
lines changed

‎doc/glossary.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,8 @@
350350
Ceph cluster. RGW bucket indexes are stored as omaps.
351351
Erasure-coded pools cannot store RADOS omap data structures.
352352

353-
Run the command ``ceph osd df`` to see your omap.
353+
Run the command ``ceph osd df`` to see the storage space used by
354+
omaps on each OSD.
354355

355356
See Eleanor Cawthon's 2012 paper `A Distributed Key-Value Store
356357
using Ceph

‎doc/rados/operations/control.rst‎

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,40 @@ To query OSD subsystem status, run the following command:
144144

145145
ceph osd stat
146146

147+
To show the ID, device class, weights, size, and usage of all OSDs in the
148+
cluster, run the following command:
149+
150+
.. prompt:: bash $
151+
152+
ceph osd df
153+
154+
The ``ceph osd df`` command appends a summary that includes OSD fullness
155+
statistics. When a cluster comprises multiple sizes and types of OSD
156+
media, this summary may be more useful by limiting the scope to a specific
157+
CRUSH device class by running a command of the following form:
158+
159+
.. prompt:: bash $
160+
161+
ceph osd df hdd
162+
ceph osd df ssd
163+
ceph osd df qlc
164+
165+
To limit the display and summary to the OSDs present on a specific host,
166+
run a command of the following form. Note that this command may be used
167+
on other CRUSH topology bucket types, for example `rack`:
168+
169+
.. prompt:: bash $
170+
171+
ceph osd df NAME
172+
173+
To display information for only a specific OSD, in this example osd.1701,
174+
run a command of the following form:
175+
176+
.. prompt:: bash $
177+
178+
ceph osd df 1701
179+
180+
147181
To write a copy of the most recent OSD map to a file (see :ref:`osdmaptool
148182
<osdmaptool>`), run the following command:
149183

@@ -319,12 +353,14 @@ command:
319353
ceph osd reweight {osd-num} {weight}
320354

321355
.. note:: Any assigned override reweight value will conflict with the balancer.
322-
This means that if the balancer is in use, all override reweight values
323-
should be ``1.0000`` in order to avoid suboptimal cluster behavior.
356+
This means that when the balancer is in use, all override reweight values
357+
must be be reset to ``1.0000`` in order to avoid unbalanced usage and
358+
full OSDs. Most clusters with no clients older than the Luminous release
359+
should use the pg-upmap balancer instead of legacy reweighting.
324360

325361
A cluster's OSDs can be reweighted in order to maintain balance if some OSDs
326362
are being disproportionately utilized. Note that override or ``reweight``
327-
weights have values relative to one another that default to 1.00000; their
363+
weights have values relative to one another that default to 1.00000. These
328364
values are not absolute, and these weights must be distinguished from CRUSH
329365
weights (which reflect the absolute capacity of a bucket, as measured in TiB).
330366
To reweight OSDs by utilization, run the following command:
@@ -335,7 +371,8 @@ To reweight OSDs by utilization, run the following command:
335371

336372
By default, this command adjusts the override weight of OSDs that have ±20% of
337373
the average utilization, but you can specify a different percentage in the
338-
``threshold`` argument.
374+
``threshold`` argument. As noted above, most clusters without pre-Luminous
375+
clients should run the balancer in pg-upmap mode instead.
339376

340377
To limit the increment by which any OSD's reweight is to be changed, use the
341378
``max_change`` argument (default: 0.05). To limit the number of OSDs that are

‎doc/rados/operations/monitoring.rst‎

Lines changed: 57 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -388,28 +388,74 @@ following command:
388388

389389
The output of ``ceph df`` resembles the following::
390390

391+
--- RAW STORAGE ---
391392
CLASS SIZE AVAIL USED RAW USED %RAW USED
392-
ssd 202 GiB 200 GiB 2.0 GiB 2.0 GiB 1.00
393-
TOTAL 202 GiB 200 GiB 2.0 GiB 2.0 GiB 1.00
394-
393+
hdd 5.4 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.58
394+
ssd 22 TiB 19 TiB 2.7 TiB 2.7 TiB 12.36
395+
TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.32
396+
395397
--- POOLS ---
396-
POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
397-
device_health_metrics 1 1 242 KiB 15 KiB 227 KiB 4 251 KiB 24 KiB 227 KiB 0 297 GiB N/A N/A 4 0 B 0 B
398-
cephfs.a.meta 2 32 6.8 KiB 6.8 KiB 0 B 22 96 KiB 96 KiB 0 B 0 297 GiB N/A N/A 22 0 B 0 B
399-
cephfs.a.data 3 32 0 B 0 B 0 B 0 0 B 0 B 0 B 0 99 GiB N/A N/A 0 0 B 0 B
400-
test 4 32 22 MiB 22 MiB 50 KiB 248 19 MiB 19 MiB 50 KiB 0 297 GiB N/A N/A 248 0 B 0 B
398+
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
399+
.mgr 11 1 558 MiB 141 1.6 GiB 0 5.8 TiB
400+
cephfs_meta 13 1024 166 GiB 14.59M 499 GiB 2.74 5.8 TiB
401+
cephfs_data 14 1024 0 B 1.17G 0 B 0 5.8 TiB
402+
cephfsECvol 19 2048 2.8 PiB 1.81G 3.5 PiB 83.79 561 TiB
403+
.nfs 20 32 9.7 KiB 61 118 KiB 0 5.8 TiB
404+
testbench 71 32 12 GiB 3.14k 37 GiB 0 234 TiB
405+
default.rgw.buckets.data 76 2048 482 TiB 132.09M 643 TiB 47.85 526 TiB
406+
.rgw.root 97 1 1.4 KiB 4 48 KiB 0 5.8 TiB
407+
default.rgw.log 98 256 3.6 KiB 209 408 KiB 0 5.8 TiB
408+
default.rgw.control 99 1 0 B 8 0 B 0 5.8 TiB
409+
default.rgw.meta 100 128 3.8 KiB 20 194 KiB 0 5.8 TiB
410+
default.rgw.buckets.index 101 256 4.2 MiB 33 13 MiB 0 5.8 TiB
411+
default.rgw.buckets.non-ec 102 128 5.6 MiB 13 17 MiB 0 5.8 TiB
412+
kubedata 104 256 63 GiB 17.65k 188 GiB 0.03 234 TiB
413+
kubemeta 105 256 241 MiB 166 724 MiB 0 5.8 TiB
401414
402-
- **CLASS:** For example, "ssd" or "hdd".
415+
- **CLASS:** Statistics for each CRUSH device class present, for example, ``ssd`` and ``hdd``.
403416
- **SIZE:** The amount of storage capacity managed by the cluster.
404417
- **AVAIL:** The amount of free space available in the cluster.
405418
- **USED:** The amount of raw storage consumed by user data (excluding
406419
BlueStore's database).
407420
- **RAW USED:** The amount of raw storage consumed by user data, internal
408421
overhead, and reserved capacity.
409422
- **%RAW USED:** The percentage of raw storage used. Watch this number in
410-
conjunction with ``full ratio`` and ``near full ratio`` to be forewarned when
423+
conjunction with ``backfillfull ratio`` and ``near full ratio`` to be forewarned when
411424
your cluster approaches the fullness thresholds. See `Storage Capacity`_.
412425

426+
Additional information may be displayed by invoking as below:
427+
428+
.. prompt:: bash $
429+
430+
ceph df detail
431+
432+
The output now resembles the below example::
433+
434+
--- RAW STORAGE ---
435+
CLASS SIZE AVAIL USED RAW USED %RAW USED
436+
hdd 5.4 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.58
437+
ssd 22 TiB 19 TiB 2.7 TiB 2.7 TiB 12.36
438+
TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.32
439+
440+
--- POOLS ---
441+
POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
442+
.mgr 11 1 558 MiB 558 MiB 0 B 141 1.6 GiB 1.6 GiB 0 B 0 5.8 TiB N/A N/A N/A 0 B 0 B
443+
cephfs_meta 13 1024 166 GiB 206 MiB 166 GiB 14.59M 499 GiB 618 MiB 498 GiB 2.74 5.8 TiB N/A N/A N/A 0 B 0 B
444+
cephfs_data 14 1024 0 B 0 B 0 B 1.17G 0 B 0 B 0 B 0 5.8 TiB N/A N/A N/A 0 B 0 B
445+
cephfsECvol 19 2048 2.8 PiB 2.8 PiB 17 KiB 1.81G 3.5 PiB 3.5 PiB 21 KiB 83.79 561 TiB N/A N/A N/A 0 B 0 B
446+
.nfs 20 32 9.7 KiB 2.2 KiB 7.5 KiB 61 118 KiB 96 KiB 22 KiB 0 5.8 TiB N/A N/A N/A 0 B 0 B
447+
testbench 71 32 12 GiB 12 GiB 2.3 KiB 3.14k 37 GiB 37 GiB 6.9 KiB 0 234 TiB N/A N/A N/A 0 B 0 B
448+
default.rgw.buckets.data 76 2048 482 TiB 482 TiB 0 B 132.09M 643 TiB 643 TiB 0 B 47.85 526 TiB N/A N/A N/A 312 MiB 623 MiB
449+
.rgw.root 97 1 1.4 KiB 1.4 KiB 0 B 4 48 KiB 48 KiB 0 B 0 5.8 TiB N/A N/A N/A 0 B 0 B
450+
default.rgw.log 98 256 3.6 KiB 3.6 KiB 0 B 209 408 KiB 408 KiB 0 B 0 5.8 TiB N/A N/A N/A 0 B 0 B
451+
default.rgw.control 99 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 5.8 TiB N/A N/A N/A 0 B 0 B
452+
default.rgw.meta 100 128 3.8 KiB 3.2 KiB 671 B 20 194 KiB 192 KiB 2.0 KiB 0 5.8 TiB N/A N/A N/A 0 B 0 B
453+
default.rgw.buckets.index 101 256 4.2 MiB 0 B 4.2 MiB 33 13 MiB 0 B 13 MiB 0 5.8 TiB N/A N/A N/A 0 B 0 B
454+
default.rgw.buckets.non-ec 102 128 5.6 MiB 0 B 5.6 MiB 13 17 MiB 0 B 17 MiB 0 5.8 TiB N/A N/A N/A 0 B 0 B
455+
kubedata 104 256 63 GiB 63 GiB 0 B 17.65k 188 GiB 188 GiB 0 B 0.03 234 TiB N/A 20 TiB N/A 0 B 0 B
456+
kubemeta 105 256 241 MiB 241 MiB 278 KiB 166 723 MiB 722 MiB 833 KiB 0 5.8 TiB N/A N/A N/A 0 B 0 B
457+
458+
413459

414460
**POOLS:**
415461

@@ -815,4 +861,4 @@ pool if needed with a command of the following form:
815861

816862
ceph osd pool clear-availability-status <pool-name>
817863

818-
Note: Clearing a score is not allowed if the feature itself is disabled.
864+
Note: Clearing a score is not allowed if the feature itself is disabled.

0 commit comments

Comments
 (0)