Skip to content

Commit 3914f86

Browse files
authored
Merge pull request ceph#61493 from zdover23/wip-doc-2025-01-23-cephfs-disaster-recovery-experts
doc/cephfs: edit disaster-recovery-experts (5 of x) Reviewed-by: Anthony D'Atri <[email protected]>
2 parents f646562 + 9af7d97 commit 3914f86

File tree

1 file changed

+74
-70
lines changed

1 file changed

+74
-70
lines changed

doc/cephfs/disaster-recovery-experts.rst

Lines changed: 74 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -229,126 +229,130 @@ recovery since the existing metadata pool would not be modified.
229229
contents of the data pool while this is the case. After recovery is
230230
complete, archive or delete the damaged metadata pool.
231231

232-
To begin, the existing file system should be taken down to prevent further
233-
modification of the data pool. Unmount all clients and then use the following
234-
command to mark the file system failed:
232+
#. To begin, the existing file system should be taken down to prevent further
233+
modification of the data pool. Unmount all clients and then use the
234+
following command to mark the file system failed:
235235

236-
.. prompt:: bash #
236+
.. prompt:: bash #
237237

238-
ceph fs fail <fs_name>
238+
ceph fs fail <fs_name>
239239

240-
.. note::
240+
.. note::
241241

242-
``<fs_name>`` here and below refers to the original, damaged file system.
242+
``<fs_name>`` here and below refers to the original, damaged file system.
243243

244-
Next, create a recovery file system in which we will populate a new metadata pool
245-
that is backed by the original data pool:
244+
#. Next, create a recovery file system in which we will populate a new metadata
245+
pool that is backed by the original data pool:
246246

247-
.. prompt:: bash #
247+
.. prompt:: bash #
248248

249-
ceph osd pool create cephfs_recovery_meta
250-
ceph fs new cephfs_recovery cephfs_recovery_meta <data_pool> --recover --allow-dangerous-metadata-overlay
249+
ceph osd pool create cephfs_recovery_meta
250+
ceph fs new cephfs_recovery cephfs_recovery_meta <data_pool> --recover --allow-dangerous-metadata-overlay
251251

252-
.. note::
252+
.. note::
253253

254-
You may rename the recovery metadata pool and file system at a future time.
255-
The ``--recover`` flag prevents any MDS daemon from joining the new file
256-
system.
254+
You may rename the recovery metadata pool and file system at a future time.
255+
The ``--recover`` flag prevents any MDS daemon from joining the new file
256+
system.
257257

258-
Next, we will create the intial metadata for the fs:
258+
#. Next, we will create the intial metadata for the fs:
259259

260-
.. prompt:: bash #
260+
.. prompt:: bash #
261261

262-
cephfs-table-tool cephfs_recovery:0 reset session
262+
cephfs-table-tool cephfs_recovery:0 reset session
263263

264-
.. prompt:: bash #
264+
.. prompt:: bash #
265265

266-
cephfs-table-tool cephfs_recovery:0 reset snap
266+
cephfs-table-tool cephfs_recovery:0 reset snap
267267

268-
.. prompt:: bash #
268+
.. prompt:: bash #
269269

270-
cephfs-table-tool cephfs_recovery:0 reset inode
270+
cephfs-table-tool cephfs_recovery:0 reset inode
271271

272-
.. prompt:: bash #
272+
.. prompt:: bash #
273273

274-
cephfs-journal-tool --rank cephfs_recovery:0 journal reset --force --yes-i-really-really-mean-it
274+
cephfs-journal-tool --rank cephfs_recovery:0 journal reset --force --yes-i-really-really-mean-it
275275

276-
Now perform the recovery of the metadata pool from the data pool:
276+
#. Now perform the recovery of the metadata pool from the data pool:
277277

278-
.. prompt:: bash #
278+
.. prompt:: bash #
279279

280-
cephfs-data-scan init --force-init --filesystem cephfs_recovery --alternate-pool cephfs_recovery_meta
280+
cephfs-data-scan init --force-init --filesystem cephfs_recovery --alternate-pool cephfs_recovery_meta
281281

282-
.. prompt:: bash #
282+
.. prompt:: bash #
283283

284-
cephfs-data-scan scan_extents --alternate-pool cephfs_recovery_meta --filesystem <fs_name>
284+
cephfs-data-scan scan_extents --alternate-pool cephfs_recovery_meta --filesystem <fs_name>
285285

286-
.. prompt:: bash #
286+
.. prompt:: bash #
287287

288-
cephfs-data-scan scan_inodes --alternate-pool cephfs_recovery_meta --filesystem <fs_name> --force-corrupt
288+
cephfs-data-scan scan_inodes --alternate-pool cephfs_recovery_meta --filesystem <fs_name> --force-corrupt
289289

290-
.. prompt:: bash #
290+
.. prompt:: bash #
291291

292-
cephfs-data-scan scan_links --filesystem cephfs_recovery
292+
cephfs-data-scan scan_links --filesystem cephfs_recovery
293293

294-
.. note::
294+
.. note::
295295

296-
Each of the scan procedures above scans through the entire data pool. This
297-
may take a long time. See the previous section on how to distribute this
298-
task among workers.
296+
Each of the scan procedures above scans through the entire data pool.
297+
This may take a long time. See the previous section on how to distribute
298+
this task among workers.
299299

300-
If the damaged file system contains dirty journal data, it may be recovered next
301-
with a command of the following form:
300+
If the damaged file system contains dirty journal data, it may be recovered
301+
next with a command of the following form:
302302

303-
.. prompt:: bash #
303+
.. prompt:: bash #
304304

305-
cephfs-journal-tool --rank=<fs_name>:0 event recover_dentries list --alternate-pool cephfs_recovery_meta
305+
cephfs-journal-tool --rank=<fs_name>:0 event recover_dentries list --alternate-pool cephfs_recovery_meta
306306

307-
After recovery, some recovered directories will have incorrect statistics.
308-
Ensure that the parameters ``mds_verify_scatter`` and ``mds_debug_scatterstat``
309-
are set to false (the default) to prevent the MDS from checking the statistics:
307+
#. After recovery, some recovered directories will have incorrect statistics.
308+
Ensure that the parameters ``mds_verify_scatter`` and
309+
``mds_debug_scatterstat`` are set to false (the default) to prevent the MDS
310+
from checking the statistics:
310311

311-
.. prompt:: bash #
312+
.. prompt:: bash #
312313

313-
ceph config rm mds mds_verify_scatter
314+
ceph config rm mds mds_verify_scatter
314315

315-
.. prompt:: bash #
316+
.. prompt:: bash #
316317

317-
ceph config rm mds mds_debug_scatterstat
318+
ceph config rm mds mds_debug_scatterstat
318319

319-
.. note::
320+
.. note::
320321

321-
Verify that the config has not been set globally or with a local ``ceph.conf`` file.
322+
Verify that the config has not been set globally or with a local
323+
``ceph.conf`` file.
322324

323-
Now, allow an MDS daemon to join the recovery file system:
325+
#. Now, allow an MDS daemon to join the recovery file system:
324326

325-
.. prompt:: bash #
327+
.. prompt:: bash #
326328

327-
ceph fs set cephfs_recovery joinable true
329+
ceph fs set cephfs_recovery joinable true
328330

329-
Finally, run a forward :doc:`scrub </cephfs/scrub>` to repair recursive statistics.
330-
Ensure that you have an MDS daemon running and issue the following command:
331+
#. Finally, run a forward :doc:`scrub </cephfs/scrub>` to repair recursive
332+
statistics. Ensure that you have an MDS daemon running and issue the
333+
following command:
331334

332-
.. prompt:: bash #
335+
.. prompt:: bash #
333336

334-
ceph tell mds.cephfs_recovery:0 scrub start / recursive,repair,force
337+
ceph tell mds.cephfs_recovery:0 scrub start / recursive,repair,force
335338

336-
.. note::
339+
.. note::
337340

338-
The `Symbolic link recovery <https://tracker.ceph.com/issues/46166>`_ is
339-
supported starting in the Quincy release.
341+
The `Symbolic link recovery <https://tracker.ceph.com/issues/46166>`_ is
342+
supported starting in the Quincy release.
340343

341-
Symbolic links were recovered as empty regular files before.
344+
Symbolic links were recovered as empty regular files before.
342345

343-
It is recommended that you migrate any data from the recovery file system as
344-
soon as possible. Do not restore the old file system while the recovery file
345-
system is operational.
346+
It is recommended that you migrate any data from the recovery file system as
347+
soon as possible. Do not restore the old file system while the recovery file
348+
system is operational.
346349

347-
.. note::
350+
.. note::
348351

349-
If the data pool is also corrupt, some files may not be restored because
350-
the backtrace information associated with them is lost. If any data objects
351-
are missing (due to issues like lost Placement Groups on the data pool),
352-
the recovered files will contain holes in place of the missing data.
352+
If the data pool is also corrupt, some files may not be restored because
353+
the backtrace information associated with them is lost. If any data
354+
objects are missing (due to issues like lost Placement Groups on the
355+
data pool), the recovered files will contain holes in place of the
356+
missing data.
353357

354358
.. _Symbolic link recovery: https://tracker.ceph.com/issues/46166

0 commit comments

Comments
 (0)