Unable to import ZFS pool with all disks online

[FallingSnow]

After my system locked up I hard reset the system. After reboot I am no longer able to import the pool. The "action" states that the pool can be imported, however when I try to it fails with insufficient replicas even though all disks are online.

➜  ~ sudo zpool import
   pool: STORAGE
     id: 13430433581951149895
  state: ONLINE
status: One or more devices contains corrupted data.
 action: The pool can be imported using its name or numeric identifier.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
 config:

        STORAGE                                              ONLINE
          mirror-0                                           ONLINE
            wwn-0x5000cca2c1c857a9                           ONLINE
            wwn-0x5000cca2c1c85c06                           ONLINE
        cache
          nvme1n1p2
          nvme0n1p2
        logs
          mirror-2                                           ONLINE
            nvme-eui.e8238fa6bf530001001b448b41d65cfd-part1  ONLINE
            nvme-eui.e8238fa6bf530001001b448b41d68250-part1  ONLINE
➜  ~ sudo zpool import STORAGE
cannot import 'STORAGE': insufficient replicas
        Destroy and re-create the pool from
        a backup source.

Debug log:

timestamp    message
1697863039   spa.c:6400:spa_tryimport(): spa_tryimport: importing STORAGE
1697863039   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1697863039   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa $import. txg 1223953
1697863039   spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=1223953
1697863039   spa.c:8543:spa_async_request(): spa=$import async request task=4096
1697863039   spa.c:8543:spa_async_request(): spa=$import async request task=4096
1697863039   spa.c:8543:spa_async_request(): spa=$import async request task=2048
1697863039   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1697863039   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1697863090   spa.c:6400:spa_tryimport(): spa_tryimport: importing STORAGE
1697863090   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1697863090   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa $import. txg 1223953
1697863090   spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=1223953
1697863090   spa.c:8543:spa_async_request(): spa=$import async request task=4096
1697863090   spa.c:8543:spa_async_request(): spa=$import async request task=4096
1697863090   spa.c:8543:spa_async_request(): spa=$import async request task=2048
1697863090   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1697863090   spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1697863090   spa.c:6252:spa_import(): spa_import: importing STORAGE
1697863090   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING
1697863090   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa STORAGE. txg 1223953
1697863090   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223953
1697863090   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863090   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863093   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52]
1697863093   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52]
1697863093   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING
1697863093   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): spa_load_retry: rewind, max txg: 1223952
1697863093   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING
1697863093   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c85c06-part1': best uberblock found for spa STORAGE. txg 1223952
1697863093   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223952
1697863093   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863093   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863096   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52]
1697863096   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52]
1697863096   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING
1697863096   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): spa_load_retry: rewind, max txg: 1223951
1697863096   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING
1697863096   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c85c06-part1': best uberblock found for spa STORAGE. txg 1223949
1697863096   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223949
1697863096   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863096   spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096
1697863099   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52]
1697863099   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52]
1697863099   spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING

I am able to import in read only mode using sudo zpool import STORAGE -o readonly=on but I'd like to fix my pool so I can use it normally.

[IvanVolosyuk]

What does 'zfs version' say?

…

On Sat, Oct 21, 2023 at 4:54 PM Ayrton Sparling ***@***.***> wrote: After my system locked up I hard reset the system. After reboot I am no longer able to import the pool. The "action" states that the pool can be imported, however when I try to it fails with insufficient replicas even though all disks are online. ➜ ~ sudo zpool import pool: STORAGE id: 13430433581951149895 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: STORAGE ONLINE mirror-0 ONLINE wwn-0x5000cca2c1c857a9 ONLINE wwn-0x5000cca2c1c85c06 ONLINE cache nvme1n1p2 nvme0n1p2 logs mirror-2 ONLINE nvme-eui.e8238fa6bf530001001b448b41d65cfd-part1 ONLINE nvme-eui.e8238fa6bf530001001b448b41d68250-part1 ONLINE ➜ ~ sudo zpool import STORAGE cannot import 'STORAGE': insufficient replicas Destroy and re-create the pool from a backup source. Debug log: timestamp message 1697863039 spa.c:6400:spa_tryimport(): spa_tryimport: importing STORAGE 1697863039 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING 1697863039 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa $import. txg 1223953 1697863039 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=1223953 1697863039 spa.c:8543:spa_async_request(): spa=$import async request task=4096 1697863039 spa.c:8543:spa_async_request(): spa=$import async request task=4096 1697863039 spa.c:8543:spa_async_request(): spa=$import async request task=2048 1697863039 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED 1697863039 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING 1697863090 spa.c:6400:spa_tryimport(): spa_tryimport: importing STORAGE 1697863090 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING 1697863090 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa $import. txg 1223953 1697863090 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=1223953 1697863090 spa.c:8543:spa_async_request(): spa=$import async request task=4096 1697863090 spa.c:8543:spa_async_request(): spa=$import async request task=4096 1697863090 spa.c:8543:spa_async_request(): spa=$import async request task=2048 1697863090 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED 1697863090 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING 1697863090 spa.c:6252:spa_import(): spa_import: importing STORAGE 1697863090 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING 1697863090 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c857a9-part1': best uberblock found for spa STORAGE. txg 1223953 1697863090 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223953 1697863090 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863090 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863093 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52] 1697863093 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52] 1697863093 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING 1697863093 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): spa_load_retry: rewind, max txg: 1223952 1697863093 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING 1697863093 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c85c06-part1': best uberblock found for spa STORAGE. txg 1223952 1697863093 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223952 1697863093 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863093 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863096 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52] 1697863096 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52] 1697863096 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING 1697863096 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): spa_load_retry: rewind, max txg: 1223951 1697863096 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): LOADING 1697863096 vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000cca2c1c85c06-part1': best uberblock found for spa STORAGE. txg 1223949 1697863096 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config untrusted): using uberblock with txg=1223949 1697863096 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863096 spa.c:8543:spa_async_request(): spa=STORAGE async request task=4096 1697863099 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52] 1697863099 spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted): FAILED: spa_ld_log_spacemaps failed [error=52] 1697863099 spa_misc.c:418:spa_load_note(): spa_load(STORAGE, config trusted): UNLOADING I am able to import in read only mode using sudo zpool import STORAGE -o readonly=on but I'd like to fix my pool so I can use it normally. — Reply to this email directly, view it on GitHub <#15429>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HOUBWY46UTDRHKERGTYANPSTAVCNFSM6AAAAAA6J5FJWGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVG42TSNBUGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[FallingSnow]

➜  / zfs version
zfs-2.2.0-1
zfs-kmod-2.2.0-1

But I just updated from 2.1.13 (which this happened on) because I though an updated zfs might help.

[FallingSnow]

I have a bunch of snapshots for all the datasets in the pool. Is there a way I could use those to recover the pool?

[justinclift]

When you import the data set read only, how much free space does it say is available?

Asking because of these lines in the debug log:

1697863093   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted):
  FAILED: spa_ld_log_sm_data(): failed at space_map_iterate(obj=1689) [error 52]
1697863093   spa_misc.c:404:spa_load_failed(): spa_load(STORAGE, config trusted):
  FAILED: spa_ld_log_spacemaps failed [error=52]

Those sound like "something" to do with the code that deals with free space.

[FallingSnow]

I'm pretty sure it does. This link is the first result when searching "space_map_iterate": https://www.klennet.com/notes/2021-04-26-zfs-and-power-failures.aspx.

I no longer have the pool so I can't tell you the free space value. I read the entire pool to a backup then recreated it to solve the issue. A process that took a few days copying around 9TB but it all worked out in the end. I didn't know ZFS could be corrupted by a power outage to make it unimportable.

[justinclift]

Oh wow. That's news to me too.

It sounds like there needs to be some mechanism to rebuild the free space map from scratch, even if time consuming, to eliminate this failure scenario.

[FallingSnow]

From my searching it seems like this has been discussed before and iirc there is an issue on here about adding the ability to rebuild free space maps. It just never went anywhere.

Found it: #3111