General reliability of ZFS with USB

[grahamperrin]

#9705 (comment):

In all my testing, I have never found ZFS to work reliably on a USB device. There's just something about the USB protocol it doesn't like. But perhaps others may have a different experience.

I do, and (if I'm not mistaken) some cases above do not involve USB, so maybe start a separate discussion about USB in general. Thanks.

---

#9705 (reply in thread):

Doesn't surprise me, all it takes to get USB to write corrupted data is to use a long extension cable, and writes succeed but when you read the files back (after clearing caches) they are corrupted. Since ZFS has a lot of checks for data corruption I imagine it picks this up very quickly, whereas other filesystems just leave you with corrupted data you never know about. I don't think it's a problem with ZFS (unless you consider it a problem that it doesn't retry the operation more times), it's more that ZFS is one of the few filesystems that notices when the storage media or cables are flaky.

#9705 (reply in thread):

I can partially confirm that. In my case, it runs fine, until it doesn't. The connection/USB must only express the slightest faint hiccup and the whole pool goes bonkers, usually requiring a node reboot....

#9705 (reply in thread):

I don't know why so many people on Linux report USB "hickups" ending up in device disconnect/reconnect, but comparing to that, I know that both SAS and SATA hardware, firmware and drivers usually implement some sort debouncing, giving "lost"/"crashed" device few seconds to recover before reporting it as gone. I don't know about this in USB, but in my opinion it is not a job of file system to handle it.

#9705 (reply in thread):

Problem is, for SAS and SATA the protocols are designed for storage, so retrying a failed read or write operation is pretty straightfoward. For USB, the USB layer doesn't know what the data it's passing around actually does, as it can encapsulate many different protocols, so after a reconnect it doesn't know whether it's safe to re-send the same data packet again or not as it doesn't know what side effects it may have.

Support is needed from elsewhere in the kernel, typically the block layer, to know what the operation was and to redo it when the device comes back.

But the problem is that the USB device often reconnects before the kernel has finished processing the disconnect message, so the kernel sees it as a new device (e.g. it moves from /dev/sda to /dev/sdb). This means the block layer loses track of it and so can't resubmit the operation, meaning the failure gets passed up the chain to the filesystem level.

On most filesystems they will just return an error saying the storage device went away, and you unmount the old device and remount the new one and keep going until the next USB dropout. But since ZFS automatically picks up storage devices, when the USB device reappears ZFS recognises the metadata on it and adds the new device node to the pool, but then it sees the last writes it attempted didn't make it to the disk, so it marks the device bad.

You can argue that it's not the job of the filesystem to handle device dropouts, but in my opinion that argument only applies if the filesystem also does not handle adding and removing devices. Since ZFS picks up drives automatically, it needs to do a few extra things around that too, like handling transient device dropouts. Compare it to the other Linux filesystems and you need to use mdraid to join multiple drives together, and mdraid is the component that adds drives to the array and so it's mdraid that handles block-level retries. The filesystem does not handle it in this case as you say, because the RAID layer is responsible for that. But again, ZFS has the RAID layer built in, so...

#9705 (reply in thread):

This means the block layer loses track of it and so can't resubmit the operation, meaning the failure gets passed up the chain to the filesystem level.

Right. And it only means it is up to USB to not report device as gone before it is really sure. It is not required to retry by itself. Block layer and ZFS should be able to handle I/O errors.

Compare it to the other Linux filesystems and you need to use mdraid to join multiple drives together

Would your pool include redundancy, it would just happily reattached the lost device and started resilver for it to make sure it is consistent. But typically users who use USB don't have redundancy, so lost device is a game over.

#9705 (reply in thread):

That sounds like USB is totally unsafe. The slightest malfunction leads to corrupt data etc. I only want to say that I have two JBODs with RAIDZ2 (8 and 5 drives) attached via USB. I have this setup since several years and never(!) had an issue. I suggest if you are experienceing frequent USB issues that you check your cables and your controllers. Specifically cheap USB controllers in cheap extern USB cases might be an issue. And turn off any type of auto suspend for USB.

Not so.

#9705 (reply in thread):

Out of interest what are the JBODS you use? I've found this a very hard to get going solution, even with SAS drives, from my research there's almost nothing reliable as a 4-5 bay unit until you get to proper SAS designed units.

#9705 (reply in thread):

I have a FANTEC Qb-X8US3-6G with 8 bays and a Qnap TL-D800C also with 8 bays. The Fantec is pretty old, purchased in 2017. The QNAP was pruchased in 2024 (replacing an old 4 bay FANTEC QB-35US3-6G).

#9705 (reply in thread):

Thanks. What OS are you running?

#9705 (reply in thread):

Linux (Endeavouros)

#9705 (reply in thread):

Thanks, I tried an earlier QNAP version and it did not work well under linux so this is great to know.

#9705 (reply in thread):

Doesn't surprise me, all it takes to get USB to write corrupted data is to use a long extension cable, and writes succeed but when you read the files back (after clearing caches) they are corrupted.

Can you please provide a proof for that statement? What you are saying is that we can not trust a USB drive that is reporting back a successful write. That is a very bolt statement. And if it is true it would render USB completely useless for storage devices. Can you provide some supporting material for this statement?

#9705 (reply in thread):

I can't provide proof as it's only anecdotal. But I have a USB 3.0 PCIe card in a Dell PowerEdge R720 connected to a bus-powered three metre active USB extension cable. If I plug a SanDisk Extreme USB 3.0 USB stick into the extension cable, then write files onto it, they will randomly become corrupted. Reading them back straight away is fine (because of the kernel cache) but removing and reattaching the drive reveals the files have different SHA256 hashes on them.

I don't know if it's the PCIe controller to blame, the active extension cable, or the SanDisk USB device itself, but there's no warning of any failed writes, and no I/O errors in dmesg. I have to plug the same USB stick into a front-panel USB port on that machine (which unfortunately is only USB 2.0) in order to copy files reliably. Using the same USB stick on a different machine's USB 3.0 ports is fine. I initially thought it could be a voltage drop at the end of the cable but I tried injecting power (and verifying it with a USB diagnostic tool and the SanDisk device connected at the same time) and the voltage was fine when the injector was in use, however that didn't help with the data corruption. I haven't experimented beyond that.

Interestingly I don't get any device disconnect/connect messages in the kernel logs like you usually do with low quality cables or dodgy connectors.

#9705 (reply in thread):

I don't know if it's the PCIe controller to blame, the active extension cable, or the SanDisk USB device itself, but there's no warning of any failed writes, and no I/O errors in dmesg.

The question here is if this is a generic USB issue or if this is hardware related. The USB protocol uses checksums (CRC) to detect transmission errors. A silent data corruption like you describe should not happen. Soemthing is odd in your case.

Check this document as example:

About the USB Protocol, Common USB Bus Errors, and How to Troubleshoot Them - Total Phase

It says:

Each endpoint is different, specified through their bandwidth requirements and the way they transfer data. The four types of USB data transfers include: Control, Isochronous, Interrupt, and Bulk transfers.

Control: Non-periodic transfers. Typically, used for device configuration, commands, and status operation. Interrupt: This is a transaction that is guaranteed to occur within a certain time interval. The device will specify the time interval at which the host should check the device to see if there is new data. This is used by input devices such as mice and keyboards. Isochronous: Periodic and continuous transfer for time-sensitive data. There is no error checking or retransmission of the data sent in these packets. This is used for devices that need to reserve bandwidth and have a high tolerance to errors. Examples include multimedia devices for audio and video. Bulk: General transfer scheme for large amounts of data. This is for contexts where it is more important that the data is transmitted without errors than for the data to arrive in a timely manner. Bulk transfers have the lowest priority. If the bus is busy with other transfers, this transaction may be delayed. The data is guaranteed to arrive without error. If an error is detected in the CRCs, the data will be retransmitted. Examples of this type of transfer are files from a mass storage device or the output from a scanner.

#9705 (reply in thread):

CRC is not perfect. It can reliably detect only certain number of errors per packet, and according to internet in some patterns it can be as low as 4 for CRC16 used in USB2. SATA, SAS, PCIe and seems USB3 use CRC32. But if user is not notified anyhow about the routinely corrected errors (users tend to react badly on them, so vendors prefer to hide them) to see that something is not great, then how can one be sure the threshold won't be reached one day? I had some experience with using random PCIe 4.0/5.0 SSDs in random PCIe risers from Amazon, not qualified for 5.0, and while I haven't noticed a data corruptions (those are a development systems, so I don't care), there were all kinds of other troubles: detection problems, command timeouts, etc.

[marshalleq]

This whole thread is now out of context. My original comment was in context of the other linked thread as I believe others were even if someone there did not want it to be. It was solely to be helpful and provide something to consider. I am sorry that you could not understand this context and felt you had to disagree with it all and create its own thread. I won't be engaging further with this one as there is no point now the context is lost. May as well be deleted to be honest.

[grahamperrin]

Please recall that when issue 9705 was reported – more than five years ago (2019), before conversion to a discussion – it was specific:

• detection of permanent errors in two metadata files

Permanent errors have been detected in the following files

• metadata, not data

– and in my experience, over the years, such errors are not permanent. The second respondent observed:

You may need to run zpool scrub twice to clear the error message buffer. …

---

… in context of the other linked thread …

You link to the comment from me (13726065), five days ago, that suggested a separate discussion.

Where you wrote about something:

In all my testing, I have never found ZFS to work reliably on a USB device. There's just something about the USB protocol it doesn't like. …

– it seemed to lose sight of the originally reported issue.

Loss of focus with so vague a comment, with some responses inevitably so far off-topic from the issue, is not helpful to people who experience the issue, people who might want discussion to be fairly relevant and fairly concise.

In that overflow of thirteen (at the time of writing), I can't guess how many are on-topic to what was originally reported …

… and so on.

---

GitHub discussions are a less than ideal venue for discussions. Please don't make things unnecessarily difficult.

Thank you.

[marshalleq]

Unsubscribed.