Ostree Corruption after update, ostree crc check in Verify Target ?

[NoelsJaan]

A small percentage of devices have corruption of large binaries in the system after an update.

bash-5.1# ostree fsck -a > /dev/null
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object bb445aa4d3652e8a7b087670c5d86151175ebdd43ff5bc1d593f08b7238a4afb: Corrupted file object; checksum expected='bb445aa4d3652e8a7b087670c5d86151175ebdd43ff5bc1d593f08b7238a4afb' actual='ed75b95c4e91c0f2daa104a48340a625331cbe5886477cce6d393b91ecc7f828' -> 42M -> /usr/lib/go/pkg/linux_arm64_dynlink/libstd.so
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object 323173b724c6e2a5572095bba472a9822a8b448656c118bfbf0868262601543a: Corrupted file object; checksum expected='323173b724c6e2a5572095bba472a9822a8b448656c118bfbf0868262601543a' actual='0ba4cbcec8c4e0d5153cca4403866b2bcd68f4829dfdfc1f1c679803c3c8b254’ -> 12M -> /usr/bin/containerd-shim-runc-v2
In commits 28e851552e4ee006cae874b1e2dcd3dfc16224ce90edc87607ca69207e535480: fsck content object ff19d31aaa80392a48f43ca5742e5e7e213559e3ae267a1675773779a15e1169: Corrupted file object; checksum expected='ff19d31aaa80392a48f43ca5742e5e7e213559e3ae267a1675773779a15e1169' actual='f490c99bb19776eeb3b6c539878c28c539080363cbcb6c66bc3f1af30e1d63e6' -> 49M -> /usr/bin/docker
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object c363ef1e3d41a9b3237a1eca7d48cfa481259acf723311e1ca86b740c6763d1d: Corrupted file object; checksum expected='c363ef1e3d41a9b3237a1eca7d48cfa481259acf723311e1ca86b740c6763d1d' actual='877dbbc9214af4235580d6df8fd8d5a3382d98ee216befdd5de6a6ec32571592' -> 61M -> /usr/bin/dockerd
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object 74974a3d70f75a2fd687feccae99e88bea01a3f9d4b8cf4eeeebbbaf37da5282: Corrupted file object; checksum expected='74974a3d70f75a2fd687feccae99e88bea01a3f9d4b8cf4eeeebbbaf37da5282' actual='b13ebd4b274a4878313f4c622cc0df8c4f62553fb53348804be1612f350cb1ae' -> 23M -> /usr/bin/containerd-ctr
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object a21ec8800cfa4b2df8372e64434cc98cd44c3467be867b83c65a55393588ca89: Corrupted file object; checksum expected='a21ec8800cfa4b2df8372e64434cc98cd44c3467be867b83c65a55393588ca89' actual='57bff6b0fbadffc1ef0760b08e0b3cfeaf557bf7df9cc69c4e1b4642b22f1fc0' -> 48M -> /bin/containerd
In commits b0092d0bb7af9b787450ed1301dff04ffc2f449164d72151ef3c5b43492aa585: fsck content object 5e360e588b0df447f8dd894ac9b521ab3d88613abba741b7ddd003ab563a9fb1: Corrupted file object; checksum expected='5e360e588b0df447f8dd894ac9b521ab3d88613abba741b7ddd003ab563a9fb1' actual='c46cf83b73509dbd8d83e6cdba829b598f8cdb266617aaf975cd1644447e4cb7' -> 53M -> /usr/lib/docker/cli-plugins/docker-compose
error: Repository corruption encountered

Checking the sha256sum of these binaries between a healthy and unhealthy system indeed show the crc's don't match

In

aktualizr-lite/src/main.cc

Line 203 in e0074a2

if (client.VerifyTarget(target) != TargetStatus::kGood) {

the contents of the pull be checked by checksum ? This verify is retraced to the https://github.com/foundriesio/aktualizr repo. I will make a issue here:
foundriesio/aktualizr#80

Thanks in advance for the help!

Links

ostreedev/ostree#2732 <- describes the exact issue

[mike-sul]

Hi @NoelsJaan! Thanks for posting this issue, and apologies for not responding sooner—I was sick last week.

The verifyTarget function is not intended for checking disk corruption. Instead, it is meant to verify the new target content—specifically, the differences between the current target's OSTree objects/files and the new target's objects that have been pulled by the "download target" call.

In the case of OSTree, the "download target" operation involves invoking ostree pull, which generates a checksum on the data written to the file system and verifies whether the resulting checksum matches the expected one. Therefore, ostree pull ensures the integrity of objects as they are pulled and written to the file system.

However, ostree pull does not verify a file or object after it has been written to the file system. If the file system becomes corrupted and a newly pulled file/object is not properly persisted due to this corruption, ostree pull will not detect the issue.

The question is: should ostree pull or the verifyTarget call be responsible for checking whether the file system is corrupted? IMO, the answer to this question: "No".

To summarize, we should clearly distinguish between two aspects:

1. Verifying the integrity (checksum) of content/data as it is pulled and written to the file system during the download phase of the OTA process.

2. Detecting file system corruption and reacting to it if it occurs.

Regarding the aspect 1: This check is already in place—aktualizr-lite and ostree pull handles it correctly (see the ostree source code, e.g. https://github.com/ostreedev/ostree/blob/364e22f19c7dcc44f76dbc8c4a844f55d0d11e39/src/libostree/ostree-repo-pull.c#L877).

Regarding the aspect 2: I believe this is outside the scope of both OSTree and the update agent client (aktualizr-lite). Instead, this is a system-level issue that should be handled by a separate system component responsible for periodically checking file system consistency and responding to disk failures when detected.

[mike-sul]

We could probably check the integrity of each file in a new target's OSTree commit (i.e. each file in the new target's root file system) just after the OTA update is completed and before the device reboots. If an integrity inconsistency is detected, a rollback should be triggered.

Additionally, the integrity check can be performed at boot time, allowing the bootloader to trigger a rollback during the boot process if necessary.

In fact, enabling fsverity in OSTree (and optionally composefs for authenticity) ensures that the above-mentioned checks are performed during ostree pull and ostree deploy (the final step of the OTA installation phase).

Furthermore, we provide functionality to support rollback in the bootloader if an integrity inconsistency is detected at boot time. Please contact our customer support team for details on enabling this functionality in your factory setup.

However, even these measures only mitigate the problem—they do not fully resolve it if disk corruption starts occurring. In many scenarios, they may not help, such as:

1. One or more files in the currently running root file system are also corrupted, causing the system to roll back from one corrupted root file system to another.

2. The aktualizr-lite binary itself is corrupted.

[mike-sul]

ostreedev/ostree#2732 <- describes the exact issue

I'm not sure this issue is exactly the same as the one you encountered. As you can see, the ostree fsck call you invoked reports inconsistency issues for multiple commits. This suggests that at least one of the objects was already corrupted before the OTA process (ostree pull) even started.

I highly doubt that ostree pull causes file corruption. Everything indicates that this is simply a case of file system corruption and is unrelated to ostree pull or the OTA process.