Problematic mount order of ZRAM

[hundertvolt1]

In some investigations about SD card writes and how ZRAM is mounted wrt. other mounts and services (details see this forum discussion from here), I found that there are two pending and closely linked issues concerning the way it is currently mounted:

<Issue 1> Wrong order configured in /etc/systemd/system/'srv-openhab\x2d<dirname>.mount.
Current state:

Before=smbd.service
After=network.target zram-config.service

This order intends to first mount the ZRAM overlays and then the bind-mount to the /srv/openhab-x directories, which leads to the bind-mount showing the contents of the physical, SD card mounted underlying directory instead of the live contents of the ZRAM overlay. ZRAM mounted folders inside /srv/openhab-x then show potentially very old data from the last graceful shutdown of the ZRAM service.

How to reproduce:
Easiest and cleanest way is to enforce the mount order manually (during boot-up it's critical due to race conditions, see <Issue 2> below!).
Stop all services:

sudo systemctl stop openhab  # just to be sure
sudo systemctl stop zram-config.service
sudo systemctl stop "srv-openhab\\x2duserdata.mount"

Start the mounts in the order "First ZRAM, then bind-mount":

sudo systemctl start zram-config.service
sudo systemctl start "srv-openhab\\x2duserdata.mount"

Then, look at the mount table - you'll get a single ZRAM overlay mount for each folder:

mount | grep overlay
overlay1 on /var/lib/openhab/foo type overlay (rw,relatime,lowerdir=/opt/zram/foo.bind,upperdir=/opt/zram/zram1/upper,workdir=/opt/zram/zram1/workdir,redirect_dir=on,uuid=on)
overlay2 on /var/lib/openhab/persistence type overlay (rw,relatime,lowerdir=/opt/zram/persistence.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on,uuid=on)

You can as well check the contents of the folders in the bind mount and at the origin. Bind mount will be older than origin and df -a will give you a "mmcblk...", while the actual ZRAM mount will be recent and will give you an "overlay...".

Desired state:
The /srv/openhab-x directories should contain live data and be exactly in the same state as their targets (e.g. /var/lib/openhab/persistence).

Suggested solution:
Start the bind mount first, then start ZRAM. zram-config.service is able to follow the bind mount and apply an additional mount of the overlay there as well. Modify /etc/systemd/system/'srv-openhab\x2d<dirname>.mount as such:

Before=smbd.service zram-config.service
After=network.target 

Also easy to manually reproduce; follow the steps in "How to reproduce", but exchange the order of the two sudo systemctl start commands. You'll see double ZRAM mounts for each folder:

overlay1 on /var/lib/openhab/foo type overlay (rw,relatime,lowerdir=/opt/zram/foo.bind,upperdir=/opt/zram/zram1/upper,workdir=/opt/zram/zram1/workdir,redirect_dir=on,uuid=on)
overlay1 on /srv/openhab-userdata/foo type overlay (rw,relatime,lowerdir=/opt/zram/foo.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on,uuid=on)

overlay2 on /var/lib/openhab/persistence type overlay (rw,relatime,lowerdir=/opt/zram/persistence.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on,uuid=on)
overlay2 on /srv/openhab-userdata/persistence type overlay (rw,relatime,lowerdir=/opt/zram/persistence.bind,upperdir=/opt/zram/zram2/upper,workdir=/opt/zram/zram2/workdir,redirect_dir=on,uuid=on)

one for the target, the other for the bind mount. Listing the contents of the ZRAM'ed folders will give you equal and up-to-date content, df -a will yield an overlay filesystem in them.

At that point, everything worked well on my Raspberry Pi 4 / 8GB OpenHABian system; the Samba shares on the /srv/ folders delivered the most recent, ZRAM contained data on the remote host as well.

But this good status is not necessarily certain, because....

<Issue 2> Ordering services to start after zram-config.service in systemd is strongly unreliable.
Some Background:
When making service starts depend on a simple mount like /etc/systemd/system/'srv-openhab\x2d<dirname>.mount which is known to systemd from the very beginning of boot-up, it is for sure that this will not happen before the mount is really usable.
So making zram-config.service wait until the bind-mount is in place, as suggested in my solution above, should be perfectly safe and stable.
Not so if waiting for zram-config.service itself. It will spawn several dynamic mounts when it's started - three per ZRAM'ed folder, as it looked to me - and then declare itself to be started. This arises two issues:

1. If you trigger further services when zram-config.service is done starting, this does not mean that the spawned mounts are ready. As a matter of fact, this starts a race condition (details see the forum post mentioned on top) of following services and ZRAM spawned mounts.
2. As these mount processes are spawned dynamically, systemd has no clue that they may even exist when booting up. Their naming follows a canonical scheme (like var-lib-openhab-persistence.mount), but although I tried many combinations to wait on them, none of them was able to solve the race condition. I tried adding all of these mounts in the After= section. Adding them in the Requires= section on top of it. Creating a zram-mounts.target file to contain all of them. Putting them in a RequiresMountsFor= which is said to be able to catch dynamic mounts. None of them worked for me - I could see actions and other services happen when the ZRAM mounts were mid-air on my system.

Conclusion:
I tried many possible solutions and did not get lucky so far. Although Solution (1) relieves the issue for the /srv/ bind-mount by putting it before even mounting ZRAM, and my system appears to work at that point, it leaves me with a very uncertain feeling.

There are many services still waiting until zram-config.service is done, but should instead wait until all ZRAM mounts are really done. The most prominent ones I found in /etc/systemd/system/zram.service.d/override.conf are:
Before=openhab.service exim4.service nmbd.service smbd.service
They, and others I didn't see yet, are potentially prone to hit a ZRAM mount which is not yet, or partially, or fully in place - depending on who wins the race.

So at that time, I have no ready solution, but strongly suggest that we find a way to make systemd wait stable and reliably on really-ready ZRAM before starting services.

[mstormi]

@ecdye fyi, this made ZRAM fail to start

I could not tag you on the forum?
https://community.openhab.org/t/sd-card-write-cycle-mitigation-my-experience-so-far/165494/68

[ecdye]

My tag on the forum changed to match my ecdye tag here. So that's why that happened.

As for the zram starting issues, yes there are many complexities with systemd and getting it to mount and behave nicely. I have had many issues to be as stable as it is. Unfortunately with systemd being what it is, I don't know if there is an easy way to resolve this issue.

I'm happy to look into it more when I have time, but to be honest, I don't know that systemd is going to allow us to solve it easily. I may have to get creative and find another strategy.

[mstormi]

Yeah I myself have been fighting this in quite a while and sorta gave up at some 'reasonable' point which now is today's state.

Then again that recent proposal seems to be working. Need to test more, though.

Just moving to OH5. Enough stuff that does not work :)

[hundertvolt1]

I found a solution with minimal changes which seems to be working at first sight; for not repeating everything, I hope it's okay to put a link to the forum post in which I described it.

For <Issue 2> I also found a way how to securely and reliably wait for ZRAM to be fully up. Here as well, I described it in detail in a forum post. It basically is creating a systemd target file with (quite rich) contents fitting to /etc/ztab and replacing dependencies to wait on zram-config.service by zram-ready.target.

In the hopeful case that the solution works as a whole and the starting order of the /srv/openhab-x and zram-config.service can be exchanged, services currently waiting for the bind-mount to be up should then be replaced by waiting for ZRAM target.

[ecdye]

Thanks @hundertvolt1 I'll take a look at it when I have a chance

[ecdye]

OK, I'm just now looking at this, and simply from a technical standpoint, you are absolutely correct on issue 1 and @mstormi I think that we are good to remove the dependency of After=network.target from those samba mounts. WDYT?

I'm reading through issue 2 now and I'll keep you posted as I get through it, it might take a bit because systemd is a PITA with ordering and I want to test it throughly before I commit to any changes outright.

[mstormi]

on my very own box I had to revert this because zram didn't start. So any changes please only after extensive testing.
@hundertvolt1 can you create a BATS test for this ?

[hundertvolt1]

on my very own box I had to revert this because zram didn't start.

@mstormi Just for understanding - do you refer to the issues you already mentioned in the forum some days ago which I could reproduce and fix on a fresh system here, or is it still not working even with the fix (-> remove After=network.target) applied?

can you create a BATS test for this

Argh... difficult. TBH, I heard from BATS the first time right now. I certainly do get the idea of it, as well as yours, but learning a new testing framework from scratch besides family and job is out of scope at the moment... especially if it's not about actually testing bash script, but forging some intricate systemd stuff into a bash test. Sorry...

[mstormi]

on my very own box I had to revert this because zram didn't start.
or is it still not working even with the fix (-> remove After=network.target) applied?

This.

can you create a BATS test for this

Argh... difficult. TBH, I heard from BATS the first time right now. I certainly do get the idea of it, as well as yours, but learning a new testing framework from scratch besides family and job is out of scope at the moment... especially if it's not about actually testing bash script, but forging some intricate systemd stuff into a bash test. Sorry...

Shouldn't be hard. On every PR update, a virtual box is deployed to the cloud incl. all SMB mounts.
All you have to test for via BATS is if zram is up and if the exported mounts are up

[hundertvolt1]

All you have to test for via BATS is if zram is up and if the exported mounts are up

Ah,, maybe I thought it too deep. So you're not thinking about extensive tests for many different config and cases, but just for checking if ZRAM service is running and the bind mounts were successful, and that's all?

[hundertvolt1]

Hi again,
I tried my best to build a BATS script which gives you information about if:

• a reqired mount service was attempted to start
• it's regarded active by systemd
• the actual mount is in place

The same for ZRAM and its internal bind mount.
The last tests are checking if ZRAM could successfully put its targets on overlayFS.
I hope this level of detail is not too high and sufficiently informative for debugging!

On my system, these tests all pass, of course ;-)

#!/usr/bin/env bats

# -------------------------------
# srv-openhab-addons.mount
# -------------------------------
@test "srv-openhab\x2daddons.mount was attempted to be started" {
  run systemctl show 'srv-openhab\x2daddons.mount' -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "srv-openhab\x2daddons.mount is active" {
  run systemctl is-active 'srv-openhab\x2daddons.mount'
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "/srv/openhab-addons is mounted" {
  run mountpoint -q "/srv/openhab-addons"
  [ "$status" -eq 0 ]
}

# -------------------------------
# srv-openhab-conf.mount
# -------------------------------
@test "srv-openhab\x2dconf.mount was attempted to be started" {
  run systemctl show 'srv-openhab\x2dconf.mount' -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "srv-openhab\x2dconf.mount is active" {
  run systemctl is-active 'srv-openhab\x2dconf.mount'
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "/srv/openhab-conf is mounted" {
  run mountpoint -q "/srv/openhab-conf"
  [ "$status" -eq 0 ]
}

# -------------------------------
# srv-openhab-sys.mount
# -------------------------------
@test "srv-openhab\x2dsys.mount was attempted to be started" {
  run systemctl show 'srv-openhab\x2dsys.mount' -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "srv-openhab\x2dsys.mount is active" {
  run systemctl is-active 'srv-openhab\x2dsys.mount'
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "/srv/openhab-sys is mounted" {
  run mountpoint -q "/srv/openhab-sys"
  [ "$status" -eq 0 ]
}

# -------------------------------
# srv-openhab-userdata.mount
# -------------------------------
@test "srv-openhab\x2duserdata.mount was attempted to be started" {
  run systemctl show 'srv-openhab\x2duserdata.mount' -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "srv-openhab\x2duserdata.mount is active" {
  run systemctl is-active 'srv-openhab\x2duserdata.mount'
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "/srv/openhab-userdata is mounted" {
  run mountpoint -q "/srv/openhab-userdata"
  [ "$status" -eq 0 ]
}


# -------------------------------
# ZRAM and persistence folder
# -------------------------------

@test "zram-config.service was attempted to be started" {
  run systemctl show zram-config.service -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "zram-config.service is active" {
  run systemctl is-active zram-config.service
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "zram-config.service exited successfully" {
  run systemctl show zram-config.service -p ExecMainStatus
  [ "$status" -eq 0 ]
  [[ "$output" = "ExecMainStatus=0" ]]
}

@test "var-lib-openhab-persistence.mount was attempted to be started" {
  run systemctl show var-lib-openhab-persistence.mount -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "var-lib-openhab-persistence.mount is active" {
  run systemctl is-active var-lib-openhab-persistence.mount
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "var-lib-openhab-persistence.mount is mounted" {
  run mountpoint -q "/var/lib/openhab/persistence"
  [ "$status" -eq 0 ]
}

@test "opt-zram-persistence.bind.mount was attempted to be started" {
  run systemctl show opt-zram-persistence.bind.mount -p ActiveEnterTimestamp
  [ "$status" -eq 0 ]
  [[ "$output" =~ ^ActiveEnterTimestamp=.+ ]]
}

@test "opt-zram-persistence.bind.mount is active" {
  run systemctl is-active opt-zram-persistence.bind.mount
  [ "$status" -eq 0 ]
  [ "$output" = "active" ]
}

@test "opt-zram-persistence.bind.mount is mounted" {
  run mountpoint -q "/opt/zram/persistence.bind"
  [ "$status" -eq 0 ]
}

@test "/var/lib/openhab/persistence is on overlay FS" {
  run stat -f -c %T /var/lib/openhab/persistence
  [ "$status" -eq 0 ]
  [[ "$output" == overlayfs ]]
}

@test "srv/openhab-userdata/persistence is on overlay FS" {
  run stat -f -c %T /srv/openhab-userdata/persistence
  [ "$status" -eq 0 ]
  [[ "$output" == overlayfs ]]
}

[hundertvolt1]

Hi again, did you already try the BATS tests? Did it work?

[ecdye]

I haven't had time to yet, but will be sure to when I have a chance