service: fix startup issues affecting some CPUs.

Startup issues were observed when running these IOCs on certain CPUs
(e.g. the Intel Xeon E-2276ML CPUs we recently acquired as spare parts).
Despite the udev rule being triggered, some/all of the resource files
hadn't been created yet when the chmod command was run. Therefore, the
correct permissions weren't applied to the resource files, so the IOC
running as the iocs user couldn't open them. When none of the files were
available, it would lead to a bad glob expansion as an additional error.

This issue hasn't been observed on the Intel Xeon E3-1505L CPUs we are
currently using in production, even though both platforms are running
the same distribution (Debian 12) and using the same kernel version
(6.1.99). It also happens on kernel 6.12.57.

This commit fixes these issues by waiting for all the resource files
before running chmod. udev itself used to support similar functionality,
but it was removed because such delays are kernel bugs which they don't
want to paper over [1]. We have also reported the bug [2].

We check for all known resource files, including resource2_wc, even
though the underlying uhal library also supports non-write-combining
BAR2, because we want to be sure we are launching the IOCs in a
consistent environment. And, since we now have a list of all needed
resource files, we can simply pass that list to chmod, instead of using
a glob.

In order to avoid an eternal loop, we wait for up to 1s (in 10 0.1s
increments) for the files to be available. Testing shows the files take
around 0.07s to be available.

[1] systemd/systemd@f2b8052
[2] https://bugzilla.kernel.org/show_bug.cgi?id=221049

Author: ericonr

service/ioc-start.sh

@@ -5,7 +5,29 @@ DEV=$1
 
 setpci -s $DEV COMMAND=0x2
 
-chmod 666 /sys/bus/pci/devices/$DEV/resource*
+resource_files=""
+for suffix in 0 2 2_wc 4; do
+    resource_files="$resource_files /sys/bus/pci/devices/$DEV/resource$suffix"
+done
+resource_files_exist() {
+    for f in $resource_files; do
+        if [ ! -f $f ]; then
+            return 1
+        fi
+    done
+
+    return 0
+}
+
+count=0
+until resource_files_exist; do
+    if [ $count -eq 10 ]; then
+        exit 1
+    fi
+    count=$((count + 1))
+    sleep .1
+done
+chmod 666 $resource_files
 
 devslot=
 for slot in /sys/bus/pci/slots/* ; do