steno,strelka-backend,suricata,zeek missing on sensor

[rserio]

So moving to the distributed version, I have setup a forward node (oracle 9 and script ran, since I was unable to run the onion iso to install) The forward node was added to the grid and I see all 3 devices Manager, Forward, IDH but in the forward node it shows the following are missing, I see the files under /nsm (link to nsm2 to use sdb) and all the permissions look right but I can't figure out why none of these are present.

I have double checked the administration-- configuration for all of these and they are false on my manager unit and true on my sensor. Which is why I'm looking for more guidance, Thanks

[cm-ops]

If you run sudo so-checkin does it complete without errors? If there are errors, look into /opt/so/log/salt/minion for errors.

[rserio]

I ran the checkin (thanks for that) and they all seem to be reporting space issues.

---

      ID: so-steno
Function: docker_container.running
  Result: False
 Comment: Image 'so-manager:5000/security-onion-solutions/so-steno:2.4.50' not present despite a docker pull raising no errors
 Started: 15:08:56.989997
Duration: 3306.598 ms
 Changes:   
          ----------
          image:
              ----------
              Errors:
                  - failed to register layer: symlink libffi.so.6.0.1 /usr/lib64/libffi.so.6: no space left on device
              Layers:
                  ----------
                  Pulled:
                      - 157bf6171731
                      - d336ac9c43fa
                      - 2d473b07cdd5
                      - 6aee36463743
              Time_Elapsed:
                  0.08090329170227051
              retcode:
                  1

---

      ID: so-suricata
Function: docker_container.running
  Result: False
 Comment: Image 'so-manager:5000/security-onion-solutions/so-suricata:2.4.50' not present despite a docker pull raising no errors
 Started: 15:09:00.941281
Duration: 4611.129 ms
 Changes:   
          ----------
          image:
              ----------
              Errors:
                  - failed to register layer: symlink librpmsign.so.9.1.3 /usr/lib64/librpmsign.so.9: no space left on device
              Layers:
                  ----------
                  Pulled:
                      - fd5367e01a11
                      - 4ee6642d4d58
                      - dea79bc0b2ca
                      - 050f06b50269
                      - 613ac2fc3f6d
                      - 3b5e23ad9b79
              Time_Elapsed:
                  0.08610725402832031
              retcode:
                  1

---

---

      ID: so-zeek
Function: docker_container.running
  Result: False
 Comment: Image 'so-manager:5000/security-onion-solutions/so-zeek:2.4.50' not present despite a docker pull raising no errors
 Started: 15:09:06.116878
Duration: 4514.235 ms
 Changes:   
          ----------
          image:
              ----------
              Errors:
                  - failed to register layer: symlink libseccomp.so.2.5.2 /usr/lib64/libseccomp.so.2: no space left on device
              Layers:
                  ----------
                  Pulled:
                      - 7cca722f8046
                      - 8e31c7bd7555
                      - fd5367e01a11
                      - 3b8fd616f24b
                      - cd4c227020bc
                      - 938f4be341f6
                      - 3b67e9bf5eee
                      - caa80b4939b6
              Time_Elapsed:
                  0.0878903865814209
              retcode:
                  1

---

      ID: strelka_backend
Function: docker_container.running
    Name: so-strelka-backend
  Result: False
 Comment: Image 'so-manager:5000/security-onion-solutions/so-strelka-backend:2.4.50' not present despite a docker pull raising no errors
 Started: 15:09:22.488422
Duration: 10824.315 ms
 Changes:   
          ----------
          image:
              ----------
              Errors:
                  - failed to register layer: write /strelka/strelka/tests/fixtures/test.jpg: no space left on device
              Layers:
                  ----------
                  Pulled:
                      - eeba71970488
                      - d0a5f9ad0890
                      - 6e3729cf69e0
                      - 31d4f354e355
                      - 45382392e03d
                      - 7af958df469e
                      - f93d1e0e198e
                      - 83d31a84a155
                      - 4f4fb700ef54
                      - 2ba2e76bfaa5
                      - e2a9695f18cf
                      - 4283b2942b55
                      - a02e6e56238d
                      - 6a40b6251f74
                      - fdd074b44567
              Time_Elapsed:
                  0.0776221752166748
              retcode:
                  1

---

but I definitely have space: (sdb is setup for /nsm)
[root@so-sensor1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 378G 8.0K 378G 1% /dev/shm
tmpfs 151G 387M 151G 1% /run
/dev/mapper/rootvg-root 189G 4.6G 185G 3% /
/dev/sda2 960M 275M 686M 29% /boot
/dev/sda1 599M 6.2M 593M 2% /boot/efi
/dev/mapper/rootvg-home 960M 40M 921M 5% /home
/dev/mapper/rootvg-var 960M 872M 89M 91% /var
/dev/mapper/rootvg-tmp 960M 40M 921M 5% /tmp
/dev/mapper/rootvg-var_cache 5.0G 450M 4.5G 9% /var/cache
/dev/mapper/rootvg-var_tmp 2.0G 47M 1.9G 3% /var/tmp
/dev/mapper/rootvg-var_log 5.0G 160M 4.8G 4% /var/log
/dev/mapper/rootvg-var_log_audit 2.0G 83M 1.9G 5% /var/log/audit
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/edbdbf6d1b5b4ead28aea4e92939c1171bbd5695d2bc2fa7a4256c83272cf987/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/3f786958e393276426e5780a9999aebb0d98a584bb77bea3921eb23b194927bf/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/fde137706f60d12b7f6472dea139633c8c91b5b39c8c06ffda6d6fad41fe3514/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/422b7e97ce6a9488cbded6883e8e250ae4f395dcbadc07a21475bc1e65781e71/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/f0502877d00f57220549fd183273412263ae297cd4091af8a3c046eebdb5d229/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/26ab981d07f25eb853e4d39fe0d0744978fce7603f29d87bd11409f0ec041f2f/merged
overlay 960M 872M 89M 91% /var/lib/docker/overlay2/a2263d7884c9928ed3bee683268f25d788caff93107af6af54d6ab042b3d8ce0/merged
tmpfs 76G 0 76G 0% /run/user/939
tmpfs 76G 0 76G 0% /run/user/0
[root@so-sensor1 ~]# lsblk -p
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
/dev/sda 8:0 0 222.6G 0 disk
├─/dev/sda1 8:1 0 600M 0 part /boot/efi
├─/dev/sda2 8:2 0 1G 0 part /boot
└─/dev/sda3 8:3 0 221G 0 part
├─/dev/mapper/rootvg-root 252:0 0 189G 0 lvm /
├─/dev/mapper/rootvg-swap 252:1 0 14.9G 0 lvm [SWAP]
├─/dev/mapper/rootvg-var_tmp 252:3 0 2G 0 lvm /var/tmp
├─/dev/mapper/rootvg-var_log_audit 252:4 0 2G 0 lvm /var/log/audit
├─/dev/mapper/rootvg-home 252:5 0 1G 0 lvm /home
├─/dev/mapper/rootvg-tmp 252:6 0 1G 0 lvm /tmp
├─/dev/mapper/rootvg-var 252:7 0 1G 0 lvm /var
├─/dev/mapper/rootvg-var_cache 252:8 0 5G 0 lvm /var/cache
└─/dev/mapper/rootvg-var_log 252:9 0 5G 0 lvm /var/log
/dev/sdb 8:16 0 7.3T 0 disk
└─/dev/sdb1 8:17 0 7.3T 0 part
└─/dev/mapper/nsmvg-nsm 252:2 0 7.3T 0 lvm
/dev/sr0 11:0 1 1024M 0 rom
[root@so-sensor1 ~]#

[rserio]

thinking this may be the /var/lib/docker needing space but I let it build itself from the github script.

[rserio]

It was the dockers needing space, I rebuilt it so I can add more space to the sensor1 as I used up too much in /

Thank you for the help. Closing this