Node age for forward nodes resets after every SO update

[Ximalas]

Why does the node age for forward nodes reset after every SO update? See attached image.

[InfosecGoon]

Are you doing anything unusual with the storage on that node? The interface pulls that number from the creation date of /nsm/pcapout on the local disk.

[Ximalas]

Are you doing anything unusual with the storage on that node? The interface pulls that number from the creation date of /nsm/pcapout on the local disk.

No. I don't meddle with our SO nodes as I believe SO should be in charge of everything.

This has happened twice. The first time was after we had installed SO for the first time, using 2.4.10 sometime early last fall. We had some issues with the manager in January, the so-elastalert container stopped working, so we used the 2.4.40 image and reinstalled all nodes in late February. Apart from the low age of the forward node, everything else is as they should be.

Please let me know if you want me to run some commands to diagnose this issue.

[Ximalas]

This was run on the forward node:

$ stat /nsm/pcapout
  File: /nsm/pcapout
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: fc02h/64514d    Inode: 132         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  939/  socore)   Gid: (  939/  socore)
Context: system_u:object_r:default_t:s0
Access: 2024-05-21 06:27:50.227667672 +0000
Modify: 2024-08-06 13:38:42.224088677 +0000
Change: 2024-08-06 13:38:42.224088677 +0000
 Birth: 2024-02-16 16:12:13.092941579 +0000

And this is from the web stats:

   Date Created: 2024-08-06 13:38:42.224 +00:00
  Earliest PCAP: 2024-05-02 01:46:06.957 +00:00
Last Heard From: 2024-08-13 06:35:09.867 +00:00

It sure looks like the modify or change timestamp is being used rather than the birth timestamp. Can anyone confirm?

As a contrast, this is the output when the same command is run on the manager node:

$ stat /nsm/pcapout
  File: /nsm/pcapout
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: fc02h/64514d    Inode: 2846450     Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:default_t:s0
Access: 2024-03-14 15:18:34.347548658 +0000
Modify: 2024-02-16 14:48:47.818249610 +0000
Change: 2024-02-16 14:48:47.818249610 +0000
 Birth: 2024-02-16 14:48:47.818249610 +0000

There were some OS updates yesterday, 2024-08-14, and before restarting the forward node, I ran this command sudo touch -t 202402161612.14 /nsm/pcapout, resulting in

$ stat /nsm/pcapout
  File: /nsm/pcapout
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: fc02h/64514d    Inode: 132         Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  939/  socore)   Gid: (  939/  socore)
Context: system_u:object_r:default_t:s0
Access: 2024-02-16 16:12:14.000000000 +0000
Modify: 2024-02-16 16:12:14.000000000 +0000
Change: 2024-08-14 17:22:57.391534746 +0000
 Birth: 2024-02-16 16:12:13.092941579 +0000

The web stats are unchanged. In other words, the OS updates are not to blame, but possibly the upgrade process for SO itself.

[InfosecGoon]

This looks like a bug - I'll refer it to the developers. Thanks!

[Ximalas]

Updating to 2.4.110 seems to have solved my issue. Note, I had to reboot the forward node twice before the correct value would appear in the grid. I also ran the touch command as shown above before each reboot.

[dougburks]

Yes, for completeness, here is the fix that was included in 2.4.110:
#13628