no alerts and logs in hunt from idh

[Taraskobilskiy]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

64

Storage for /

500

Storage for /nsm

500

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

when installing the idh module and deploying it to the manager node, there are no logs from opencanary, playbooks are active, http bait is available, but nothing happens when entering the login and password. I tried to do everything that was described in the community but there was no result.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[foxman0719]

I have the same problem, I do TCPDUMP to the interfaces and there is traffic without problem, but it does not appear reflected in the tool or in alerts or in the GRID

please help

[Taraskobilskiy]

UPD: logs

[Taraskobilskiy]

The problem still persists, could you please suggest steps to troubleshoot since it’s not clear what to do about it

[Taraskobilskiy]

I’ll add: when installing node idh, the installation occurs correctly, the node itself is visible in the grid, there are no errors, but when accessing in IDH port 22 ssh or 80 http and entering login and password, opencanary events are not generated (at least they are not visible in hunt)

[reyesj2]

Do you see logs generated in /nsm/idh/opencanary.log on the idh node?
What is the output of elastic-agent status on the idh node?

[Taraskobilskiy]

Maybe I have too many integrations and the extra ones need to be removed from the policy?

[reyesj2]

No that looks correct, the IDH node uses the so-grid-nodes_general policy. The 'idh-logs' integration is what will pick up the logs if found.

You can click 'logs' in that screenshot and see if you can find what the error causing the unhealthy state is.

Is this your only idh node? If it is just go to hunt and search tags: idh you'll see all idh logs. If you have more than one idh node you'll need to look for the node_id and make sure 'so-idh4' is included in the logs shown

You can also try reinstalling the elastic agent on that idh node
For that from your manager run

sudo salt 'so-idh4*' cmd.run 'elastic-agent uninstall -f'
sudo salt 'so-idh4*' state.apply elasticfleet.install_agent_grid
sudo salt 'so-idh4*' cmd.run 'elastic-agent status'

[Taraskobilskiy]

The logs are also not displayed, I tried to run diagnostics and downloaded an archive with logs, but I didn’t find anything there to help solve the problem:

I deployed more than 5 IDH, but in the end not one of them works. Searching by tag also does not give results,

---

---

although if you take a very long period, you can see that at first there were events:

---

---

[Taraskobilskiy]

UPD:
after reinstalling the agent everything looks like this, but there are still no events:

---

[sysadmin@so-manager ~]$ sudo salt 'so-idh4*' cmd.run 'elastic-agent uninstall -f'
[sudo] password for sysadmin:
so-idh4_idh:
Elastic Agent has been uninstalled.
[sysadmin@so-manager ~]$ sudo salt 'so-idh4*' state.apply elasticfleet.install_agent_grid
so-idh4_idh:

      ID: run_installer
Function: cmd.script
    Name: salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64
  Result: True
 Comment: Command 'salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64' run
 Started: 04:25:59.639441
Duration: 13953.109 ms
 Changes:
          ----------
          pid:
              315458
          retcode:
              0
          stderr:
          stdout:

              Installation initiated, view install log for further details.


              Installation completed successfully.

              Summary for so-idh4_idh

---

Succeeded: 1 (changed=1)
Failed: 0

Total states run: 1
Total run time: 13.953 s

[sysadmin@so-manager ~]$ sudo salt 'so-idh4*' cmd.run 'elastic-agent status'
so-idh4_idh:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (HEALTHY) Running
├─ filestream-monitoring
│ ├─ status: (STARTING) Starting: spawned pid '318133'
│ ├─ filestream-monitoring
│ │ └─ status: (STARTING) Starting: spawned pid '318133'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting: spawned pid '318133'
├─ log-default
│ ├─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-elasticsearch-90a626e3-36d8-4399-a9b9-39361f7448da
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-15b78604-37fb-46cc-9cd4-cd4692f53122
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-1b2c77b7-36bf-4212-88b1-895ffc5afdf1
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-1fd4c726-bc9d-4c21-9f84-3e78a519f2a8
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-5cc786b7-a311-4f5a-9738-dbad9b1ab643
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-689c5fcf-d70d-445a-ac71-fb5a407d04e1
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-81d11465-6f80-43e1-b289-42678e41142e
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-8a6b5755-62e8-4fab-89aa-fd7a9916ff8b
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-9e01c157-1694-4cdf-abfa-dd4682ce401e
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-a646f5db-391f-429c-857d-948eb6a781b2
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-db6fed93-57de-4e4d-99f5-4192207f2b2f
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-dea3e095-79b6-477f-9a59-ad7334a1016e
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-e56ad89f-5856-4eb4-ace1-43177c8957a1
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-logs-zeek-logs
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ ├─ log-default-logfile-redis-9b473127-9a2e-4628-86a9-2f8579a93d05
│ │ └─ status: (STARTING) Starting: spawned pid '318118'
│ └─ log-default-logfile-system-21d7b3bb-c260-45d5-b736-f6b47741e224
│ └─ status: (STARTING) Starting: spawned pid '318118'
├─ osquery-default
│ ├─ status: (STARTING) Starting: spawned pid '318104'
│ ├─ osquery-default
│ │ └─ status: (STARTING) Starting: spawned pid '318104'
│ └─ osquery-default-727f3426-a6ec-446d-8e40-104f1cddaaad
│ └─ status: (STARTING) Starting: spawned pid '318104'
├─ tcp-default
│ ├─ status: (STARTING) Starting: spawned pid '318127'
│ ├─ tcp-default
│ │ └─ status: (STARTING) Starting: spawned pid '318127'
│ └─ tcp-default-tcp-tcp-8ec50926-73e4-463f-a048-ae94abb68b75
│ └─ status: (STARTING) Starting: spawned pid '318127'
└─ udp-default
├─ status: (STARTING) Starting: spawned pid '318097'
├─ udp-default
│ └─ status: (STARTING) Starting: spawned pid '318097'
└─ udp-default-udp-udp-c4e5e9fc-9b1c-4166-b3bd-6fdf96528b08
└─ status: (STARTING) Starting: spawned pid '318097'

---

[Taraskobilskiy]

upd:
after some time, some services switched to DEGRADED status and the error "ERROR: Minions returned with non-zero exit code" appeared:

---

[sysadmin@so-manager ~]$ sudo salt 'so-idh4*' cmd.run 'elastic-agent status'
so-idh4_idh:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a degraded state
├─ filestream-monitoring
│ ├─ status: (DEGRADED) Degraded: pid '320445' missed 2 check-ins
│ ├─ filestream-monitoring
│ │ └─ status: (STARTING) Starting: spawned pid '320445'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting: spawned pid '320445'
├─ log-default
│ ├─ status: (DEGRADED) Degraded: pid '320433' missed 2 check-ins
│ ├─ log-default
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-elasticsearch-90a626e3-36d8-4399-a9b9-39361f7448da
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-15b78604-37fb-46cc-9cd4-cd4692f53122
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-1b2c77b7-36bf-4212-88b1-895ffc5afdf1
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-1fd4c726-bc9d-4c21-9f84-3e78a519f2a8
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-5cc786b7-a311-4f5a-9738-dbad9b1ab643
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-689c5fcf-d70d-445a-ac71-fb5a407d04e1
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-81d11465-6f80-43e1-b289-42678e41142e
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-8a6b5755-62e8-4fab-89aa-fd7a9916ff8b
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-9e01c157-1694-4cdf-abfa-dd4682ce401e
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-a646f5db-391f-429c-857d-948eb6a781b2
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-db6fed93-57de-4e4d-99f5-4192207f2b2f
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-dea3e095-79b6-477f-9a59-ad7334a1016e
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-e56ad89f-5856-4eb4-ace1-43177c8957a1
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-logs-zeek-logs
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ ├─ log-default-logfile-redis-9b473127-9a2e-4628-86a9-2f8579a93d05
│ │ └─ status: (STARTING) Starting: spawned pid '320433'
│ └─ log-default-logfile-system-21d7b3bb-c260-45d5-b736-f6b47741e224
│ └─ status: (STARTING) Starting: spawned pid '320433'
├─ osquery-default
│ ├─ status: (DEGRADED) Degraded: pid '320427' missed 2 check-ins
│ ├─ osquery-default
│ │ └─ status: (STARTING) Starting: spawned pid '320427'
│ └─ osquery-default-727f3426-a6ec-446d-8e40-104f1cddaaad
│ └─ status: (STARTING) Starting: spawned pid '320427'
├─ tcp-default
│ ├─ status: (DEGRADED) Degraded: pid '320439' missed 2 check-ins
│ ├─ tcp-default
│ │ └─ status: (STARTING) Starting: spawned pid '320439'
│ └─ tcp-default-tcp-tcp-8ec50926-73e4-463f-a048-ae94abb68b75
│ └─ status: (STARTING) Starting: spawned pid '320439'
└─ udp-default
├─ status: (DEGRADED) Degraded: pid '320405' missed 2 check-ins
├─ udp-default
│ └─ status: (STARTING) Starting: spawned pid '320405'
└─ udp-default-udp-udp-c4e5e9fc-9b1c-4166-b3bd-6fdf96528b08
└─ status: (STARTING) Starting: spawned pid '320405'
ERROR: Minions returned with non-zero exit code

---

[Taraskobilskiy]

Can you suggest any solution to this problem? or other troubleshooting actions?

[reyesj2]

From elastic fleet when you click on the 'so-idh4' agent you should be able to pull up logs about that agent and maybe why its degraded.

Click on 'Dataset' and unselect any that are selected so it shows all logs.

The so-idh4 node status in SOC -> Grid is still OK/green?

[Taraskobilskiy]

unfortunately, the logs are not displayed for any selection in the dataset:

Yes, the grid status is green:

[Taraskobilskiy]

In general, we managed to solve the problem by reinstalling IDH on a separate network and setting up a firewall rule in the admin panel.

but no matter how much I deployed the idh module on the same network where the manager node and other sec onion elements were, the problem with the elastic agent remained.

@reyesj2 thank you for your feedback and help in this matter!