Osquery - Windows endpoints - do not display results in Manager, but they are available in Discover

[presianbg]

Version

2.4.140

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

42

RAM

128

Storage for /

300GB

Storage for /nsm

1TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

Detail

Hi,

Just installed our shiny new distributed SO 2.4 and noticed that the OSqury results from Windows endpoints are not visualized in the OSquery manager (agent has responded, no osquery data has been reported.), but they are available in the Discovery.

Also the Linux (nodes) is OK:

This is the same issue reported in #13891 and #13819 .

Similar issues were reported elsewhere, but they seems to be fixed with latest versions:
https://discuss.elastic.co/t/osquery-has-results-but-not-displaying-them/334606/2
https://discuss.elastic.co/t/osquery-issue/371994

Any ideas how to proceed with the troubleshooting?

Cheers!

[reyesj2]

I ran the query and seem to be getting results from windows endpoints. This is windows 10, 11, and server 2019

Does fleet show your agents as healthy? Under the agent -> logs if you select 'osquery' do you see any errors related to the osquery process?

[presianbg]

Also, not sure if its related, but just found that the FleetServer-somanager agent has a second entry with Inactive status:

EDIT:
Nahh, found one fairly recent report about the same behavior:
elastic/kibana#183142

[reyesj2]

If you do choose to share note that it can have debug level logs that may be sensitive / contain secrets configured via an integration

I don't seem to be able to reproduce, are you able to share an elastic-agent diagnostic log ? I can take a look or if you pull it using fleet you can look through there for anything that indicates what the issue is

via fleet when you click an agent -> diagnostics -> request diagnostics .zip

If you do choose to share note that it can have debug level logs that may be sensitive / contain secrets configured via an integration

[presianbg]

@reyesj2 , thanks for investing time for resolving this issue.
Here are the requested diagnostic logs with debug verbosity from my lab:

elastic-agent-diagnostics-2025-03-31T20-58-23Z-00.zip

Relevant snippet from my last osquery job with debug verbosity:

23:54:45.127
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][info] 140 events sent to index logs-osquery_manager.result-default
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][debug] close connection
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][debug] closing
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][debug] 141 events out of 141 events sent to logstash host somanager:5055. Continue sending
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][debug] handle error: write tcp 10.20.1.166:62093->10.20.1.215:5055: wsasend: An existing connection was forcibly closed by the remote host.
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][debug] close connection
23:54:55.128
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][error] Failed to publish events caused by: write tcp 10.20.1.166:62093->10.20.1.215:5055: wsasend: An existing connection was forcibly closed by the remote host.
23:54:56.440
elastic_agent.osquerybeat
osquery-so-manager_logstash
[elastic_agent.osquerybeat][error] failed to publish events: write tcp 10.20.1.166:62093->10.20.1.215:5055: wsasend: An existing connection was forcibly closed by the remote host.

[reyesj2]

Okay, so I expanded my setup a bit to also use a windows server 2022 and I am still not able to reproduce

What does your distributed grid consist of ? Any receiver nodes? Any fleet nodes? Is it just a lab environment with low amounts of data being monitored or is the load high? Are you sure the health of the agent is 'healthy' and remains healthy or is it possible its going healthly/unhealthy? (IIRC that is the only time I've experienced similar odd issues like this)

Do you any different results if you open hunt and query for something like

log.level: error AND event.module: "elastic_agent" | groupby event.module* event.dataset | groupby "host.hostname" | groupby log.source* | table soc_timestamp event.dataset message

You could then click on one of the problematic hosts and 'include' so you only get the logs for that agent. Then looking for more info around failing to send events to logstash

[presianbg]

Strange...
Its a test distributed env consisting of 1 manager, 1 storage, 1 forward (sensor) node.
Only the node members and 1 test servers are reporting, so the event volume is minimum.

Yes, all agents are reporting Healthy status:

The hunt you suggested is returning similar error messages:

I will re-install this anyway, hope it will work afterwards. I will keep you posted.

[Sahale]

Same problem after upgrading to 2.4.141 from 2.3.130.
Can't find solution.

[Sahale]

Fixed!
The SecOnion node’s system clock was running 2 seconds fast despite active NTP sync with public servers. Replacing the NTP server pool resolved the time drift instantly.
Time magic ✨