[elastic_agent.filebeat][debug] File didn't change: /usr/local/nginx/logs/upstream.log why?? /usr/local/nginx/logs/upstream.log this is live log. Please suggest!

[sandraimmaculate]

Hello I'm trying to parse the upstream logs in security onion so i have created an pipeline and it started parsing the logs in security onion but after the disk issues it stopped storing the logs then i cleared some space it started parsing nginx logs audit logs etc but not upstream logs
it showing [elastic_agent.filebeat][debug] File didn't change: /usr/local/nginx/logs/upstream.log but the new logs are getting store in /usr/local/nginx/logs/upstream.log path it is live
i'm getting the below one every time in logs

14:53:25.731
elastic_agent.filebeat
[elastic_agent.filebeat][debug] Check file for harvesting: /usr/local/nginx/logs/upstream.log
14:53:25.731
elastic_agent.filebeat
[elastic_agent.filebeat][debug] Update existing file for harvesting: /usr/local/nginx/logs/upstream.log, offset: 0
14:53:25.731
elastic_agent.filebeat
[elastic_agent.filebeat][debug] Start next scan
14:53:25.731
elastic_agent.filebeat
[elastic_agent.filebeat][debug] Run input
14:53:25.731
elastic_agent.filebeat
[elastic_agent.filebeat][debug] File didn't change: /usr/local/nginx/logs/upstream.log

But why the file upstream.log will change every second with new log entry

[defensivedepth]

Please respond with the information as found in Provide sufficient technical info:

#1720

[sandraimmaculate]

My Security Onion version is 2.4.100, deployed in the cloud. I performed a network installation on a standalone instance with 4 vCPUs, 16 GB RAM, and a 256 GB root volume. The so-status command shows that all services are running, and sudo salt-call state.highstate executed without any errors.

The issue is that I created a pipeline to parse the upstream logs from Nginx, which doesn’t have a system integration. I added the pipeline to a custom log integration in the configuration, attached the integration to the policy, and assigned it to the agent. Initially, it collected logs as expected. However, on October 24, I encountered an issue where the root volume was only 24% available, causing logs to stop parsing in Security Onion. After freeing up space, making 80% available, all logs resumed parsing except for the upstream logs.

When I checked for errors, it showed: "File didn't change: /usr/local/nginx/logs/upstream.log." and "Ignore file because ignore_older reached: /usr/local/nginx/logs/upstream.log" However, this file is active, with logs being added every second. So, I’m not sure why the upstream logs are not being parsed in Security Onion.

[sandraimmaculate]

Even though the root volume still has 24 GB available, the logs are not being stored in Security Onion. If there is a default threshold causing this, how can we increase or decrease it?

[sandraimmaculate]

Now im getting the below warn in logs

[elastic_agent.filebeat][warn] Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
15:05:39.677
elastic_agent.filebeat
[elastic_agent.filebeat][info] creating new InputManager

why? what this means ? what steps i need to take to resolve this

[sandraimmaculate]

hello team ,
can you please suggest
If there is a default threshold causing this, how can we increase or decrease it? Because the logs are getting parsed even though the root volume is 24gb available