Filebeat Module Log Duplication

[EddieN17]

Hello,

I have a on-prem distributed setup of SO 2.3.230 using the SO ISO. I have 4 search nodes, a manager, a receiver node, and many forward nodes. I do not use true elasticsearch clustering.

I am currently trying to implement an Okta integration with my setup using the filebeat module. I am able to successfully intake Okta logs by following this documentation. The only difference in my config is that I do not have direct internet access, so I have an additional configuration of var.proxy_url:

The logs do experience duplication though. They have identical _id and all of the data is the same except for the event.ingested and event.created. So, my assumption of the problem is: The filebeat module does the API call and sends the log to Manager and Manager sends the log out to a search node with the typical Redis process. But since the API call is on a time interval, it keeps grabbing the same most latest log. If the search node that Redis chooses already has the log, no issue, no duplication will happen. But if Redis chooses a search node that doesn't have the log locally stored, it will also store that same identical log on this new search node. So, the identical log now appears on both search nodes. Then this process just keeps repeating until all 4 search nodes have that same log, and then no more duplication happens. The same log just gets stored on all 4 search nodes, so I see all 4 of the same logs on Kibana. I have come to this assumption as event.ingested is not always identical with my var.interval: 60s. Sometimes the ingested timestamp is exactly the minute different, sometimes its 3 minutes, but it's not consistent with every minute.

The duplication also only occurs when no new Okta logs have happened in the var.interval. It does not duplicate every log, it just duplicates the most recent log it grabs from Okta. Also, note I do use other filebeat modules that intakes logs via syslog and no duplication occurs there. The problem is most likely with the API call on its time interval.

I have tried configuring the filebeat module on a receiver, search, manager, and forward node, and no matter what the duplication issue still exists. Does anyone have an idea of what my issue is?

Thanks.

[weslambert]

It might be beneficial to consult the Elastic Filebeat repo to see if there are any issues open for the Okta module that are related to this issue.