[M365 Defender] - Add a new data stream to support vulnerability logs

[ShourieG]

We require a new data stream in the m365 defender integration that is capable of pulling vulnerability logs, using either the standard vulnerability api documented here or by using the latest graph apis documented here. The graph apis have limitations at the moment as they are not capable of fetching a paginated vulnerability list similar to the older standard REST apis. The approach needs to be decided.

This feature enhancement is tied to a recent customer request & support ticket linked here

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[nicpenning]

This is an integration I would like to see. Bumping for awareness.

https://learn.microsoft.com/en-us/defender-endpoint/api/get-all-vulnerabilities

https://learn.microsoft.com/en-us/defender-endpoint/api/get-all-vulnerabilities-by-machines

[davidson01]

+1.

[nicpenning]

Pinging again here for awareness and the need for this integration.

[kcreddy]

Hey @piyush-elastic @janvi-elastic,

Regarding your question:

Since a full sync brings in a large amount of data, there’s a risk it could eventually lead to index overflow or high storage usage. To handle this efficiently, we can set up an Index Lifecycle Management (ILM) policy to automatically clean up older data. Could you please let us know the value of the data retention period , so we can configure the ILM policy accordingly?
Additionally, We’re currently encountering the error exceeding maximum number of CEL executions due to a child call being made for each vulnerability to retrieve its affected machines.
To address this, we can go with the following approach:
We will introduce a configuration parameter called Max Executions, with a default value based on the value of batch size (current maximum is 8000). If the batch size changes in the future, users will have the flexibility to update the Max Executions value through the advanced configuration parameters.
Please let us know your thoughts on this approach.

From #13455 (comment), the data needs to be available for atleast 24h and current Elastic CNVM makes it available for 72h. We can start with a retention of 7 days and adjust based on the feedback if need be.

Regarding the Max Executions issue, we might be doing something thats not ideal. If you have live data, how long does a full ingestion take to finish, say for a batch of 10 or 100 hosts? Maybe we can correlate to 50k+ hosts and see if this approach even scales. We may want to finish a full ingest atleast inside 24h for 50k hosts at the bear minimum.

[piyush-elastic]

@kcreddy - We recently tried fetching the full dataset from our instance, which included around 255,831 vulnerabilities and 2,213 machines/products. The process took approximately 16 hours and ingested about 257,545 logs.
To implement a full sync approach moving forward, we’ll need to set the data collection interval to more than 16 hours.
Could you please suggest a suitable interval for this scenario?

[kcreddy]

2,213 machines/products. The process took approximately 16 hours and ingested about 257,545 logs.

16 hours for 2000 machines is definitely lower performance than I expected.
Are we making any API calls per host? If so, we should avoid such calls when doing a full ingestion. What data points will be missed out if we avoid per host API calls?

Could you please suggest a suitable interval for this scenario?

I think 4h ingestion maybe a good starting point just like Qualys VMDR which also does a full ingestion.