[rapid7_insightvm] Add asset_vulnerability data stream for Cloud Detection and Response (CDR) workflow #14079

brijesh-elastic · 2025-05-30T10:48:56Z

Proposed commit message

rapid7_insightvm: Add asset_vulnerability data stream for Cloud Detection and Response (CDR) workflow

This adds a breaking change, as it involves deprecating asset data stream using HTTPJSON input and
adding new data_stream asset_vulnerability using CEL input.

The new data stream, asset_vulnerability, will retrieve a list of assets and the vulnerabilities contained within each.
Additionally, it will enrich the vulnerability details by querying vulnerability endpoint.
The agent will publish one document per vulnerability per host.

ECS mapping and transforms have also been added to facilitate
the Cloud Native Vulnerability Management (CNVM)[1] workflow.

Enabled agentless deployment for rapid7 insightvm
Upgraded the format_version to 3.3.5
Updated Kibana version constraints to ^8.19.0 || ^9.1.0

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html

Note

To Reviewers:

Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/rapid7_insightvm directory.
Run the following command to run tests.

elastic-package test

Related issues

elastic-vault-github-plugin-prod · 2025-05-30T11:12:43Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/rapid7_insightvm/manifest.yml

packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

elasticmachine · 2025-06-07T17:32:56Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

maxcold · 2025-06-11T13:15:36Z

...s/rapid7_insightvm/data_stream/asset_vulnerability/elasticsearch/ingest_pipeline/default.yml

+      tag: set_vulnerability_enumeration
+      value: CVE
+  # Remove cloud.* fields populated by beat.
+  # These fields correspond to EA rather than Tenable hosts and could be misleading.


nit: comment mentions Tenable, should be Rapid7 I guess

Addressed in 9bd0b7b

maxcold · 2025-06-11T13:27:44Z

packages/rapid7_insightvm/data_stream/asset_vulnerability/manifest.yml

+    template_path: cel.yml.hbs
+    enabled: false
+    vars:
+      - name: interval


There is no initial interval as far as I can see. Do we do full scan periodically? In that case we need to set the @timestamp of the documents to the ingestion time, like we did for AWS SecurityHub full posture data stream https://github.com/elastic/integrations/pull/13372/files#diff-2aebbf76c4f4587a0d3701264a03075f4886df078bb04d4b41e40ec329902488R592 This way we won't have the problem of old documents outside of the retention period

For Rapid7, it will collect all the vulnerabilities present in all the assets during the first interval. After subsequent intervals, it will collect only the new and remediated vulnerabilities that have introduced since the last interval.

cc: @kcreddy

@maxcold, we are going with incremental load to keep lower memory footprint as @brijesh-elastic observed agentless agent restarts in Serverless environments.

But I see your point. We need to match the transform retention with initial_interval, otherwise even if we ingest 2 years worth of data if our transform retention is much lower, all the documents are deleted inside destination indices. Is this correct understanding?
If this is the case, we can add initial_interval like with other incremental integrations.

Yes, if we go with incremental I think we need to follow other integraitons approach and have initial_interval. It will at least give users the option to ingest some initial data for the CDR flows. But I'm not sure I understand how do we then have the data from 2023 in the envs if you do incremental updates only?

But I'm not sure I understand how do we then have the data from 2023 in the envs if you do incremental updates only?

In the absence of initial_interval, the first fetch ingests all historical data. All fetches after this will be incremental.

@brijesh-elastic is already working on adding initial_interval option, so that first fetch only has user-defined initial interval data instead of ingesting all historical data.

maxcold

tested cloud security UIs with the changes, everything works well as far as i can tell

packages/rapid7_insightvm/_dev/build/docs/README.md

packages/rapid7_insightvm/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

packages/rapid7_insightvm/changelog.yml

packages/rapid7_insightvm/data_stream/asset_vulnerability/manifest.yml

packages/rapid7_insightvm/manifest.yml

packages/rapid7_insightvm/_dev/build/docs/README.md

packages/rapid7_insightvm/data_stream/asset/manifest.yml

packages/rapid7_insightvm/manifest.yml

packages/rapid7_insightvm/data_stream/asset_vulnerability/agent/stream/cel.yml.hbs

...ata_stream/asset_vulnerability/_dev/test/pipeline/test-asset-vulnerability.log-expected.json

packages/rapid7_insightvm/data_stream/asset_vulnerability/agent/stream/cel.yml.hbs

efd6

I think the behaviour is better, but I still have concerns about #14079 (comment).

packages/rapid7_insightvm/data_stream/asset_vulnerability/agent/stream/cel.yml.hbs

...ata_stream/asset_vulnerability/_dev/test/pipeline/test-asset-vulnerability.log-expected.json

chemamartinez

Regarding comments I added, LGTM.

kcreddy

nit only. LGTM for my comments 👍🏼 . Please wait for @efd6 approval.

kcreddy · 2025-06-17T09:31:40Z

packages/rapid7_insightvm/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

+  - remove:
+      field:
+        - organization
+        - division
+        - team
+      ignore_missing: true
+      if: ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String
+      tag: remove_agentless_tags
+      description: >-
+        Removes the fields added by Agentless as metadata,
+        as they can collide with ECS fields.


Can you also add a changelog entry for this change (bugfix) similar to #14172

efd6

Nice work. Nit only, then LGTM

efd6 · 2025-06-19T05:48:10Z

packages/rapid7_insightvm/data_stream/asset_vulnerability/agent/stream/cel.yml.hbs

+          ?"asset": has(state.?cursor.last_interval_time) ?
+            optional.none()
+          :
+            optional.of(("last_scan_end > " + string((timestamp(interval_time) - duration(state.initial_interval)).format(time_layout.RFC3339)))),


Suggested change

optional.of(("last_scan_end > " + string((timestamp(interval_time) - duration(state.initial_interval)).format(time_layout.RFC3339)))),

optional.of(("last_scan_end > " + (timestamp(interval_time) - duration(state.initial_interval)).format(time_layout.RFC3339))),

timestamp.format returns a string.

elasticmachine · 2025-06-19T06:27:15Z

💚 Build Succeeded

Buildkite Build
Commit: d1ae436

History

💚 Build #27363 succeeded 04f7bbc
💚 Build #27325 succeeded 5fbe5cf
💚 Build #27282 succeeded 132cb48
💚 Build #27190 succeeded 471e7fe
💚 Build #27181 succeeded 1ff28b2
💚 Build #27134 succeeded 9bd0b7b

cc @brijesh-elastic

elastic-sonarqube · 2025-06-19T06:27:22Z

Quality Gate failed

Failed conditions
75.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-06-19T06:51:16Z

Package rapid7_insightvm - 2.0.0 containing this change is available at https://epr.elastic.co/package/rapid7_insightvm/2.0.0/

brijesh-elastic added 4 commits May 29, 2025 15:43

add support of asset_vulnerability datastream

8ed1900

add package.* mappings

1a45072

add transform

a908189

add agentless deployment option

697f7e1

brijesh-elastic self-assigned this May 30, 2025

kcreddy mentioned this pull request May 30, 2025

Add rapid7_insightvm.asset_vulnerability source indices to kibana_system role permissions elastic/elasticsearch#128661

Merged

brijesh-elastic and others added 4 commits May 30, 2025 16:45

Remove unwanted code and update changelog entry

4d6902a

Add grok description

68e3b42

Add vuln_id filters

46a452b

Update CEL program to publish events as they become available

13fff44

kcreddy reviewed Jun 7, 2025

View reviewed changes

packages/rapid7_insightvm/manifest.yml Outdated Show resolved Hide resolved

kcreddy reviewed Jun 7, 2025

View reviewed changes

packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml Outdated Show resolved Hide resolved

kcreddy marked this pull request as ready for review June 7, 2025 17:32

kcreddy requested a review from a team as a code owner June 7, 2025 17:32

kcreddy requested a review from a team June 7, 2025 17:33

kcreddy and others added 4 commits June 7, 2025 23:05

remove debug from cel

3283605

fix transform version.

d78abe0

Increase memory for agentless deployment

130e2b7

Increase memory to 4gb

15e4652

maxcold reviewed Jun 11, 2025

View reviewed changes

Address PR comments from @maxcold

9bd0b7b

maxcold approved these changes Jun 12, 2025

View reviewed changes

adding support of initial_interval option

1ff28b2

deprecate asset data stream and update readme

471e7fe

kcreddy reviewed Jun 13, 2025

View reviewed changes

chemamartinez reviewed Jun 13, 2025

View reviewed changes

efd6 reviewed Jun 15, 2025

View reviewed changes

address pr comments

132cb48

efd6 reviewed Jun 17, 2025

View reviewed changes

chemamartinez approved these changes Jun 17, 2025

View reviewed changes

resolve comments

5fbe5cf

brijesh-elastic requested review from efd6 and kcreddy June 17, 2025 09:10

kcreddy approved these changes Jun 17, 2025

View reviewed changes

kcreddy mentioned this pull request Jun 17, 2025

Change Rapid7 InsightVM integration to one doc per vulnerability #9354

Closed

update changelog entry

04f7bbc

This was referenced Jun 18, 2025

[rapid7_insightvm] expand documents to map each vulnerability per asset #13878

Closed

Rapid7 InsightVM: Implement transform for Cloud Security Workflows #13776

Closed

Rapid7 Insight VM: Implement mappings for Cloud Security Workflows #13775

Closed

efd6 approved these changes Jun 19, 2025

View reviewed changes

resolve nits

d1ae436

efd6 approved these changes Jun 19, 2025

View reviewed changes

brijesh-elastic merged commit 8d53d83 into elastic:main Jun 19, 2025
6 of 7 checks passed

This was referenced Jun 19, 2025

[meta][CDR] Update Rapid7 integration to Leverage Native CDR Workflows #13647

Closed

[aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306

Merged

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

nick-alayil mentioned this pull request Jul 10, 2025

[Internal]: Add docs for Rapid7 Integration elastic/docs-content#2092

Closed

1 task

	optional.of(("last_scan_end > " + string((timestamp(interval_time) - duration(state.initial_interval)).format(time_layout.RFC3339)))),
	optional.of(("last_scan_end > " + (timestamp(interval_time) - duration(state.initial_interval)).format(time_layout.RFC3339))),

[rapid7_insightvm] Add asset_vulnerability data stream for Cloud Detection and Response (CDR) workflow #14079

[rapid7_insightvm] Add asset_vulnerability data stream for Cloud Detection and Response (CDR) workflow #14079

Uh oh!

Conversation

brijesh-elastic commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elastic-vault-github-plugin-prod bot commented May 30, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jun 7, 2025

Uh oh!

maxcold Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

maxcold Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

brijesh-elastic Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chemamartinez left a comment

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Jun 19, 2025

Choose a reason for hiding this comment

brijesh-elastic commented May 30, 2025 •

edited

Loading

kcreddy Jun 11, 2025 •

edited

Loading

kcreddy Jun 12, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading