qualys_vmdr: Add latest transform for Asset Host Detections #13455

kcreddy · 2025-04-08T09:27:58Z

Proposed commit message

Adds a latest transform for vulnerabilities which allows data
from Qualys VMDR to be displayed in Elastic Security CNVM workflow.

Ref: https://github.com/elastic/integrations/issues/11673

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Related issues

Closes Qualys VMDR: Implement transform for enhancing cloud security workflow #12718

elasticmachine · 2025-04-08T09:57:07Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-04-08T21:25:26Z

packages/qualys_vmdr/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

I know that having these in separate files reflects the situation in the data stream's defs, but it makes me sad. I the rationale for this diff collision avoidance?

The fields in these files package.yml, resource.yml, vulnerability.yml are meant for extended ECS fields that are not in ECS definitions, but are used by the Elastic workflows.
Here both package.name, package.version are present in ECS: https://www.elastic.co/guide/en/ecs/current/ecs-package.html. But field package.fixed_version isn't. Hence it was separated.
It could also help checking diff between source fields and destination fields: https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/fields/package.yml as they are now identical.

kcreddy · 2025-04-09T07:55:09Z

The PR will be moved to ready for review once live testing is finished.
cc: @maxcold @alexreal1314

alexreal1314 · 2025-04-09T10:16:18Z

The PR will be moved to ready for review once live testing is finished. cc: @maxcold @alexreal1314

still waiting for credentials to test it.

maxcold · 2025-04-09T14:47:11Z

packages/qualys_vmdr/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - vulnerability.id


this might be causing duplication of the documents (we see duplication in the test env). I remember we had this conversation before, but I need to find out the outcome. If having fields with multiple values as unique_key causing duplication we might need to find another way, eg. use QID+resource.id+namespace as a key. We need to understand better our options here.

@kcreddy @maxcold IMO we should use event.id+resource.id+namespace since we set event.id based on unique_vuln_id in host detection, or use qualys_vmdr.knowledge_base.qid field in knowledge_base ingest pipeline.

Thanks for the suggestion @alexreal1314.
I have implemented it here: 591a0f0. Removed vulnerability.package.name and vulnerability.package.version as well.

maxcold · 2025-04-09T14:53:21Z

packages/qualys_vmdr/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+retention_policy:
+  time:
+    field: "@timestamp"
+    max_age: 4h


@kcreddy am I correct that Qualys integration implemented in a way that it ingests the whole data every cycle?
If yes, we need to think about the value here. 4h is the default value but also configurable. For example InfoSec team set it to 12h now. this means that after 4h the findings will be gone for the next 8h. Until we implement the connection between the params and this value in the transform, setting higher value might be needed. Say 24h as a good middle ground or maybe even more (native CNVM has 72h for example). In this case for some time there might be some old vulnerabilities lingering around in the env, but it's better than having no findings

am I correct that Qualys integration implemented in a way that it ingests the whole data every cycle?

Yes it's correct, it's the default behaviour. However, there is the option to enable an incremental scan.

If yes, we need to think about the value here. 4h is the default value but also configurable. For example InfoSec team set it to 12h now. this means that after 4h the findings will be gone for the next 8h. Until we implement the connection between the params and this value in the transform, setting higher value might be needed. Say 24h as a good middle ground or maybe even more (native CNVM has 72h for example). In this case for some time there might be some old vulnerabilities lingering around in the env, but it's better than having no findings

I agree with your analysis. I would like to add details that will likely influence the value:

If Qualys is configured with authenticated scan (either through the Qualys agent, either through network scan), Qualys will see if the vulnerability is fixed and will change the status as fixed.

If Qualys is configured to scan with the Qualys agent, the minimal scanning interval is 4h and the default is also 4h

~~To sum up, we'll see old vulnerabilities lingering around only on non-authenticated network scans. As it's not a recommended practice to run unauthenticated scans, we should be fine most of the time.~~ EDIT: I think it's inaccurate unless the CNVM functionality is filtering fixed or ignored vulnerabilities.

Thanks for the suggestion @maxcold and @clement-fouque
I updated the retention to 24h here: 591a0f0

@clement-fouque thanks for sharing more context! We will work toward connecting the data retention param on the transform with the params of the integration, but for now, I think we should proceed with the longer retention period

elastic-vault-github-plugin-prod · 2025-04-13T16:30:37Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

maxcold · 2025-04-14T09:34:22Z

@kcreddy we probably need to update kibana.version condition https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/manifest.yml#L12 to the version: "^8.19.0 || ^9.1.0" . @alexreal1314 as far as I can see you didn't backport the changes of supporting multiple CVEs and packages to other branches. We might need to backport at least to 8.x so the changes go out in 8.19 as well.

kcreddy · 2025-04-14T10:21:35Z

@kcreddy we probably need to update kibana.version condition https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/manifest.yml#L12 to the version: "^8.19.0 || ^9.1.0" . @alexreal1314 as far as I can see you didn't backport the changes of supporting multiple CVEs and packages to other branches. We might need to backport at least to 8.x so the changes go out in 8.19 as well.

@maxcold, Okay. Do you think merging this PR closer to 8.19 or after 8.19 makes sense?
Prior to this, you can test the integration on 8.19 using the fleet API.

maxcold · 2025-04-14T11:55:54Z

@nick-alayil can you provide your input on

Do you think merging this PR closer to 8.19 or after 8.19 makes sense?

This would mean that the Qualys data won't be available in Serverless before 8.19/9.1 release in June. Are we ok with that? do we have customers in Serverless we would like to enable with this feature?

kcreddy · 2025-04-16T16:10:31Z

If the changes look solid and testing is good, I think we should merge this right away. We're following the serverless first approach as an org now anyway, so this aligns with that direction.

However, I agree with the statement: 'We're following a serverless-first approach as an organization now anyway, so this aligns with that direction.' In my opinion, this technical limitation within the integrations ecosystem shouldn't block our serverless-first approach.

Thanks for the input @nick-alayil and @maxcold.
I've updated the min stack version as per your suggestion -> 89b8323. We can proceed merging the PR now. We will be following the integration backporting guidelines to backport any bugs in the meantime.

kcreddy · 2025-04-16T16:41:22Z

The CI error should be fixed by elastic/elasticsearch#126417.

efd6 · 2025-04-17T02:37:24Z

Running this locally on v8.19.0-SNAPSHOT, I have minimised this to

---
description: Pipeline for processing Asset Host Detection data.
processors:
  - set:
      field: message
      value: '{"ATTRIBUTE":[{"OTHER_FIELD":""}]}'
  - json:
      field: message
      target_field: json
  - remove:
      tag: remove_json
      field:
        - json.ATTRIBUTE.LAST_ERROR_DATE
      ignore_missing: true
on_failure:
  - append:
      field: error.message
      value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

This ceases to fail when ATTRIBUTE is not an array.

Using the dev tools gives an indication why this might be:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "json": {
          "field": "message",
          "target_field": "json"
        }
      },
     {
        "remove": {
          "field": [
            "json.ATTRIBUTE.LAST_ERROR_DATE"
          ],
          "ignore_missing": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "idx",
      "_id": "id",
      "_source": {
        "message": """{"ATTRIBUTE":[{"OTHER_FIELD":""}]}"""
      }
    }
  ]
}

gives

{
  "docs": [
    {
      "error": {
        "root_cause": [
          {
            "type": "illegal_argument_exception",
            "reason": "[LAST_ERROR_DATE] is not an integer, cannot be used as an index as part of path [json.ATTRIBUTE.LAST_ERROR_DATE]"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "[LAST_ERROR_DATE] is not an integer, cannot be used as an index as part of path [json.ATTRIBUTE.LAST_ERROR_DATE]",
        "caused_by": {
          "type": "number_format_exception",
          "reason": "For input string: \"LAST_ERROR_DATE\""
        }
      }
    }
  ]
}

while when the document is {"ATTRIBUTE":{"OTHER_FIELD":""}} there is no error.

If I run the same failing simulate request on v8.17.3, I get no failure.

The image for the v8.19.0-SNAPSHOT is 3c3225efef7e8285bb2c5de5cf0b15fe4747677fc84473a2b3db53fe2391212e, which reports {"@timestamp":"2025-04-17T02:29:50.155Z", "log.level": "INFO", "message":"version[8.19.0-SNAPSHOT], pid[233], build[docker/c070c976010455cf7d9b45b6270cb6709e84c1ae/2025-04-15T17:38:52.824533143Z], OS[Linux/6.10.14-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/24/24+36-3646]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"3c3225efef7e","elasticsearch.cluster.name":"elasticsearch"} which is here and should contain the fix in elastic/elasticsearch#126417. So I have to conclude that the issue here is not the same as the one that was fixed in that PR — this is not completely surprising since the error messages are subtly different.

kcreddy · 2025-04-21T02:42:36Z

Thanks @efd6 for the analysis. It is indeed a variation of elastic/elasticsearch#126417 but for lists.

@joegallo made the fix in elastic/elasticsearch#127006 and should be available in v8.19.0 stack version.

kcreddy · 2025-04-21T02:42:39Z

/test

kcreddy · 2025-04-21T11:49:15Z

/test

kcreddy · 2025-04-23T02:43:37Z

/test

elasticmachine · 2025-04-23T03:08:15Z

💚 Build Succeeded

Buildkite Build
Commit: 89b8323

History

💔 Build #24928 failed 89b8323
💔 Build #24916 failed 89b8323
💔 Build #24840 failed 89b8323
💚 Build #24718 succeeded 591a0f0
💚 Build #24541 succeeded 859d27e

cc @kcreddy

elastic-sonarqube · 2025-04-23T03:08:21Z

Quality Gate failed

Failed conditions
5.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

alexreal1314 · 2025-04-28T07:28:22Z

Hi @kcreddy @maxcold I've tested latest changes on my env https://alex-dev91.kb.us-west2.gcp.elastic-cloud.com/ and seems that after we changed the unique key of the transform there are no duplicates.

this is the query i ran:

POST security_solution-qualys_vmdr.vulnerability_latest-v1/_search
{
  "size": 0,
  "aggs": {
    "duplicate_check": {
      "composite": {
        "size": 10000,
        "sources": [
          { "event_id": { "terms": { "field": "event.id.keyword" } } },
          { "resource_id": { "terms": { "field": "resource.id.keyword" } } },
          { "namespace": { "terms": { "field": "data_stream.namespace.keyword" } } }
        ]
      },
      "aggs": {
        "docs_per_key": {
          "bucket_script": {
            "buckets_path": {
              "docCount": "_count"
            },
            "script": "params.docCount"
          }
        },
        "duplicate_filter": {
          "bucket_selector": {
            "buckets_path": {
              "docCount": "_count"
            },
            "script": "params.docCount > 1"
          }
        }
      }
    }
  }
}

so from my side we are good to go.

maxcold

lgtm, thanks for pushing this forward @kcreddy !

elastic-vault-github-plugin-prod · 2025-04-28T12:59:59Z

Package qualys_vmdr - 6.6.0 containing this change is available at https://epr.elastic.co/package/qualys_vmdr/6.6.0/

Add latest transform for Asset Host Detections.

50f11cb

kcreddy self-assigned this Apr 8, 2025

kcreddy added 3 commits April 8, 2025 14:58

update PR number

75309cc

Remove undefined elastic_agent ECS fields

df87794

Remove overridden ECS fields

06dad55

kcreddy added enhancement New feature or request Integration:qualys_vmdr Qualys VMDR Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Apr 8, 2025

kcreddy marked this pull request as ready for review April 8, 2025 09:57

kcreddy requested a review from a team as a code owner April 8, 2025 09:57

kcreddy requested a review from a team April 8, 2025 09:58

kcreddy added 2 commits April 8, 2025 16:48

update latest transform index

933427a

update readme

859d27e

kcreddy marked this pull request as draft April 8, 2025 15:30

kcreddy mentioned this pull request Apr 8, 2025

[meta] Qualys VMDR: Enhancement to leverage cloud workflows #11673

Closed

11 tasks

efd6 approved these changes Apr 8, 2025

View reviewed changes

maxcold reviewed Apr 9, 2025

View reviewed changes

update transform unique key and retention

591a0f0

kcreddy marked this pull request as ready for review April 14, 2025 02:36

kcreddy requested review from alexreal1314 and maxcold April 14, 2025 02:36

kcreddy mentioned this pull request Apr 14, 2025

[M365 Defender] - Add a new data stream to support vulnerability logs #7482

Closed

efd6 mentioned this pull request Apr 17, 2025

[Serverless security] [ti_misp] Failing test daily: pipeline test: test-misp-sample-ndjson.log in ti_misp.threat #13294

Closed

joegallo mentioned this pull request Apr 17, 2025

Correctly handle non-integers in nested paths in the remove processor elastic/elasticsearch#127006

Merged

alexreal1314 approved these changes Apr 28, 2025

View reviewed changes

maxcold approved these changes Apr 28, 2025

View reviewed changes

kcreddy merged commit c0cf6f2 into elastic:main Apr 28, 2025
5 of 7 checks passed

kcreddy mentioned this pull request May 8, 2025

[M365 Defender][Microsoft Defender Endpoint] Add support of vulnerability data-stream #13595

Merged

5 tasks

nick-alayil mentioned this pull request Jul 10, 2025

[Internal]: Add docs for Qualys Integration elastic/docs-content#2090

Closed

1 task

qualys_vmdr: Add latest transform for Asset Host Detections #13455

qualys_vmdr: Add latest transform for Asset Host Detections #13455

Uh oh!

Conversation

kcreddy commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Apr 8, 2025

Uh oh!

efd6 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexreal1314 commented Apr 9, 2025

Uh oh!

maxcold Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

alexreal1314 Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

clement-fouque Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Apr 13, 2025

Choose a reason for hiding this comment

Uh oh!

maxcold Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 13, 2025

🚀 Benchmarks report

Uh oh!

maxcold commented Apr 14, 2025

Uh oh!

kcreddy commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxcold commented Apr 14, 2025

Uh oh!

kcreddy commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kcreddy commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Apr 17, 2025

Uh oh!

kcreddy commented Apr 21, 2025

Uh oh!

kcreddy commented Apr 21, 2025

Uh oh!

kcreddy commented Apr 21, 2025

Uh oh!

kcreddy commented Apr 23, 2025

Uh oh!

elasticmachine commented Apr 23, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Apr 23, 2025

Quality Gate failed

Uh oh!

kcreddy commented Apr 8, 2025 •

edited

Loading

kcreddy commented Apr 9, 2025 •

edited

Loading

kcreddy Apr 13, 2025 •

edited

Loading

clement-fouque Apr 10, 2025 •

edited

Loading

kcreddy commented Apr 14, 2025 •

edited

Loading

kcreddy commented Apr 16, 2025 •

edited

Loading

kcreddy commented Apr 16, 2025 •

edited

Loading

alexreal1314 commented Apr 28, 2025 •

edited

Loading