[qualys_gav] Initial release of the Qualys GAV #14644

janvi-elastic · 2025-07-22T13:25:50Z

Proposed commit message

The initial release includes asset data stream, associated dashboards 
and visualizations.

Qualys GAV fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from documentation and live data samples, 
which were subsequently sanitized.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/qualys_gav directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: qualys_gav - START ---
╭────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ qualys_gav │             │ asset     │ dashboard qualys_gav-e7e0529f-6cb1-4b01-b5f8-568cfb07c306 is loaded │ PASS   │       1.63µs │
│ qualys_gav │ asset       │ asset     │ index_template logs-qualys_gav.asset is loaded                      │ PASS   │        257ns │
│ qualys_gav │ asset       │ asset     │ ingest_pipeline logs-qualys_gav.asset-0.1.0 is loaded               │ PASS   │        189ns │
╰────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_gav - END   ---
Done
--- Test results for package: qualys_gav - START ---
╭────────────┬─────────────┬───────────┬───────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ qualys_gav │ asset       │ pipeline  │ (ingest pipeline warnings test-asset.log) │ PASS   │ 346.797409ms │
│ qualys_gav │ asset       │ pipeline  │ test-asset.log                            │ PASS   │ 434.950727ms │
╰────────────┴─────────────┴───────────┴───────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_gav - END   ---
Done
--- Test results for package: qualys_gav - START ---
╭────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ qualys_gav │ asset       │ static    │ Verify sample_event.json │ PASS   │ 170.773573ms │
╰────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_gav - END   ---
Done
--- Test results for package: qualys_gav - START ---
╭────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ qualys_gav │ asset       │ system    │ default   │ PASS   │ 46.423833195s │
╰────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: qualys_gav - END   ---
Done

Related issues

Screenshot

elastic-vault-github-plugin-prod · 2025-07-22T13:52:54Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

janvi-elastic · 2025-07-24T12:51:06Z

@ShourieG - As discussed, I have created an issue in the Elasticsearch repository to request the necessary index deletion permissions using ILM policy.

packages/qualys_gav/data_stream/asset/agent/stream/cel.yml.hbs

packages/qualys_gav/data_stream/asset/elasticsearch/ilm/default_policy.json

packages/qualys_gav/_dev/build/docs/README.md

packages/qualys_gav/data_stream/asset/agent/stream/cel.yml.hbs

efd6 · 2025-07-30T02:56:13Z

packages/qualys_gav/_dev/deploy/docker/files/config.yml

+                              "lastVMScan": 0,
+                              "lastComplianceScan": 0,
+                              "lastFullScan": 0,
+                              "lastVmScanDateScanner": 0,
+                              "lastVmScanDateAgent": 0,
+                              "lastPcScanDateScanner": 0,
+                              "lastPcScanDateAgent": 0,


These are likely not epoch(0) (1970-01-01T00:00:00.000Z), but rather absent data, so the date processors should probably test for 0 and not retain as a timestamp if they are. Do qualys document this?

No we do not have Documentation For Fields by Qualys GAV, although we agree with your point that 0 indicated the absence of data.
So should we process to remove the field?

Yes, please.

efd6 · 2025-07-30T03:05:38Z

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

+        // Parse times into long
+        long lastBootMillis = parseTime(lastBootStr);
+        long createdDateMillis = parseTime(createdDateStr);
+
+        // Early exit
+        if(lastBootMillis < createdDateMillis) {
+          return;
+        }
+
+        // Calculate uptime
+        long uptimeMillis = lastBootMillis - createdDateMillis;
+        long uptimeSeconds =  uptimeMillis / 1000;
+
+        // Set uptime
+        ctx.host = ctx.host ?: [:];
+        ctx.host.uptime = uptimeSeconds;


I'm confused by all of this.

what is the meaning of the created_date field?

what relationship does the creation date have with the uptime? surely uptime is now - last_boot_time?

what is the meaning of the invariant assertion with the // Early exit comment? what does it mean if it fails?

created_date refers to the creation time of the asset.

We're currently mapping host.uptime as last_boot_time - created_date.

The // Early exit condition handles a corner case where last_boot_time < created_date, which would result in a negative uptime. In such cases, not to set host.uptime.

We're currently mapping host.uptime as last_boot_time - created_date.

Why? This does not seem sensible. If I create an asset a year ago, maybe booting it at that time, maybe not, and then shut it down until today. This would give me an uptime of a year. This is obviously not correct.

The // Early exit condition handles a corner case where last_boot_time < created_date, which would result in a negative uptime. In such cases, not to set host.uptime.

This "corner case" is an indication that the invariant is not correctly assumed, as indicated in my insane case above.

Sure we will remove host.uptime mapping.

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

ShourieG · 2025-08-08T12:50:03Z

@janvi-elastic, I've merged the PR and backported it to 9.1,9.0,8.19.

8.18 is still having some backport failures.

ShourieG · 2025-08-11T08:23:38Z

@janvi-elastic, @efd6, the ILM permissions for kibana_system is now updated in ES for 8.18 and above, so all good from my end on this. We can go ahead with the approval if all else looks good.

janvi-elastic · 2025-08-11T09:09:55Z

@janvi-elastic, I've merged the PR and backported it to 9.1,9.0,8.19.

8.18 is still having some backport failures.

So do we need to update the version or we can go ahead with 8.18?

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

ShourieG · 2025-08-12T06:18:58Z

@janvi-elastic, I've merged the PR and backported it to 9.1,9.0,8.19.
8.18 is still having some backport failures.

So do we need to update the version or we can go ahead with 8.18?

@janvi-elastic, 8.18 is merged now so no need to update

efd6

Thanks. Please wait for @ShourieG.

ShourieG

Updated review status due to version constraints

packages/qualys_gav/manifest.yml

elasticmachine · 2025-08-13T10:42:06Z

💚 Build Succeeded

Buildkite Build
Commit: eda4110

History

💚 Build #29921 succeeded 08c19dd
💚 Build #29593 succeeded 1c81efc
💚 Build #29402 succeeded d33683b
💚 Build #29310 succeeded 701a79e
💚 Build #29214 succeeded 6ad6749
💚 Build #29026 succeeded 758ebdc

elastic-sonarqube · 2025-08-13T10:42:11Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
83.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-08-13T11:48:51Z

Package qualys_gav - 0.1.0 containing this change is available at https://epr.elastic.co/package/qualys_gav/0.1.0/

The initial release includes asset data stream, associated dashboards and visualizations. Qualys GAV fields are mapped to their corresponding ECS fields where possible. Test samples were derived from documentation and live data samples, which were subsequently sanitized.

Initial release

0a0116f

janvi-elastic requested a review from a team as a code owner July 22, 2025 13:25

Update changelog

78675fc

Add ILM Policy

758ebdc

ShourieG requested a review from a team July 25, 2025 09:45

ShourieG reviewed Jul 25, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/agent/stream/cel.yml.hbs Outdated Show resolved Hide resolved

ShourieG reviewed Jul 25, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/agent/stream/cel.yml.hbs Outdated Show resolved Hide resolved

ShourieG reviewed Jul 25, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/elasticsearch/ilm/default_policy.json Show resolved Hide resolved

ShourieG reviewed Jul 28, 2025

View reviewed changes

packages/qualys_gav/_dev/build/docs/README.md Outdated Show resolved Hide resolved

Resolved review comments

6ad6749

janvi-elastic requested a review from ShourieG July 29, 2025 06:18

efd6 reviewed Jul 30, 2025

View reviewed changes

Resolved review comments

701a79e

janvi-elastic requested a review from efd6 July 31, 2025 05:13

ShourieG reviewed Jul 31, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

efd6 reviewed Jul 31, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Remove on_failure processor on json processor

d33683b

ShourieG reviewed Aug 1, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

remove mapping and remove date fields having 0 value

1c81efc

janvi-elastic requested review from ShourieG and efd6 August 11, 2025 09:08

efd6 reviewed Aug 12, 2025

View reviewed changes

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

packages/qualys_gav/data_stream/asset/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Update script to remove 0 values

08c19dd

efd6 approved these changes Aug 13, 2025

View reviewed changes

ShourieG approved these changes Aug 13, 2025

View reviewed changes

ShourieG requested changes Aug 13, 2025

View reviewed changes

packages/qualys_gav/manifest.yml Outdated Show resolved Hide resolved

Upgrade package version.

eda4110

ShourieG approved these changes Aug 13, 2025

View reviewed changes

ShourieG merged commit ef3b198 into elastic:main Aug 13, 2025
9 checks passed

[qualys_gav] Initial release of the Qualys GAV #14644

[qualys_gav] Initial release of the Qualys GAV #14644

Uh oh!

Conversation

janvi-elastic commented Jul 22, 2025 • edited by ShourieG Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshot

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 22, 2025

🚀 Benchmarks report

Uh oh!

janvi-elastic commented Jul 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

janvi-elastic Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

janvi-elastic Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

janvi-elastic Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShourieG commented Aug 8, 2025

Uh oh!

ShourieG commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janvi-elastic commented Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

ShourieG commented Aug 12, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

ShourieG left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Aug 13, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Aug 13, 2025

Quality Gate passed

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 13, 2025

Uh oh!

Reviewers

janvi-elastic commented Jul 22, 2025 •

edited by ShourieG

Loading

ShourieG commented Aug 11, 2025 •

edited

Loading

ShourieG left a comment •

edited

Loading