Qualys KB Data Connector: Migrate from deprecated Data Collector API to Logs Ingestion API (DCR/DCE)

[Lodewyk-Git]

Is your feature request related to a problem? Please describe.

The Qualys VM KnowledgeBase (KB) data connector under Solutions/Qualys VM Knowledgebase/Data Connectors still uses the legacy Log Analytics HTTP Data Collector API (/api/logs) with shared-key authentication (WorkspaceID + WorkspaceKey). This API is deprecated and scheduled for retirement in September 2026. The connector also uses filesystem-based checkpointing (CSV on C:\home\site), which is unreliable on Consumption plan due to ephemeral storage during scale-out and restarts.

Additionally, the current connector has several functional issues:

• Checkpoint updates are commented out in the "no new records" branches, causing repeated re-fetching of the same time window
• ConvertTo-Json is called without -Depth, truncating nested objects at depth 2
• $mbytes calculation encodes the raw array instead of the JSON string, producing incorrect size checks
• Multi-value fields (CVE_ID, Vendor References, Software) only capture the first value instead of all entries
• No retry logic for transient API failures
• No error handling on session logout (can crash the function)
• Filter parameters are passed unsanitized to the API

Describe the solution you'd like

Migrate the Qualys KB connector to the Azure Monitor Logs Ingestion API with Data Collection Rules (DCRs) and Data Collection Endpoints (DCEs). A fully working replacement is available at: https://github.com/Lodewyk-Git/AzureFunctionQualysKB

Key changes in the replacement:

• Auth: Managed Identity with Entra ID tokens (resource https://monitor.azure.com) instead of shared keys
• Ingestion: Logs Ingestion API with DCR/DCE instead of deprecated Data Collector API
• Schema: Explicit DCR-based custom table with typed columns (no _s/_d suffixes)
• Checkpoint: Azure Blob Storage instead of filesystem CSV
• Chunking: 1 MB payload chunking with 950 KB safety margin (Logs Ingestion API limit) instead of the legacy 30 MB check
• Retry: Exponential backoff with jitter, Retry-After header support for 429/5xx responses
• Multi-value fields: Full array extraction for CVE_ID, CVE_URL, Vendor References, Software, Auth Types
• Filter validation: Input sanitization via Format-KBFilterParameters
• Code structure: Modular design with run.ps1 (orchestrator) and QualysKBHelpers.psm1 (10 exported helper functions)
• ARM template: One-click azuredeploy.json that deploys Function App, Storage, DCE, DCR, custom table, RBAC, and blob containers
• RBAC: Monitoring Metrics Publisher scoped to the DCR only (least privilege)
• Transform KQL: source | extend TimeGenerated = iif(isnotempty(DateValue), todatetime(DateValue), now()) | project-away DateValue

Describe alternatives you've considered

• Patching the existing connector to only fix the commented-out checkpoint lines and depth issues, but this does not address the deprecated API or shared-key auth model
• Waiting for September 2026 deprecation deadline, but this risks production outages for organisations still using the connector
• Using a different ingestion method (e.g. Azure Monitor Agent), but the Qualys KB API is a pull-based REST API that requires an Azure Function - AMA is not applicable here

Additional context

• The replacement connector has been tested in production and handles first-run (no checkpoint blob), empty responses, oversized payloads, and partial chunk failures
• The azuredeploy.json includes a Deploy to Azure button for portal deployment
• The modular ARM templates in infra/ support scripted deployment via Deploy-Solution.ps1 with a pre-flight check that auto-migrates classic tables to DCR-based
• PowerShell 7.4 runtime, Azure Functions v4, Consumption Y1 plan
• Repository: https://github.com/Lodewyk-Git/AzureFunctionQualysKB