Security monitoring feature parity with Agent V2

[kamalchaturvedi]

Proposed changes

Commit 1:
The securityviolationsprocessor now processes NGINX App Protect WAF syslog messages, and transforms them into SecurityViolationEvent protobuf messages. This protobuf definition replaces the existing struct definition in /internal folder. This was done to allow management-plane to import this schema as a contract for handling security violations.

Commit 2:
Additionally, added the following capabilities to the parsing the details extraction from raw violations, to bring the feature in parity with Agent V2 implementation:

1. Parses XML violation details with context extraction (parameter, header, cookie, uri, request)
2. Extracts attack signature details

Commit 3:
These changes were thoroughly tested with addition of /testdata in Agent V2 implementation (https://github.com/nginx/agent/tree/dev-v2/src/extensions/nginx-app-protect/monitoring/processor/testdata) and additional variety of violations, to ensure robust coverage.

Commit 4:
Added temporarily the hardcoded docker0 interface IP, required to enable NAP V5 integration.

Agent Config Modifications for Test

features:
  - certificates
  - configuration
  - metrics
  - file-watcher
  - api-action
  - logs-nap 

collector: 
  exporters:
    debug: {}
  processors:
    batch:
      "logs":
        send_batch_size: 1000
        timeout: 30s
        send_batch_max_size: 1000
  pipelines:
   logs:
     "default-security-events":
       receivers: ["tcplog/nginx_app_protect"]
       processors: ["securityviolations/default","batch/logs"]
       exporters: ["debug","otlp/default"]

Testing

Violations Triggered

1. curl -v 'http://127.0.0.1/myfile.tmp'

Expected Violations: VIOL_FILETYPE, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008429009\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"GET\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/myfile.tmp\"  request:\"GET /myfile.tmp HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33336  violations:\"HTTP protocol compliance failed::Illegal file type::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:2  sig_set_names:\"N/A\"  sig_cves:\"N/A\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_FILETYPE\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}"}

2. curl -X SEARCH -k -v 'http://127.0.0.1/hello'

Expected Violations: VIOL_METHOD, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008429519\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"SEARCH\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/hello\"  request:\"SEARCH /hello HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33344  violations:\"HTTP protocol compliance failed::Illegal method::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:2  sig_set_names:\"N/A\"  sig_cves:\"N/A\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_METHOD\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}"}

• curl -k -v 'http://127.0.0.1/a=<script>getAllMoney()</script>'

Expected Violations: VIOL_ATTACK_SIGNATURE, VIOL_HTTP_PROTOCOL, VIOL_BOT_CLIENT, VIOL_URL_METACHAR, VIOL_RATING_THREAT
Expected Signature IDs ": 200000099, 200000093

Output Payload:

{"resource": {"service.instance.id": "f904b8d3-4d6f-46b8-9b77-ba3926cc3407", "service.name": "otel-nginx-agent", "service.version": "v3.6.2"}, "otelcol.component.id": "securityviolations/default", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/default-security-events", "otelcol.signal": "logs", "protobuf": "policy_name:\"nms_app_protect_strict_policy\"  support_id:\"2378510662008430029\"  request_outcome:REQUEST_OUTCOME_REJECTED  request_outcome_reason:SECURITY_WAF_VIOLATION  blocking_exception_reason:\"N/A\"  method:\"GET\"  protocol:\"HTTP\"  xff_header_value:\"N/A\"  uri:\"/a=<script>getAllMoney()</script>\"  request:\"GET /a=<script>getAllMoney()</script> HTTP/1.1\\\\r\\\\nHost: 127.0.0.1\\\\r\\\\nUser-Agent: curl/8.5.0\\\\r\\\\nAccept: */*\\\\r\\\\n\\\\r\\\\n\"  request_status:REQUEST_STATUS_BLOCKED  vs_name:\"1-localhost:1-/\"  remote_addr:\"127.0.0.1\"  destination_port:80  server_port:33358  violations:\"HTTP protocol compliance failed::Illegal meta character in URL::Attack signature detected::Violation Rating Threat detected::Bot Client Detected\"  sub_violations:\"HTTP protocol compliance failed:Host header contains IP address\"  violation_rating:5  sig_set_names:\"{High Accuracy Signatures;Cross Site Scripting Signatures;Generic Detection Signatures (High/Medium Accuracy)}\"  sig_cves:\"{High Accuracy Signatures;Cross Site Scripting Signatures;Generic Detection Signatures (High/Medium Accuracy)}\"  client_class:\"Untrusted Bot\"  client_application:\"N/A\"  client_application_version:\"N/A\"  severity:SEVERITY_CRITICAL  threat_campaign_names:\"N/A\"  bot_anomalies:\"N/A\"  bot_category:\"HTTP Library\"  enforced_bot_anomalies:\"N/A\"  bot_signature_name:\"curl\"  system_id:\"ea6197e981ac\"  parent_hostname:\"ea6197e981ac\"  violations_data:{violation_data_name:\"VIOL_ATTACK_SIGNATURE\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"/a=<script>getAllMoney()</script>\"}  violation_data_signatures:{sig_data_id:200000099  sig_data_blocking_mask:\"3\"  sig_data_buffer:\"/a=<script>getAllMoney()</script>\"  sig_data_offset:3  sig_data_length:7}  violation_data_signatures:{sig_data_id:200000093  sig_data_blocking_mask:\"3\"  sig_data_buffer:\"/a=<script>getAllMoney()</script>\"  sig_data_offset:4  sig_data_length:7}}  violations_data:{violation_data_name:\"VIOL_HTTP_PROTOCOL\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_URL_METACHAR\"  violation_data_context:\"uri\"  violation_data_context_data:{context_data_name:\"uri\"  context_data_value:\"L2E9PHNjcmlwdD5nZXRBbGxNb25leSgpPC9zY3JpcHQ+\"}}  violations_data:{violation_data_name:\"VIOL_BOT_CLIENT\"  violation_data_context_data:{}}  violations_data:{violation_data_name:\"VIOL_RATING_THREAT\"  violation_data_context_data:{}}"}

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING document
• I have run make install-tools and have attached any dependency changes to this pull request
• If applicable, I have added tests that prove my fix is effective or that my feature works
• If applicable, I have checked that any relevant tests pass after adding my changes
• If applicable, I have updated any relevant documentation (README.md)
• If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

[github-actions]

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[nginx-nickc]

can we prebuild this map? If we expect to be high-volume, rebuilding the map log parse is probably not needed?

[nginx-nickc]

nit, you can preallocate with length

trimmedParts := make([]string, 0, len(parts))

[nginx-nickc]

did we explore the golang csv parser? https://pkg.go.dev/encoding/csv