[New Integration] StatsD OpenTelemetry Input Package #17109
Conversation
This adds a new input package for collecting StatsD metrics using the OpenTelemetry StatsD receiver. The package enables collection of metrics from StatsD-compatible applications through the EDOT collector. Features: - Configurable endpoint and transport (UDP, TCP, unixgram) - Adjustable aggregation interval - Support for metric type labels - Monotonic counter option - Supports DogStatsD, InfluxDB, and Graphite tag formats Closes: #17108 Co-authored-by: Julien Lind <jlind23@users.noreply.github.com>
|
@claude please do a pr review |
|
Claude finished @strawgate's task in 3m 52s —— View job PR Review CompleteI've submitted my review requesting changes. See the review for details. Summary of Findings: 🟠 HIGH: CODEOWNERS missing entry - The new package needs to be added to 🟠 HIGH: Directory/package name mismatch - Directory is 🟡 MEDIUM: Boolean template rendering - Variables Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
|
@claude could you please provide testing instructions? Ideally tests are based on elastic-package usage. |
|
Claude finished @jlind23's task in 1m 6s —— View job Testing Instructions for StatsD OpenTelemetry Input Package
Prerequisites
Policy TestsPolicy tests verify that the integration policy generates the expected Elastic Agent configuration. The package includes policy test files at:
# Start the Elastic stack (Kibana and Elasticsearch only needed)
elastic-package stack up -d --services kibana,elasticsearch
# Navigate to the package directory
cd packages/statsd_input_otel
# Run policy tests
elastic-package test policy
# Optional: Run with verbose output
elastic-package test policy -vSystem TestsSystem tests validate the complete data flow from the StatsD service to Elasticsearch. The package includes:
# Start the full Elastic stack
elastic-package stack up -d
# Navigate to the package directory
cd packages/statsd_input_otel
# Run system tests
elastic-package test system
# Optional: Run with verbose output and human-readable report
elastic-package test system -v --report-format human
# Optional: Keep services running for debugging (pauses cleanup for 10 minutes)
elastic-package test system --defer-cleanup 10mStatic TestsStatic tests validate package structure, formatting, and manifest correctness: cd packages/statsd_input_otel
elastic-package test staticRun All TestsTo run the complete test suite: # Ensure stack is running
elastic-package stack up -d
cd packages/statsd_input_otel
# Run all test types
elastic-package test
# Cleanup
elastic-package stack downManual Testing (Sending StatsD Metrics)If you want to manually test the integration:
Quick Validation Checklist
Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds a new StatsD OpenTelemetry Input Package for collecting StatsD metrics. I've identified several issues that need to be addressed:
Critical Issues:
- CODEOWNERS missing entry - The new package at
/packages/statsd_input_otelis not registered in.github/CODEOWNERS - Directory/package name mismatch - The package directory is
statsd_input_otelbut thenamein manifest.yml isstatsd_otel_input. This should be consistent (typically directory and name match).
Template Issues:
3. The boolean variables (enable_metric_type, is_monotonic_counter) with default value false will not render in the template because {{#if false_value}} evaluates to false. Consider always including these fields rather than conditionally including them.
CI Failure:
The Buildkite failure in "Check go sources" appears to be related to the missing CODEOWNERS entry - the mage check command validates that all packages have corresponding CODEOWNERS entries.
Positive Aspects:
- Good documentation with clear explanations of supported metric types and tag formats
- Proper test infrastructure with policy tests and system tests
- Follows the existing OTel input package patterns (similar to nginx_input_otel)
- Configuration options align with upstream statsdreceiver
This review was generated by Claude. Feedback? React with 👍 or 👎.
There was a problem hiding this comment.
Two issues need to be addressed before merge:
-
Template boolean rendering (HIGH): In
input.yml.hbs, the{{#if enable_metric_type}}and{{#if is_monotonic_counter}}conditionals will skip rendering when these values arefalsebecause Handlebars treatsfalseas falsy. This means the StatsD receiver won't receive these configuration values when set to their defaults. Remove the conditionals and render these fields unconditionally. -
System test endpoint configuration (HIGH): The
test-default-config.ymluses{{Hostname}}:{{Port}}which derives from the docker service. However, the agent is a UDP listener that needs to bind to its own address (e.g.,0.0.0.0:8125), not connect to the docker service. The docker service sends UDP to the agent, not the other way around. Compare withpackages/statsd_input/_dev/test/system/test-default-config.ymlwhich correctly uses explicitlisten_address: 0.0.0.0andlisten_port: 8125.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| {{#if aggregation_interval}} | ||
| aggregation_interval: {{aggregation_interval}} | ||
| {{/if}} | ||
| {{#if enable_metric_type}} |
There was a problem hiding this comment.
🟠 HIGH Boolean fields with default false won't be rendered
Why: Handlebars {{#if false}} evaluates to false, so enable_metric_type and is_monotonic_counter won't appear when set to false.
| {{#if enable_metric_type}} | |
| receivers: | |
| statsd: | |
| endpoint: {{endpoint}} | |
| {{#if transport}} | |
| transport: {{transport}} | |
| {{/if}} | |
| {{#if aggregation_interval}} | |
| aggregation_interval: {{aggregation_interval}} | |
| {{/if}} | |
| enable_metric_type: {{enable_metric_type}} | |
| is_monotonic_counter: {{is_monotonic_counter}} | |
| processors: | |
| resourcedetection/system: | |
| detectors: ["system"] | |
| service: | |
| pipelines: | |
| metrics: | |
| receivers: [statsd] | |
| processors: [resourcedetection/system] |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
packages/statsd_input_otel/_dev/test/system/test-default-config.yml
Outdated
Show resolved
Hide resolved
packages/statsd_input_otel/_dev/deploy/docker/docker-compose.yml
Outdated
Show resolved
Hide resolved
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…g.yml Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
@claude please do a pr review |
There was a problem hiding this comment.
The package structure follows OTel input package conventions and passes lint/build checks.
One clarification needed: The handlebars template uses {{#if enable_metric_type}} and {{#if is_monotonic_counter}} conditional blocks, but these won't render when the values are false (the defaults). However, the test expected output shows these values present as false. Please verify the test expected file is correctly generated - either the conditionals should be removed from the template to always render these options, or the expected test output may need adjustment if false values shouldn't appear in the config.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| title: Endpoint | ||
| description: Address and port to listen on for StatsD metrics. | ||
| default: localhost:8125 | ||
| - name: transport |
There was a problem hiding this comment.
⚪ LOW Consider using type: select for transport validation
Why: Using select with defined options prevents invalid transport values at configuration time.
| - name: transport | |
| - name: transport | |
| type: select | |
| required: false | |
| title: Transport Protocol | |
| description: Protocol used by the StatsD server. | |
| options: | |
| - value: udp | |
| text: UDP | |
| - value: tcp | |
| text: TCP | |
| - value: unixgram | |
| text: Unix Datagram Socket | |
| default: udp | |
| show_user: false |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
Claude finished @jlind23's task in 1m 32s —— View job PR Review Complete
I've submitted my review requesting changes. See the review for details. Summary of Findings: 🔴 CRITICAL: docker-compose.yml duplicate content - The file contains duplicate content (lines 9-15 repeat lines 1-7), making it invalid YAML. 🔴 CRITICAL: test-default-config.yml invalid YAML - The file has duplicate 🟠 HIGH: Boolean template rendering - In Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
There was a problem hiding this comment.
Two YAML files have syntax errors that will cause tests to fail:
docker-compose.ymlcontains duplicate content (lines 9-15 repeat lines 1-7)test-default-config.ymlhas duplicatevars:declarations
Additionally, the Handlebars template conditionals for enable_metric_type and is_monotonic_counter won't render when these boolean values are false. The policy test expected output shows these values present, but the template won't produce them with the current logic.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| @@ -0,0 +1,15 @@ | |||
| version: '2.3' | |||
There was a problem hiding this comment.
🔴 CRITICAL File contains duplicate content
Why: The docker-compose.yml file has been corrupted - it contains duplicate content starting at line 9, which makes it invalid YAML.
| version: '2.3' | |
| version: '2.3' | |
| services: | |
| statsd_input_otel: | |
| image: docker.elastic.co/observability/stream:v0.18.0 | |
| volumes: | |
| - ./sample_logs:/sample_logs:ro | |
| command: log --start-signal=SIGHUP --delay=10s --addr elastic-agent:8125 -p=udp /sample_logs/test-statsd.log |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| @@ -0,0 +1,3 @@ | |||
| vars: | |||
There was a problem hiding this comment.
🔴 CRITICAL Duplicate vars key makes YAML invalid
Why: The file contains two 'vars:' declarations (lines 1-2), making it invalid YAML. This will cause test failures.
| vars: | |
| vars: | |
| endpoint: "0.0.0.0:8125" |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| {{#if aggregation_interval}} | ||
| aggregation_interval: {{aggregation_interval}} | ||
| {{/if}} | ||
| {{#if enable_metric_type}} |
There was a problem hiding this comment.
🟠 HIGH Boolean false values won't render in template
Why: Handlebars {{#if false}} evaluates to false, so enable_metric_type and is_monotonic_counter won't appear in the generated config when set to false. The policy test expects these values but the template won't produce them.
| {{#if enable_metric_type}} | |
| receivers: | |
| statsd: | |
| endpoint: {{endpoint}} | |
| {{#if transport}} | |
| transport: {{transport}} | |
| {{/if}} | |
| {{#if aggregation_interval}} | |
| aggregation_interval: {{aggregation_interval}} | |
| {{/if}} | |
| enable_metric_type: {{enable_metric_type}} | |
| is_monotonic_counter: {{is_monotonic_counter}} | |
| processors: | |
| resourcedetection/system: | |
| detectors: ["system"] | |
| service: | |
| pipelines: | |
| metrics: | |
| receivers: [statsd] | |
| processors: [resourcedetection/system] |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
Two files contain duplicate YAML content that will cause parsing failures:
- docker-compose.yml - The entire service definition is duplicated, making the file invalid
- test-default-config.yml - The
vars:key is duplicated
These must be fixed before merge as they will break the test infrastructure.
Additionally, there's an inconsistency between the package name (statsd_input_otel) and the dataset name used in the expected test output (statsd_otel_input). While the tests may pass if the expected file reflects actual system behavior, this naming inconsistency could cause confusion.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| @@ -0,0 +1,15 @@ | |||
| version: '2.3' | |||
There was a problem hiding this comment.
🔴 CRITICAL Duplicate docker-compose content causes invalid YAML
Why: The file contains the entire docker-compose definition duplicated, making it invalid YAML that will fail to parse.
| version: '2.3' | |
| version: '2.3' | |
| services: | |
| statsd_input_otel: | |
| image: docker.elastic.co/observability/stream:v0.18.0 | |
| volumes: | |
| - ./sample_logs:/sample_logs:ro | |
| command: log --start-signal=SIGHUP --delay=10s --addr elastic-agent:8125 -p=udp /sample_logs/test-statsd.log |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| @@ -0,0 +1,3 @@ | |||
| vars: | |||
There was a problem hiding this comment.
🔴 CRITICAL Duplicate 'vars:' key causes invalid YAML
Why: The file has 'vars:' defined twice, which is invalid YAML and will fail to parse during tests.
| vars: | |
| vars: | |
| endpoint: "0.0.0.0:8125" |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| - context: datapoint | ||
| statements: | ||
| - set(attributes["data_stream.type"], "metrics") | ||
| - set(attributes["data_stream.dataset"], "statsd_otel_input.statsdreceiver") |
There was a problem hiding this comment.
🟡 MEDIUM Dataset name inconsistent with package name
Why: The package is named 'statsd_input_otel' but the dataset uses 'statsd_otel_input'. This inconsistency may cause confusion with metric routing.
| - set(attributes["data_stream.dataset"], "statsd_otel_input.statsdreceiver") | |
| - set(attributes["data_stream.dataset"], "statsd_input_otel.statsdreceiver") |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
The expected policy test file references the old package name statsd_otel_input instead of statsd_input_otel after the rename in commit 2531099. This will cause policy tests to fail.
Additionally, the input template uses {{#if}} conditionals around boolean values enable_metric_type and is_monotonic_counter. In Handlebars, {{#if false}} evaluates as falsy and won't render the block. However, the expected test output includes these properties with false values. Either remove the conditionals to always include these properties (matching the filelog_otel pattern), or update the expected file to not include them when false.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| - context: datapoint | ||
| statements: | ||
| - set(attributes["data_stream.type"], "metrics") | ||
| - set(attributes["data_stream.dataset"], "statsd_otel_input.statsdreceiver") |
There was a problem hiding this comment.
🟠 HIGH data_stream.dataset uses old package name
Why: Package was renamed from statsd_otel_input to statsd_input_otel but expected file wasn't updated. Policy tests will fail.
| - set(attributes["data_stream.dataset"], "statsd_otel_input.statsdreceiver") | |
| - set(attributes["data_stream.dataset"], "statsd_input_otel.statsdreceiver") |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| {{#if aggregation_interval}} | ||
| aggregation_interval: {{aggregation_interval}} | ||
| {{/if}} | ||
| {{#if enable_metric_type}} |
There was a problem hiding this comment.
🟡 MEDIUM Boolean conditionals may not work as expected
Why: Using #if on boolean false values won't render the block, but test-default.expected shows these properties included with false values.
| {{#if enable_metric_type}} | |
| enable_metric_type: {{enable_metric_type}} | |
| is_monotonic_counter: {{is_monotonic_counter}} |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| title: Endpoint | ||
| description: Address and port to listen on for StatsD metrics. | ||
| default: localhost:8125 | ||
| - name: transport |
There was a problem hiding this comment.
⚪ LOW Consider using select type for transport
Why: A select type provides validation and better UX by showing only valid options (udp, tcp, unixgram) in the Fleet UI.
| - name: transport | |
| - name: transport | |
| type: select | |
| required: false | |
| title: Transport Protocol | |
| description: Protocol used by the StatsD server. | |
| default: udp | |
| show_user: false | |
| options: | |
| - text: UDP | |
| value: udp | |
| - text: TCP | |
| value: tcp | |
| - text: Unix Datagram | |
| value: unixgram |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
The package structure follows existing OTel input package patterns well. One non-blocking improvement: the transport field would benefit from using a select type instead of text to validate against the fixed set of valid values (udp, tcp, unixgram) and provide a better user experience in the Fleet UI.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| title: Endpoint | ||
| description: Address and port to listen on for StatsD metrics. | ||
| default: localhost:8125 | ||
| - name: transport |
There was a problem hiding this comment.
🟡 MEDIUM Use select type for transport field
Why: Using text type allows arbitrary invalid values; select type provides validation and better UX.
| - name: transport | |
| - name: transport | |
| type: select | |
| required: false | |
| title: Transport Protocol | |
| description: Protocol used by the StatsD server. | |
| default: udp | |
| show_user: false | |
| options: | |
| - value: udp | |
| text: UDP | |
| - value: tcp | |
| text: TCP | |
| - value: unixgram | |
| text: Unix Socket |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Installed successfully
|
|
@claude please do a pr review |
|
Claude finished @jlind23's task in 2m 1s —— View job PR Review Complete
I've approved this PR. See the review for details. Summary: All previously identified issues have been addressed:
The package passes Non-blocking suggestion: Consider using a Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
There was a problem hiding this comment.
The package passes lint and build checks. All previously identified issues (duplicate YAML content, boolean template rendering, naming consistency) have been addressed. The package follows existing OTel input package patterns and includes proper test infrastructure.
Non-blocking suggestion:
- The transport field could use a
selecttype instead oftextfor better validation (see inline comment)
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
There was a problem hiding this comment.
The package has a critical YAML validation error that prevents it from passing elastic-package check.
The transport variable in manifest.yml contains duplicate mapping keys (type, required, title, description, default, show_user appear twice at lines 38-43 and 51-56). This causes YAML parsing failures:
yaml: unmarshal errors:
line 51: mapping key "type" already defined at line 38
line 52: mapping key "required" already defined at line 39
...
The duplicate fields should be removed - keep only one set of field definitions for the transport variable, consolidating with the select type and options array.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| value: tcp | ||
| - text: Unix Socket | ||
| value: unixgram | ||
| type: text |
There was a problem hiding this comment.
🔴 CRITICAL Duplicate YAML keys in transport variable
Why: Duplicate mapping keys cause YAML parsing errors and fail elastic-package validation.
| type: text | |
| - name: transport | |
| type: select | |
| required: false | |
| title: Transport Protocol | |
| description: Protocol used by the StatsD server. Supported values are udp, tcp, and unixgram. | |
| default: udp | |
| show_user: false | |
| options: | |
| - text: UDP | |
| value: udp | |
| - text: TCP | |
| value: tcp | |
| - text: Unix Socket | |
| value: unixgram |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
The manifest.yml contains duplicate YAML keys (type and required at lines 51-52 duplicate lines 38-39), which causes the file to fail YAML parsing. Running elastic-package check confirms this breaks package validation.
Remove lines 51-52 to fix.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| - text: TCP | ||
| value: tcp | ||
| - text: Unix Socket | ||
| value: unixgram |
There was a problem hiding this comment.
🔴 CRITICAL Duplicate YAML keys cause parse failure
Why: Lines 51-52 duplicate 'type' and 'required' already defined at lines 38-39, causing YAML parse errors and failing elastic-package check.
| value: unixgram | |
| - text: Unix Socket | |
| value: unixgram | |
| - name: aggregation_interval |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
@claude please do a pr review |
There was a problem hiding this comment.
The StatsD OpenTelemetry Input Package is well-structured and follows the established patterns for OTel input packages. Key observations:
- Package structure is consistent with other _otel input packages (nginx_input_otel, etc.)
- Configuration options correctly map to the upstream statsdreceiver
- Test infrastructure is in place with valid StatsD sample data
- CODEOWNERS entry is consistent with the existing statsd_input package team
- elastic-package check passes successfully
No blocking issues found.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
Claude finished @jlind23's task in 1m 49s —— View job PR Review Complete
I've approved this PR. See the review for details. Summary: All previously identified issues have been resolved:
CI checks are passing. No blocking issues found. Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
There was a problem hiding this comment.
The StatsD OpenTelemetry Input Package passes validation and follows established patterns for OTel input packages.
Verified:
- ✅ Package passes
elastic-package check(lint + build) - ✅ CODEOWNERS entry added correctly
- ✅ Package naming is consistent (
statsd_input_otel) - ✅ docker-compose.yml is valid YAML
- ✅ test-default-config.yml is valid YAML
- ✅ Boolean template rendering fixed (unconditional rendering)
- ✅ Manifest YAML structure is valid with select type for transport
- ✅ Documentation includes clear explanation of supported metrics and tags
- ✅ Test infrastructure in place with sample StatsD data
- ✅ CI checks passing
No blocking issues found.
This review was generated by Claude. Feedback? React with 👍 or 👎.
💔 Build Failed
Failed CI StepsHistory
|
|
@nimarezainia @perk this one will be blocked up until the statsD receiver is added to EDOT - https://www.elastic.co/docs/reference/edot-collector/components could you please follow up on this? |
andrewkroh
added
New Integration
Summary
Adds a new input package for collecting StatsD metrics using the OpenTelemetry StatsD receiver.
Changes
statsd_input_otelpackage with OTel receiver configurationCloses #17108
Generated with Claude Code