Skip to content

Latest commit

 

History

History
145 lines (103 loc) · 6.3 KB

zero-code-ebpf-instrumentation.md

File metadata and controls

145 lines (103 loc) · 6.3 KB

Zero-Code Instrumentation with eBPF

Important

OBI support is a work in progress 🚧. Some features may not be fully supported on all platforms or configurations. OBI is currently only tested on Ubuntu nodes that meet the upstream documentation requirements. Always refer to the upstream documentation for the latest supported Ubuntu versions and platform information.

OpenTelemetry eBPF Instrumentation (OBI) is a lightweight and efficient way to collect telemetry data using eBPF for user-space applications. It provides many features for applications using a zero-code instrumentation approach:

  • Context Propagation: Propagate trace context across services automatically
  • Broad language support: Java, .NET, Go, Python, Ruby, Node.js, C, C++, and Rust
  • Lightweight: No code changes required, no libraries to install, no restarts needed
  • Efficient instrumentation: Traces and metrics are captured by eBPF probes with minimal overhead
  • Kubernetes-native: Provides configuration-free auto-instrumentation for Kubernetes applications
  • Visibility into encrypted communications: Capture transactions over TLS/SSL without decryption
  • Protocol support: HTTP/S, gRPC, SQL, Redis, MongoDB, and more
  • Low cardinality metrics: Prometheus-compatible metrics with low cardinality for cost reduction
  • Network observability: Capture network flows between services
  • Database traces: Capture database queries and connections

Tip

OBI provides broad observability for most services within a cluster.

Use the language specific auto-instrumentation to get deeper insights into applications where supported (Java, Node.js, Python, .NET). OBI will auto-detect if auto-instrumentation is present and avoid duplicating traces.

For more detailed observability, you can also combine OBI with application-level instrumentation using OpenTelemetry SDKs.

Getting Started

OBI can be enabled by setting the obi.enabled value to true during installation or upgrades.

helm install my-splunk-otel-collector \
  splunk-otel-collector-chart/splunk-otel-collector \
  --set="splunkObservability.realm=${SPLUNK_REALM}" \
  --set="splunkObservability.accessToken=${SPLUNK_ACCESS_TOKEN}" \
  --set="obi.enabled=true"

Configuration Options

For basic usage, no additional configuration is required.

Additional configuration options are available to customize OBI features. Refer to the OBI chart's documentation for more details.

Verification

After installation, verify OBI pods are up and running:

# Check if eBPF instrumentation pods are running
kubectl get pods -l app.kubernetes.io/name=obi

# Check pod logs for any warnings or errors
kubectl logs -l app.kubernetes.io/name=obi -f

Platform Requirements

Warning

OBI only works on Linux nodes. Windows is not supported.

  • Kubernetes 1.24+
  • Helm 3.9+
  • Linux kernel version 5.8 or later (or 4.18 for Red Hat Enterprise Linux)
  • x86_64 or arm64 processor
  • Runtime support for eBPF (most modern Linux distributions)

Security and Capabilities

OBI requires elevated privileges and specific Linux kernel capabilities to function. For complete security considerations and configuration options, refer to the OBI Security Documentation.

Default Permissions

By default, OBI is configured with the following elevated permissions:

  • Privileged mode (privileged: true) for unrestricted access

Reducing Permissions

If your cluster policies require it, you can reduce permissions by:

  1. Modifying container permissions
    • Unprivileged mode: privileged: false
    • Run as root user: runAsUser: 0
    • Disable privilege escalation (optional): allowPrivilegeEscalation: false
    • Read-only filesystem (optional): readOnlyRootFilesystem: true
  2. Modifying Linux capabilities (Remove unnecessary capabilities as needed)
    • BPF: Core eBPF for functionality (REQUIRED)
    • NET_RAW: Network observability
    • NET_ADMIN: Network and application observability
    • PERFMON: Performance monitoring
    • DAC_READ_SEARCH: Kernel introspection
    • CHECKPOINT_RESTORE: Process introspection
    • SYS_PTRACE: Allow access to container namespaces and ability to inspect executables

Consult the security documentation for detailed guidance on adjusting permissions and information about how each capability is used.

Note

Reducing permissions may limit OBI features.

Troubleshooting

Pods not running on Windows nodes

This is expected. The nodeSelector: kubernetes.io/os: linux prevents scheduling on Windows.

Missing capability warnings

If you see warnings like "Required system capabilities not present", either:

  1. Ensure nodes have required kernel support
  2. Adjust kernel.perf_event_paranoid sysctl on nodes (for AKS/EKS)
  3. Review and, if necessary, adjust the granular securityContext.capabilities as described in the security documentation. You may consider granting SYS_ADMIN, but understand this is highly privileged and effectively gives root‑equivalent access to the host.

Pod fails to start

  1. Check security policies (Pod Security Standards, Network Policies)
  2. Ensure adequate CPU/memory resources (default: 100m CPU, 100Mi memory)

Further Reading