Feature Request: dtctl send event — Unified CLI for Agent Observability Lifecycle

[WulfAI]

Problem Statement

Agents as First-Class OpenPipeline Data Sources with Custom Ingest Endpoints

dtctl is excellent for configuring observability (pipelines, dashboards, SLOs, workflows), but agents—as autonomous systems that emit telemetry—currently have no native way to send data to their custom-configured OpenPipeline endpoints via the same CLI.

This creates friction in closed-loop agentic systems where agents:

1. Operators configure observability infrastructure (dtctl create settings ...) ✓
2. Agents query telemetry for context (dtctl query "fetch bizevents | filter ...") ✓
3. Agents execute workflows for self-remediation (dtctl exec workflow ...) ✓
4. Agents emit telemetry to their custom endpoint ... ❌ (requires a different tool)

Current Flow (Fragmented)

Agent has telemetry
  ↓
SDK or raw API call (curl, Python client)
  ↓
Dynatrace built-in endpoint
  ↓
Generic processing (shared with all other events)

Desired Flow (Unified dtctl + Custom Endpoints)

Agent has telemetry
  ↓
dtctl send event --endpoint agent-activity (same tool)
  ↓
Custom-configured OpenPipeline endpoint
  ↓
Operator-configured processing rules → Dynatrace

---

Architecture: OpenPipeline Custom Endpoints

The OpenPipeline Capability

OpenPipeline supports three categories of ingest endpoints:

Built-in Endpoints (Generic, No Config)

POST /platform/ingest/v1/events                    (generic events)
POST /api/v2/bizevents/ingest                     (business events)
POST /api/v2/logs/ingest                          (logs)
POST /api/v2/metrics/ingest                       (metrics)

Custom Configured Endpoints (Operator-Defined)

POST /platform/ingest/custom/events/<endpoint-name>
POST /platform/ingest/custom/events.sdlc/<endpoint-name>
POST /platform/ingest/custom/security.events/<endpoint-name>

Why Custom Endpoints for Agents

Instead of all agents emitting to generic endpoints, each agent type gets:

• Its own ingest endpoint (named, discoverable)
• Its own processing pipeline (custom rules per agent type)
• Built-in source tracking (events tagged by endpoint)
• Operator control (enrichment, routing, filtering per endpoint)

Configuration Model

Step 1: Operator creates custom ingest endpoint

dtctl create settings \
  --schema builtin:openpipeline.custom-events \
  --name agent-activity \
  --description "Ingest endpoint for agent activity events"

# Result: POST /platform/ingest/custom/events/agent-activity becomes available

Step 2: Operator configures processing pipeline for that endpoint

dtctl create settings \
  --schema builtin:openpipeline.bizevents.pipelines \
  --scope custom.events.agent-activity \
  -f agent-pipeline-config.yaml

# Pipeline defines:
#   - Enrichment rules (add computed fields)
#   - Routing rules (send to specific storage per agent type)
#   - Filtering rules (include/exclude records)
#   - Transformation rules (reshape events)

Step 3: Agent emits to its configured endpoint

dtctl send event -f activity.json --endpoint agent-activity

# Targets: POST /platform/ingest/custom/events/agent-activity
# OpenPipeline processes via configured rules
# Result: enriched, routed, transformed events in Dynatrace

Complete Data Flow

Operator Configuration Phase:
  dtctl create settings --schema openpipeline.custom-events --name agent-activity
  dtctl create settings --schema openpipeline.bizevents.pipelines --scope custom.events.agent-activity

Agent Runtime Phase:
  Agent Logic → dtctl send event --endpoint agent-activity → /platform/ingest/custom/events/agent-activity

OpenPipeline Processing:
  ├─ Enrichment (add fields, compute values)
  ├─ Routing (route to specific BizEvents stream per agent type)
  ├─ Filtering (include/exclude, sample if needed)
  ├─ Transformation (reshape, extract fields)
  ↓
Result in Dynatrace:
  Events queryable by agent.id, endpoint, computed fields, etc.

---

Custom Endpoints vs. Built-in Endpoints

Aspect	Built-in	Custom Configured
Endpoint	Generic shared	Dedicated per agent/use-case
Configuration	None (hard-coded)	Full OpenPipeline customization
Source tracking	No built-in separation	Automatic via endpoint name
Processing rules	Same for all events	Unique rules per endpoint
Operator control	Limited	Full (enrichment, routing, filtering)
Agent identity	Mixed in generic stream	Preserved, queryable
Scaling	All agents compete	Clean separation

---

dtctl send event — Implementation

Proposed Commands

# Operator lists available custom endpoints
dtctl get settings --schema builtin:openpipeline.custom-events

# Agent emits to specific endpoint
dtctl send event -f activity.json --endpoint agent-activity

# Or with inline attributes
dtctl send event --type agent.activity \
  --set agent.id=problem-resolver \
  --set operation=investigate \
  --endpoint agent-activity

# Batch emit
dtctl send events -f activities.jsonl --batch --endpoint agent-activity

# Dry-run validation (schema check without sending)
dtctl send event -f activity.json --endpoint agent-activity --dry-run

# Auto-discover endpoint from config (optional)
dtctl send event -f activity.json
# → Looks up default agent endpoint from dtctl profile

Token Scopes

For agents to emit to custom endpoints:

• openpipeline.events.custom — Required to send to /platform/ingest/custom/events/<name>

For operators to configure:

• openpipeline.settings.create — Configure custom endpoints
• openpipeline.settings.manage — Manage pipeline rules

---

Use Cases Enabled

• ✅ Agents emitting to operator-configured custom endpoints
• ✅ Different agent types with different processing pipelines
• ✅ Custom enrichment per agent type (add computed fields)
• ✅ Custom routing per agent type (route to different storage)
• ✅ Full audit trail (events tracked by endpoint + agent.id)
• ✅ Closed-loop workflows (emit → query → decide → act, all via dtctl)
• ✅ Fully autonomous agents (no SDK, pure CLI, fully observable)

---

Real-World Reference: Agentic Enterprise Operating Model

For detailed context: https://github.com/wlfghdr/agentic-enterprise

The agentic enterprise model uses dtctl for the complete observability lifecycle:

1. Operator configuration: dtctl create settings (custom endpoints + pipelines)
2. Agent querying: dtctl query (grounding context for decisions)
3. Agent execution: dtctl exec workflow (self-remediation)
4. Agent emission: dtctl send event --endpoint <name> (complete the loop)

Result: Fully autonomous agents with first-class observability, all via unified dtctl CLI.

---

Acceptance Criteria

• Support custom endpoints: POST /platform/ingest/custom/events/<name>
• Support custom SDLC endpoints: POST /platform/ingest/custom/events.sdlc/<name>
• Support custom security endpoints: POST /platform/ingest/custom/security.events/<name>
• dtctl send event -f <file> --endpoint <name> sends to custom endpoint
• dtctl send event --type <type> --set <key=value> --endpoint <name> works
• dtctl send events --batch --endpoint <name> handles JSONL batch ingest
• dtctl send event --dry-run --endpoint <name> validates without sending
• Optional auto-discovery: dtctl get settings --schema openpipeline.custom-events
• Token scope documentation (openpipeline.events.custom)
• Help text explains: operator config → custom endpoint → pipeline processing
• Documentation includes operator + agent workflow examples
• Agents can use custom endpoints without SDK dependency

---

Priority

High — Enables fully autonomous agents with operator-controlled observability pipelines, completing the dtctl-native observability lifecycle for agentic systems.

[wlfghdr]

@WulfAI , is this proposal considering the right ingest path when openpipeline configs would open additional endpoints for agent ingest, to not use bizevents but something different?

[WulfAI]

Excellent point, @wlfghdr!

You're identifying a critical API design decision I hadn't fully considered:

Current Proposal: BizEvents Ingest Path

Your Proposal: OpenPipeline-Native Path

Why this matters:

If OpenPipeline configs expose dedicated ingest endpoints for agents (not just bizevents), we should use those directly. This would:

✅ Better alignment: Agent data flows directly through the pipeline agent configured, not as a detour through BizEvents
✅ Pipeline-native: Respects the pipeline as first-class ingest target, not secondary
✅ Schema flexibility: Different agent event types could have different pipelines
✅ Cleaner architecture: No BizEvents intermediary layer if not needed

API Coverage Needed

Before implementing , we should clarify:

1. Does OpenPipeline expose an ingest endpoint for agents?

   • If yes: Should agents target that directly?
   • If no: Is this something Dynatrace should expose?

2. Multi-pipeline support:

   • Can an agent target a specific pipeline?
   • Or does auto-route based on event type + OpenPipeline config?

3. Backward compatibility:

   • Should we support both BizEvents path (for existing setup) AND pipeline-native path?

Suggested API Coverage for Issue Update

Ingest Path Options:

Option	Endpoint	Use Case	Pros	Cons
BizEvents		Structured business events	Works today, well-established	Indirect routing through pipeline
Pipeline-native	(hypothetical)	Agent-originated events	Direct pipeline targeting, cleaner	Needs to be exposed by Dynatrace
OTLP	(Dynatrace OTLP endpoint)	Traces/metrics/logs	Full OpenTelemetry standard	Overkill for simple events

Recommendation: Clarify OpenPipeline ingest capabilities with Dynatrace team, then propose the right path to dtctl maintainers. If pipeline-native endpoints exist or are feasible, that's the superior architecture.

Should I update the issue text to explore both paths before proposing implementation?

[wlfghdr]

@WulfAI , check OpenPipeline capabilities and options for configuring the processing of any events ingest. Should be doable without technical blockers I think.

[WulfAI]

Perfect! You're right—OpenPipeline is designed exactly for this:

From the docs: OpenPipeline is "the Dynatrace data handling solution to seamlessly ingest and process data from different sources, at any scale, and in any format."

This clearly includes agents as a data source.

Key Insight: OpenPipeline-First Design

Instead of:

Agent → /api/v2/bizevents/ingest → OpenPipeline processing

We should do:

Agent → OpenPipeline ingest endpoint → Configured pipeline processing

This makes agents true OpenPipeline citizens:

1. Configuration: Agent operator configures a pipeline via dtctl create settings to receive agent events
2. Processing: Pipeline enriches, filters, routes agent telemetry
3. Ingest: Agent emits directly to OpenPipeline ingest endpoint (not BizEvents intermediary)
4. Result: Agent data flows through first-class pipeline, not generic event sink

API Coverage Update Needed

dtctl send event should target OpenPipeline ingest endpoint, not BizEvents:

# Agent emits to configured OpenPipeline
dtctl send event -f agent-activity.json

# Under the hood:
# → Targets OpenPipeline ingest endpoint
# → Event flows through configured pipeline
# → Enrichment, routing, processing happens
# → Results in Dynatrace

Implementation Questions

1. Which OpenPipeline endpoint? Does OpenPipeline expose an ingest endpoint (like /api/v2/openpipeline/ingest) that accepts agent events?
2. Pipeline routing: Does the agent specify target pipeline, or does OpenPipeline auto-route based on event schema/type?
3. Schema: Can agents emit arbitrary event structures, or specific schemas?

Updated Recommendation

Option: OpenPipeline-Native (Preferred)

• Agent emits to OpenPipeline ingest endpoint
• Event flows through configured pipeline
• First-class OpenPipeline citizen (not BizEvents workaround)
• Scales to "any format" per OpenPipeline design

This is the clean, scalable architecture for agent observability in Dynatrace.

Should we pivot the issue to focus on OpenPipeline ingest + propose the necessary API endpoints/endpoints if they don't exist?

[WulfAI]

🎯 EVEN BETTER! OpenPipeline has custom endpoints!

From the API Ingestion Reference: OpenPipeline supports custom ingest endpoints for different event types.

Key Discovery: Custom Endpoints

Built-in Generic Events:
  POST /platform/ingest/v1/events

Custom Generic Events:  ← AGENTS CAN USE THIS
  POST /platform/ingest/custom/events/<custom-endpoint-name>
  
Custom SDLC Events:
  POST /platform/ingest/custom/events.sdlc/<custom-endpoint-name>

Custom Security Events:
  POST /platform/ingest/custom/security.events/<custom-endpoint-name>

Implication for Agents

Agents don't just emit to a generic endpoint. They can have their own custom-configured ingest point:

# Agent operator configures custom endpoint for agent telemetry
dtctl create settings --schema openpipeline.custom-events \
  --name agent-activity-endpoint

# Agent emits directly to its custom endpoint
dtctl send event -f activity.json --endpoint agent-activity-endpoint
# → POST /platform/ingest/custom/events/agent-activity-endpoint

Architecture Implication

This is even cleaner:

Agent emits to custom-configured endpoint
  ↓
OpenPipeline processes via configured custom endpoint
  ↓
Custom processing rules (enrichment, routing, transformation)
  ↓
Dynatrace storage

Result: Agents are completely first-class in OpenPipeline. Each agent deployment can have its own custom ingest point + custom processing rules.

Updated Recommendation

API for dtctl send event:

# Auto-route based on configuration
dtctl send event -f activity.json

# Or explicit endpoint
dtctl send event -f activity.json --endpoint agent-activity

# Inline attributes
dtctl send event --type agent.activity \
  --set agent.id=resolver \
  --endpoint agent-activity

Token Scopes

Based on API reference, custom endpoints use token scope:

• openpipeline.events.custom (for custom generic events)
• openpipeline.sdlc.custom (for custom SDLC)
• openpipeline.events_security.custom (for custom security)

Revised Acceptance Criteria

• Support built-in endpoint: /platform/ingest/v1/events
• Support custom endpoints: /platform/ingest/custom/events/<endpoint-name>
• dtctl send event can target built-in or custom endpoints
• Optional --endpoint flag for custom endpoint selection
• Auto-discovery of configured custom endpoints (optional)

This makes agents true first-class OpenPipeline citizens with fully customizable processing pipelines.

[WulfAI]

🎯 DEEP DIVE: OpenPipeline Custom Endpoints via Configuration

Based on the OpenPipeline API Ingestion Reference, I've identified the full capability:

OpenPipeline Endpoint Hierarchy

1. Built-in Generic Endpoints (No Config Needed)

POST /platform/ingest/v1/events
POST /api/v2/bizevents/ingest
POST /api/v2/logs/ingest
POST /api/v2/metrics/ingest

2. Custom Configured Endpoints (OpenPipeline Config)

POST /platform/ingest/custom/events/<custom-endpoint-name>
POST /platform/ingest/custom/events.sdlc/<custom-endpoint-name>
POST /platform/ingest/custom/security.events/<custom-endpoint-name>

The Configuration Model

Operators configure custom ingest endpoints via OpenPipeline settings:

# Create custom endpoint named 'agent-activity'
# (This is a named ingestion point, not a full pipeline yet)
dtctl create settings   --schema builtin:openpipeline.custom-events   --name agent-activity

# This makes available:
#   POST /platform/ingest/custom/events/agent-activity

Then operators configure the processing pipeline for that endpoint:

# Configure processing rules for agent-activity endpoint
dtctl create settings   --schema builtin:openpipeline.bizevents.pipelines   --scope custom.events.agent-activity   -f agent-pipeline.yaml

# Pipeline rules define:
#   - Enrichment (add computed fields)
#   - Routing (send to specific storage)
#   - Filtering (include/exclude records)
#   - Transformation (reshape events)

Result: Agent-Specific Data Flow

Agent Application
  ↓ dtctl send event --endpoint agent-activity
  ↓
Dynatrace Ingest
  ↓ POST /platform/ingest/custom/events/agent-activity
  ↓
OpenPipeline (custom.events.agent-activity scope)
  ├─ Enrichment rules
  ├─ Filtering rules
  ├─ Routing rules
  ├─ Transformation rules
  ↓
Dynatrace Storage

Key Advantages Over Generic Endpoints

Aspect	Generic (/api/v2/bizevents)	Custom (/platform/ingest/custom)
Endpoint type	Shared across all event types	Dedicated per agent/use-case
Configuration scope	No customization	Full pipeline customization per endpoint
Agent identity	Lost in generic stream	Preserved via endpoint name
Processing rules	Same for all events	Unique rules per agent type
Audit trail	Hard to filter by source	Built-in via endpoint routing
Scale	All agents mixed	Clean separation of concerns

dtctl send event Implementation

# Operator configures custom endpoint
dtctl create settings --schema builtin:openpipeline.custom-events --name agent-activity

# Agent emits to custom endpoint  
dtctl send event -f activity.json --endpoint agent-activity

# Under the hood:
# 1. dtctl resolves --endpoint agent-activity
# 2. Constructs: POST /platform/ingest/custom/events/agent-activity
# 3. Sends event to that endpoint
# 4. OpenPipeline applies configured processing rules

Configuration Example

Step 1: Define custom endpoint

# agent-endpoint-config.yaml
name: agent-activity
type: custom.events
description: "Ingest endpoint for agent activity telemetry"

dtctl create settings \
  --schema builtin:openpipeline.custom-events \
  -f agent-endpoint-config.yaml

Step 2: Configure processing pipeline

# agent-pipeline.yaml
source:
  scope: custom.events.agent-activity

enrichment:
  - add_computed_fields:
      - name: env
        value: production
      - name: region
        value: eu-west-1

routing:
  - filter: agent.type == "problem-resolver"
    target: bizevents.agent-resolver
  - filter: agent.type == "devops-agent"
    target: bizevents.agent-devops
  - default: bizevents.agent-default

dtctl create settings \
  --schema builtin:openpipeline.bizevents.pipelines \
  -f agent-pipeline.yaml

Step 3: Agent emits

dtctl send event \
  --type agent.activity \
  --set agent.id=problem-resolver \
  --set operation=investigate \
  --endpoint agent-activity

Token Scopes

For custom endpoint ingest, agents need:

• openpipeline.events.custom — Send to custom generic events endpoints

Operators need (for configuration):

• openpipeline.settings.create — Configure custom endpoints
• openpipeline.settings.manage — Manage pipeline rules

Why This Is Optimal for Agents

1. Per-agent-type endpoints — Each agent deployment gets its own ingest point
2. Operator-configured processing — Rules defined once, applied to all agent events
3. Clean audit trail — Events labeled by endpoint (built-in source identification)
4. Full customization — Operators can enrich, filter, route per agent type
5. First-class citizen — Agents are data sources with dedicated pipelines, not generic producers

Updated dtctl send event Acceptance Criteria

• Support custom endpoints: POST /platform/ingest/custom/events/<name>
• dtctl send event --endpoint <name> resolves configured endpoint
• Auto-discovery: dtctl send event can list available endpoints
• Batch support: dtctl send events --batch --endpoint <name>
• Dry-run: dtctl send event --dry-run --endpoint <name>
• Token scope documentation (openpipeline.events.custom)
• Help text explains the flow: config → endpoint → pipeline → storage
• Documentation includes operator + agent workflow examples

This makes the full agentic-enterprise observability lifecycle complete: operators configure the pipeline, agents emit to it, OpenPipeline processes, and everything is auditable and observable.

[wlfghdr]

Additionally, I think the SDLC event ingest via CLI would make it super convenient for any CI/CD pipeline / task / runner to also do a one-liner ingest of the according SDLC events.

[wlfghdr]

Cross-Repository Link: Observability Contract

The normative observability contract that defines the event schema for agent telemetry is tracked in wlfghdr/agentic-enterprise#36 — "Unified Observability Contract (OTel-first): spans + derived events + correlation."

How they connect

agentic-enterprise#36     → Defines WHAT agents emit (required spans, correlation attributes, event types)
This issue (dtctl#44)     → Defines HOW agents emit (CLI: dtctl send event --endpoint <name>)

Key alignment points:

• The agent.activity event attributes referenced in this issue's examples align with the required attribute set from agentic-enterprise#36 (agent identity, OTel GenAI operation, grounding evidence, verification, mission correlation)
• The custom endpoint architecture (--endpoint flag → /platform/ingest/custom/events/<name>) enables operator-controlled processing pipelines per agent type, matching the governance model in the agentic-enterprise framework
• --dry-run validation could check conformance against the contract's required attribute set

[discostu105]

Generally "ingest" capabilities would make sense.

However, needs to go towards the right APIs (live domain) and those are right now not yet Platform Token enabled. This will come soon. As soon as it's available, we can add it.

Also, this needs somewhat careful design (telemetry types spans, logs, metrics, events, ...).

[WulfAI]

Totally agree on “right APIs + auth first”. A couple design thoughts that might keep dtctl’s UX stable while the Platform Token enablement catches up:

1. Make the CLI schema-driven, not endpoint-driven

   • dtctl send event --type <sdlc|custom|...> (or --schema) and then map that to the correct ingestion API.
   • That way the user-facing contract stays stable even if the backend endpoint changes from “early/custom” → “official/platform-token”.

2. Prefer SDLC event vocab for Git correlation (when possible)

   • If this lands on /platform/ingest/v1/events.sdlc, using the SDLC semantic dictionary (vcs.*, pipeline.*, task.*) gives immediate compatibility with Dynatrace’s enrichment/correlation.
   • We’ve been drafting an OTel-first “agent activity” contract that intentionally reuses SDLC vcs.* fields for PR/MR context (see wlfghdr/agentic-enterprise Change azure_connection and azure_monitoring_config to cloud_connection and cloud_monitoring_config #36 for the attribute list + mapping notes).

3. Auth / environment gating as a first-class UX

   • If the “live domain” endpoint isn’t Platform Token enabled yet: explicit preflight (dtctl send event --dry-run / dtctl capabilities) that tells users why it won’t work (token type / tenant / API availability).
   • Keeps the feature shippable without implying it’s universally available.

4. Telemetry types: keep “events” narrow; bridge spans/logs separately

   • For agent observability lifecycle, “events” are a good first milestone (cheap + correlatable). Spans/logs/metrics can be separate subcommands to avoid an over-general ‘ingest everything’ interface.

Happy to help sanity-check a minimal v0 that just does SDLC events + dry-run validation.