-
Notifications
You must be signed in to change notification settings - Fork 1.5k
DynamicD First PR #22328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
DynamicD First PR #22328
Changes from all commits
e1d9ab8
bbf7c9e
d061e8d
8b69120
1de397e
e23e897
f1701e4
f34398e
7e2ab0e
e48aa7a
5134f80
d64ad07
68e648b
a73a7eb
1ce3ac6
a03098f
0388427
3e54cc7
2f65a35
99bc92e
f4119b4
d3241d2
a3a0a25
826074f
6359a3f
9fbc2df
112a848
5bb8c6c
938424c
5bc14c6
1ac2a9a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Add DynamicD tool for generating realistic fake telemetry data using AI |
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. request: this will be completely invisible for anyone using this command. If you want full visibility add it to the docs here. If you think this is more an internal tool, which I guess it is because it is part of |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # DynamicD | ||
|
|
||
| Generate realistic fake telemetry data for Datadog integrations using AI. | ||
|
|
||
| DynamicD uses Claude to analyze an integration's metrics, service checks, and dashboards, then generates a Python script that simulates realistic, scenario-aware data. | ||
|
|
||
| ## Setup | ||
|
|
||
| ### Required: Anthropic API Key (for script generation) | ||
|
|
||
| ```bash | ||
| # Option 1: Environment variable | ||
| export ANTHROPIC_API_KEY="your-anthropic-key" | ||
|
|
||
| # Option 2: ddev config | ||
| ddev config set dynamicd.llm_api_key "your-anthropic-key" | ||
| ``` | ||
|
|
||
| ### Required: Datadog API Key (for sending data) | ||
|
|
||
| ```bash | ||
| ddev config set orgs.<your-org>.api_key "your-dd-api-key" | ||
| ``` | ||
|
|
||
| ### Optional: Datadog App Key (for org name validation) | ||
|
|
||
| ```bash | ||
| ddev config set orgs.<your-org>.app_key "your-dd-app-key" | ||
| ``` | ||
|
|
||
nubtron marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ### Optional: Datadog Site (for non-US users) | ||
|
|
||
| ```bash | ||
| # Default is datadoghq.com (US1) | ||
| ddev config set orgs.<your-org>.site datadoghq.eu # EU | ||
| ddev config set orgs.<your-org>.site us3.datadoghq.com # US3 | ||
| ddev config set orgs.<your-org>.site us5.datadoghq.com # US5 | ||
| ddev config set orgs.<your-org>.site ap1.datadoghq.com # AP1 | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ```bash | ||
| # Interactive scenario selection | ||
| ddev meta scripts dynamicd <integration> | ||
|
|
||
| # Specific scenario | ||
| ddev meta scripts dynamicd celery --scenario incident | ||
|
|
||
| # Save script for later use | ||
| ddev meta scripts dynamicd redis --scenario healthy --save | ||
|
|
||
| # Preview without executing | ||
| ddev meta scripts dynamicd postgres --show-only | ||
|
|
||
| # Custom duration (default: run forever) | ||
| ddev meta scripts dynamicd kafka --duration 300 | ||
| ``` | ||
|
|
||
| ## Scenarios | ||
|
|
||
| | Scenario | Description | | ||
| |----------|-------------| | ||
| | `healthy` | Normal operation with baseline metrics. | | ||
| | `degraded` | Performance issues, increased latency, some errors. | | ||
| | `incident` | Active incident with failures and alerts. | | ||
| | `recovery` | System recovering, metrics returning to normal. | | ||
| | `peak_load` | High traffic, elevated but healthy metrics. | | ||
| | `maintenance` | Scheduled maintenance, reduced capacity. | | ||
|
|
||
| ## What Gets Generated | ||
|
|
||
| DynamicD creates scripts that send: | ||
|
|
||
| - **Metrics**: All dashboard metrics with realistic, correlated values | ||
| - **Logs**: Scenario-appropriate log messages (INFO/WARN/ERROR) | ||
| - **Service Checks**: Health status matching the scenario (if integration defines them) | ||
| - **Events**: Significant state changes (incidents, recoveries) | ||
|
|
||
| All telemetry is tagged with `env:dynamicd` for easy filtering in Datadog. | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Context Building**: Reads integration metadata, metrics, dashboards, and service checks | ||
| 2. **Stage 1 (Analysis)**: LLM analyzes the service type and operational patterns | ||
| 3. **Stage 2 (Generation)**: LLM generates a self-contained Python simulator | ||
| 4. **Execution**: Runs the script, auto-fixes errors if needed (up to 3 retries) | ||
|
|
||
| ## Options | ||
|
|
||
| | Option | Description | | ||
| |--------|-------------| | ||
| | `--scenario`, `-s` | Scenario to simulate (healthy, degraded, incident, etc.) | | ||
| | `--duration, -d` | Duration in seconds (0 = run forever, default) | | ||
| | `--rate, -r` | Target metrics per batch (default: 100) | | ||
| | `--save` | Save script to integration's fake_data/ directory | | ||
| | `--show-only` | Display generated script without executing | | ||
mahipdeora25 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| | `--timeout` | Execution timeout in seconds (for testing) | | ||
| | `--all-metrics` | Generate ALL metrics, not just dashboard metrics | | ||
| | `--sandbox/--no-sandbox` | Run in Docker container for isolation (default: enabled) | | ||
|
|
||
| **Note**: Sandbox mode is enabled by default and requires Docker Desktop to be running. Start Docker with `open -a Docker` on macOS. Use `--no-sandbox` to run directly on host. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # (C) Datadog, Inc. 2024-present | ||
| # All rights reserved | ||
| # Licensed under a 3-clause BSD style license (see LICENSE) | ||
| """DynamicD utilities - Smart fake data generator for Datadog integrations.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion: can we have this dependency on a separate feature, i.e.
optional-dependenciesunderaior something like that? Not sure how bit it is but most of the users might not actually need this and if it is a client it might be a lot of boiler plate we can avoid.You can then load the command only if the library exist. That way if
ddevis installed without theaifeature, the command would not even be visible.I am marking this as a suggestion though, I am not sure the impact of this but maybe worth considering.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a follow up on this comment (as requested offline). The issue I am trying to address with the comment is not startup time when importing it, is installation overhead for people that won't use this command (one of the reasons why we mentioned we could have this being a ddev plugin).
We install
ddevevery time we open a PR in the repo and that pulls all its dependencies. For a dependency that exist to serve a single command, having it being a global dependency when we are never going to use that command in CI seems a bit too much. If this dependency was an option dependency declared asIf you want ddev to have that command you just install it as
pipx install ddev[ai]and it will pull that dependency as well. We can avoid installing it in CI. You can even deefine the command to only exist if the dependency is there, since we won't be able to run without it. Instead of failing without this dependency the commadn would not even show.When loading the command one could do
The importlib import is free, it is part of the bootstrap and the
find_specis a lookup on table so this has negligible impact on the end user but runningddev meta scripts --helpwon't even show the monitor command if the dependency is not installed.Again, this is not a hard requirement because I checked and the anthropic dependency is less than half a megabyte
But worth considering.