This guide provides a step-by-step process to enable observability in a NeMo Agent Toolkit workflow using Weights and Biases (W&B) Weave for tracing using just a few lines of code in your workflow configuration file.
An account on Weights & Biases is required to use Weave.
You can create an account on Weights & Biases by clicking on the "Sign Up" button in the top right corner of the website.
Under the "Account" section, you can find your API key. Click on the "Show" button to reveal the API key. Take note of this API key as you will need it to run the workflow.
export WANDB_API_KEY=<your_api_key>To install the Weave plugin, run one of the following commands, depending on whether you installed the NeMo Agent Toolkit from source or from a package.
::::{tab-set} :sync-group: install-tool
:::{tab-item} source :selected: :sync: source
uv pip install -e ".[weave]":::
:::{tab-item} package :sync: package
uv pip install "nvidia-nat[weave]":::
::::
Pick an example from the list of available workflows. In this guide, we will be using the simple_calculator example.
uv pip install -e examples/observability/simple_calculator_observabilityUpdate your workflow configuration file to include the weave telemetry settings. For example, examples/observability/simple_calculator_observability/configs/config-weave.yml has the following weave settings:
general:
telemetry:
tracing:
weave:
_type: weave
project: "nat-demo"This setup enables logging trace data to W&B weave. The weave integration only requires the project parameter to be set.
| Parameter | Description | Example |
|---|---|---|
project |
The name of your W&B Weave project | "nat-demo" |
entity (deprecated) |
Your W&B username or team name | "your-wandb-username-or-teamname" |
Install simple_calculator example using the instructions in the examples/observability/simple_calculator_observability/README.md guide.
Run the workflow using config-weave.yml configuration file:
nat run --config_file examples/observability/simple_calculator_observability/configs/config-weave.yml --input "Is the product of 2 * 4 greater than the current hour of the day?"If it is your first time running the workflow, you will be prompted to login to W&B Weave.
As the workflow runs, you will find a Weave URL (starting with a 🍩 emoji). Click on the URL to access your logged trace timeline.
Note how the integration captures not only the nat intermediate steps but also the underlying framework. This is because Weave has integrations with many of your favorite frameworks.
When tracing LLM workflows, you may be processing sensitive information like personal identifiers, credit card numbers, or API keys. NeMo Agent Toolkit Weave integration supports automatic redaction of Personally Identifiable Information (PII) and sensitive keys from your traces.
To enable PII redaction, you need presidio-analyzer and presidio-anonymizer installed. Installing the weave plugin will install these packages for you.
Update your workflow configuration to enable PII redaction:
general:
telemetry:
tracing:
weave:
_type: weave
project: "nat-demo"
redact_pii: true # Enable PII redaction
redact_pii_fields: # Optional: specify which entity types to redact
- EMAIL_ADDRESS
- PHONE_NUMBER
- CREDIT_CARD
- US_SSN
- PERSON
redact_keys: # Optional: specify additional keys to redact
- custom_secret
- api_key
- auth_tokenThe Weave integration supports the following redaction options:
| Parameter | Description | Required |
|---|---|---|
redact_pii |
Enable PII redaction (true/false) | No (default: false) |
redact_pii_fields |
List of PII entity types to redact | No (default: all supported entities) |
redact_keys |
List of additional keys to redact beyond the defaults | No |
When redact_pii is enabled, common PII entities like email addresses, phone numbers, credit cards, and more are automatically redacted from your traces before they are sent to Weave. The redact_pii_fields parameter allows you to customize which entity types to redact.
See the Microsoft Presidio documentation for a full list of supported entity types.
Additionally, the redact_keys parameter allows you to specify custom keys that should be redacted beyond the default sensitive keys (api_key, auth_headers, authorization).
When using Weave telemetry with the FastAPI front end, you can enable a /feedback endpoint that allows users to provide thumbs-up and thumbs-down feedback on agent responses. This feedback is linked to specific traces in your Weave project for analysis.
To enable the feedback endpoint, configure your workflow to use the WeaveFastAPIPluginWorker:
general:
front_end:
_type: fastapi
runner_class: nat.plugins.weave.fastapi_plugin_worker.WeaveFastAPIPluginWorker
telemetry:
tracing:
weave:
_type: weave
project: "nat-demo"The WeaveFastAPIPluginWorker registers the /feedback endpoint when Weave telemetry is configured. For more details on the feedback API, see the API Server Endpoints documentation.
To associate traces and feedback with a specific user, set one or more of the following fields on the NeMo Agent Toolkit context metadata from within your authentication callback: trace_user_name, trace_user_email, or trace_user_id. These trace_-prefixed fields are an explicit opt-in, so general identity fields set on the metadata for other purposes will not affect Weave attribution.
Any fields that are set will be written to the Weave call summary. When a user submits feedback through the /feedback endpoint, the first available value is used in priority order (trace_user_name → trace_user_email → trace_user_id), falling back to anonymous.
The following example shows how to set these fields from within an authentication callback:
from nat.runtime.context import Context
def my_auth_callback(request):
user_info = authenticate(request) # returns user data from your identity provider
context = Context.get()
context.metadata.trace_user_name = user_info.get("name")
context.metadata.trace_user_email = user_info.get("email")