You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/explorer/api/annotations.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ Annotations provide additional context, facilitating collaboration and agent err
16
16
17
17
## Annotation Format
18
18
19
-
You can add annotations to traces at upload time. For this, both during [file upload](./Uploading_Traces/file_uploads.md) and via the [Push API](./Uploading_Traces/push_api.md), you can include an `annotations` field in the trace data. This field should be an array of objects, each representing an annotation. Each annotation object should have the following fields:
19
+
You can add annotations to traces at upload time. For this, both during [file upload](../uploading-traces/file-uploads) and via the [Push API](../uploading-traces/push-api), you can include an `annotations` field in the trace data. This field should be an array of objects, each representing an annotation. Each annotation object should have the following fields:
Copy file name to clipboardExpand all lines: docs/explorer/api/uploading-traces/file-uploads.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,7 @@ The explorer supports two types of trace formats: _raw event lists_ and _annotat
20
20
21
21
#### Raw Event Lists
22
22
23
-
Raw event lists are `jsonl` files where each line is a JSON array of events. Each event is a dictionary with at least a `role` and `content` field according to the trace format described [in this chapter](../2_traces.md).
23
+
Raw event lists are `jsonl` files where each line is a JSON array of events. Each event is a dictionary with at least a `role` and `content` field according to the trace format described [in this chapter](../../trace-format).
Copy file name to clipboardExpand all lines: docs/explorer/api/uploading-traces/push-api.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,7 +24,7 @@ The `PushTracesRequest` class holds the request data for a trace upload request.
24
24
25
25
This represents the traces in a dataset. Each `List[Dict]` is a single trace within the dataset. Each `dict` is a single message within a trace - these can represet a user prompt, a tool call, a tool output, etc.
26
26
27
-
Must be in the [required trace format](../2_traces.md). Must not be empty.
27
+
Must be in the [required trace format](../../trace-format). Must not be empty.
Copy file name to clipboardExpand all lines: docs/explorer/benchmarks.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@ Once you have prepared and ensured that your agent traces are in a compatible fo
38
38
39
39
### Step 3: Associate Your Agent Dataset with a Benchmark and Score
40
40
41
-
[Instructions on updating the datasets's metadata](./Explorer_API/Dataset_Metadata/update_dataset_metadata_api.md) to include the `benchmark`, `name` and `accuracy` fields, that will associate your agent dataset with a specific benchmark and score.
41
+
[Instructions on updating the datasets's metadata](./api/dataset-metadata/update.md) to include the `benchmark`, `name` and `accuracy` fields, that will associate your agent dataset with a specific benchmark and score.
42
42
43
43
For instance, to associate your `gpt-4o` agent dataset with the `webarena` benchmark and an accuracy score of `0.5`, you can update the dataset's metadata as follows:
Next, install the Invariant SDK in your Python environment, by running the following command. See [Installation](Explorer_API/installation.md) for alternative methods using different package managers.
34
+
Next, install the Invariant SDK in your Python environment, by running the following command. See [Installation](api/sdk-installation) for alternative methods using different package managers.
This will upload your traces to the Invariant Explorer under a new dataset named `my-first-dataset`. If the corresponding dataset already exists, the traces will be appended to it.
80
80
81
-
To learn more about the expected trace format see the chapter on the [trace format requirements](Explorer_API/2_traces.md).
81
+
To learn more about the expected trace format see the chapter on the [trace format requirements](api/trace-format.md).
Copy file name to clipboardExpand all lines: docs/explorer/self-hosted.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,15 +40,15 @@ This will pull and launch the required Explorer Docker containers on your machin
40
40
41
41
## Using the Self-Hosted Explorer
42
42
43
-
The self-hosted version of Explorer is configured to run on `http://localhost`. You can access it in your browser at this address. The local instance provides the same API as the managed cloud instance, so you can use the [Invariant SDK](./Explorer_API/1_client_setup.md) to connect to it.
43
+
The self-hosted version of Explorer is configured to run on `http://localhost`. You can access it in your browser at this address. The local instance provides the same API as the managed cloud instance, so you can use the [Invariant SDK](./api/client-setup) to connect to it.
44
44
45
45
### Storage
46
46
47
47
The self-hosted version of Explorer will create a `data/` directory in the current working directory to store traces and other data.
48
48
49
49
### Usage and Access
50
50
51
-
You can access the self-hosted version of Explorer at `http://localhost`. To use it with the [Invariant SDK](./Explorer_API/1_client_setup.md) you can set the `INVARIANT_API_ENDPOINT` environment variable to `http://localhost/`. For security reasons, you'll still need to create and provide an API key to access the self-hosted version of Explorer.
51
+
You can access the self-hosted version of Explorer at `http://localhost`. To use it with the [Invariant SDK](./api/client-setup) you can set the `INVARIANT_API_ENDPOINT` environment variable to `http://localhost/`. For security reasons, you'll still need to create and provide an API key to access the self-hosted version of Explorer.
52
52
53
53
### Updates
54
54
`invariant explorer` will automatically check for updates and pull the latest stable version of the Explorer Docker images. If you want to try the latest development version, you can use the `--version=main` flag. Note however, that database migrations may be required when switching between versions, which may not always be backwards compatible.
The test result provides information about which assertion failed but also [localizes the assertion failure precisely](./Writing_Tests/tests.md) in the provided list of agent messages.
85
+
The test result provides information about which assertion failed but also [localizes the assertion failure precisely](writing/tests) in the provided list of agent messages.
86
86
87
87
**Visual Test Viewer (Explorer):**
88
88
89
-
As an alternative to the command line, you can also [visualize test results](./Running_Tests/Visual_Debugger.md) on the [Invariant Explorer](https://explorer.invariantlabs.ai/):
89
+
As an alternative to the command line, you can also [visualize test results](running/visual-debugging) on the [Invariant Explorer](https://explorer.invariantlabs.ai/):
90
90
91
91
```py
92
92
$ invariant test --push
@@ -98,11 +98,11 @@ Like the terminal output, the Explorer highlights the relevant ranges, but does
98
98
99
99
## Features
100
100
101
-
* Comprehensive [`Trace` API](Writing_Tests/1_Traces.ipynb) for easily navigating and checking agent traces.
102
-
*[Assertions library](Writing_Tests/2_Assertions.md) to check agent behavior, including fuzzy checkers such as _Levenshtein distance_, _semantic similarity_ and _LLM-as-a-judge_ pipelines.
103
-
* Full [`pytest` compatibility](Running_Tests/PyTest_Compatibility.md) for easy integration with existing test and CI/CD pipelines.
104
-
* Parameterized tests for [testing multiple scenarios](Writing_Tests/parameterized-tests) with a single test function.
105
-
*[Visual test viewer](Writing_Tests/4_Visual_Test_Viewer.md) for exploring large traces and debugging test failures.
101
+
* Comprehensive [`Trace` API](writing/traces) for easily navigating and checking agent traces.
102
+
*[Assertions library](writing/matchers) to check agent behavior, including fuzzy checkers such as _Levenshtein distance_, _semantic similarity_ and _LLM-as-a-judge_ pipelines.
103
+
* Full [`pytest` compatibility](running/pytest-compatibility) for easy integration with existing test and CI/CD pipelines.
104
+
* Parameterized tests for [testing multiple scenarios](writing/parameterized-tests) with a single test function.
105
+
*[Visual test viewer](running/visual-debugging) for exploring large traces and debugging test failures.
Copy file name to clipboardExpand all lines: docs/testing/running/pytest-compatibility.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,7 +24,7 @@ results. This name will be used to derive a fresh dataset name on each run
24
24
--push (optional)
25
25
```
26
26
27
-
Flag to indicate whether to [push data to Explorer](./Visual_Debugger.md). If set to true,
27
+
Flag to indicate whether to [push data to Explorer](./visual-debugging.md). If set to true,
28
28
the `INVARIANT_API_KEY` environment variable must be set. Visit the [Explorer Documentation](https://explorer.invariantlabs.ai/docs/explorer) to learn how to obtain your own API key.
0 commit comments