Skip to content

Commit 316f220

Browse files
lbeurerkellnerlbeurerkellner
committed
Merge branch 'main' of github.com:invariantlabs-ai/docs
2 parents 6048d27 + f02668d commit 316f220

32 files changed

+333
-213
lines changed

docs/explorer/Explorer_API/3_annotations.md renamed to docs/explorer/api/annotations.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Annotations provide additional context, facilitating collaboration and agent err
1616

1717
## Annotation Format
1818

19-
You can add annotations to traces at upload time. For this, both during [file upload](./Uploading_Traces/file_uploads.md) and via the [Push API](./Uploading_Traces/push_api.md), you can include an `annotations` field in the trace data. This field should be an array of objects, each representing an annotation. Each annotation object should have the following fields:
19+
You can add annotations to traces at upload time. For this, both during [file upload](uploading-traces/file-uploads.md) and via the [Push API](uploading-traces/push-api.md), you can include an `annotations` field in the trace data. This field should be an array of objects, each representing an annotation. Each annotation object should have the following fields:
2020

2121
##### `content` <span class='type'>string</span> <span class='required'/>
2222

docs/explorer/Explorer_API/1_client_setup.md renamed to docs/explorer/api/client-setup.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ export INVARIANT_API_KEY=YourAPIKey
2121

2222
## Creating a Client
2323

24-
In your Python code, you can create an `AsyncClient` or a `Client` object. This object will use the environment variables you set up earlier to authenticate your uploads.
24+
In your Python code, you can create an `AsyncClient` (which exposes asynchronous methods) or a `Client` (which exposes synchronous methods) object. This object will use the environment variables you set up earlier to authenticate your uploads.
2525

2626
```python
2727
from invariant_sdk.async_client import AsyncClient

docs/explorer/Explorer_API/Dataset_Metadata/get_dataset_metadata_api.md renamed to docs/explorer/api/dataset-metadata/get.md

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -40,14 +40,4 @@ The response object from the Invariant API.
4040
client = AsyncClient()
4141

4242
dataset_metadata = await client.get_dataset_metadata(dataset_name="some_dataset_name")
43-
```
44-
45-
> Client Example
46-
```python
47-
from invariant_sdk.client import Client
48-
from invariant_sdk.types.push_traces import PushTracesRequest
49-
50-
client = Client()
51-
52-
dataset_metadata = client.get_dataset_metadata(dataset_name="some_dataset_name")
53-
```
43+
```

docs/explorer/Explorer_API/Dataset_Metadata/update_dataset_metadata_api.md renamed to docs/explorer/api/dataset-metadata/update.md

Lines changed: 1 addition & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -134,60 +134,6 @@ The response object from the Invariant API.
134134
response = await client.update_dataset_metadata(request)
135135
```
136136

137-
> Client Example
138-
```python
139-
from invariant_sdk.client import Client
140-
from invariant_sdk.types.update_dataset_metadata import UpdateDatasetMetadataRequest, MetadataUpdate
141-
142-
client = Client()
143-
144-
# Metadata state: {}
145-
146-
request_1 = UpdateDatasetMetadataRequest(
147-
dataset_name="some_name",
148-
metadata=MetadataUpdate(benchmark="some_benchmark")
149-
)
150-
response_1 = client.update_dataset_metadata(request_1)
151-
152-
# Metadata state: {"benchmark": "some_benchmark"}
153-
154-
request_2 = UpdateDatasetMetadataRequest(
155-
dataset_name="some_name",
156-
metadata=MetadataUpdate(accuracy=5, name="xyz")
157-
)
158-
159-
response_2 = client.update_dataset_metadata(request_2)
160-
161-
# Metadata state: {"benchmark": "some_benchmark", "accuracy": 5, "name": "xyz"}
162-
163-
request_3 = UpdateDatasetMetadataRequest(
164-
dataset_name="some_name",
165-
replace_all=True
166-
metadata=MetadataUpdate(benchmark="new_benchmark")
167-
)
168-
169-
response_3 = client.update_dataset_metadata(request_3)
170-
171-
# Metadata state: {"benchmark": "new_benchmark"}
172-
173-
```
174-
175-
> Client Example to clear all previously set metadata
176-
```python
177-
from invariant_sdk.client import Client
178-
from invariant_sdk.types.update_dataset_metadata import UpdateDatasetMetadataRequest, MetadataUpdate
179-
180-
client = Client()
181-
182-
request = UpdateDatasetMetadataRequest(
183-
dataset_name="some_name",
184-
replace_all=True,
185-
metadata=MetadataUpdate()
186-
)
187-
188-
response = client.update_dataset_metadata(request)
189-
```
190-
191137
### `create_request_and_update_dataset_metadata`
192138

193139
The `create_request_and_update_dataset_metadata` method is used to update the metadata for a dataset.
@@ -254,47 +200,4 @@ The response object from the Invariant API.
254200
dataset_name="some_name"
255201
replace_all=True,
256202
)
257-
```
258-
259-
> Client Example
260-
```python
261-
from invariant_sdk.client import Client
262-
263-
client = Client()
264-
265-
# Metadata state: {}
266-
267-
response_1 = client.create_request_and_update_dataset_metadata(
268-
dataset_name="some_name",
269-
metadata={"benchmark": "some_benchmark"}
270-
)
271-
272-
# Metadata state: {"benchmark": "some_benchmark"}
273-
274-
response_2 = client.create_request_and_update_dataset_metadata(
275-
dataset_name="some_name",
276-
metadata={"accuracy": 5, "name": "xyz"}
277-
)
278-
279-
# Metadata state: {"benchmark": "some_benchmark", "accuracy": 5, "name": "xyz"}
280-
281-
response_3 = client.create_request_and_update_dataset_metadata(
282-
dataset_name="some_name",
283-
replace_all=True,
284-
metadata={"benchmark": "new_benchmark"}
285-
)
286-
287-
# Metadata state: {"benchmark": "new_benchmark"}
288-
```
289-
290-
> Client Example to clear all previously set metadata
291-
```python
292-
from invariant_sdk.client import Client
293-
294-
client = Client()
295-
296-
response = client.create_request_and_update_dataset_metadata(
297-
dataset_name="some_name"
298-
replace_all=True,
299-
)
300-
```
203+
```

docs/explorer/Explorer_API/installation.md renamed to docs/explorer/api/sdk-installation.md

File renamed without changes.

docs/explorer/Explorer_API/2_traces.md renamed to docs/explorer/api/trace-format.md

File renamed without changes.

‎docs/explorer/api/uploading-traces/append-messages.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
hide:
3+
- toc
4+
---
5+
6+
# Append Messages API
7+
8+
<div class='subtitle'>Append new messages to an existing trace</div>
9+
10+
The Append Messages API allows you to append new messages to an existing trace in a programmatic way. Within a trace, all the messages are stored as a list of messages.
11+
12+
The SDK includes a timestamp field by default (set to the current time) when calling the API. This timestamp field is used to perform a sorted insert of the new messages into the list of existing messages in the trace.
13+
14+
It is possible some of the existing messages in the trace don't have any timestamp - in that case the trace creation timestamp is used for the comparison.
15+
16+
This API works on traces within datasets and on standalone traces (snippets) without any datasets.
17+
18+
## Data Types
19+
20+
### `AppendMessagesRequest`
21+
22+
The `AppendMessagesRequest` class holds the request data for a messages append request.
23+
24+
##### `messages` <span class='type'>List[Dict]</span> <span class='required'/>
25+
26+
This represents the new messages to append to the trace. Each `dict` is a single message within a trace - these can represet a user prompt, a tool call, a tool output, etc.
27+
28+
Must be in the [required trace format](../trace-format.md). Must not be empty.
29+
30+
##### `trace_id` <span class='type'>str</span> <span class='required'/>
31+
32+
The id of the trace to which you want to append messages. This has to exist before the Append Messages API can be called and the caller must be the owner of the trace or the dataset containing the trace.
33+
34+
35+
## Pushing Traces
36+
37+
There are two SDK methods to push traces: `append_messages` and `create_request_and_append_messages`. The former accepts the `AppendMessagesRequest` type as an argument and the latter accepts Python-native types as arguments.
38+
39+
### `append_messages`
40+
The `append_messages` method is used to append messages to an existing trace using a pre-constructed request object.
41+
42+
##### `request` <span class='type'>AppendMessagesRequest</span> <span class='required'/>
43+
44+
The request object containing new messages and the trace id.
45+
46+
##### `request_kwargs` <span class='type'>Optional[Dict[str, Any]]</span> <span class='optional'/>
47+
48+
Additional keyword arguments to pass to the requests method. Default is `None`.
49+
50+
##### Return Value
51+
52+
##### <span class='type'>Dict</span>
53+
54+
The response object from the Invariant API.
55+
56+
> AsyncClient Example
57+
```python
58+
from invariant_sdk.async_client import AsyncClient
59+
from invariant_sdk.types.append_messages import PushTracesRequest
60+
61+
client = AsyncClient()
62+
63+
request = PushTracesRequest(
64+
messages=[
65+
{"role": "user", "content": "one"},
66+
{"role": "assistant", "content": "two \n three"},
67+
],
68+
trace_id="some_trace_id"
69+
)
70+
71+
response = await client.append_messages(request)
72+
```
73+
74+
### `create_request_and_append_messages`
75+
76+
The `create_request_and_append_messages` method is used to append messages to an existing trace. It creates a request object from the provided messages and trace id and pushes this data to the API.
77+
78+
##### `messages` <span class='type'>List[Dict]</span> <span class='required'/>
79+
80+
This represents the new messages to append to the trace. Each `dict` is a single message within a trace - these can represet a user prompt, a tool call, a tool output, etc.
81+
82+
Must be in the [required trace format](../trace-format.md). Must not be empty.
83+
84+
##### `trace_id` <span class='type'>str</span> <span class='required'/>
85+
86+
The id of the trace to which you want to append messages. This has to exist before the Append Messages API can be called and the caller must be the owner of the trace or the dataset containing the trace.
87+
88+
##### `request_kwargs` <span class='type'>Optional[Mapping]</span> <span class='optional'/>
89+
90+
Additional keyword arguments to pass to the requests method. Default is `None`.
91+
92+
##### Return Value
93+
94+
##### <span class='type'>Dict</span>
95+
96+
The response object from the Invariant API.
97+
98+
> AsyncClient Example
99+
```python
100+
from invariant_sdk.async_client import AsyncClient
101+
102+
client = AsyncClient()
103+
104+
messages = [
105+
{"role": "user", "content": "one"},
106+
{"role": "assistant", "content": "two \n three"},
107+
]
108+
109+
response = await client.create_request_and_push_trace(
110+
messages=messages,
111+
trace_id="some_trace_id"
112+
)
113+
```

docs/explorer/Explorer_API/Uploading_Traces/file_uploads.md renamed to docs/explorer/api/uploading-traces/file-uploads.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ The explorer supports two types of trace formats: _raw event lists_ and _annotat
2020

2121
#### Raw Event Lists
2222

23-
Raw event lists are `jsonl` files where each line is a JSON array of events. Each event is a dictionary with at least a `role` and `content` field according to the trace format described [in this chapter](../2_traces.md).
23+
Raw event lists are `jsonl` files where each line is a JSON array of events. Each event is a dictionary with at least a `role` and `content` field according to the trace format described [in this chapter](../trace-format.md).
2424

2525
```json
2626
[{ "role": "user", "content": "Hello, world!" }, { "role": "assistant", "content": "Hi!" }]
@@ -36,7 +36,7 @@ To include trace-level metadata in raw event lists, include a metadata JSON obje
3636

3737
#### Annotated Event Lists
3838

39-
Annotated event lists support the same format as raw event lists, but also include [annotations](../3_annotations.md).
39+
Annotated event lists support the same format as raw event lists, but also include [annotations](../annotations.md).
4040

4141
For this, each line is a JSON object with the fields `messages`, `annotations` (optional) and `metadata` (optional, for trace-level metadata).
4242

docs/explorer/Explorer_API/Uploading_Traces/push_api.md renamed to docs/explorer/api/uploading-traces/push-api.md

Lines changed: 3 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ The `PushTracesRequest` class holds the request data for a trace upload request.
2424

2525
This represents the traces in a dataset. Each `List[Dict]` is a single trace within the dataset. Each `dict` is a single message within a trace - these can represet a user prompt, a tool call, a tool output, etc.
2626

27-
Must be in the [required trace format](../2_traces.md). Must not be empty.
27+
Must be in the [required trace format](../trace-format.md). Must not be empty.
2828

2929
##### `annotations` <span class='type'>Optional[List[List[AnnotationCreate]]</span> <span class='optional'/>
3030

@@ -48,13 +48,13 @@ Must be a list of dictionaries if provided and must be the same length as messag
4848

4949
Each metadata dictionary can have arbitrary keys and values for storing additional information about the trace.
5050

51-
See [File Uploads](../Uploading_Traces/file_uploads.md) for more information on metadata.
51+
See [File Uploads](file-uploads.md) for more information on metadata.
5252

5353
### `AnnotationCreate`
5454

5555
The `AnnotationCreate` class holds the data for an annotation to be created.
5656

57-
See [Annotations](../3_annotations.md) for more information on annotations.
57+
See [Annotations](../annotations.md) for more information on annotations.
5858

5959
##### `content` <span class='type'>str</span> <span class='required'/>
6060

@@ -104,36 +104,6 @@ Additional keyword arguments to pass to the requests method. Default is `None`.
104104

105105
The response object from the Invariant API.
106106

107-
> AsyncClient Example
108-
```python
109-
from invariant_sdk.async_client import AsyncClient
110-
from invariant_sdk.types.push_traces import PushTracesRequest
111-
112-
client = AsyncClient()
113-
114-
request = PushTracesRequest(
115-
messages=[
116-
[
117-
{"role": "user", "content": "one"},
118-
{"role": "assistant", "content": "two \n three"},
119-
]
120-
],
121-
annotations=[
122-
[
123-
{
124-
"content": "annotating one",
125-
"address": "messages[0].content:L0",
126-
"extra_metadata": {"key1": "value1"},
127-
}
128-
]
129-
],
130-
metadata=[{"meta_key_1": "meta_value_1"}],
131-
dataset="dataset_name"
132-
)
133-
134-
response = await client.push_trace(request)
135-
```
136-
137107
> Client Example
138108
```python
139109
from invariant_sdk.client import Client
@@ -194,39 +164,6 @@ Additional keyword arguments to pass to the requests method. Default is `None`.
194164

195165
The response object from the Invariant API.
196166

197-
> AsyncClient Example
198-
```python
199-
from invariant_sdk.async_client import AsyncClient
200-
201-
client = AsyncClient()
202-
203-
messages = [
204-
[
205-
{"role": "user", "content": "one"},
206-
{"role": "assistant", "content": "two \n three"},
207-
]
208-
]
209-
210-
annotations = [
211-
[
212-
{
213-
"content": "annotating one",
214-
"address": "messages[0].content:L0",
215-
"extra_metadata": {"key1": "value1"},
216-
}
217-
]
218-
]
219-
220-
metadata = [{"meta_key_1": "meta_value_1"}]
221-
222-
response = await client.create_request_and_push_trace(
223-
messages=messages,
224-
annotations=annotations,
225-
metadata=metadata,
226-
dataset="dataset_name"
227-
)
228-
```
229-
230167
> Client Example
231168
```python
232169
from invariant_sdk.client import Client

‎docs/explorer/benchmarks.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ Once you have prepared and ensured that your agent traces are in a compatible fo
3838

3939
### Step 3: Associate Your Agent Dataset with a Benchmark and Score
4040

41-
[Instructions on updating the datasets's metadata](./Explorer_API/Dataset_Metadata/update_dataset_metadata_api.md) to include the `benchmark`, `name` and `accuracy` fields, that will associate your agent dataset with a specific benchmark and score.
41+
[Instructions on updating the datasets's metadata](./api/dataset-metadata/update.md) to include the `benchmark`, `name` and `accuracy` fields, that will associate your agent dataset with a specific benchmark and score.
4242

4343
For instance, to associate your `gpt-4o` agent dataset with the `webarena` benchmark and an accuracy score of `0.5`, you can update the dataset's metadata as follows:
4444

0 commit comments

Comments
 (0)