initial commit

Author: lbeurerkellner

.DS_Store

@@ -0,0 +1,23 @@
+name: Documentation Docker Image
+
+on:
+  push:
+    branches: [ "main" ]
+
+jobs:
+
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Build the Docker image
+      working-directory: docs
+      run: docker build . --tag invariant-docs:main
+    - name: Login to Invariant Container Registry
+      run: echo ${{ secrets.INVARIANT_REGISTRY_PASSWORD }} | docker login -u ${{ secrets.INVARIANT_REGISTRY_USERNAME }} --password-stdin https://images.invariantnet.com
+    - name: Rename the Docker image
+      run: docker tag invariant-docs:main images.invariantnet.com/invariant-docs:main
+    - name: Push the Docker image
+      run: docker push images.invariantnet.com/invariant-docs:main

.github/workflows/docs.yml

@@ -0,0 +1,15 @@
+FROM python:3.12
+
+# install mkdocs
+COPY requirements.txt /tmp
+RUN pip install --no-cache-dir -r /tmp/requirements.txt
+
+# build the documentation
+COPY . /docs
+WORKDIR /docs
+
+RUN mkdocs build --clean
+
+# serve the documentation if DEV_MODE is set, otherwise, serve built documentation
+# for prod serving, we copy everything to root/docs/, as this is where the docs are actually served from in the container
+CMD if [ "$DEV_MODE" = "true" ]; then mkdocs serve -a 0.0.0.0:8000; else mkdocs build; mkdir -p root; mv site root/docs; cd root; python -m http.server 8000 --bind 0.0.0.0; fi

Dockerfile

@@ -0,0 +1,16 @@
+# Invariant SDK documentation
+
+Documentation for the Invariant SDK. 
+
+To locally serve a live version of the documentation, run the following command in this directory:
+
+```bash
+docker build -t invariant-docs .
+docker run -it -p 8000:8000 -e DEV_MODE=true -v .:/docs/ invariant-docs 
+```
+
+With `DEV_MODE=true` and the mounted volume, the documentation will reflect changes made to the markdown files in real-time.
+
+For deployment, we build the Docker image and push it to the registry:
+
+The actual documentation can be found in the subfolder `docs/`.

README.md

@@ -0,0 +1 @@
+.DS_Store

docs/.gitignore

@@ -0,0 +1,46 @@
+# Client Setup
+
+The SDK exposes a `Client` class. To create an object of this type, you need two variables: the Invariant API endpoint URL and the API key.
+
+## Getting the API Key
+Navigate to the <img class='inline-invariant' src="../assets/logo.svg"/> [Invariant Explorer](https://explorer.invariantlabs.ai) and create an account via GitHub Sign-In.
+
+Once you have created an account, go to your [User Settings](https://explorer.invariantlabs.ai/settings) and generate an API key.
+
+Make note of your API key, as you will need it to authenticate your uploads. If you're running in a shell, you can export the API key now as an environment variable:
+
+## Setting Up Environment Variables
+
+Navigate to your shell and export the API key as an environment variable. You can optionally set the API endpoint URL as an environment variable as well, which allows you to use private instances of Explorer.
+
+```bash
+export INVARIANT_API_ENDPOINT=https://explorer.invariantlabs.ai
+# Add the API key here.
+export INVARIANT_API_KEY=YourAPIKey
+```
+
+## Creating a Client
+
+In your Python code, you can create a `Client` object. This object will use the environment variables you set up earlier to authenticate your uploads.
+
+```python
+from invariant_sdk.client import Client
+
+client = Client()
+```
+
+Without parameters, the `Client` object will automatically use the environment variables you set up earlier and the default Explorer instance at `https://explorer.invariantlabs.ai`.
+
+## Overriding Environment Configuration
+
+If you want to override the environment configuration or use a different API key, you can also pass the API endpoint URL and API key as arguments to the `Client` constructor directly.
+
+```python
+from invariant_sdk.client import Client
+
+client = Client(
+    api_url="https://explorer.invariantlabs.ai",
+    # Add the API key here.
+    api_key="YourAPIKey",
+)
+```

docs/Explorer_API/1_client_setup.md

@@ -0,0 +1,173 @@
+# Trace Format
+
+The Explorer supports agent traces made up of sequences of events like messages, tool calls and environment outputs. The required format is compatible with the [OpenAI chat data structure](https://platform.openai.com/docs/api-reference/chat/create) with [function calling](https://platform.openai.com/docs/guides/function-calling) support.
+
+This documents shows pseudo-code based class definitions to specify the format more formally, but trace data is expected to be JSON serialized.
+
+### `Event`
+
+```python
+class Event:
+    role: str
+    content: Optional[str]
+    tool_calls: Optional[list[ToolCall]]
+```
+
+##### `role` <span class='type'>string</span> <span class='required'/>
+
+The role of the event, e.g., `user`, `assistant`, `system` or something else.
+
+##### `content` <span class='type'>string</span> <span class='optional'/>
+
+The content of the event, e.g., agent reasoning and intermediate results.
+
+##### `tool_calls` <span class='type'>list[ToolCall]</span> <span class='optional'/>
+
+A list of tool calls made by the agent in this event.
+
+
+> Examples <br/><br/>
+    Simple Message
+    ```json
+    { "role": "user", "content": "Hello, how are you?" }
+    ```
+    Message with Tool Call
+    ```json
+    { 
+        "role": "assistant", 
+        "content": "Checking your inbox...", 
+        "tool_calls": [
+            { 
+                "id": "1", 
+                "type": "function", 
+                "function": { 
+                    "name": "get_inbox", 
+                    "arguments": {
+                        "n": 10
+                    }
+                }
+            }
+        ]
+    }
+    ```
+
+### `ToolCall`
+
+```python
+class ToolCall:
+    id: str
+    type: str
+    function: Function
+
+class Function:
+    name: str
+    arguments: dict
+```
+
+<!-- * `id (str)`: A unique identifier for the tool call.
+* `type (str)`: The type of the tool call, e.g., `function`.
+* `function (Function)`: The function call made by the agent.
+    * `name (str)`: The name of the function called.
+    * `arguments (Dict[str, Any])`: The arguments passed to the function, encoded as a JSON dictionary. -->
+
+##### `id` <span class='type'>string</span> <span class='required'/>
+
+A unique identifier for the tool call.
+
+##### `type` <span class='type'>string</span> <span class='required'/>
+
+The type of the tool call, e.g., `function`.
+
+##### `function` <span class='type'>Function</span> <span class='required'/>
+
+The function call made by the agent.
+
+* ##### `name` <span class='type'>string</span> <span class='required'/>
+
+    The name of the function called.
+
+* ##### `arguments` <span class='type'>dict</span> <span class='required'/>
+
+    The arguments passed to the function, encoded as a JSON dictionary.
+
+> Example
+    ```json
+    {
+        "id": "1",
+        "type": "function",
+        "function": {
+            "name": "get_inbox",
+            "arguments": {
+                "n": 10
+            }
+        }
+    }
+    ```
+
+### `ToolOutput`
+
+A special event type for tool outputs, associated with a previous `ToolCall`.
+
+```python
+class ToolOutput(Event):
+    role: str
+    content: str
+    tool_call_id: Optional[str]
+```
+
+##### `role` <span class='type'>string</span> <span class='required'/>
+
+The role of the event, e.g., `tool`.
+
+##### `content` <span class='type'>string</span> <span class='required'/>
+
+The content of the tool output, e.g., the result of a function call.
+
+##### `tool_call_id` <span class='type'>string</span> <span class='optional'/>
+
+The identifier of a previous ToolCall that this output corresponds to.
+
+> Example
+    ```json
+    {
+        "role": "tool",
+        "tool_call_id": "1",
+        "content": "1. Subject: Hello, From: Alice, Date: 2024-01-01, 2. Subject: Meeting, From: Bob, Date: 2024-01-02"
+    }
+    ```
+
+### Full Trace Example
+
+The format suitable for the Invariant SDK is a list of `Event` objects. Here is an example of a trace with a user asking for their inbox, the assistant fetching the inbox, and the assistant listing the inbox contents.
+
+```json
+[
+    {
+        "role": "user", 
+        "content": "What's in my inbox?"
+    }, 
+    {
+        "role": "assistant",
+        "content": "Here are the latest emails.", 
+        "tool_calls": [
+            {
+                "id": "1",
+                "type": "function",
+                "function": {
+                    "name": "get_inbox",
+                    "arguments": {}
+                }
+            }
+        ]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "1",
+        "content": "1. Subject: Hello, From: Alice, Date: 2024-01-0, 2. Subject: Meeting, From: Bob, Date: 2024-01-02"
+    },
+    {
+        "role": "assistant",
+        "content": "You have 2 new emails."
+    }
+]
+```

docs/Explorer_API/2_traces.md

@@ -0,0 +1,44 @@
+# Annotations
+
+<div class='subtitle'>Learn how to annotate traces in Explorer to add context and structure</div>
+
+In addition to hosting agent traces, Explorer can also be used to annotate traces with additional information, as created in error analysis, human labeling or by other means. 
+
+Annotations provide additional context, facilitating collaboration and agent error and security analysis.
+
+<figure class='wide'>
+
+<img src="../assets/annotated-trace.png" alt="An annotated trace" style="width: 100%;">
+
+<figcaption>An annotated trace in Explorer</figcaption>
+
+</figure>
+
+## Annotation Format
+
+You can add annotations to traces at upload time. For this, both during [file upload](./Uploading_Traces/file_uploads.md) and via the [Push API](./Uploading_Traces/push_api.md), you can include an `annotations` field in the trace data. This field should be an array of objects, each representing an annotation. Each annotation object should have the following fields:
+
+##### `content` <span class='type'>string</span> <span class='required'/>
+
+The content of the annotation.
+
+##### `address` <span class='type'>string</span> <span class='required'/>
+
+The address of the span to annotate. This should be in the format `<path>:<start>-<end>`, where `<path>` is the path to an object in the event log, and `<start>` and `<end>` are the start and end indices of the character range to annotate.
+
+##### `extra_metadata` <span class='type'>Optional[Dict[Any, Any]]</span> <span class='optional'/>
+
+Additional metadata for the annotation.
+
+Each metadata dictionary can have arbitrary keys and values for storing additional information about the annotation.
+
+> Example Annotation
+    ```json
+    {
+        "content": "This is an example annotation",
+        "address": "messages.0.content:5-10",
+        "extra_metadata": {
+            "source": "my-analyzer"
+        }
+    }
+    ```

docs/Explorer_API/3_annotations.md

@@ -0,0 +1,40 @@
+---
+hide:
+    - toc
+---
+
+# GetDatasetMetadata API 
+
+<div class='subtitle'>Get the metadata associated with your Dataset</div>
+
+The GetDatasetMetadata API allows you to get the metadata associated with a dataset in a programmatic way.
+
+---
+### `get_dataset_metadata` 
+
+The `get_dataset_metadata method` is used to get the metadata for a dataset from the Invariant API using the `dataset_name`.
+
+##### `request` <span class='type'>dataset_name</span> <span class='required'/>
+
+The name of the dataset.
+
+##### `request_kwargs` <span class='type'>Optional[Dict[str, Any]]</span> <span class='optional'/>
+
+Additional keyword arguments to pass to the requests method. Default is `None`.
+
+##### Return Value
+
+##### <span class='type'>Dict</span>
+
+The response object from the Invariant API.
+
+
+> Example
+    ```python
+    from invariant_sdk.client import Client
+    from invariant_sdk.types.push_traces import PushTracesRequest
+
+    client = Client()
+
+    dataset_metadata = client.get_dataset_metadata(dataset_name="some_dataset_name")
+    ```