feat(ai-monitoring): Add AI Monitoring docs (#9807)

* Add AI Monitoring docs

Co-authored-by: Liza Mock <[email protected]>

Author: colin-sentry

docs/product/ai-monitoring/getting-started/index.mdx

@@ -0,0 +1,84 @@
+---
+title: Set Up
+sidebar_order: 0
+description: "Learn how to set up Sentry AI Monitoring"
+---
+
+Sentry AI Monitoring is easiest to use with the Python SDK and an official integration like OpenAI.
+
+![AI Monitoring User Interface](../img/pipelines-view.png)
+
+
+To start sending AI data to Sentry, make sure you've created a Sentry project for your AI-enabled repository and follow one of the guides below:
+
+## Official AI Integrations
+
+- [OpenAI](/platforms/python/integrations/openai/)
+- [Langchain](/platforms/python/integrations/langchain/)
+
+<Alert level="note" title="Don't see your platform?">
+
+We'll be adding AI integrations continuously. You can also instrument AI manually with the Sentry Python SDK.
+
+</Alert>
+
+
+## Pipelines and LLMs
+
+The Sentry AI Monitoring feature relies on the fact that you have an orchestrator (like langchain) creating pipelines of one or more AI models (such as gpt-4). In the AI Monitoring dashboard, we show you a table of the AI pipelines and pull the token usage from your AI models.
+
+If you're using OpenAI without langchain, you'll need to manually create pipelines with the `@ai_track` annotation. If you're using langchain without OpenAI, you might have to manually record token usage with `record_token_usage()`. Both manual helpers are documented below.
+
+### Python SDK Decorators
+
+The [Python SDK](/platforms/python) includes an `@ai_track` decorator which will mark functions as AI-related and
+cause them to show up in the AI Monitoring dashboard.
+
+```python
+
+import time
+from sentry_sdk.ai_monitoring import ai_track, record_token_usage
+import sentry_sdk
+import requests
+
+@ai_track(description="AI tool")
+def some_workload_function():
+    """
+    This function is an example of calling arbitrary code with @ai_track so that it shows up in the Sentry trace
+    """
+    time.sleep(5)
+
+@ai_track(description="LLM")
+def some_llm_call():
+    """
+    This function is an example of calling an LLM provider that isn't officially supported by Sentry.
+    """
+    with sentry_sdk.start_span(op="ai.chat_completions.create.examplecom", description="Example.com LLM") as span:
+        result = requests.get('https://example.com/api/llm-chat?question=say+hello').json()
+        # this annotates the tokens used by the LLM so that they show up in the graphs in the dashboard
+        record_token_usage(span, total_tokens=result["usage"]["total_tokens"])
+        return result["text"]
+
+@ai_track(description="My AI pipeline")
+def some_pipeline():
+    """
+    The topmost level function with @ai_track gets the operation "ai.pipeline", which makes it show up
+    in the table of AI pipelines in the Sentry AI Monitoring dashboard.
+    """
+    client = OpenAI()
+    some_workload_function()
+    some_llm_call()
+    response = (
+        client.chat.completions.create(
+            model="some-model", messages=[{"role": "system", "content": "say hello"}]
+        )
+        .choices[0]
+        .message.content
+    )
+    print(response)
+
+with sentry_sdk.start_transaction(op="ai-inference", name="The result of the AI inference"):
+    some_pipeline()
+
+```
+

docs/product/ai-monitoring/getting-started/the-dashboard.mdx

@@ -0,0 +1,34 @@
+---
+title: AI Monitoring Dashboard
+sidebar_order: 100
+description: "Learn how to use Sentry's AI Monitoring Dashboard."
+---
+
+
+Once you've [configured the Sentry SDK](/product/ai-monitoring/getting-started/) for your AI project, you'll start
+receiving data in the Sentry AI Monitoring dashboard.
+
+![AI Monitoring Dashboard](../img/pipelines-view.png)
+
+
+## The Per-pipeline Dashboard
+In the example below, there are two LangChain pipelines (whose `.name` is the name that shows up in the table).
+One has used 58,000 tokens in the past hour, and the other has used 44,900 tokens. When you click one of the pipelines in the table, you can see details about that particular pipeline.
+
+![AI Monitoring for a specific pipeline](../img/details-view.png)
+
+As you can see in the example above, the "Ask Sentry" pipeline has used 59 thousand tokens and taken 3.2 seconds on average.
+
+<Note>
+Creating an AI pipeline is different than calling an LLM. If you're creating AI pipelines by calling LLMs directly (without using a tool like LangChain), consider using [manual AI instrumentation](/product/ai-monitoring/getting-started/#manually-instrumenting-ai-workloads).
+</Note>
+
+## Where AI Data Shows Up In the Trace View
+
+If configured to include PII, the Sentry SDK will add prompts and responses to LLMs and other AI models to spans in the trace view.
+
+![AI Monitoring trace example](../img/trace-view.png)
+
+In the example above, you can see input messages and LLM responses related to the `ai.chat_completions.create.langchain` span. Other spans like `ai.chat_completions.create.openai` show the number of tokens used for that particular chat completion.
+
+This view can show other data as well. For example, if you call your LLM from a webserver, the trace will include details about the webserver through other integrations, and you'll get a holistic view of all the related parts.

docs/product/ai-monitoring/img/details-view.png

@@ -0,0 +1,21 @@
+---
+title: "AI Monitoring"
+sidebar_order: 62
+description: "Sentry AI monitoring helps you understand your LLM calls."
+---
+
+<Include name="feature-stage-alpha.mdx" />
+
+Sentry's AI Monitoring tools help you understand what's going on with your AI pipelines. They automatically collect information about prompts, tokens, and models from providers like OpenAI.
+
+## Example AI Monitoring Use Cases
+
+- Users are reporting issues with an AI workflow, and you want to investigate responses from the relevant large language models.
+- Workflows have been failing due to high token usage, and you want to understand the cause of the higher token usage.
+- Users report that AI workflows are taking longer than usual, and you want to understand what steps in a workflow are slowest.
+
+To use AI Monitoring, you must have an existing Sentry account and project set up. If you don't have one, [create an account here](https://sentry.io/signup/).
+
+![AI Monitoring User Interface](./img/details-view.png)
+
+- Learn how to [set up Sentry's AI Monitoring](/product/ai-monitoring/getting-started/).

docs/product/ai-monitoring/img/pipelines-view.png

@@ -37,6 +37,12 @@ Giving Sentry visibility into your [**releases**](/product/releases/) makes it p
 
 Releases are integrated with the rest of Sentry so you can directly see how an error or performance issue was affected by a release, in addition to being able to resolve release-specific issues.
 
+### AI Monitoring
+
+Our [**AI Monitoring**](/product/ai-monitoring/) feature gives you insights into your AI pipelines within the broader context of your app. When you `pip install sentry` into a project that's also using an AI provider like OpenAI, Sentry will automatically pick up useful metrics like token usage, prompts, and model IDs, and send them to our AI Monitoring dashboard.
+
+
+
 ### Recurring Job Monitoring
 
 [**Cron Monitors**](/product/crons/) allows you to monitor the uptime and performance of any scheduled, recurring job in Sentry. Once implemented, it'll allow you to get alerts and metrics to help you solve errors, detect timeouts, and prevent disruptions to your service.