Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/ko/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
sections:
- local: tutorials/building_good_agents
title: ์ข‹์€ ์—์ด์ „ํŠธ ๊ตฌ์ถ•ํ•˜๊ธฐ
# - local: tutorials/inspect_runs
# title: ๐Ÿ“Š Inspect your agent runs using telemetry
- local: tutorials/inspect_runs
title: ๐Ÿ“Š ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ๋กœ ์—์ด์ „ํŠธ ์‹คํ–‰ ๊ฒ€์‚ฌํ•˜๊ธฐ
# - local: tutorials/tools
# title: ๐Ÿ› ๏ธ Tools - in-depth guide
# - local: tutorials/secure_code_execution
Expand Down
179 changes: 179 additions & 0 deletions docs/source/ko/tutorials/inspect_runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# OpenTelemetry๋กœ ์‹คํ–‰ ๊ฒ€์‚ฌํ•˜๊ธฐ[[inspecting-runs-with-opentelemetry]]

[[open-in-colab]]

> [!TIP]
> ์—์ด์ „ํŠธ ๊ตฌ์ถ•์ด ์ฒ˜์Œ์ด๋ผ๋ฉด ๋จผ์ € [์—์ด์ „ํŠธ ์†Œ๊ฐœ](../conceptual_guides/intro_agents)์™€ [์•ˆ๋‚ด์„œ](../guided_tour)๋ฅผ ์ฝ์–ด๋ณด์„ธ์š”.

## ์—์ด์ „ํŠธ ์‹คํ–‰์„ ๋กœ๊น…ํ•˜๋Š” ์ด์œ ๋Š”?[[why-log-your-agent-runs?]]

์—์ด์ „ํŠธ ์‹คํ–‰์„ ๋””๋ฒ„๊น…ํ•˜๋Š” ๊ฒƒ์€ ๋ณต์žกํ•œ ์ž‘์—…์ž…๋‹ˆ๋‹ค.

์‹คํ–‰์ด ์ œ๋Œ€๋กœ ์ง„ํ–‰๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜๊ธฐ ์–ด๋ ต์Šต๋‹ˆ๋‹ค. ์—์ด์ „ํŠธ ์›Œํฌํ”Œ๋กœ์šฐ๋Š” ์„ค๊ณ„์ƒ ์˜ˆ์ธก ๋ถˆ๊ฐ€๋Šฅํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค(๋งŒ์•ฝ ์˜ˆ์ธก ๊ฐ€๋Šฅํ–ˆ๋‹ค๋ฉด ์ผ๋ฐ˜์ ์ธ ์ฝ”๋“œ๋ฅผ ์‚ฌ์šฉํ–ˆ์„ ๊ฒƒ์ž…๋‹ˆ๋‹ค).

์‹คํ–‰ ๊ณผ์ •์„ ์‚ดํŽด๋ณด๋Š” ๊ฒƒ๋„ ์‰ฝ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹ค๋‹จ๊ณ„ ์—์ด์ „ํŠธ๋Š” ์ฝ˜์†”์„ ๋กœ๊ทธ๋กœ ๋น ๋ฅด๊ฒŒ ์ฑ„์šฐ๋Š” ๊ฒฝํ–ฅ์ด ์žˆ์œผ๋ฉฐ, ๋Œ€๋ถ€๋ถ„์˜ ์˜ค๋ฅ˜๋Š” ๋‹จ์ˆœํ•œ "LLM์˜ ์‹ค์ˆ˜" ์œ ํ˜•์œผ๋กœ, LLM์ด ๋‹ค์Œ ๋‹จ๊ณ„์—์„œ ๋” ๋‚˜์€ ์ฝ”๋“œ๋‚˜ ๋„๊ตฌ ํ˜ธ์ถœ์„ ์ž‘์„ฑํ•˜์—ฌ ์Šค์Šค๋กœ ๊ต์ •ํ•ฉ๋‹ˆ๋‹ค.

๋”ฐ๋ผ์„œ ๋‚˜์ค‘์— ๊ฒ€์‚ฌํ•˜๊ณ  ๋ชจ๋‹ˆํ„ฐ๋งํ•  ์ˆ˜ ์žˆ๋„๋ก ๊ณ„์ธก์„ ํ†ตํ•ด ์—์ด์ „ํŠธ ์‹คํ–‰์„ ๊ธฐ๋กํ•˜๋Š” ๊ฒƒ์ด ํ”„๋กœ๋•์…˜ ํ™˜๊ฒฝ์—์„œ๋Š” ํ•„์ˆ˜์ž…๋‹ˆ๋‹ค!

์—์ด์ „ํŠธ ์‹คํ–‰์„ ๊ณ„์ธกํ•˜๊ธฐ ์œ„ํ•ด [OpenTelemetry](https://opentelemetry.io/) ํ‘œ์ค€์„ ๋„์ž…ํ–ˆ์Šต๋‹ˆ๋‹ค.

์ฆ‰, ๊ณ„์ธก ์ฝ”๋“œ๋ฅผ ์‹คํ–‰ํ•œ ํ›„ ์—์ด์ „ํŠธ๋ฅผ ํ‰์†Œ์ฒ˜๋Ÿผ ์‹คํ–‰ํ•˜๋ฉด ๋ชจ๋“  ๋‚ด์šฉ์ด ํ”Œ๋žซํผ์— ์ž๋™์œผ๋กœ ๋กœ๊น…๋ฉ๋‹ˆ๋‹ค. ๋‹ค์–‘ํ•œ OpenTelemetry ๋ฐฑ์—”๋“œ์—์„œ ์ด๋ฅผ ๊ตฌํ˜„ํ•˜๋Š” ๋ฐฉ๋ฒ•์˜ ์˜ˆ์‹œ๋ฅผ ์•„๋ž˜์— ์ œ์‹œํ•ฉ๋‹ˆ๋‹ค.

ํ”Œ๋žซํผ์—์„œ์˜ ์‹ค์ œ ๋ชจ์Šต์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:

<div class="flex justify-center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolagents/inspect_run_phoenix.gif"/>
</div>

## Arize AI Phoenix๋กœ ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ ์„ค์ •[[setting-up-telemetry-with-arize-ai-phoenix]]

๋จผ์ € ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋ฅผ ์„ค์น˜ํ•˜์„ธ์š”. ์—ฌ๊ธฐ์„œ๋Š” ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•˜๊ณ  ๊ฒ€์‚ฌํ•˜๋Š” ์ข‹์€ ์†”๋ฃจ์…˜์ธ [Arize AI์˜ Phoenix](https://github.com/Arize-ai/phoenix)๋ฅผ ์„ค์น˜ํ•˜์ง€๋งŒ, ์ด ์ˆ˜์ง‘ ๋ฐ ๊ฒ€์‚ฌ ๋ถ€๋ถ„์— ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๋‹ค๋ฅธ OpenTelemetry ํ˜ธํ™˜ ํ”Œ๋žซํผ๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

```shell
pip install 'smolagents[telemetry,toolkit]'
```

๊ทธ ๋‹ค์Œ ๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ์ˆ˜์ง‘๊ธฐ๋ฅผ ์‹คํ–‰ํ•˜์„ธ์š”.

```shell
python -m phoenix.server.main serve
```

๋งˆ์ง€๋ง‰์œผ๋กœ `SmolagentsInstrumentor`๋ฅผ ์„ค์ •ํ•˜์—ฌ ์—์ด์ „ํŠธ๋ฅผ ์ถ”์ ํ•˜๊ณ  Phoenix ๊ธฐ๋ณธ ์—”๋“œํฌ์ธํŠธ๋กœ ์ถ”์ ์„ ๋ณด๋‚ด์„ธ์š”.

```python
from phoenix.otel import register
from openinference.instrumentation.smolagents import SmolagentsInstrumentor

register()
SmolagentsInstrumentor().instrument()
```
๊ทธ๋Ÿฌ๋ฉด ์—์ด์ „ํŠธ๋ฅผ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

```py
from smolagents import (
CodeAgent,
ToolCallingAgent,
WebSearchTool,
VisitWebpageTool,
InferenceClientModel,
)

model = InferenceClientModel()

search_agent = ToolCallingAgent(
tools=[WebSearchTool(), VisitWebpageTool()],
model=model,
name="search_agent",
description="This is an agent that can do web search.",
)

manager_agent = CodeAgent(
tools=[],
model=model,
managed_agents=[search_agent],
)
manager_agent.run(
"If the US keeps its 2024 growth rate, how many years will it take for the GDP to double?"
)
```
๋์ž…๋‹ˆ๋‹ค!
์ด์ œ `http://0.0.0.0:6006/projects/`๋กœ ์ด๋™ํ•˜์—ฌ ์‹คํ–‰์„ ๊ฒ€์‚ฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolagents/inspect_run_phoenix.png">

CodeAgent๊ฐ€ ๊ด€๋ฆฌํ•˜๋Š” ToolCallingAgent๋ฅผ ํ˜ธ์ถœํ•˜์—ฌ(์ฐธ๊ณ ๋กœ ๊ด€๋ฆฌ๋˜๋Š” ์—์ด์ „ํŠธ๋Š” CodeAgent๊ฐ€ ๋  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค) ๋ฏธ๊ตญ 2024๋…„ ์„ฑ์žฅ๋ฅ ์„ ์›น์—์„œ ๊ฒ€์ƒ‰ํ•˜๋„๋ก ์š”์ฒญํ•œ ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ดํ›„ ๊ด€๋ฆฌ๋˜๋Š” ์—์ด์ „ํŠธ๊ฐ€ ๊ฒฐ๊ณผ๋ฅผ ๋ณด๊ณ ํ•˜๋ฉด, ๊ด€๋ฆฌ์ž ์—์ด์ „ํŠธ๊ฐ€ ์ด ์ •๋ณด๋ฅผ ํ™œ์šฉํ•˜์—ฌ ๊ฒฝ์ œ ๋ฐฐ์ฆ ์‹œ๊ฐ„์„ ๊ณ„์‚ฐํ–ˆ์Šต๋‹ˆ๋‹ค! ํฅ๋ฏธ๋กญ์ฃ ?

## ๐Ÿชข Langfuse๋กœ ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ ์„ค์ •[[setting-up-telemetry-with-๐Ÿชข-langfuse]]

์ด ๋ถ€๋ถ„์€ `SmolagentsInstrumentor`๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ **Langfuse**๋กœ Hugging Face **smolagents**๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ณ  ๋””๋ฒ„๊น…ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

> **Langfuse๋ž€?** [Langfuse](https://langfuse.com)๋Š” LLM ์—”์ง€๋‹ˆ์–ด๋ง์„ ์œ„ํ•œ ์˜คํ”ˆ์†Œ์Šค ํ”Œ๋žซํผ์ž…๋‹ˆ๋‹ค. AI ์—์ด์ „ํŠธ๋ฅผ ์œ„ํ•œ ์ถ”์  ๋ฐ ๋ชจ๋‹ˆํ„ฐ๋ง ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•˜์—ฌ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ œํ’ˆ์„ ๋””๋ฒ„๊น…ํ•˜๊ณ , ๋ถ„์„ํ•˜๊ณ , ์ตœ์ ํ™”ํ•  ์ˆ˜ ์žˆ๋„๋ก ๋„์™€์ค๋‹ˆ๋‹ค. Langfuse๋Š” ๋„ค์ดํ‹ฐ๋ธŒ ํ†ตํ•ฉ, OpenTelemetry, SDK๋ฅผ ํ†ตํ•ด ๋‹ค์–‘ํ•œ ๋„๊ตฌ์™€ ํ”„๋ ˆ์ž„์›Œํฌ์™€ ํ†ตํ•ฉ๋ฉ๋‹ˆ๋‹ค.

### 1๋‹จ๊ณ„: ์˜์กด์„ฑ ์„ค์น˜[[step-1:-install-dependencies]]

```python
%pip install langfuse 'smolagents[telemetry]' openinference-instrumentation-smolagents
```

### 2๋‹จ๊ณ„: ํ™˜๊ฒฝ ๋ณ€์ˆ˜ ์„ค์ •[[step-2:-set-up-environment-variables]]

Langfuse API ํ‚ค๋ฅผ ์„ค์ •ํ•˜๊ณ  Langfuse๋กœ ์ถ”์ ์„ ๋ณด๋‚ด๋„๋ก OpenTelemetry ์—”๋“œํฌ์ธํŠธ๋ฅผ ๊ตฌ์„ฑํ•˜์„ธ์š”. [Langfuse Cloud](https://cloud.langfuse.com)์— ๊ฐ€์ž…ํ•˜๊ฑฐ๋‚˜ [Langfuse๋ฅผ ์ž์ฒด ํ˜ธ์ŠคํŒ…](https://langfuse.com/self-hosting)ํ•˜์—ฌ Langfuse API ํ‚ค๋ฅผ ์–ป์œผ์„ธ์š”.

๋˜ํ•œ [Hugging Face ํ† ํฐ](https://huggingface.co/settings/tokens) (`HF_TOKEN`)์„ ํ™˜๊ฒฝ ๋ณ€์ˆ˜๋กœ ์ถ”๊ฐ€ํ•˜์„ธ์š”.

```python
import os
# Get keys for your project from the project settings page: https://cloud.langfuse.com
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # ๐Ÿ‡ช๐Ÿ‡บ EU region
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # ๐Ÿ‡บ๐Ÿ‡ธ US region

# your Hugging Face token
os.environ["HF_TOKEN"] = "hf_..."
```

ํ™˜๊ฒฝ ๋ณ€์ˆ˜๊ฐ€ ์„ค์ •๋˜๋ฉด ์ด์ œ Langfuse ํด๋ผ์ด์–ธํŠธ๋ฅผ ์ดˆ๊ธฐํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. `get_client()`๋Š” ํ™˜๊ฒฝ ๋ณ€์ˆ˜์— ์ œ๊ณต๋œ ์ž๊ฒฉ ์ฆ๋ช…์„ ์‚ฌ์šฉํ•˜์—ฌ Langfuse ํด๋ผ์ด์–ธํŠธ๋ฅผ ์ดˆ๊ธฐํ™”ํ•ฉ๋‹ˆ๋‹ค.

```python
from langfuse import get_client

langfuse = get_client()

# Verify connection
if langfuse.auth_check():
print("Langfuse client is authenticated and ready!")
else:
print("Authentication failed. Please check your credentials and host.")
```

### 3๋‹จ๊ณ„: `SmolagentsInstrumentor` ์ดˆ๊ธฐํ™”[[step-3:-initialize-the-`smolagentsinstrumentor`]]

์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ฝ”๋“œ ์•ž์—์„œ `SmolagentsInstrumentor`๋ฅผ ์ดˆ๊ธฐํ™”ํ•˜์„ธ์š”.

```python
from openinference.instrumentation.smolagents import SmolagentsInstrumentor

SmolagentsInstrumentor().instrument()
```

### 4๋‹จ๊ณ„: smolagent ์‹คํ–‰[[step-4:-run-your-smolagent]]

```python
from smolagents import (
CodeAgent,
ToolCallingAgent,
WebSearchTool,
VisitWebpageTool,
InferenceClientModel,
)

model = InferenceClientModel(
model_id="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
)

search_agent = ToolCallingAgent(
tools=[WebSearchTool(), VisitWebpageTool()],
model=model,
name="search_agent",
description="This is an agent that can do web search.",
)

manager_agent = CodeAgent(
tools=[],
model=model,
managed_agents=[search_agent],
)
manager_agent.run(
"How can Langfuse be used to monitor and improve the reasoning and decision-making of smolagents when they execute multi-step tasks, like dynamically adjusting a recipe based on user feedback or available ingredients?"
)
```

### 5๋‹จ๊ณ„: Langfuse์—์„œ ์ถ”์  ๋ณด๊ธฐ[[step-5:-view-traces-in-langfuse]]

์—์ด์ „ํŠธ๋ฅผ ์‹คํ–‰ํ•œ ํ›„ Langfuse์—์„œ smolagents ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์—์„œ ์ƒ์„ฑ๋œ ์ถ”์  ์ •๋ณด๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. AI ์—์ด์ „ํŠธ ๋””๋ฒ„๊น…๊ณผ ์ตœ์ ํ™”์— ๋„์›€์ด ๋˜๋Š” LLM ์ƒํ˜ธ์ž‘์šฉ์˜ ์ƒ์„ธํ•œ ๊ณผ์ •์„ ์‚ดํŽด๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

![smolagents example trace](https://langfuse.com/images/cookbook/integration-smolagents/smolagent_example_trace.png)

_[Langfuse์˜ ์ถ”์  ์˜ˆ์‹œ](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/ce5160f9bfd5a6cd63b07d2bfcec6f54?timestamp=2025-02-11T09%3A25%3A45.163Z&display=details)_