Skip to content

Commit 64bd7d4

Browse files
lisadunlaplisadunlap
committed
fixed some trace representation
1 parent dafa3d9 commit 64bd7d4

38 files changed

+2807
-463
lines changed

‎.github/workflows/nightly.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: Nightly Build and Publish
2+
3+
on:
4+
schedule:
5+
- cron: "0 3 * * *" # Run daily at 3:00 AM UTC
6+
workflow_dispatch: # Allow manual triggering
7+
8+
jobs:
9+
build-and-publish-nightly:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout code
13+
uses: actions/checkout@v4
14+
15+
- name: Set up Python
16+
uses: actions/setup-python@v5
17+
with:
18+
python-version: '3.10'
19+
20+
- name: Install build dependencies
21+
run: |
22+
python -m pip install --upgrade pip
23+
pip install build twine tomli
24+
25+
- name: Update version with dev suffix
26+
run: |
27+
python << 'EOF'
28+
import re
29+
from datetime import datetime
30+
31+
# Read current pyproject.toml
32+
with open('pyproject.toml', 'r') as f:
33+
content = f.read()
34+
35+
# Extract current version
36+
version_match = re.search(r'version = "([^"]+)"', content)
37+
if not version_match:
38+
raise ValueError("Could not find version in pyproject.toml")
39+
40+
current_version = version_match.group(1)
41+
42+
# Remove any existing dev suffix if present
43+
base_version = re.sub(r'\.dev\d+$', '', current_version)
44+
45+
# Generate dev version with date (YYYYMMDD format)
46+
date_suffix = datetime.utcnow().strftime('%Y%m%d')
47+
dev_version = f"{base_version}.dev{date_suffix}"
48+
49+
# Replace version in content
50+
new_content = re.sub(
51+
r'version = "[^"]+"',
52+
f'version = "{dev_version}"',
53+
content
54+
)
55+
56+
# Write back to pyproject.toml
57+
with open('pyproject.toml', 'w') as f:
58+
f.write(new_content)
59+
60+
print(f"Updated version from {current_version} to {dev_version}")
61+
EOF
62+
63+
- name: Build package
64+
run: |
65+
python -m build
66+
67+
- name: Publish to PyPI
68+
env:
69+
TWINE_USERNAME: __token__
70+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
71+
run: |
72+
twine upload dist/*
73+

‎README.md‎

Lines changed: 81 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,80 @@
1-
<div align="center">
1+
<p align="center">
2+
<img src="stringsight_github.png" alt="StringSight logo" width="600">
3+
</p>
4+
5+
<h1 align="center">StringSight</h1>
6+
7+
<p align="center">
8+
<em>Extract, cluster, and analyze behavioral properties from Large Multimodal Models</em>
9+
</p>
10+
11+
<p align="center">
12+
<a href="https://www.python.org/downloads/">
13+
<img src="https://img.shields.io/badge/python-3.8+-blue.svg" alt="Python 3.8+">
14+
</a>
15+
<a href="LICENSE">
16+
<img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT License">
17+
</a>
18+
<a href="https://lisadunlap.github.io/StringSight/">
19+
<img src="https://img.shields.io/badge/docs-Documentation-blue" alt="Docs">
20+
</a>
21+
<a href="https://blog.stringsight.com">
22+
<img src="https://img.shields.io/badge/blog-blog.stringsight.com-orange" alt="Blog">
23+
</a>
24+
<a href="https://stringsight.com">
25+
<img src="https://img.shields.io/badge/website-stringsight.com-green" alt="Website">
26+
</a>
27+
</p>
28+
29+
<p align="center">
30+
<strong>Annoyed at having to look through your long model conversations or agentic traces? Fear not, StringSight has come to ease your woes. Understand and compare model behavior by automatically extracting behavioral properties from their responses, grouping similar behaviors together, and quantifying how important these behaviors are.</strong>
31+
</p>
232

3-
# StringSight
4-
### *Extract, cluster, and analyze behavioral properties from Large Language Models*
5-
6-
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
7-
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
33+
## Installation
834

9-
[![Docs](https://img.shields.io/badge/docs-Documentation-blue)](https://stringsight.com/docs)
10-
[![Website](https://img.shields.io/badge/website-stringsight.com-green)](https://stringsight.com)
35+
```bash
36+
# (Optional) create and activate a dedicated environment
37+
conda create -n stringsight python=3.11
38+
conda activate stringsight
1139

12-
**Understand how different generative models behave by automatically extracting behavioral properties from their responses, grouping similar behaviors together, and quantifying how important these behaviors are.**
40+
# Install the core library from PyPI
41+
pip install stringsight
1342

14-
</div>
43+
# Install with all optional extras (recommended for notebooks and advanced workflows)
44+
pip install "stringsight[full]"
45+
```
1546

16-
## Installation
47+
For local development or contributing, you can install from source in editable mode:
1748

1849
```bash
19-
# Create conda environment
50+
# Clone the repository
51+
git clone https://github.com/lisabdunlap/stringsight.git
52+
cd stringsight
53+
54+
# (Optional) create and activate a dedicated environment
2055
conda create -n stringsight python=3.11
2156
conda activate stringsight
2257

23-
# Install StringSight
58+
# Install StringSight in editable mode with full extras
2459
pip install -e ".[full]"
2560

26-
2761
# Install StringSight in editable mode with dev dependencies
2862
pip install -e ".[dev]"
63+
```
2964

30-
# Set API keys
65+
Set your API keys (required for running LLM-backed pipelines):
66+
67+
```bash
3168
export OPENAI_API_KEY="your-openai-key"
32-
export ANTHROPIC_API_KEY="your-anthropic-key" # optional
33-
export GOOGLE_API_KEY="your-google-key" # optional
69+
export ANTHROPIC_API_KEY="your-anthropic-key"
70+
export GOOGLE_API_KEY="your-google-key"
3471
```
3572

73+
vLLM support coming soon, I promise!
74+
3675
## Quick Start
3776

38-
For a comprehensive tutorial with detailed explanations, see [starter_notebook.ipynb](starter_notebook.ipynb).
77+
For a comprehensive tutorial with detailed explanations, see [starter_notebook.ipynb](starter_notebook.ipynb) or open it directly in [Google Colab](https://colab.research.google.com/drive/1XBQqDqTK6-9wopqRB51j8cPfnTS5Wjqh?usp=drive_link).
3978

4079
### 1. Extract and Cluster Properties with `explain()`
4180

@@ -221,30 +260,7 @@ Use the React frontend or other visualization tools to explore your results.
221260

222261
### Side-by-Side Comparisons
223262

224-
**Option 1: Pre-paired Data**
225-
226-
**Required Columns:**
227-
| Column | Description | Example |
228-
|--------|-------------|---------|
229-
| `prompt` | Question given to both models | `"What is machine learning?"` |
230-
| `model_a` | First model name | `"gpt-4"` |
231-
| `model_b` | Second model name | `"claude-3"` |
232-
| `model_a_response` | First model's response | `"Machine learning is..."` |
233-
| `model_b_response` | Second model's response | `"ML involves..."` |
234-
235-
**Optional Columns:**
236-
| Column | Description | Example |
237-
|--------|-------------|---------|
238-
| `score` | Winner and metrics | `{"winner": "model_a", "helpfulness_a": 4.2, "helpfulness_b": 3.8}` |
239-
| `score_columns` | Alternative: separate columns for each metric with `_a` and `_b` suffixes (e.g., `accuracy_a`, `accuracy_b`) | `score_columns=["accuracy_a", "accuracy_b", "helpfulness_a", "helpfulness_b"]` |
240-
| `prompt_column` | Name of the prompt column in your dataframe (default: `"prompt"`) | `prompt_column="query"` |
241-
| `model_a_column` | Name of the model_a column (default: `"model_a"`) | `model_a_column="model_1"` |
242-
| `model_b_column` | Name of the model_b column (default: `"model_b"`) | `model_b_column="model_2"` |
243-
| `model_a_response_column` | Name of the model_a_response column (default: `"model_a_response"`) | `model_a_response_column="response_1"` |
244-
| `model_b_response_column` | Name of the model_b_response column (default: `"model_b_response"`) | `model_b_response_column="response_2"` |
245-
| `question_id_column` | Name of the question_id column (default: `"question_id"` if column exists) | `question_id_column="qid"` |
246-
247-
**Option 2: Tidy Data (Auto-pairing)**
263+
**Option 1: Tidy Data (Auto-pairing)**
248264

249265
If your data is in tidy single-model format with multiple models, StringSight can automatically pair them:
250266

@@ -268,6 +284,29 @@ clustered_df, model_stats = explain(
268284

269285
The pipeline will automatically pair rows where both models answered the same prompt.
270286

287+
**Option 2: Pre-paired Data**
288+
289+
**Required Columns:**
290+
| Column | Description | Example |
291+
|--------|-------------|---------|
292+
| `prompt` | Question given to both models | `"What is machine learning?"` |
293+
| `model_a` | First model name | `"gpt-4"` |
294+
| `model_b` | Second model name | `"claude-3"` |
295+
| `model_a_response` | First model's response | `"Machine learning is..."` |
296+
| `model_b_response` | Second model's response | `"ML involves..."` |
297+
298+
**Optional Columns:**
299+
| Column | Description | Example |
300+
|--------|-------------|---------|
301+
| `score` | Winner and metrics | `{"winner": "model_a", "helpfulness_a": 4.2, "helpfulness_b": 3.8}` |
302+
| `score_columns` | Alternative: separate columns for each metric with `_a` and `_b` suffixes (e.g., `accuracy_a`, `accuracy_b`) | `score_columns=["accuracy_a", "accuracy_b", "helpfulness_a", "helpfulness_b"]` |
303+
| `prompt_column` | Name of the prompt column in your dataframe (default: `"prompt"`) | `prompt_column="query"` |
304+
| `model_a_column` | Name of the model_a column (default: `"model_a"`) | `model_a_column="model_1"` |
305+
| `model_b_column` | Name of the model_b column (default: `"model_b"`) | `model_b_column="model_2"` |
306+
| `model_a_response_column` | Name of the model_a_response column (default: `"model_a_response"`) | `model_a_response_column="response_1"` |
307+
| `model_b_response_column` | Name of the model_b_response column (default: `"model_b_response"`) | `model_b_response_column="response_2"` |
308+
| `question_id_column` | Name of the question_id column (default: `"question_id"` if column exists) | `question_id_column="qid"` |
309+
271310
## Outputs
272311

273312
### `clustered_df` (DataFrame)

‎docs/assets/stringsight_logo.png‎

104 KB
Loading

‎docs/getting-started/installation.md‎

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,27 +26,25 @@ conda activate stringsight
2626
### 2. Install StringSight
2727

2828
```bash
29-
# Clone the repository
30-
git clone https://github.com/lisabdunlap/stringsight.git
31-
cd stringsight
29+
# From PyPI (recommended): install core package + optional ML/embedding tooling
30+
pip install "stringsight[full]"
3231

33-
# Install in development mode with all dependencies
34-
pip install -e ".[full]"
32+
# Or, for local development from source:
33+
# git clone https://github.com/lisabdunlap/stringsight.git
34+
# cd stringsight
35+
# pip install -e ".[full]"
3536
```
3637

37-
### 3. Set API Key
38+
### 3. Set API Key(s)
3839

39-
**Option A: Environment Variable**
4040
```bash
4141
# Add to your shell profile (.bashrc, .zshrc, etc.)
42+
# We dafult to OAI but we support any litellm compatable api
4243
export OPENAI_API_KEY="your-api-key-here"
44+
... # any other provider keys
4345
```
4446

45-
**Option B: .env File**
46-
```bash
47-
# Create .env file in project root
48-
echo "OPENAI_API_KEY=your-api-key-here" > .env
49-
```
47+
vLLM support coming very soon. I promise!
5048

5149
### 4. Verify Installation
5250

@@ -66,21 +64,28 @@ curl http://127.0.0.1:8000/health
6664

6765
### Core Package Only
6866
```bash
69-
pip install -e .
70-
```
67+
# From PyPI
68+
pip install stringsight
7169

72-
### With Visualization Tools
73-
```bash
74-
pip install -e ".[viz]"
70+
# From a local clone (development)
71+
pip install -e .
7572
```
7673

7774
### With Development Tools
7875
```bash
76+
# From PyPI
77+
pip install "stringsight[dev]"
78+
79+
# From a local clone (development)
7980
pip install -e ".[dev]"
8081
```
8182

8283
### All Features
8384
```bash
85+
# From PyPI (recommended for most users)
86+
pip install "stringsight[full]"
87+
88+
# From a local clone (development)
8489
pip install -e ".[full]"
8590
```
8691

0 commit comments

Comments
 (0)