Skip to content

Commit 007341d

Browse files
Move GCE based data generation to its own package (#628)
1 parent d00184d commit 007341d

File tree

16 files changed

+40
-42
lines changed

16 files changed

+40
-42
lines changed

libraries/python/assistant-evaluations/.vscode/launch.json renamed to libraries/python/assistant-data-gen/.vscode/launch.json

File renamed without changes.

libraries/python/assistant-evaluations/.vscode/settings.json renamed to libraries/python/assistant-data-gen/.vscode/settings.json

File renamed without changes.
File renamed without changes.

libraries/python/assistant-evaluations/README.md renamed to libraries/python/assistant-data-gen/README.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
1-
# Evaluation
2-
This is a work-in-progress tool for evaluating Semantic Workbench Assistants for quality.
1+
# Data Generation
32

3+
This is a tool for generating data for testing Semantic Workbench assistants.
44

5-
## Automation and Data Generation
6-
There is currently one part to this which is automation to populate a Workbench conversation automatically without human intervention.
5+
The core functionality of this library is an automation to populate a Workbench conversation automatically without human intervention.
76
This is implemented using a specialized version of the guided conversation engine (GCE).
87
The GCE here focuses on the agenda and using an exact resource constraint to force the GCE to have a long running conversation.
98

@@ -12,7 +11,7 @@ There is also a quick `generate_scenario.py` script that can be used to generate
1211
### Setup
1312

1413
1. Run the workbench service running locally (at http://127.0.0.1:3000), an assistant service, and create the assistant you want to test.
15-
2. Have LLM provider configured. Check [pydantic_ai_utils.py](./assistant_evaluations/pydantic_ai_utils.py) for an example of how it is configured for Pydantic AI.
14+
2. Have LLM provider configured. Check [pydantic_ai_utils.py](./assistant_data_gen/pydantic_ai_utils.py) for an example of how it is configured for Pydantic AI.
1615
1. For example, create a `.env` file with your Azure OpenAI endpoint set as `ASSISTANT__AZURE_OPENAI_ENDPOINT=<your_endpoint>`
1716
3. Create a configuration file. See [document_assistant_example_config.yaml](./configs/document_assistant_example_config.yaml) for an example.
1817
1. The scenarios field is a list that allows you to specify multiple test scenarios (different conversation paths).
@@ -33,4 +32,3 @@ python scripts/generate_scenario.py --config path/to/custom_config.yaml
3332

3433
### Recommendations
3534
1. Be as specific as possible with your conversation flows. Generic conversation flows and/or resource constraints that are too high can lead to the agents getting stuck in a thank you loop.
36-

libraries/python/assistant-evaluations/assistant_evaluations/__init__.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/__init__.py

File renamed without changes.

libraries/python/assistant-evaluations/assistant_evaluations/assistant_api.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/assistant_api.py

File renamed without changes.

libraries/python/assistant-evaluations/assistant_evaluations/config.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/config.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
import yaml
77
from pydantic import BaseModel, Field
88

9-
from assistant_evaluations.gce.gce_agent import ResourceConstraintMode
9+
from assistant_data_gen.gce.gce_agent import ResourceConstraintMode
1010

1111

1212
class ScenarioConfig(BaseModel):

libraries/python/assistant-evaluations/assistant_evaluations/gce/__init__.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/gce/__init__.py

File renamed without changes.

libraries/python/assistant-evaluations/assistant_evaluations/gce/gce_agent.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/gce/gce_agent.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@
4040
from pydantic_ai.providers.openai import OpenAIProvider
4141
from pydantic_ai.tools import ToolDefinition
4242

43-
from assistant_evaluations.gce.prompts import (
43+
from assistant_data_gen.gce.prompts import (
4444
AGENDA_SYSTEM_PROMPT,
4545
CONVERSATION_SYSTEM_PROMPT,
4646
FIRST_USER_MESSAGE,
@@ -49,7 +49,7 @@
4949
TERMINATION_INSTRUCTIONS_EXACT,
5050
TERMINATION_INSTRUCTIONS_MAXIMUM,
5151
)
52-
from assistant_evaluations.pydantic_ai_utils import create_model
52+
from assistant_data_gen.pydantic_ai_utils import create_model
5353

5454

5555
class ResourceConstraintMode(Enum):

libraries/python/assistant-evaluations/assistant_evaluations/gce/prompts.py renamed to libraries/python/assistant-data-gen/assistant_data_gen/gce/prompts.py

File renamed without changes.

0 commit comments

Comments
 (0)