Commit 1660c9a
authored
Docs/howto-texttosqlagent (#2264)
New things added in this PR are:
- A new how-to guide `docs/howtos/applications/text2sql.md`
- A new example module `examples/ragas_examples/text2sql/` to accompany
the guide
## Changes Made
- **Added "How to evaluate a Text to SQL Agent" Guide**: A new how-to
guide (`docs/howtos/applications/text2sql.md`) was created. It walks
users through setting up, evaluating, and iteratively improving a
text-to-SQL system.
- **Created a Full Text-to-SQL Example**: A new, installable example
module (`ragas_examples.text2sql`) has been added. This module is
self-contained and provides all the necessary components to follow the
guide:
- **`Text2SQLAgent`**: A configurable agent (`text2sql_agent.py`) that
uses OpenAI models to convert natural language to SQL, featuring
comprehensive tracing.
- **Evaluation Pipeline**: An evaluation script (`evals.py`) built with
the Ragas experimentation framework, including custom metrics for
`sql_validity` and `execution_accuracy`.
- **Error Analysis Tool**: An automated script (`analyze_errors.py`)
that uses an LLM to categorize failure modes in evaluation results,
helping to guide system improvements.
- **Data and DB Utilities**: Scripts to download and prepare the BookSQL
dataset (`data_utils.py`) and a robust database utility module
(`db_utils.py`) for query execution and schema handling.
- **Iterative Prompts**: Includes multiple prompt versions
(`prompt.txt`, `prompt_v2.txt`, `prompt_v3.txt`) to demonstrate the
iterative improvement process described in the guide.
## Testing
### How to Test
- [ ] Manual testing steps:
1. Follow the instructions in the new
`docs/howtos/applications/text2sql.md` guide.
2. Install the required dependencies with `uv pip install
"ragas-examples[text2sql]"`.
3. Set the `OPENAI_API_KEY` environment variable.
4. Run the data preparation commands, evaluation commands, and error
analysis commands as specified in the guide.
5. Verify that the commands execute successfully and the outputs are
consistent with the examples provided in the documentation.
## References
- Documentation: `docs/howtos/applications/text2sql.md`1 parent 6549737 commit 1660c9a
File tree
20 files changed
+3563
-3
lines changed- .cursor
- commands
- rules
- docs/howtos/applications
- examples
- ragas_examples
- text2sql
- datasets
20 files changed
+3563
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
47 | 49 | | |
48 | 50 | | |
49 | 51 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
| 192 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
0 commit comments