Skip to content

Commit 1660c9a

Browse files
authored
Docs/howto-texttosqlagent (#2264)
New things added in this PR are: - A new how-to guide `docs/howtos/applications/text2sql.md` - A new example module `examples/ragas_examples/text2sql/` to accompany the guide ## Changes Made - **Added "How to evaluate a Text to SQL Agent" Guide**: A new how-to guide (`docs/howtos/applications/text2sql.md`) was created. It walks users through setting up, evaluating, and iteratively improving a text-to-SQL system. - **Created a Full Text-to-SQL Example**: A new, installable example module (`ragas_examples.text2sql`) has been added. This module is self-contained and provides all the necessary components to follow the guide: - **`Text2SQLAgent`**: A configurable agent (`text2sql_agent.py`) that uses OpenAI models to convert natural language to SQL, featuring comprehensive tracing. - **Evaluation Pipeline**: An evaluation script (`evals.py`) built with the Ragas experimentation framework, including custom metrics for `sql_validity` and `execution_accuracy`. - **Error Analysis Tool**: An automated script (`analyze_errors.py`) that uses an LLM to categorize failure modes in evaluation results, helping to guide system improvements. - **Data and DB Utilities**: Scripts to download and prepare the BookSQL dataset (`data_utils.py`) and a robust database utility module (`db_utils.py`) for query execution and schema handling. - **Iterative Prompts**: Includes multiple prompt versions (`prompt.txt`, `prompt_v2.txt`, `prompt_v3.txt`) to demonstrate the iterative improvement process described in the guide. ## Testing ### How to Test - [ ] Manual testing steps: 1. Follow the instructions in the new `docs/howtos/applications/text2sql.md` guide. 2. Install the required dependencies with `uv pip install "ragas-examples[text2sql]"`. 3. Set the `OPENAI_API_KEY` environment variable. 4. Run the data preparation commands, evaluation commands, and error analysis commands as specified in the guide. 5. Verify that the commands execute successfully and the outputs are consistent with the examples provided in the documentation. ## References - Documentation: `docs/howtos/applications/text2sql.md`
1 parent 6549737 commit 1660c9a

File tree

20 files changed

+3563
-3
lines changed

20 files changed

+3563
-3
lines changed

.cursor/commands/git-pr.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
Review the PR and create PR description in this format:
2+
3+
## Issue Link / Problem Description
4+
<!-- Link to related issue or describe the problem this PR solves -->
5+
- Fixes #[issue_number]
6+
- OR describe the issue: What problem does this solve? How can it be replicated?
7+
8+
## Changes Made
9+
<!-- Describe what you changed and why -->
10+
-
11+
-
12+
-
13+
14+
## Testing
15+
<!-- Describe how this should be tested -->
16+
### How to Test
17+
- [ ] Automated tests added/updated
18+
- [ ] Manual testing steps:
19+
1.
20+
2.
21+
3.
22+
23+
## References
24+
<!-- Link to related issues, discussions, forums, or external resources -->
25+
- Related issues:
26+
- Documentation:
27+
- External references:
28+
29+
## Screenshots/Examples (if applicable)
30+
<!-- Add screenshots or code examples showing the change -->
31+
32+
---
33+
<!--
34+
Thank you for contributing to Ragas!
35+
Please fill out the sections above as completely as possible.
36+
The more information you provide, the faster your PR can be reviewed and merged.
37+
-->

.cursor/rules/docs-diataxis-guidelines.mdc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,11 @@ When writing or editing documentation, categorise each page as **one** of the fo
4141
Choose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.
4242

4343
### Writing Style
44-
• Use second-person (you) and active voice.
44+
• Use second-person ("you") and active voice.
4545
• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).
4646
• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.
47+
• Use `??? "Click to expand"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.
48+
• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.
4749

4850
### Cross-linking Between Modes
4951
• End tutorials with pointers to relevant how-to guides for further exploration.

.cursor/rules/update-guide.mdc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
alwaysApply: false
3+
---
4+
We are writing a how to guide for Ragas docs and Ragas users.
5+
6+
So after any coding step we complete, or after any succesful runs, always update the guide with what was done.

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,3 +189,4 @@ examples/dist/
189189
examples/build/
190190
examples/*.egg-info/
191191
examples/ragas_examples/_version.py
192+
examples/ragas_examples/text2sql/experiments/*

docs/howtos/applications/index.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,8 @@ usecases to solve problems you might encounter when you're building.
1616
- [Single-hop Query Testset](singlehop_testset_gen.md)
1717

1818
## Benchmarking
19-
- [Benchmarking Gemini models](gemini_benchmarking.md)
19+
- [Benchmarking Gemini models](gemini_benchmarking.md)
20+
21+
## Agent Evaluation
22+
23+
- [Evaluate a Text-to-SQL Agent](text2sql.md)

0 commit comments

Comments
 (0)