Make llm-complete-guide work again
#164
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a new function `run_llm_judged_tests` to perform end-to-end tests on RAG systems using LLM evaluation. The implementation includes: - Parallel processing of test cases - Scoring for toxicity, faithfulness, helpfulness, and relevance - Retry logic for robust test execution - Detailed logging of test results
Enhance the evaluation visualization step by logging detailed metrics to ZenML, including: - Retrieval performance metrics - Generation failure rates - Quality scores (toxicity, faithfulness, helpfulness, relevance) - Composite scores for overall quality and retrieval effectiveness
Refactor import statements in eval_retrieval.py and eval_visualisation.py to: - Remove unused imports - Organize imports consistently - Simplify import statements
Simplify the dev/rag.yaml configuration by removing the commented "environment configuration" line, keeping the configuration clean and concise.
Modify the default temperature parameter in get_completion_from_messages() from 0.4 to 0, ensuring more deterministic and focused model responses.
Modify Hugging Face space deployment to ensure ZenML store secrets are converted to strings before adding, preventing potential type-related errors during deployment.
htahir1
requested changes
Jan 31, 2025
Contributor
htahir1
left a comment
htahir1
left a comment
There was a problem hiding this comment.
postgres is a bit hard locally - is there a way we can change this to a leaner DB? even something like sqlite?
Contributor
Author
I'll see what we can do. Problem is that sqlite doesn't really support vector search etc like Postgres does. But maybe I can refactor that out in a separate PR? |
Contributor
Author
|
I guess I could try https://alexgarcia.xyz/sqlite-vec/ but I'd like to do it as a completely self-contained PR, not in this one please. WDYT @htahir1 ? |
Update project dependencies to include: - Elasticsearch for potential search and indexing functionality - Tenacity for improved retry handling in various components
Contributor
|
Yes @strickvl lets just merge this one so its fixed on main |
- Add explicit constants for ZenML chatbot model name and version - Enhance find_vectorstore_name() function with error handling and fallback mechanism - Improve logging for vector store metadata retrieval
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Quite a few small fixes.
Also made Postgresql the default DB again.
And parallelise the evals.