Note: In order to most efficiently utilize GitHub actions please FORK the repository as opposed to cloning
This is a starting guide designed to accelerate the development of a robust LLMOPs framework with Microsoft Azure.
The repository aligns with the three main pillars of LLMOps:
- Evaluation/Testing - pre-deployment processes to provide confidence for release to end users
- Monitoring - near real-time capture of deployment metrics and security threats
- Feedback - post-interaction analysis of user behavior and model responses to drive future improvements
Checkout this Blog post for a more in depth overview: Is my LLM Chatbot Ready for Production?
This repository contains Demo Notebooks with accompanying source code for all stages of an LLMOps implementaiton
- Demo Notebooks: Step-by-Step interactive walkthroughs of various eval techniques
- 00_setup: Environment and development setup
- 01_evaluation: Use PromptFlow to orchestrate evaluation of a chatbot. Then use that evaluation framework with GitHub Actions for CI/CD
- 02_monitoring: Setup a system to monitor a chatbot for secutiry and availability in real time
- 03_feedback: Evaluate production inputs/responses captured with Model Data Collector. Use an LLM to sythesize and categorize mock user survey feedback