* Add logging

* Add README
* Add walkthrough notebook

Author: subramen

recipes/use_cases/github_triage/README.md

@@ -0,0 +1,60 @@
+# Automatic Issues Triaging with Llama
+
+This tool utilizes an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository. It serves as a reference implementation for using Llama to develop custom reporting and data analytics applications.
+
+## Features
+
+The tool performs the following tasks:
+
+* Fetches issue threads from a specified repository
+* Analyzes issue discussions and generates annotations such as category, severity, component affected, etc.
+* Categorizes all issues by theme
+* Synthesizes key challenges faced by users, along with probable causes and remediations
+* Generates a high-level executive summary providing insights on diagnosing and improving the developer experience
+
+For a step-by-step look, check out the [walkthrough notebook](walkthrough.ipynb).
+
+## Getting Started
+
+
+### Installation
+
+```bash
+pip install -r requirements.txt
+```
+
+### Setup
+
+1. **API Keys and Model Service**: Set your GitHub token for API calls. Some privileged information may not be available if you don't have push-access to the target repository.
+2. **Model Configuration**: Set the appropriate values in the `model` section of [config.yaml](config.yaml) for using Llama via VLLM or Groq.
+3. **JSON Schemas**: Edit the output JSON schemas in [config.yaml](config.yaml) to ensure consistency in outputs. VLLM supports JSON-decoding via the `guided_json` generation argument, while Groq requires passing the schema in the system prompt.
+
+### Running the Tool
+
+```bash
+python triage.py --repo_name='meta-llama/llama-recipes' --start_date='2024-08-14' --end_date='2024-08-27'
+```
+
+### Output
+
+The tool generates:
+
+* CSV files with `annotations`, `challenges`, and `overview` data, which can be persisted in SQL tables for downstream analyses and reporting.
+* Graphical matplotlib plots of repository traffic, maintenance activity, and issue attributes.
+* A PDF report for easier reading and sharing.
+
+## Config
+
+The tool's configuration is stored in [config.yaml](config.yaml). The following sections can be edited:
+
+* **Github Token**: Use a token that has push-access on the target repo.
+* **model**: Specify the model service (`vllm` or `groq`) and set the endpoints and API keys as applicable.
+* **prompts**: For each of the 3 tasks Llama does in this tool, we specify a prompt and an output JSON schema:
+  * `parse_issue`: Parsing and generating annotations for the issues 
+  * `assign_category`: Assigns each issue to a category specified in an enum in the corresponding JSON schema
+  * `get_overview`: Generates a high-level executive summary and analysis of all the parsed and generated data
+
+## Troubleshooting
+
+* If you encounter issues with API calls, ensure that your GitHub token is set correctly and that you have the necessary permissions.
+* If you encounter issues with the model service, check the configuration values in [config.yaml](config.yaml).

recipes/use_cases/github_triage/config.yaml

@@ -1,18 +1,16 @@
-tokens:
-  github: <github token>
+github_token: <github token>
 model:
   use: groq
   vllm:
       endpoint: "http://localhost:8000/v1"
-      key: token
       model_id: "meta-llama/Meta-Llama-3.1-70B-Instruct"
   groq:
       key: <groq token>
       model_id: llama-3.1-70b-versatile
 
 prompts:
   parse_issue:
-    system: You are an expert open-source maintainer of an AI open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can.
+    system: You are an expert maintainer of an open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can.
     json_schema: '{
         "type": "object",
         "properties": {
@@ -60,7 +58,7 @@ prompts:
         "required": ["summary", "possible_causes", "remediations", "component", "sentiment", "issue_type", "severity", "op_expertise"]
       }'
   assign_category:
-    system: "You are the lead maintainer of an AI open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme. Some themes you should use if applicable: Cloud Compute, Installation and Environment, Model Loading, Model Fine-tuning and Training, Model Conversion, Model Inference, Distributed Training and Multi-GPU, Performance and Optimization, Quantization and Mixed Precision, Documentation, CUDA Compatibility, Model Evaluation and Benchmarking, Miscellaneous, Invalid."
+    system: "You are the lead maintainer of an open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme."
     json_schema: '{
         "type": "object",
         "properties": {

recipes/use_cases/github_triage/llm.py

@@ -7,7 +7,8 @@
 from openai import OpenAI
 import groq
 
-log = logging.getLogger(__name__)
+logger = logging.getLogger(__name__)
+logger.addHandler(logging.StreamHandler())
 CFG = yaml.safe_load(open("config.yaml", "r"))
 
 class LlamaVLLM():
@@ -47,7 +48,7 @@ class LlamaGroq():
     def __init__(self, key, model_id):
         self.model_id = model_id
         self.client = groq.Groq(api_key=key)
-        print(f"Using Groq:{self.model_id} for inference")
+        logger.debug(f"Using Groq:{self.model_id} for inference")
 
     def chat(
         self, 
@@ -78,13 +79,13 @@ def chat(
                 output = completion.choices[0].message.content
                 break
             except groq.RateLimitError as e:
-                wait = response.headers['X-Ratelimit-Reset']
+                wait = e.response.headers['X-Ratelimit-Reset']
                 response = e.response
                 print(e)
-                print(f"waiting for {wait} to prevent ratelimiting")
+                print(f"[groq] waiting for {wait} to prevent ratelimiting")
                 time.sleep(wait)
-            except:
-                print(f"inference failed for input: {inputs}")
+            except Exception as e:
+                logger.error(f"INFERENCE FAILED with Error: {e.response.status_code}! for input:\n{inputs[-1]['content'][:300]}")
 
         return output
 
@@ -107,7 +108,6 @@ def run_llm_inference(
     Returns:
     - Union[str, List[str]]: The response(s) from the LLM.
     """
-    log.info(f"[run_llm_inference] {prompt_name}")
 
     # initialize appropriate LLM accessor
     if CFG['model']['use'] == 'vllm':
@@ -117,6 +117,8 @@ def run_llm_inference(
     else:
         raise ValueError("Invalid model type in config.yaml")
 
+    logger.debug(f"Running `{prompt_name}` inference with {CFG['model']['use']}")
+    
     _batch = True
     if isinstance(inputs, str):
         _batch = False
@@ -150,7 +152,7 @@ def run_llm_inference(
                     responses_json.append(json.loads(r, strict=False))
                     continue
                 except json.JSONDecodeError:
-                    log.error(f"Error decoding JSON: {r}")
+                    logger.error(f"Error decoding JSON: {r}")
             responses_json.append(None)
         responses = responses_json