Add recipe for Llama Triaging & Reporting Tool (meta-llama#651)

Author: subramen

.github/scripts/spellcheck_conf/wordlist.txt

@@ -1451,6 +1451,10 @@ openhathi
 sarvam
 subtask
 acc
+Triaging
+matplotlib
+remediations
+walkthrough
 OCRVQA
 OCRVQADataCollator
 ocrvqa

recipes/use_cases/README.md

@@ -1,3 +1,6 @@
+## [Automatic Triaging of Github Repositories](./github_triage/walkthrough.ipynb): Use Llama to automatically triage issues in an OSS repository and generate insights to improve community experience
+This tool utilizes an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository. It serves as a reference implementation for using Llama to develop custom reporting and data analytics applications.
+
 ## [VideoSummary](video_summary.ipynb): Ask Llama 3 to Summarize a Long YouTube Video (using Replicate or [OctoAI](../3p_integrations/octoai/video_summary.ipynb))
 This demo app uses Llama 3 to return a text summary of a YouTube video. It shows how to retrieve the caption of a YouTube video and how to ask Llama to summarize the content in different ways, from the simplest naive way that works for short text to more advanced methods of using LangChain's map_reduce and refine to overcome the 8K context length limit of Llama 3.
 

recipes/use_cases/github_triage/README.md

@@ -0,0 +1,60 @@
+# Automatic Issues Triaging with Llama
+
+This tool utilizes an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository. It serves as a reference implementation for using Llama to develop custom reporting and data analytics applications.
+
+## Features
+
+The tool performs the following tasks:
+
+* Fetches issue threads from a specified repository
+* Analyzes issue discussions and generates annotations such as category, severity, component affected, etc.
+* Categorizes all issues by theme
+* Synthesizes key challenges faced by users, along with probable causes and remediations
+* Generates a high-level executive summary providing insights on diagnosing and improving the developer experience
+
+For a step-by-step look, check out the [walkthrough notebook](walkthrough.ipynb).
+
+## Getting Started
+
+
+### Installation
+
+```bash
+pip install -r requirements.txt
+```
+
+### Setup
+
+1. **API Keys and Model Service**: Set your GitHub token for API calls. Some privileged information may not be available if you don't have push-access to the target repository.
+2. **Model Configuration**: Set the appropriate values in the `model` section of [config.yaml](config.yaml) for using Llama via VLLM or Groq.
+3. **JSON Schemas**: Edit the output JSON schemas in [config.yaml](config.yaml) to ensure consistency in outputs. VLLM supports JSON-decoding via the `guided_json` generation argument, while Groq requires passing the schema in the system prompt.
+
+### Running the Tool
+
+```bash
+python triage.py --repo_name='meta-llama/llama-recipes' --start_date='2024-08-14' --end_date='2024-08-27'
+```
+
+### Output
+
+The tool generates:
+
+* CSV files with `annotations`, `challenges`, and `overview` data, which can be persisted in SQL tables for downstream analyses and reporting.
+* Graphical matplotlib plots of repository traffic, maintenance activity, and issue attributes.
+* A PDF report for easier reading and sharing.
+
+## Config
+
+The tool's configuration is stored in [config.yaml](config.yaml). The following sections can be edited:
+
+* **Github Token**: Use a token that has push-access on the target repo.
+* **model**: Specify the model service (`vllm` or `groq`) and set the endpoints and API keys as applicable.
+* **prompts**: For each of the 3 tasks Llama does in this tool, we specify a prompt and an output JSON schema:
+  * `parse_issue`: Parsing and generating annotations for the issues 
+  * `assign_category`: Assigns each issue to a category specified in an enum in the corresponding JSON schema
+  * `get_overview`: Generates a high-level executive summary and analysis of all the parsed and generated data
+
+## Troubleshooting
+
+* If you encounter issues with API calls, ensure that your GitHub token is set correctly and that you have the necessary permissions.
+* If you encounter issues with the model service, check the configuration values in [config.yaml](config.yaml).

recipes/use_cases/github_triage/config.yaml

@@ -0,0 +1,144 @@
+github_token: <github token>
+model:
+  use: groq
+  vllm:
+      endpoint: "http://localhost:8000/v1"
+      model_id: "meta-llama/Meta-Llama-3.1-70B-Instruct"
+  groq:
+      key: <groq token>
+      model_id: llama-3.1-70b-versatile
+
+prompts:
+  parse_issue:
+    system: You are an expert maintainer of an open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can.
+    json_schema: '{
+        "type": "object",
+        "properties": {
+          "summary": {
+            "description": "Summary of the issue and discussion along with any details about platform or tooling.",
+            "type": "string"
+          },
+          "possible_causes": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "remediations": {
+            "description": "How we can improve the code or documentation to prevent this issue.",
+            "type": "array",
+            "maxItems": 2,
+            "items": {
+              "type": "string"
+            }
+          },
+          "component": {
+            "description": "The specific module or component affected by the issue",
+            "type": "string"
+          },
+          "sentiment": {
+            "type": "string",
+            "enum": ["positive", "negative", "neutral"]
+          },
+          "issue_type": {
+            "description": "Any issue not related to LLMs, Llama or code in this repository should be marked as \"invalid\"",
+            "type": "string",
+            "enum": ["bug_report", "feature_request", "documentation", "installation", "discussion", "invalid"]
+          },
+          "severity": {
+            "type": "string",
+            "enum": ["critical", "major", "minor", "trivial"]
+          },
+          "op_expertise": {
+            "description": "Assess the reporters level of expertise.",
+            "type": "string",
+            "enum": ["beginner", "intermediate", "advanced"]
+          }
+        },
+        "required": ["summary", "possible_causes", "remediations", "component", "sentiment", "issue_type", "severity", "op_expertise"]
+      }'
+  assign_category:
+    system: "You are the lead maintainer of an open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme."
+    json_schema: '{
+        "type": "object",
+        "properties": {
+          "report": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "theme": {
+                  "description": "key theme identified from the issues",
+                  "type": "string",
+                  "enum": ["Cloud Compute", "Installation and Environment", "Model Loading", "Model Fine-tuning and Training", "Model Conversion", "Model Inference", "Distributed Training and Multi-GPU", "Performance and Optimization", "Quantization and Mixed Precision", "Documentation", "CUDA Compatibility", "Model Evaluation and Benchmarking", "Miscellaneous", "Invalid"]
+                },
+                "description": {
+                  "type": "string"
+                },
+                "related_issues": {
+                  "description": "Issue numbers related to this theme",
+                  "type": "array",
+                  "items": {
+                    "type": "number"
+                  }
+                }
+              },
+              "required": ["theme", "description", "related_issues"]
+            }
+          }
+        },
+        "required": ["report"]
+      }'
+  get_overview:
+    system: You are not only an experienced Open Source maintainer, but also an expert at paraphrasing raw data into clear succinct reports. Draft a concise report about the issues in this open source repository. Include an executive summary that provides an overview of the challenges faced, any open questions or decisions to be made, or actions that we can take. Group issues together if they ladder up to the same overall challenge, summarize the challenges and include any actionable resolutions we can take (more information in the \"remediations\" sections). Use your experience and judgement to ignore issues that are clearly unrelated to the open source project. Ensure the output is in JSON.
+    json_schema: '{
+        "type": "object",
+        "properties": {
+          "executive_summary": {
+            "description": "An executive summary of the analysis",
+            "type": "string"
+          },
+          "open_questions": {
+            "description": "Any open questions or decisions that the product team needs to make in light of these issues",
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "issue_analysis": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "key_challenge": {
+                  "description": "A description of the challenge reported in these issues",
+                  "type": "string"
+                },
+                "affected_issues": {
+                  "description": "A list of issues that are related to this challenge",
+                  "type": "array",
+                  "items": {
+                    "type": "number"
+                  }
+                },
+                "possible_causes": {
+                  "description": "A list of possible causes or reasons for this challenge to occur",
+                  "type": "array",
+                  "items": {
+                    "type": "string"
+                  }
+                },
+                "remediations": {
+                  "description": "Steps we can take to address this challenge",
+                  "type": "array",
+                  "items": {
+                    "type": "string"
+                  }
+                }
+              },
+              "required": ["key_challenge", "affected_issues", "possible_causes", "remediations"]
+            }
+          }
+        },
+        "required": ["issue_analysis", "open_questions", "actions", "executive_summary"]
+      }'

recipes/use_cases/github_triage/llm.py

@@ -0,0 +1,165 @@
+import logging 
+from typing import Any, Dict, List, Optional, Union
+import yaml
+import time
+import json
+
+from tqdm import tqdm
+from openai import OpenAI
+import groq
+
+logger = logging.getLogger(__name__)
+logger.addHandler(logging.StreamHandler())
+CFG = yaml.safe_load(open("config.yaml", "r"))
+
+class LlamaVLLM():
+    def __init__(self, endpoint, model_id):
+        self.model_id = model_id
+        self.client = OpenAI(base_url=endpoint, api_key='token')
+
+    def chat(
+        self,
+        inputs: List[Dict[str, str]],
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        guided_decode_json_schema: Optional[str] = None
+    ) -> List[str]:
+
+        if generation_kwargs is None:
+            generation_kwargs = {}
+            
+        try:
+            response = self.client.chat.completions.create(
+                model=self.model,
+                messages=inputs,
+                extra_body={
+                    "guided_json": guided_decode_json_schema
+                },
+                **generation_kwargs,
+            )
+            output = response.choices[0].message
+        except Exception as e:
+            logger.error(
+                f"FAILED to generate inference for input {inputs}\nError: {str(e)}"
+            )
+            output = None
+        return output
+    
+
+class LlamaGroq():
+    def __init__(self, key, model_id):
+        self.model_id = model_id
+        self.client = groq.Groq(api_key=key)
+        logger.debug(f"Using Groq:{self.model_id} for inference")
+
+    def chat(
+        self, 
+        inputs: List[Dict[str, str]], 
+        generation_kwargs: Optional[Dict[str, Any]] = None,
+        guided_decode_json_schema: Optional[str] = None
+    ) -> str:
+        
+        if generation_kwargs is None:
+            generation_kwargs = {}
+            
+        # Currently Groq doesn't support guided JSON decoding. Workaround:
+        if guided_decode_json_schema is not None:
+            inputs[0]['content'] += f"\n\nEnsure your response aligns with the following JSON schema:\n{guided_decode_json_schema}\n\n"
+        
+        output = None
+        
+        while True:
+            try:
+                response = self.client.chat.completions.with_raw_response.create(
+                    model=self.model_id,
+                    messages=inputs,
+                    stream=False,
+                    **generation_kwargs,
+                    response_format={"type": 'json_object' if guided_decode_json_schema is not None else 'text'}
+                )
+                completion = response.parse()
+                output = completion.choices[0].message.content
+                break
+            except groq.RateLimitError as e:
+                wait = e.response.headers['X-Ratelimit-Reset']
+                response = e.response
+                print(e)
+                print(f"[groq] waiting for {wait} to prevent ratelimiting")
+                time.sleep(wait)
+            except Exception as e:
+                logger.error(f"INFERENCE FAILED with Error: {e.response.status_code} for input:\n{inputs[-1]['content'][:300]}")
+                break
+
+        return output
+
+
+def run_llm_inference(
+    prompt_name: str,
+    inputs: Union[str, List[str]],
+    generation_kwargs: Optional[Dict] = None,
+    guided_decode_json_schema=None,
+) -> Union[List[str], List[Dict[str, Any]]]:
+    """
+    Run the LLM inference on the given inputs.
+
+    Args:
+    - prompt_name (str): The name of the prompt to use.
+    - inputs (str or List[str]): The input(s) to the LLM.
+    - generation_kwargs (Dict): Additional keyword arguments to pass to the LLM.
+    - guided_decode_json_schema (str): The JSON schema to use for guided decoding.
+
+    Returns:
+    - Union[str, List[str]]: The response(s) from the LLM.
+    """
+    
+    # initialize appropriate LLM accessor
+    if CFG['model']['use'] == 'vllm':
+        LLM = LlamaVLLM(**CFG['model']['vllm'])
+    elif CFG['model']['use'] == 'groq':
+        LLM = LlamaGroq(**CFG['model']['groq'])
+    else:
+        raise ValueError("Invalid model type in config.yaml")
+    
+    logger.debug(f"Running `{prompt_name}` inference with {CFG['model']['use']}")
+    
+    _batch = True
+    if isinstance(inputs, str):
+        _batch = False
+        inputs = [inputs]
+
+    inputs = [
+        [
+            {"role": "system", "content": CFG["prompts"][prompt_name]["system"]},
+            {"role": "user", "content": i},
+        ]
+        for i in inputs
+    ]
+
+    if (
+        guided_decode_json_schema is None
+        and "json_schema" in CFG["prompts"][prompt_name]
+    ):
+        guided_decode_json_schema = " ".join(
+            CFG["prompts"][prompt_name]["json_schema"].split()
+        )
+
+    responses = [
+        LLM.chat(i, generation_kwargs, guided_decode_json_schema) 
+        for i in tqdm(inputs, desc=f"Inference[{prompt_name}]")
+    ]
+
+    if guided_decode_json_schema is not None:
+        responses_json = []
+        for r in responses:
+            if r is not None:
+                try:
+                    responses_json.append(json.loads(r, strict=False))
+                    continue
+                except json.JSONDecodeError:
+                    logger.error(f"Error decoding JSON: {r}")
+            responses_json.append(None)
+        responses = responses_json
+
+    if not _batch:
+        responses = responses[0]
+
+    return responses