Initial commit

Author: Jinyu Wang

.gitignore

@@ -0,0 +1,150 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+###################################
+## Specific setting for this repo
+###################################
+.vscode/
+env_configs/
+data/
+logs/
+
+# May enable in the future
+notebooks/
+test/

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns

LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

RAI_TRANSPARENCY.md

@@ -0,0 +1,55 @@
+# PIKE-RAG: Responsible AI FAQ
+
+## What is PIKE-RAG?
+
+PIKE-RAG, or PrIvate KnowledgE and Rationale Augmentation Generation, represents an advanced evolution in AI-assisted content interpretation, tailored for industrial applications requiring private knowledge and domain-specific rationale. Unlike conventional Retrieval-Augmented Generation (RAG) systems which primarily rely on retrieval to inform large language models (LLMs), PIKE-RAG introduces a methodology that integrates private knowledge extraction with the generation of a coherent rationale. This technique enables the LLMs to progressively navigate towards more accurate and contextually relevant responses. By parsing data to create detailed knowledge structures akin to a heterogeneous knowledge graph, and guiding LLMs to construct coherent rationale in a knowledge-aware manner, PIKE-RAG surpasses traditional models by extracting, understanding, linking and applying private knowledge that is not readily accessible through standard retrieval processes.
+
+## What can PIKE-RAG do?
+
+The strength of PIKE-RAG lies in its sophisticated ability to link disparate pieces of information across extensive data sets, thereby facilitating the answering of complex queries that are beyond the reach of typical keyword and vector-based search methods. By constructing a heterogeneous knowledge graph from a user-provided private dataset, PIKE-RAG can adeptly navigate through a sea of information to locate and synthesize content that addresses user questions, even when the answers are scattered across multiple documents. Moreover, PIKE-RAG's advanced functionality enables it to tackle thematic inquiries, such as identifying prevailing motifs or patterns within a dataset. This capacity to handle both specific, cross-referenced questions and broader, thematic ones makes PIKE-RAG a powerful tool for extracting actionable insights from large volumes of data.
+
+## What is/are PIKE-RAG's intended use(s)?
+
+PIKE-RAG is an advanced system crafted to revolutionize how large language models assist in complex industrial tasks. Here are the its intended uses:
+
+- PIKE-RAG is specifically engineered to enhance large language models in sectors where the need for deep, domain-specific knowledge is paramount, and where conventional retrieval-augmented systems fall short. Its intended use is in complex industrial applications where the extraction and application of specialized knowledge are critical for achieving accurate and insightful outcomes. By incorporating private knowledge and constructing coherent rationales, PIKE-RAG is adept at guiding LLMs to provide precise responses tailored to intricate queries.
+
+- Aimed at addressing the multifaceted challenges encountered in industrial environments, PIKE-RAG is poised for deployment in scenarios requiring a high level of logical reasoning and the ability to navigate through specialized corpora. It is ideal for use cases that demand not just information retrieval but also a deep understanding of that information and its application in a logical, reasoned manner. This makes PIKE-RAG particularly valuable for tasks where decision-making is heavily reliant on exclusive, industry-specific insights.
+
+- With a focus on the incremental improvement of RAG systems, PIKE-RAG's deployment strategy is designed to systematically tackle tasks based on their knowledge complexity. This approach ensures that as industrial demands evolve, PIKE-RAG can develop alongside, continuously enhancing the decision-making capabilities of LLMs and maintaining relevance and accuracy in the context of the industry's specific needs.
+
+- PIKE-RAG is structured to operate within the framework of existing industry protocols, but users must ensure that it is used in conjunction with responsible analytic practices. The system is designed to augment, not replace, human expertise, and therefore it is essential for domain experts to critically evaluate and interpret PIKE-RAG's outputs, verifying their validity and applicability to the specific context.
+
+- While PIKE-RAG does not independently collect user data, it functions within the larger ecosystem of data processing and management. Users are advised to be mindful of data privacy concerns and are encouraged to review the privacy policies associated with the large language models and data storage solutions interfacing with PIKE-RAG. It is the responsibility of the user to ensure that the use of PIKE-RAG complies with all relevant data protection regulations and organizational guidelines.
+
+## How was PIKE-RAG evaluated? What metrics are used to measure performance?
+
+PIKE-RAG's performance and utility were rigorously assessed through a comprehensive evaluation strategy that employed multiple metrics to measure different aspects of its capabilities. Here's how PIKE-RAG was evaluated and the metrics used to gauge its performance:
+
+- Accuracy Metrics: One fundamental way PIKE-RAG was evaluated is by its performance on public datasets. Metrics such as Exact Match (EM), F1 score, Precision, Recall, and an automatically LLM-powered Accuracy metric were utilized for datasets that have ground-truth labels. These metrics are critical as they offer an objective measure of PIKE-RAG's ability to provide correct answers. Manual checking over random samples from these testing data underscores the reliability of these metrics.
+
+- Transparency and Rationale Assessment: To evaluate the transparency and reliability of the responses generated by PIKE-RAG, the system's outputs were closely examined in conjunction with the reference corpus they were derived from. This evaluation focused on the coherence and relevance of the rationale embedded within the responses, ensuring that the system's logic is traceable and grounded in the source material.
+
+- Hallucination Rate Measurement: The credibility of the rationales provided in PIKE-RAG's responses was subjected to a meticulous manual review, where each sentence was evaluated to ascertain whether the underlying logic could be sourced to any part of the reference corpus.
+
+- Resilience Testing: PIKE-RAG's robustness against various forms of adversarial attacks was tested, including prompt and data corpus injection attacks. These tests, which encompassed both manual and semi-automated techniques, aimed to probe the system's defenses against user prompt injection attacks ("jailbreaks") and cross prompt injection attacks ("data attacks").
+
+## What are the limitations of PIKE-RAG? How can users minimize the impact of PIKE-RAG’s limitations when using the system?
+
+PIKE-RAG, like any advanced system, comes with certain limitations that users should be aware of in order to effectively utilize the system within its operational boundaries. The open-source version of PIKE-RAG is crafted with a general-purpose approach in mind. As a result, while it outperforms other untrained and unadjusted methods across the datasets being tested, it may not deliver the optimal performance possible within a specific domain. This is because it is not fine-tuned to the unique nuances and specialized requirements that certain domains may present.
+
+Users can mitigate the limitations posed by PIKE-RAG's generalist nature by introducing domain-specific customizations. For instance, when deploying PIKE-RAG in a particular domain, users can enhance performance by tailoring the corpus pre-processing steps to better reflect the domain's specificities, thereby ensuring that the knowledge base PIKE-RAG draws from is highly relevant and fine-tuned. Additionally, during the rationale generation phase, users can incorporate domain-specific demonstrations or templates that guide the system to construct rationales that are more aligned with domain expertise and practices. This bespoke approach can significantly improve the relevance and accuracy of the system's outputs, making them more actionable and trustworthy for domain-specific applications.
+
+By understanding and addressing these limitations through targeted domain-specific adjustments, users can effectively harness the power of PIKE-RAG and reduce the impact of its constraints, thus maximizing the system's utility and performance in specialized contexts.
+
+## What operational factors and settings allow for effective and responsible use of PIKE-RAG?
+
+To ensure effective and responsible use of PIKE-RAG, it is important to consider the following operational factors and settings:
+
+- User Expertise: The system is intended for use by trusted individuals who possess the necessary domain sophistication. These users are expected to have experience in handling intricate information tasks, which is crucial for the proper interpretation and analysis of PIKE-RAG's outputs.
+
+- Human Analysis and Provenance Tracking: While PIKE-RAG is generally robust against injection attacks and is capable of identifying conflicting information sources, it is imperative that human analysts perform a thorough review of the system's responses. Reliable insights are generated when users critically evaluate the provenance of the information and ensure that the inferences made by PIKE-RAG during answer generation align with human judgment and domain-specific knowledge.
+
+- Resilience and Safety Considerations: Although PIKE-RAG has undergone rigorous testing for its resilience to various types of adversarial attacks, there is still the possibility that the LLM configured with PIKE-RAG might produce content that is inappropriate or offensive. This is especially relevant when deploying the system in sensitive contexts. To mitigate such risks, developers must carefully assess the outputs within the context of their specific use case. Implementing additional safeguards is crucial, and these may include using safety classifiers, model-specific safety features, and services like [Azure AI Content Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety), or developing bespoke solutions that are tailored to the unique requirements of the application.
+
+By acknowledging these operational factors and incorporating the appropriate settings and checks, users can leverage PIKE-RAG in a manner that is both effective and aligned with ethical and responsible AI practices. This careful approach ensures that the system's capabilities are harnessed to their full potential while minimizing the risk of misuse or the propagation of harmful content.

README.md

@@ -0,0 +1,112 @@
+# PIKE-RAG: PrIvate KnowledgE and Rationale Augmented Generation
+
+## Quick Start
+
+Please set your `PYTHONPATH` before running the scripts:
+
+### Windows
+
+```powershell
+$Env:PYTHONPATH=PATH-TO-THIS-REPO
+
+# If you exactly under this repository directory, you can do it by
+$Env:PYTHONPATH=$PWD
+```
+
+### Linux / Mac OS
+
+```sh
+export PYTHONPATH=PATH-TO-THIS-REPO
+
+# If you are exactly under the repository directory, you can do it by
+export PYTHONPATH=$PWD
+```
+
+## .env File
+
+Please follow below environment configuration variable names to create your *.env* file, we suggest you put it under
+`PIKE-RAG/env_configs/` which has already been added to *.gitignore* file:
+
+### For Azure OpenAI Client
+
+```sh
+AZURE_OPENAI_ENDPOINT = "YOUR-ENDPOINT(https://xxx.openai.azure.com/)"
+OPENAI_API_TYPE = "azure"
+OPENAI_API_VERSION = "2023-07-01-preview"
+```
+
+*Note that the way to access GPT API with key is disabled in Azure now.*
+
+To access GPT resource from Azure, please remember to login to Azure CLI using your *SC-* account:
+
+```sh
+# Install Azure-CLI and other dependencies. Sudo permission is required.
+bash scripts/install_az.sh
+
+# Login Azure CLI using device code.
+bash scripts/login_az.sh
+```
+
+### For Azure Meta LlaMa Client
+
+Since the endpoint and API keys varied among different LlaMa models, you can add multiple
+(`llama_endpoint_name`, `llama_key_name`) pairs you want to use into the *.env* file, and specify the names when
+initializing `AzureMetaLlamaClient` (you can modify the llm client args in the YAML files). If `null` is set to be the
+name, the (`LLAMA_ENDPOINT`, `LLAMA_API_KEY`) would be used as the default environment variable name.
+
+```sh
+# Option 1: Set only one pair in one time, update these variables every time you want to change the LlaMa model.
+LLAMA_ENDPOINT = "YOUR-LLAMA-ENDPOINT"
+LLAMA_API_KEY = "YOUR-API-KEY"
+
+# Option 2: Add multiple pairs into the .env file, for example:
+LLAMA3_8B_ENDPOINT = "..."
+LLAMA3_8B_API_KEY = "..."
+
+LLAMA3_70B_ENDPOINT = "..."
+LLAMA3_70B_API_KEY = "..."
+```
+
+#### Ways to Get the Available Azure Meta LLaMa **Endpoints**, **API Keys** and **Model Names**
+
+The way we have implemented the LLaMa model so far involves requesting the deployed model on the GCR server. You can
+find the available settings follow the steps below:
+
+1. Open [Azure Machine Learning Studio](https://ml.azure.com/home), sign in may be required;
+2. Click *Workspaces* on the left side (expand the menu by clicking the three horizontal lines in the top left corner if
+you cannot find it);
+3. Choose and click on a valid workspace, e.g., *gcrllm2ws*;
+4. Click *Endpoints* on the left side (expand the menu by clicking the three horizontal lines in the top left corner if
+you cannot find it), You can find the available model list in this page;
+5. Choose and click the model you want to use, e.g., *gcr-llama-3-8b-instruct*:
+    - **model** name: in tab "Details", scroll to find "Deployment summary", the *Live traffic allocation* string (e.g.,
+        *meta-llama-3-8b-instruct-4*) is the model name you need to set up in your YAML file;
+    - **LLAMA_ENDPOINT** & **LLAMA_API_KEY**: can be found in tab "Consume".
+
+#### Handling the Issue "Specified deployment could not be found"
+
+If you get error message "Specified deployment could not be found", it indicates that the GCR team has changed the
+server deployment location. In this case, you need to check the available model list in
+[Azure Machine Learning Studio](https://ml.azure.com/home) and update the YAML config again.
+
+## Contributing
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+
+When you submit a pull request, a CLA bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
+trademarks or logos is subject to and must follow
+[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
+Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
+Any use of third-party trademarks or logos are subject to those third-party's policies.