diff --git a/docs/genai/01_getting_started/01_intro.mdx b/docs/genai/01_getting_started/01_intro.mdx new file mode 100644 index 0000000000..12aaf3136c --- /dev/null +++ b/docs/genai/01_getting_started/01_intro.mdx @@ -0,0 +1,14 @@ +# Pythia + +:::warning[Non-research workflows] +If you're looking to harness Generative AI for administrative or classroom use, please reach out to genai-support@nyu.edu +::: + +Welcome to Pythia, the generative AI platform for research workflows. As part of the Pythia platform, the following capabilities are offered: +- [Access to externally hosted LLMs](../02_external_llms/01_llm_access.mdx) +- [HPC resources for fine tuning LLMs](../03_llm_fine_tuning/01_intro.md) +- [Milvus vector database](../04_vector_databases/01_intro.md) + +:::tip[Personal use] +If you want to access NYU provided LLMs for personal use, proceed to https://gemini.google.com/app with your NYU credentials. +::: diff --git a/docs/genai/01_getting_started/_category_.json b/docs/genai/01_getting_started/_category_.json new file mode 100644 index 0000000000..41f4c00e71 --- /dev/null +++ b/docs/genai/01_getting_started/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Getting Started" +} diff --git a/docs/genai/02_external_llms/01_llm_access.mdx b/docs/genai/02_external_llms/01_llm_access.mdx new file mode 100644 index 0000000000..829359c62c --- /dev/null +++ b/docs/genai/02_external_llms/01_llm_access.mdx @@ -0,0 +1,50 @@ +# Portkey + +[Portkey](https://portkey.ai/docs/introduction/what-is-portkey) is an enterprise-grade LLM gateway running on-prem: + +![Portkey Architecture](./static/portkey_arch.png) + +Broadly it facilitates: +- **Route requests** via configurations that can allow for fallbacks, load balance, etc +- **Observability** via the control pane that displays your usage statistics and logs. You can retrive your logs via an API. +- **Prompt management** via the prompt playground. You can compare the respone for a prompt across multiple LLMs, collaborate with your team and export your prompts. +- **Guardrails** usage by allowing you to define them or integrate with third party guardrails. +- **Agentic workflows** by integrating with various agentic frameworks like langchain, llamaindex, etc. +- **Security & Governance** by allowing you to set budget and rate limits on the API keys created for your workspace + +:::tip[Gateway URL] +Whenever you instantiate a Portkey client, the `base_url` must be set to `base_url="https://ai-gateway.apps.cloud.rt.nyu.edu/v1/"`. If you miss this parameter you would be connecting to the vendor's SaaS platform and NYU provisioned virtual keys will not work. +::: + +## Onboarding +Send an email to `genai-support@nyu.edu` to start the onboarding process. + +## Getting started with Portkey +As part of the onboarding process, you would have received an invite which gives you access to a workspace. We will also add virtual keys for LLMs to your workspace as part of the onboarding process. Once you've accepted it, head over to `https://app.portkey.ai/` and select the sign-in with Single Sign-On option and proceed with your NYU email address. + +:::danger[Access to Portkey is only permitted via NYU VPN] +You need to be connected to the NYU VPN to access the Portkey LLM gateway. If you are not, your requests will timeout and result in connection errors. +::: + +You will now be able to create an API key for yourself by access the `API Keys` item on the left sidebar. With an API key and a virtual key at your disposal, you can now run the following script: +```python +from portkey_ai import Portkey + +portkey = Portkey( + base_url="https://ai-gateway.apps.cloud.rt.nyu.edu/v1/", + api_key="", # Replace with your Portkey API key + virtual_key="", # Replace with your virtual key for Google +) + +completion = portkey.chat.completions.create( + messages=[ + {"role": "system", "content": "You are not a helpful assistant"}, + {"role": "user", "content": "Say this is a test"}, + ], + model="gemini-2.0-flash-lite-preview-02-05", +) + +print(completion) +``` + +Once the script is executed, you can head back to `app.portkey.ai` to view the logs for the call! diff --git a/docs/genai/02_external_llms/02_catalogue.md b/docs/genai/02_external_llms/02_catalogue.md new file mode 100644 index 0000000000..6e94829d37 --- /dev/null +++ b/docs/genai/02_external_llms/02_catalogue.md @@ -0,0 +1,13 @@ +# LLM Catalogue + +We currently facilitate access to the following externally hosted LLMs: + +## OpenAI +- gpt-4o-mini +- o3-mini +- text-embedding-3-small + +## VertexAI +- Gemini-2.0 models (flash, flash-lite) +- Gemini-1.5 models (flash, pro) +For a comprehensive list, please [refer to the VertexAI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models). diff --git a/docs/genai/02_external_llms/03_playground.md b/docs/genai/02_external_llms/03_playground.md new file mode 100644 index 0000000000..f2f1c5b4db --- /dev/null +++ b/docs/genai/02_external_llms/03_playground.md @@ -0,0 +1,3 @@ +# OpenWebUI + +We are working on providing an NYU hosted instance of [OpenWebUI](https://docs.openwebui.com/). More details about this will be provided soon. diff --git a/docs/genai/02_external_llms/_category_.json b/docs/genai/02_external_llms/_category_.json new file mode 100644 index 0000000000..e0e1effe87 --- /dev/null +++ b/docs/genai/02_external_llms/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Access externally hosted LLMs" +} diff --git a/docs/genai/02_external_llms/static/portkey_arch.png b/docs/genai/02_external_llms/static/portkey_arch.png new file mode 100644 index 0000000000..0cdcf8cb08 Binary files /dev/null and b/docs/genai/02_external_llms/static/portkey_arch.png differ diff --git a/docs/genai/03_llm_fine_tuning/01_intro.md b/docs/genai/03_llm_fine_tuning/01_intro.md new file mode 100644 index 0000000000..fad487c474 --- /dev/null +++ b/docs/genai/03_llm_fine_tuning/01_intro.md @@ -0,0 +1,3 @@ +# Fine tuning + +You can use HPC for this. But for most cases, RAG might suffice. Please look into harnessing RAG before attempting to fine-tune a model. diff --git a/docs/genai/03_llm_fine_tuning/_category_.json b/docs/genai/03_llm_fine_tuning/_category_.json new file mode 100644 index 0000000000..3b49e8d5ea --- /dev/null +++ b/docs/genai/03_llm_fine_tuning/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Fine-tuning LLMs" +} diff --git a/docs/genai/04_vector_databases/01_intro.md b/docs/genai/04_vector_databases/01_intro.md new file mode 100644 index 0000000000..1e3d5d47d1 --- /dev/null +++ b/docs/genai/04_vector_databases/01_intro.md @@ -0,0 +1,3 @@ +# Vector Databases + +What is it? How is it different from a regular database? diff --git a/docs/genai/04_vector_databases/_category_.json b/docs/genai/04_vector_databases/_category_.json new file mode 100644 index 0000000000..81514c64bb --- /dev/null +++ b/docs/genai/04_vector_databases/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Vector Databases" +} diff --git a/docs/genai/intro.md b/docs/genai/intro.md deleted file mode 100644 index 65c46f2fc4..0000000000 --- a/docs/genai/intro.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Generative AI for Research workflows - -