Merge branch 'main' into amyles

Author: amyles

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/README.md

@@ -0,0 +1,99 @@
+# Calling multiple vLLM inference servers using LiteLLM
+
+In this tutorial we explain how to use a LiteLLM Proxy Server to call multiple LLM inference endpoints from a single interface. LiteLLM interacts will 100+ LLMs such as OpenAI, Cohere, NVIDIA Triton and NIM, etc. Here we will use two vLLM inference servers.
+
+<!-- ![Hybrid shards](assets/images/litellm.png "LiteLLM") -->
+
+# When to use this asset?
+
+To run the inference tutorial with local deployments of Mistral 7B Instruct v0.3 using a vLLM inference server powered by an NVIDIA A10 GPU and a LiteLLM Proxy Server on top. 
+
+# How to use this asset?
+
+These are the prerequisites to run this tutorial:
+* An OCI tenancy with A10 quota
+* A Huggingface account with a valid Auth Token
+* A valid OpenAI API Key
+
+## Introduction
+
+LiteLLM provides a proxy server to manage auth, loadbalancing, and spend tracking across 100+ LLMs. All in the OpenAI format.
+vLLM is a fast and easy-to-use library for LLM inference and serving.
+The first step will be to deploy two vLLM inference servers on NVIDIA A10 powered virtual machine instances. In the second step, we will create a LiteLLM Proxy Server on a third no-GPU instance and explain how we can use this interface to call the two LLM from a single location. For the sake of simplicity, all 3 instances will reside in the same public subnet here.
+
+![Hybrid shards](assets/images/litellm-architecture.png "LiteLLM")
+
+## vLLM inference servers deployment
+
+For each of the inference nodes a VM.GPU.A10.2 instance (2 x NVIDIA A10 GPU 24GB) is used in combination with the NVIDIA GPU-Optimized VMI image from the OCI marketplace. This Ubuntu-based image comes with all the necessary libraries (Docker, NVIDIA Container Toolkit) preinstalled. It is a good practice to deploy two instances in two different fault domains to ensure a higher availability.
+
+The vLLM inference server is deployed using the vLLM official container image.
+```
+docker run --gpus all \
+    -e HF_TOKEN=$HF_TOKEN -p 8000:8000 \
+    --ipc=host \
+    vllm/vllm-openai:latest \
+    --host 0.0.0.0 \
+    --port 8000 \
+    --model mistralai/Mistral-7B-Instruct-v0.3 \
+    --tensor-parallel-size 2 \
+    --load-format safetensors \
+    --trust-remote-code \
+    --enforce-eager
+```
+where `$HF_TOKEN` is a valid HuggingFace token. In this case we use the 7B Instruct version of Mistral LLM. The vLLM endpoint can be directly called for verification with:
+```
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "mistralai/Mistral-7B-Instruct-v0.3",
+        "messages": [
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ]
+    }' | jq
+```
+
+## LiteLLM server deployment
+
+No GPU are required for LiteLLM. Therefore, a CPU based VM.Standard.E4.Flex instance (4 OCPUs, 64 GB Memory) with a standard Ubuntu 22.04 image is used. Here LiteLLM is used as a proxy server calling a vLLM endpoint. Install LiteLLM using `pip`:
+```
+pip install 'litellm[proxy]'
+```
+Edit the `config.yaml` file (OpenAI-Compatible Endpoint):
+```
+model_list:
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://xxx.xxx.xxx.xxx:8000/v1
+      api_key: sk-0123456789
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://xxx.xxx.xxx.xxx:8000/v1
+      api_key: sk-0123456789
+```
+where `sk-0123456789` is a valid OpenAI API key and `xxx.xxx.xxx.xxx` are the two GPU instances public IP addresses.
+
+Start the LiteLLM Proxy Server with the following command:
+```
+litellm --config /path/to/config.yaml
+```
+Once the the Proxy Server is ready call the vLLM endpoint through LiteLLM with:
+```
+curl http://localhost:4000/chat/completions \
+    -H 'Authorization: Bearer sk-0123456789' \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Mistral-7B-Instruct",
+        "messages": [
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ]
+    }' | jq
+```
+
+## Documentation
+
+* [LiteLLM documentation](https://litellm.vercel.app/docs/providers/openai_compatible)
+* [vLLM documentation](https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html)
+* [MistralAI](https://mistral.ai/)

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/assets/images/litellm-architecture.png

@@ -0,0 +1,11 @@
+model_list:
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://public_ip_1:8000/v1
+      api_key: sk-0123456789
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://public_ip_2:8000/v1
+      api_key: sk-0123456789

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/config.yaml

@@ -0,0 +1,31 @@
+
+# OpenShift on OCI
+ 
+Red Hat OpenShift can be hosted on OCI as a self-run platform. Oracle provides terraform templates for easy implementation and platform integration.
+
+ 
+# Useful Links
+
+- [Red Hat OpenShift documentation - installing on OCI](https://docs.openshift.com/container-platform/4.16/installing/installing_oci/installing-oci-assisted-installer.html))
+- [Oracle Cloud documentation - Getting started with OpenShift on OCI](https://docs.oracle.com/en-us/iaas/Content/openshift-on-oci/overview.htm)
+
+# Team Publications
+
+- [Using OCI Object storage for the OpenShift Internal Registry](enable-image-registry/README.md)
+  
+
+# Reusable Assets Overview
+
+- [Terraform script to provision OpenShift on OCI](https://github.com/oracle-quickstart/oci-openshift)
+  
+
+# License
+
+Copyright (c) 2024 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.
+
+[def]: #useful-links
+[def2]: def

cloud-infrastructure/vmware-solutions/openshift-on-oci/README.md

@@ -0,0 +1,90 @@
+# Setting up OpenShift Image Registry to use OCI Object Storage Bucket
+
+## Prerequisites
+You need to have the OpenShift CLI tool installed and properly configured.
+
+https://docs.openshift.com/container-platform/4.16/cli_reference/openshift_cli/getting-started-cli.html
+
+## 1. What is the OpenShift Image Registry?
+The OpenShift Image Registry is a built-in, containerized, enterprise-grade registry that stores Docker-formatted container images in a Red Hat OpenShift Container Platform cluster. It is a critical component for managing container images within the OpenShift environment, providing secure storage and efficient retrieval of container images required for deployments.
+
+After you have created an OpenShift Cluster on OCI, the image registry is not yet configured with the right storage settings. This will result in errors when you are trying to deploy your projects. You will see error messages like 
+
+```Error starting build: an image stream cannot be used as build output because the integrated image registry is not configured```
+
+<img src="files/1.NoRegistrySetup.png" width=600x>
+
+## 2. Configure OCI Object Storage for S3 Compatibility
+Oracle Cloud Infrastructure (OCI) Object Storage can be configured to work as an S3-compatible storage backend for the OpenShift Image Registry. This compatibility allows OpenShift to store container images directly in an OCI Object Storage bucket
+
+### a. Setup the correct compartment you want to use for Object Storage S3 compatability
+OCI Object Storage is S3-compatible by default, so no additional configuration is needed for basic S3 API operations. However, you may need to set the right compartment you want to use for S3 compatible buckets. 
+
+Go to your Tenancy Details in the Governance & Administration menu and click on <b>Edit Object Storage settings</b>
+
+<img src="files/2.OCI-setup-OS-AWS-Compartment.png" width=500x>
+
+Create a bucket in the selected compartment. 
+
+<img src="files/3.OCICreateBucket.png" width=500x>
+
+### b. Create a S3 Access and Secret key
+In the OCI console navigate to your profile (top right corner) and go to the <B>Customer Secret Keys</b> section.
+
+Create a new secret and make sure you note the Secret shown, as this is only one time displayed! After the sectet is created you will also see the access key.
+
+## 3. Create a secret for the Image Registry
+Now that you have you S3 Compatible Access and Secret key, you can create this secret for image registry. This secret needs to have the name of <b>image-registry-private-configuration-user</b>
+
+You can create the secret by running the following command, using the OpenShift CLI
+
+```oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=[your_access_key] --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=[your_secret_key] --namespace openshift-image-registry```
+
+## 4. Configure the Image Registry to use the S3 Object Storage
+Last you need to configure the OpenShift internal image registry to use the OCI S3 Compatible object storage. 
+
+You can do this by running:
+
+```oc edit configs.imageregistry.operator.openshift.io/cluster```
+
+You should see that currently your storage is not configured.
+
+<img src="files/4.Config_default.png" width=500x>
+
+Remove the {} behind the storage item and create the fields for S3 object storage
+```
+  storage:
+    s3:
+      bucket: os-cluster
+      region: [your-oci-region]
+      regionEndpoint: https://[yournamespace].compat.objectstorage.[your-oci-region].oraclecloud.com
+```
+
+Replace the [yournamespace] with your own object storage name space. You can find this namespace on the OCI Tenancy Details page.
+
+Replace the 2x [your-oci-region] with the OCI region you are using, for example: eu-frankfurt-1
+
+Finally, change the <b>managementState</b> from <b>Removed</b> to <b>Managed</b>
+
+<img src="files/5.Config_OCI-objectstorage.png" width=500x>
+
+Save and close the file and OpenShift will automatically update the image registry.
+
+## 5. Check the Image Registry operator
+You can now check if the image registry is properly configured. You can rerun the ```oc edit configs.imageregistry.operator.openshift.io/cluster``` and scroll down to the status section. You should see there a reference to the S3 opbject storage.
+
+Alternatively you can navigate to the cluster settings page under administration on your OpenShift console. Click on <b>ClusterOperators</b> and select the <b>image-registry</b>.
+
+Under the conditions you should see that the registry is ready.
+
+<img src="files/6.Configured.png" width=500x>
+
+## 6. Ready for deployment of your applications
+Your image registry should now be able to store images and you are now ready to start deploying applications and templates.
+
+
+
+# License
+Copyright (c) 2024 Oracle and/or its affiliates.
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.