Merge branch 'main' into dependabot/pip/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/rag-langchain-vllm-mistral/files/llama-index-core-0.10.38

Author: AlexanderHodicke

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/README.md

@@ -0,0 +1,99 @@
+# Calling multiple vLLM inference servers using LiteLLM
+
+In this tutorial we explain how to use a LiteLLM Proxy Server to call multiple LLM inference endpoints from a single interface. LiteLLM interacts will 100+ LLMs such as OpenAI, Cohere, NVIDIA Triton and NIM, etc. Here we will use two vLLM inference servers.
+
+<!-- ![Hybrid shards](assets/images/litellm.png "LiteLLM") -->
+
+# When to use this asset?
+
+To run the inference tutorial with local deployments of Mistral 7B Instruct v0.3 using a vLLM inference server powered by an NVIDIA A10 GPU and a LiteLLM Proxy Server on top. 
+
+# How to use this asset?
+
+These are the prerequisites to run this tutorial:
+* An OCI tenancy with A10 quota
+* A Huggingface account with a valid Auth Token
+* A valid OpenAI API Key
+
+## Introduction
+
+LiteLLM provides a proxy server to manage auth, loadbalancing, and spend tracking across 100+ LLMs. All in the OpenAI format.
+vLLM is a fast and easy-to-use library for LLM inference and serving.
+The first step will be to deploy two vLLM inference servers on NVIDIA A10 powered virtual machine instances. In the second step, we will create a LiteLLM Proxy Server on a third no-GPU instance and explain how we can use this interface to call the two LLM from a single location. For the sake of simplicity, all 3 instances will reside in the same public subnet here.
+
+![Hybrid shards](assets/images/litellm-architecture.png "LiteLLM")
+
+## vLLM inference servers deployment
+
+For each of the inference nodes a VM.GPU.A10.2 instance (2 x NVIDIA A10 GPU 24GB) is used in combination with the NVIDIA GPU-Optimized VMI image from the OCI marketplace. This Ubuntu-based image comes with all the necessary libraries (Docker, NVIDIA Container Toolkit) preinstalled. It is a good practice to deploy two instances in two different fault domains to ensure a higher availability.
+
+The vLLM inference server is deployed using the vLLM official container image.
+```
+docker run --gpus all \
+    -e HF_TOKEN=$HF_TOKEN -p 8000:8000 \
+    --ipc=host \
+    vllm/vllm-openai:latest \
+    --host 0.0.0.0 \
+    --port 8000 \
+    --model mistralai/Mistral-7B-Instruct-v0.3 \
+    --tensor-parallel-size 2 \
+    --load-format safetensors \
+    --trust-remote-code \
+    --enforce-eager
+```
+where `$HF_TOKEN` is a valid HuggingFace token. In this case we use the 7B Instruct version of Mistral LLM. The vLLM endpoint can be directly called for verification with:
+```
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "mistralai/Mistral-7B-Instruct-v0.3",
+        "messages": [
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ]
+    }' | jq
+```
+
+## LiteLLM server deployment
+
+No GPU are required for LiteLLM. Therefore, a CPU based VM.Standard.E4.Flex instance (4 OCPUs, 64 GB Memory) with a standard Ubuntu 22.04 image is used. Here LiteLLM is used as a proxy server calling a vLLM endpoint. Install LiteLLM using `pip`:
+```
+pip install 'litellm[proxy]'
+```
+Edit the `config.yaml` file (OpenAI-Compatible Endpoint):
+```
+model_list:
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://xxx.xxx.xxx.xxx:8000/v1
+      api_key: sk-0123456789
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://xxx.xxx.xxx.xxx:8000/v1
+      api_key: sk-0123456789
+```
+where `sk-0123456789` is a valid OpenAI API key and `xxx.xxx.xxx.xxx` are the two GPU instances public IP addresses.
+
+Start the LiteLLM Proxy Server with the following command:
+```
+litellm --config /path/to/config.yaml
+```
+Once the the Proxy Server is ready call the vLLM endpoint through LiteLLM with:
+```
+curl http://localhost:4000/chat/completions \
+    -H 'Authorization: Bearer sk-0123456789' \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Mistral-7B-Instruct",
+        "messages": [
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ]
+    }' | jq
+```
+
+## Documentation
+
+* [LiteLLM documentation](https://litellm.vercel.app/docs/providers/openai_compatible)
+* [vLLM documentation](https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html)
+* [MistralAI](https://mistral.ai/)

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/assets/images/litellm-architecture.png

@@ -0,0 +1,11 @@
+model_list:
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://public_ip_1:8000/v1
+      api_key: sk-0123456789
+  - model_name: Mistral-7B-Instruct
+    litellm_params:
+      model: openai/mistralai/Mistral-7B-Instruct-v0.3
+      api_base: http://public_ip_2:8000/v1
+      api_key: sk-0123456789

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/litellm/config.yaml

@@ -27,6 +27,7 @@ This page contains information and useful links regarding Compute services that
 - [Script to install and mount OCI bucket as Filesystem using Fuse S3FS](https://github.com/Olygo/OCI_S3FS)
 - [Mount a boot volume from one compute instance (or VM) onto another compute instance in order to replace lost ssh keys](https://gitlab.com/ms76152/system-administration)
 - [Transfer data to and from Oracle Cloud Infrastructure using OS tools such as sftp, scp, oci cli, curl](https://github.com/mariusscholtz/Oracle-Cloud-Infrastructure-resources/blob/main/VM-shapes/data%20transfer%20to%20OCI%20v1.0.pdf)
+- [Quering Compute Capacity using CloudShell](https://github.com/Olygo/OCI_ComputeCapacityReport)
 
 # Useful Links
 

cloud-infrastructure/compute-including-hpc/compute-software/README.md

@@ -0,0 +1,97 @@
+# C3 Hosting Service Provider - IAM Policies for Isolation
+
+The Hosting Service Provider (HSP) model on Compute Cloud@Customer (C3) allows
+hosting for multiple end customers, each isolated in a dedicated compartment
+with separate VCN(s) per customer. To ensure the end customer can only
+create resources in just their own compartment, a set of IAM policies are
+required.
+
+The HSP documentation suggests the following policies per end customer
+based on an example with two hosting customers, A & B. They assume that
+each end customer will have two roles for their
+staff: Customer Administrator and Customer End User. 
+
+## Example Policies for Customer Administrator
+```
+Allows the group specified to use all C3 services in the compartment
+listed:
+
+Allow group CustA-Admin-grp to manage all-resources in compartment
+path:to:CustA
+
+Allow group CustB-Admin-grp to manage all-resources in compartment
+path:to:CustB
+```
+Note that the above policy grants permissions in the CustA and CustB
+compartments of the C3 but **also in the same compartment in the OCI
+tenancy**! To prevent permissions being granted in the OCI tenancy
+append a condition such as:
+
+```Allow group CustA-Admin-grp to manage all-resources in compartment
+path:to:CustA where all {request.region != 'LHR',request.region !=
+'FRA'}
+
+Allow group CustB-Admin-grp to manage all-resources in compartment
+path:to:CustB where all {request.region != 'LHR',request.region !=
+'FRA'}
+```
+In the example above the condition prevents resource creation in London
+and Frankfurt regions. Adjust the list to include all regions the
+tenancy is subscribed to.
+
+The path to the end user compartment must be explicitly stated, using
+the comma format, relative to the compartment where the policy is
+created. 
+
+## Example Policies for Customer End User
+```
+Allow group CustA-Users-grp to manage instance-family in compartment
+path:to:CustA  
+Allow group CustA-Users-grp to use volume-family in compartment
+path:to:CustA  
+Allow group CustA-Users-grp to use virtual-network-family in compartment
+path:to:CustA  
+Allow group CustB-Users-grp to manage instance-family in compartment
+path:to:CustB  
+Allow group CustB-Users-grp to use volume-family in compartment
+path:to:CustB  
+Allow group CustB-Users-grp to use virtual-network-family in compartment
+path:to:CustB
+```
+As above append a condition to limit permissions to the C3 and prevent
+resource creation in OCI regions:
+```
+Allow group CustA-Users-grp to manage instance-family in compartment
+path:to:CustA where all {request.region != 'LHR',request.region !=
+'FRA'}  
+Allow group CustA-Users-grp to use volume-family in compartment
+path:to:CustA where all {request.region != 'LHR',request.region !=
+'FRA'}  
+Allow group CustA-Users-grp to use virtual-network-family in compartment
+path:to:CustA where all {request.region != 'LHR',request.region !=
+'FRA'}  
+Allow group CustB-Users-grp to manage instance-family in compartment
+path:to:CustB where all {request.region != 'LHR',request.region !=
+'FRA'}  
+Allow group CustB-Users-grp to use volume-family in compartment
+path:to:CustB where all {request.region != 'LHR',request.region !=
+'FRA'}  
+Allow group CustB-Users-grp to use virtual-network-family in compartment
+path:to:CustB where all {request.region != 'LHR',request.region !=
+'FRA'}
+```
+## Common Policy
+
+Currently any user of a C3 needs access to certain resources located at
+the tenancy level to use IaaS resources in the web UI.
+Backup policies, tag namespaces, platform images, all reside at the
+tenancy level and need a further policy to allow normal use of C3 IaaS
+services. Note that this is a subtle difference to the behaviour on OCI. 
+
+An extra policy as below is required (where CommonGroup contains **all**
+HSP users on the C3):
+```
+allow group CommonGroup to read all-resources in tenancy where
+target.compartment.name='root-compartment-name'
+```
+

cloud-infrastructure/private-cloud-and-edge/hsp/README.md

@@ -0,0 +1,31 @@
+
+# OpenShift on OCI
+ 
+Red Hat OpenShift can be hosted on OCI as a self-run platform. Oracle provides terraform templates for easy implementation and platform integration.
+
+ 
+# Useful Links
+
+- [Red Hat OpenShift documentation - installing on OCI](https://docs.openshift.com/container-platform/4.16/installing/installing_oci/installing-oci-assisted-installer.html))
+- [Oracle Cloud documentation - Getting started with OpenShift on OCI](https://docs.oracle.com/en-us/iaas/Content/openshift-on-oci/overview.htm)
+
+# Team Publications
+
+- [Using OCI Object storage for the OpenShift Internal Registry](enable-image-registry/README.md)
+  
+
+# Reusable Assets Overview
+
+- [Terraform script to provision OpenShift on OCI](https://github.com/oracle-quickstart/oci-openshift)
+  
+
+# License
+
+Copyright (c) 2024 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.
+
+[def]: #useful-links
+[def2]: def

cloud-infrastructure/vmware-solutions/openshift-on-oci/README.md

@@ -0,0 +1,90 @@
+# Setting up OpenShift Image Registry to use OCI Object Storage Bucket
+
+## Prerequisites
+You need to have the OpenShift CLI tool installed and properly configured.
+
+https://docs.openshift.com/container-platform/4.16/cli_reference/openshift_cli/getting-started-cli.html
+
+## 1. What is the OpenShift Image Registry?
+The OpenShift Image Registry is a built-in, containerized, enterprise-grade registry that stores Docker-formatted container images in a Red Hat OpenShift Container Platform cluster. It is a critical component for managing container images within the OpenShift environment, providing secure storage and efficient retrieval of container images required for deployments.
+
+After you have created an OpenShift Cluster on OCI, the image registry is not yet configured with the right storage settings. This will result in errors when you are trying to deploy your projects. You will see error messages like 
+
+```Error starting build: an image stream cannot be used as build output because the integrated image registry is not configured```
+
+<img src="files/1.NoRegistrySetup.png" width=600x>
+
+## 2. Configure OCI Object Storage for S3 Compatibility
+Oracle Cloud Infrastructure (OCI) Object Storage can be configured to work as an S3-compatible storage backend for the OpenShift Image Registry. This compatibility allows OpenShift to store container images directly in an OCI Object Storage bucket
+
+### a. Setup the correct compartment you want to use for Object Storage S3 compatability
+OCI Object Storage is S3-compatible by default, so no additional configuration is needed for basic S3 API operations. However, you may need to set the right compartment you want to use for S3 compatible buckets. 
+
+Go to your Tenancy Details in the Governance & Administration menu and click on <b>Edit Object Storage settings</b>
+
+<img src="files/2.OCI-setup-OS-AWS-Compartment.png" width=500x>
+
+Create a bucket in the selected compartment. 
+
+<img src="files/3.OCICreateBucket.png" width=500x>
+
+### b. Create a S3 Access and Secret key
+In the OCI console navigate to your profile (top right corner) and go to the <B>Customer Secret Keys</b> section.
+
+Create a new secret and make sure you note the Secret shown, as this is only one time displayed! After the sectet is created you will also see the access key.
+
+## 3. Create a secret for the Image Registry
+Now that you have you S3 Compatible Access and Secret key, you can create this secret for image registry. This secret needs to have the name of <b>image-registry-private-configuration-user</b>
+
+You can create the secret by running the following command, using the OpenShift CLI
+
+```oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=[your_access_key] --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=[your_secret_key] --namespace openshift-image-registry```
+
+## 4. Configure the Image Registry to use the S3 Object Storage
+Last you need to configure the OpenShift internal image registry to use the OCI S3 Compatible object storage. 
+
+You can do this by running:
+
+```oc edit configs.imageregistry.operator.openshift.io/cluster```
+
+You should see that currently your storage is not configured.
+
+<img src="files/4.Config_default.png" width=500x>
+
+Remove the {} behind the storage item and create the fields for S3 object storage
+```
+  storage:
+    s3:
+      bucket: os-cluster
+      region: [your-oci-region]
+      regionEndpoint: https://[yournamespace].compat.objectstorage.[your-oci-region].oraclecloud.com
+```
+
+Replace the [yournamespace] with your own object storage name space. You can find this namespace on the OCI Tenancy Details page.
+
+Replace the 2x [your-oci-region] with the OCI region you are using, for example: eu-frankfurt-1
+
+Finally, change the <b>managementState</b> from <b>Removed</b> to <b>Managed</b>
+
+<img src="files/5.Config_OCI-objectstorage.png" width=500x>
+
+Save and close the file and OpenShift will automatically update the image registry.
+
+## 5. Check the Image Registry operator
+You can now check if the image registry is properly configured. You can rerun the ```oc edit configs.imageregistry.operator.openshift.io/cluster``` and scroll down to the status section. You should see there a reference to the S3 opbject storage.
+
+Alternatively you can navigate to the cluster settings page under administration on your OpenShift console. Click on <b>ClusterOperators</b> and select the <b>image-registry</b>.
+
+Under the conditions you should see that the registry is ready.
+
+<img src="files/6.Configured.png" width=500x>
+
+## 6. Ready for deployment of your applications
+Your image registry should now be able to store images and you are now ready to start deploying applications and templates.
+
+
+
+# License
+Copyright (c) 2024 Oracle and/or its affiliates.
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.