Initial commit

Author: Yuri-Rassokhin

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/torchserve-ai-examples/LICENSE

@@ -0,0 +1,35 @@
+#Copyright (c) 2024 Oracle and/or its affiliates.
+#
+#The Universal Permissive License (UPL), Version 1.0
+#
+#Subject to the condition set forth below, permission is hereby granted to any
+#person obtaining a copy of this software, associated documentation and/or data
+#(collectively the "Software"), free of charge and under any and all copyright
+#rights in the Software, and any and all patent rights owned or freely
+#licensable by each licensor hereunder covering either (i) the unmodified
+#Software as contributed to or provided by such licensor, or (ii) the Larger
+#Works (as defined below), to deal in both
+#
+#(a) the Software, and
+#(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+#one is included with the Software (each a "Larger Work" to which the Software
+#is contributed by such licensors),
+#
+#without restriction, including without limitation the rights to copy, create
+#derivative works of, display, perform, and distribute the Software and make,
+#use, sell, offer for sale, import, export, have made, and have sold the
+#Software and the Larger Work(s), and to sublicense the foregoing rights on
+#either these or other terms.
+#
+#This license is subject to the following condition:
+#The above copyright notice and either this complete permission notice or at
+#a minimum a reference to the UPL must be included in all copies or
+#substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/torchserve-ai-examples/README.md

@@ -0,0 +1,25 @@
+### How to Configure AI model as HTTP Endpoint Using TorchServe
+
+1. Download and test the model using download script ../basic-utility-scripts/download\_and\_test\_huggingface\_model.py
+
+2. Go to the folder where you downloaded model is located, for the sake of example, it'll be /share/app/llama3-8b-instruct folder, and do the following as a regular user.
+
+3. Put handler.py to this folder. The handler is a python file that implements semantics of inference; in particular, it implements methods 'preprocess', 'inference', 'postprocess', and an umbrella method 'handle' that invokes previous methods sequentially.
+
+4. Put config.properties to this folder. This file name is fixed and recognized by TorchServe. This file defines performance & scalability characteristics of a given model, such as batch size and number of 'workers', that is, processes, associated with the inference of a given model.
+
+5. Archive the model along with all required file into .mar file recognizable by TorchServe using this command:
+
+torch-model-archiver --model-name llama3-8b-instruct --version 1.0 --serialized-file /share/app/llama3-8b-instruct/model-00001-of-00004.safetensors --handler /share/app/llama3-8b-instruct/handler.py --extra-files "/share/app/llama3-8b-instruct/config.json,/share/app/llama3-8b-instruct/generation\_config.json,/share/app/llama3-8b-instruct/model-00002-of-00004.safetensors,/share/app/llama3-8b-instruct/model-00003-of-00004.safetensors,/share/app/llama3-8b-instruct/model-00004-of-00004.safetensors,/share/app/llama3-8b-instruct/model.safetensors.index.json,/share/app/llama3-8b-instruct/tokenizer.json,/share/app/llama3-8b-instruct/tokenizer\_config.json,/share/app/llama3-8b-instruct/special\_tokens\_map.json" --export-path model\_store --force
+
+6. Start TorchServe:
+
+torchserve --start --ts-config /share/app/llama3-8b-instruct/config.properties --model-store model\_store --models llama3-8b-instruct.mar
+
+7. You can use the following basic commands with the running TorchServe:
+    * How to stop TorchServe: torchserve --stop
+    * How to send inference request: curl -X POST http://127.0.0.1:8080/predictions/llama3-8b-instruct -T input\_example.json
+    * How to directly pass payload to inference request: curl -X POST http://localhost:8080/predictions/llama -H "Content-Type: application/json" -d '{ "input": "Hello, who are you?" }'
+    * How to monitor GPU utilization during inference processing: watch -n 1 nvidia-smi
+    * How to check if TorchServe is healthy: curl http://localhost:8080/ping
+    * How to check the list of models exposed via TorchServe: curl http://localhost:8081/models

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/torchserve-ai-examples/config.properties

@@ -0,0 +1,54 @@
+#Copyright (c) 2024 Oracle and/or its affiliates.
+#
+#The Universal Permissive License (UPL), Version 1.0
+#
+#Subject to the condition set forth below, permission is hereby granted to any
+#person obtaining a copy of this software, associated documentation and/or data
+#(collectively the "Software"), free of charge and under any and all copyright
+#rights in the Software, and any and all patent rights owned or freely
+#licensable by each licensor hereunder covering either (i) the unmodified
+#Software as contributed to or provided by such licensor, or (ii) the Larger
+#Works (as defined below), to deal in both
+#
+#(a) the Software, and
+#(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+#one is included with the Software (each a "Larger Work" to which the Software
+#is contributed by such licensors),
+#
+#without restriction, including without limitation the rights to copy, create
+#derivative works of, display, perform, and distribute the Software and make,
+#use, sell, offer for sale, import, export, have made, and have sold the
+#Software and the Larger Work(s), and to sublicense the foregoing rights on
+#either these or other terms.
+#
+#This license is subject to the following condition:
+#The above copyright notice and either this complete permission notice or at
+#a minimum a reference to the UPL must be included in all copies or
+#substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.
+
+# Path where models are stored
+model_store=/share/app/llama3-8b-instruct/
+
+# Number of workers per model
+default_workers_per_model=1
+
+# Inference API settings
+inference_address=http://0.0.0.0:8080
+management_address=http://0.0.0.0:8081
+metrics_address=http://0.0.0.0:8082
+
+# Model snapshot settings
+model_snapshot={"name":"startup.cfg","modelCount":1,"models":{"llama3-8b-instruct":{"1.0":{"defaultVersion":true,"marName":"llama3-8b-instruct.mar","minWorkers":1,"maxWorkers":1,"batchSize":1,"maxBatchDelay":100,"responseTimeout":120}}}}
+
+#model_name=llama3-8b-instruct
+#initial_workers=1
+#max_workers=1
+#llama3-8b-instruct.mar=1

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/torchserve-ai-examples/handler.py

@@ -0,0 +1,97 @@
+#Copyright (c) 2024 Oracle and/or its affiliates.
+#
+#The Universal Permissive License (UPL), Version 1.0
+#
+#Subject to the condition set forth below, permission is hereby granted to any
+#person obtaining a copy of this software, associated documentation and/or data
+#(collectively the "Software"), free of charge and under any and all copyright
+#rights in the Software, and any and all patent rights owned or freely
+#licensable by each licensor hereunder covering either (i) the unmodified
+#Software as contributed to or provided by such licensor, or (ii) the Larger
+#Works (as defined below), to deal in both
+#
+#(a) the Software, and
+#(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+#one is included with the Software (each a "Larger Work" to which the Software
+#is contributed by such licensors),
+#
+#without restriction, including without limitation the rights to copy, create
+#derivative works of, display, perform, and distribute the Software and make,
+#use, sell, offer for sale, import, export, have made, and have sold the
+#Software and the Larger Work(s), and to sublicense the foregoing rights on
+#either these or other terms.
+#
+#This license is subject to the following condition:
+#The above copyright notice and either this complete permission notice or at
+#a minimum a reference to the UPL must be included in all copies or
+#substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.
+
+import logging
+import json
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+
+logger = logging.getLogger(__name__)
+
+class SimpleTextGenerationHandler:
+    def __init__(self):
+        self.initialized = False
+
+    def initialize(self, context):
+        logger.info("YR: model initialization")
+        self.model_dir = context.system_properties.get("model_dir")
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+        logger.info("YR: loading model")
+        # Load the tokenizer and model from the local directory
+        self.model = AutoModelForCausalLM.from_pretrained(
+            self.model_dir,
+            torch_dtype=torch.bfloat16
+        ).to(self.device)
+
+        self.tokenizer = AutoTokenizer.from_pretrained(self.model_dir)
+
+        logger.info("YR: setting pipeline")
+        # Use the Hugging Face pipeline for text generation
+        self.pipe = pipeline(
+            "text-generation",
+            model=self.model,
+            tokenizer=self.tokenizer,
+            device=0  # Use GPU
+        )
+
+        self.initialized = True
+
+    def preprocess(self, data):
+        input_text = data[0].get("body")
+        return input_text.decode('utf-8') if isinstance(input_text, (bytes, bytearray)) else input_text
+
+    def inference(self, input_text):
+        try:
+            result = self.pipe(input_text)[0]['generated_text']
+            return [result]
+        except Exception as e:
+            raise RuntimeError(f"Error during inference: {str(e)}")
+
+    def postprocess(self, inference_output):
+        return [json.dumps({"generated_text": inference_output[0]})]
+
+_service = SimpleTextGenerationHandler()
+
+def handle(data, context):
+    if not _service.initialized:
+        _service.initialize(context)
+    if data is None:
+        return None
+
+    data = _service.preprocess(data)
+    predictions = _service.inference(data)
+    return _service.postprocess(predictions)

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/torchserve-ai-examples/input_example.json

@@ -0,0 +1,39 @@
+#Copyright (c) 2024 Oracle and/or its affiliates.
+#
+#The Universal Permissive License (UPL), Version 1.0
+#
+#Subject to the condition set forth below, permission is hereby granted to any
+#person obtaining a copy of this software, associated documentation and/or data
+#(collectively the "Software"), free of charge and under any and all copyright
+#rights in the Software, and any and all patent rights owned or freely
+#licensable by each licensor hereunder covering either (i) the unmodified
+#Software as contributed to or provided by such licensor, or (ii) the Larger
+#Works (as defined below), to deal in both
+#
+#(a) the Software, and
+#(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+#one is included with the Software (each a "Larger Work" to which the Software
+#is contributed by such licensors),
+#
+#without restriction, including without limitation the rights to copy, create
+#derivative works of, display, perform, and distribute the Software and make,
+#use, sell, offer for sale, import, export, have made, and have sold the
+#Software and the Larger Work(s), and to sublicense the foregoing rights on
+#either these or other terms.
+#
+#This license is subject to the following condition:
+#The above copyright notice and either this complete permission notice or at
+#a minimum a reference to the UPL must be included in all copies or
+#substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.
+
+{
+        "input_text": "Once upon a long ago..."
+}