Merge pull request #506 from VipulMascarenhas/main

Advanced deployment and fine-tuning config options

Author: darenr

ai-quick-actions/fine-tuning-tips.md

@@ -139,6 +139,81 @@ The accuracy metric reflects the proportion of correct completions made by the m
 
 As the training progresses, monitoring both accuracy and loss provides insights into the model's learning dynamics. A decreasing loss alongside increasing accuracy suggests that the model is learning effectively. However, it's important to watch for signs of over-fitting, where the model performs exceptionally well on the training data but fails to generalize to new, unseen data. This can be detected if the validation loss stops decreasing or starts increasing, even as training loss continues to decline.
 
+### Advanced Configuration Update Options
+
+The available shapes for models in AI Quick Actions are pre-configured for fine-tuning for the models available in the Fine-Tuned model tab. 
+However, if you need to add more shapes to the list of 
+available options, you can do so by updating the relevant configuration file. Currently, this 
+update option is only available for models that users can register.
+
+#### For Custom Models:
+To add shapes for custom models, follow these steps:
+
+1. **Register the model**: Ensure the model is registered via AI Quick Actions UI or CLI.
+
+2. **Navigate to the model's artifact directory**: After registration, locate the directory where the model's artifacts are stored in the object storage. 
+
+3. **Create a configuration folder**: Inside the artifact directory, create a new folder named config. For example, if the model path is `oci://<bucket>@namespace/path/to/model/`
+then create a folder `oci://<bucket>@namespace/path/to/model/config`.
+
+4. **Add a fine-tuning configuration file**: Within the config folder, create a file named `ft_config.json` with the following content:
+
+
+```
+{
+    "shape":
+    {
+        "VM.GPU.A10.1":
+        {
+            "batch_size": 1,
+            "replica": "1-10"
+        },
+        "VM.GPU.A10.2":
+        {
+            "batch_size": 1,
+            "replica": "1-10"
+        },
+        "BM.GPU.A10.4":
+        {
+            "batch_size": 1,
+            "replica": 1
+        },
+        "BM.GPU4.8":
+        {
+            "batch_size": 4,
+            "replica": 1
+        },
+        "BM.GPU.L40S-NC.4":
+        {
+            "batch_size": 4,
+            "replica": 1
+        },
+        "BM.GPU.A100-v2.8":
+        {
+            "batch_size": 6,
+            "replica": 1
+        },
+        "BM.GPU.H100.8":
+        {
+            "batch_size": 6,
+            "replica": 1
+        }
+    },
+    "finetuning_params": "--trust_remote_code True"
+}
+```
+This JSON file lists all available shapes that can be used to fine-tune a model via AI Quick Actions. It also has additional
+configuration that can be set via the configuration field.
+
+5. Modify shapes or configuration as needed: If you want to add or remove any 
+[shapes supported](https://docs.oracle.com/en-us/iaas/data-science/using/supported-shapes.htm) by 
+the OCI Data Science platform, you can directly edit this `ft_config.json` file. 
+6. In addition to the shapes list, you can also use other configuration listed in the above json.  Another field called 
+`finetuning_params` is available if you want to explicitly pass additional parameters to the launch of the fine-tuning container. 
+For example, the field `"finetuning_params": "--trust_remote_code True"` might be required if the model
+needs to execute the code that resides on the Hugging Face Hub rather than natively in the Transformers library.  
+
+
 Table of Contents:
 
 - [Home](README.md)

ai-quick-actions/model-deployment-tips.md

@@ -304,6 +304,94 @@ public class RestExample {
 
 ```
 
+### Advanced Configuration Update Options
+
+The available shapes for models in AI Quick Actions are pre-configured for both registration and 
+deployment for models available in the Model Explorer. However, if you need to add more shapes to the list of 
+available options, you can do so by updating the relevant configuration file. Currently, this 
+update option is only available for models that users can register.
+
+#### For Custom Models:
+To add shapes for custom models, follow these steps:
+
+1. **Register the model**: Ensure the model is registered via AI Quick Actions UI or CLI.
+
+2. **Navigate to the model's artifact directory**: After registration, locate the directory where the model's artifacts are stored in the object storage. 
+
+3. **Create a configuration folder**: Inside the artifact directory, create a new folder named config. For example, if the model path is `oci://<bucket>@namespace/path/to/model/`
+then create a folder `oci://<bucket>@namespace/path/to/model/config`.
+
+4. **Add a deployment configuration file**: Within the config folder, create a file named `deployment_config.json` with the following content:
+
+
+```
+{
+  "configuration": {
+    "VM.Standard.A1.Flex": {
+      "parameters": {},
+      "shape_info": {
+        "configs": [
+          {
+            "memory_in_gbs": 128,
+            "ocpu": 20
+          },
+          {
+            "memory_in_gbs": 256,
+            "ocpu": 40
+          },
+          {
+            "memory_in_gbs": 384,
+            "ocpu": 60
+          },
+          {
+            "memory_in_gbs": 512,
+            "ocpu": 80
+          }
+        ],
+        "type": "CPU"
+      }
+    }
+  },
+  "shape": [
+    "VM.GPU.A10.1",
+    "VM.GPU.A10.2",
+    "BM.GPU.A10.4",
+    "BM.GPU4.8",
+    "BM.GPU.L40S-NC.4",
+    "BM.GPU.A100-v2.8",
+    "BM.GPU.H100.8",
+    "VM.Standard.A1.Flex"
+  ]
+}
+```
+
+This JSON file lists all available GPU and CPU shapes for AI Quick Actions. 
+The CPU shapes include additional configuration details required for model deployment, 
+such as memory and OCPU settings.
+
+5. Modify shapes as needed: If you want to add or remove any 
+[shapes supported](https://docs.oracle.com/en-us/iaas/data-science/using/supported-shapes.htm) by 
+the OCI Data Science platform, you can directly edit this `deployment_config.json` file.
+
+6. The `configuration` field in this json file can also support parameters for vLLM and TGI inference containers. For example,
+if a model can be deployed by either one of these containers, and you want to set the server parameters through configuration file, then 
+you can add the corresponding shape along with the parameter value inside the `configuration` field. You can achieve the same
+using [Advanced Deployment Options](#advanced-deployment-options) from AI Quick Actions UI as well. 
+
+
+```
+  "configuration": {
+    "VM.GPU.A10.1": {
+      "parameters": {
+        "TGI_PARAMS": "--max-stop-sequences 6",
+        "VLLM_PARAMS": "--max-model-len 4096"
+      }
+    }
+    ...
+    ...
+  }
+```
+
 
 ### Troubleshooting