Merge pull request #202 from mrbullwinkle/mrb_09_09_2024_pm_fine_tuning

prmerger-automator[bot] · web-flow · commit 13fb5ddc8cf7 · 2024-09-09T21:10:05.000Z
[Azure OpenAI] Fine-tuning tutorial updates
diff --git a/articles/ai-services/openai/tutorials/fine-tune.md b/articles/ai-services/openai/tutorials/fine-tune.md
@@ -1,21 +1,21 @@
 ---
-title: Azure OpenAI Service fine-tuning gpt-3.5-turbo
+title: Azure OpenAI Service fine-tuning gpt-4o-mini
 titleSuffix: Azure OpenAI
-description: Learn how to use Azure OpenAI's latest fine-tuning capabilities with gpt-3.5-turbo.
+description: Learn how to use Azure OpenAI's latest fine-tuning capabilities with gpt-4o-mini-2024-07-18
 #services: cognitive-services
 manager: nitinme
 ms.service: azure-ai-openai
 ms.topic: tutorial
-ms.date: 05/15/2024
+ms.date: 09/09/2024
 author: mrbullwinkle
 ms.author: mbullwin
 recommendations: false
 ms.custom: devx-track-python
 ---
 
-# Azure OpenAI GPT-3.5 Turbo fine-tuning tutorial
+# Azure OpenAI GPT-4o-mini fine-tuning tutorial
 
-This tutorial walks you through fine-tuning a `gpt-35-turbo-0613` model.
+This tutorial walks you through fine-tuning a `gpt-4o-mini-2024-07-18` model.
 
 In this tutorial you learn how to:
 
@@ -24,7 +24,7 @@ In this tutorial you learn how to:
 > * Create environment variables for your resource endpoint and API key.
 > * Prepare your sample training and validation datasets for fine-tuning.
 > * Upload your training file and validation file for fine-tuning.
-> * Create a fine-tuning job for `gpt-35-turbo-0613`.
+> * Create a fine-tuning job for `gpt-4o-mini-2024-07-18`.
 > * Deploy a custom fine-tuned model.
 
 ## Prerequisites
@@ -33,13 +33,12 @@ In this tutorial you learn how to:
 - Python 3.8 or later version
 - The following Python libraries: `json`, `requests`, `os`, `tiktoken`, `time`, `openai`, `numpy`.
 - [Jupyter Notebooks](https://jupyter.org/)
-- An Azure OpenAI resource in a [region where `gpt-35-turbo-0613` fine-tuning is available](../concepts/models.md). If you don't have a resource the process of creating one is documented in our resource [deployment guide](../how-to/create-resource.md).
+- An Azure OpenAI resource in a [region where `gpt-4o-mini-2024-07-18` fine-tuning is available](../concepts/models.md). If you don't have a resource the process of creating one is documented in our resource [deployment guide](../how-to/create-resource.md).
 - Fine-tuning access requires **Cognitive Services OpenAI Contributor**.
 - If you do not already have access to view quota, and deploy models in Azure OpenAI Studio you will require [additional permissions](../how-to/role-based-access-control.md).
 
-
 > [!IMPORTANT]
-> We strongly recommend reviewing the [pricing information](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/#pricing) for fine-tuning prior to beginning this tutorial to make sure you are comfortable with the associated costs. In testing, this tutorial resulted in one training hour billed, in addition to the costs that are associated with fine-tuning inference, and the hourly hosting costs of having a fine-tuned model deployed. Once you have completed the tutorial, you should delete your fine-tuned model deployment otherwise you will continue to incur the hourly hosting cost.
+> We recommend reviewing the [pricing information](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/#pricing) for fine-tuning to familiarize yourself with the associated costs. In testing, this tutorial resulted in 48,000 tokens being billed (4,800 training tokens * 10 epochs of training). Training costs are in addition to the costs that are associated with fine-tuning inference, and the hourly hosting costs of having a fine-tuned model deployed. Once you have completed the tutorial, you should delete your fine-tuned model deployment otherwise you will continue to incur the hourly hosting cost.
 
 ## Set up
 
@@ -106,7 +105,7 @@ source /etc/environment
 
 ### Create a sample dataset
 
-Fine-tuning `gpt-35-turbo-0613` requires a specially formatted JSONL training file. OpenAI provides the following example in their documentation:
+Fine-tuning `gpt-4o-mini-2024-07-18` requires a specially formatted JSONL training file. OpenAI provides the following example in their documentation:
 
 ```json
 {"messages": [{"role": "system", "content": "Marv is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the capital of France?"}, {"role": "assistant", "content": "Paris, as if everyone doesn't know that already."}]}
@@ -206,7 +205,10 @@ First example in validation set:
 
 In this case we only have 10 training and 10 validation examples so while this will demonstrate the basic mechanics of fine-tuning a model this in unlikely to be a large enough number of examples to produce a consistently noticeable impact.
 
-Now you can then run some additional code from OpenAI using the tiktoken library to validate the token counts. Individual examples need to remain under the `gpt-35-turbo-0613` model's input token limit of 4096 tokens.
+Now you can then run some additional code from OpenAI using the tiktoken library to validate the token counts. Token counting using this method is not going to give you the exact token counts that will be used for fine-tuning, but should provide a good estimate.
+
+> [!NOTE]
+> Individual examples need to remain under the `gpt-4o-mini-2024-07-18` model's current training example context legnth of: 64,536 tokens. The model's input token limit remains 128,000 tokens.
 
 ```python
 # Validate token counts
@@ -216,7 +218,7 @@ import tiktoken
 import numpy as np
 from collections import defaultdict
 
-encoding = tiktoken.get_encoding("cl100k_base") # default encoding used by gpt-4, turbo, and text-embedding-ada-002 models
+encoding = tiktoken.get_encoding("o200k_base") # default encoding for gpt-4o models. This requires the latest version of tiktoken to be installed.
 
 def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):
     num_tokens = 0
@@ -268,27 +270,27 @@ for file in files:
 Processing file: training_set.jsonl
 
 #### Distribution of total tokens:
-min / max: 47, 62
-mean / median: 52.1, 50.5
-p5 / p95: 47.9, 57.5
+min / max: 46, 59
+mean / median: 49.8, 48.5
+p5 / p95: 46.0, 53.599999999999994
 
 #### Distribution of assistant tokens:
-min / max: 13, 30
-mean / median: 17.6, 15.5
-p5 / p95: 13.0, 21.9
+min / max: 13, 28
+mean / median: 16.5, 14.0
+p5 / p95: 13.0, 19.9
 **************************************************
 Processing file: validation_set.jsonl
 
 #### Distribution of total tokens:
-min / max: 43, 65
-mean / median: 51.4, 49.0
-p5 / p95: 45.7, 56.9
+min / max: 41, 64
+mean / median: 48.9, 47.0
+p5 / p95: 43.7, 54.099999999999994
 
 #### Distribution of assistant tokens:
 min / max: 8, 29
-mean / median: 15.9, 13.5
-p5 / p95: 11.6, 20.9
-**************************************************
+mean / median: 15.0, 12.5
+p5 / p95: 10.7, 19.999999999999996
+****************************
 ```
 
 ## Upload fine-tuning files
@@ -304,7 +306,7 @@ from openai import AzureOpenAI
 client = AzureOpenAI(
   azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"),
   api_key = os.getenv("AZURE_OPENAI_API_KEY"),
-  api_version = "2024-05-01-preview"  # This API version or later is required to access seed/events/checkpoint features
+  api_version = "2024-08-01-preview"  # This API version or later is required to access seed/events/checkpoint features
 )
 
 training_file_name = 'training_set.jsonl'
@@ -381,7 +383,7 @@ In this example we're also passing the seed parameter. The seed controls the rep
 response = client.fine_tuning.jobs.create(
     training_file = training_file_id,
     validation_file = validation_file_id,
-    model = "gpt-35-turbo-0613", # Enter base model name. Note that in Azure OpenAI the model name contains dashes and cannot contain dot/period characters.
+    model = "gpt-4o-mini-2024-07-18", # Enter base model name. Note that in Azure OpenAI the model name contains dashes and cannot contain dot/period characters.
     seed = 105 # seed parameter controls reproducibility of the fine-tuning job. If no seed is specified one will be generated automatically.
 )
 
@@ -404,7 +406,7 @@ print(response.model_dump_json(indent=2))
 response = openai.FineTuningJob.create(
     training_file = training_file_id,
     validation_file = validation_file_id,
-    model = "gpt-35-turbo-0613",
+    model = "gpt-4o-mini-2024-07-18",
 )
 
 job_id = response["id"]
@@ -435,7 +437,7 @@ Status: pending
     "batch_size": -1,
     "learning_rate_multiplier": 1
   },
-  "model": "gpt-35-turbo-0613",
+  "model": "gpt-4o-mini-2024-07-18",
   "object": "fine_tuning.job",
   "organization_id": null,
   "result_files": null,
@@ -540,7 +542,7 @@ Status: pending
     "batch_size": -1,
     "learning_rate_multiplier": 1
   },
-  "model": "gpt-35-turbo-0613",
+  "model": "gpt-4o-mini-2024-07-18",
   "object": "fine_tuning.job",
   "organization_id": null,
   "result_files": null,
@@ -564,7 +566,7 @@ Found 4 fine-tune jobs.
 
 ## List fine-tuning events
 
-API version: `2024-05-01-preview` or later is required for this command.
+API version: `2024-08-01-preview` or later is required for this command.
 
 While not necessary to complete fine-tuning it can be helpful to examine the individual fine-tuning events that were generated during training. The full training results can also be examined after training is complete in the [training results file](../how-to/fine-tuning.md#analyze-your-customized-model).
 
@@ -728,7 +730,7 @@ This command isn't available in the 0.28.1 OpenAI Python library. Upgrade to the
 
 ## List checkpoints
 
-API version: `2024-05-01-preview` or later is required for this command.
+API version: `2024-08-01-preview` or later is required for this command.
 
 When each training epoch completes a checkpoint is generated. A checkpoint is a fully functional version of a model which can both be deployed and used as the target model for subsequent fine-tuning jobs. Checkpoints can be particularly useful, as they can provide a snapshot of your model prior to overfitting having occurred. When a fine-tuning job completes you will have the three most recent versions of the model available to deploy. The final epoch will be represented by your fine-tuned model, the previous two epochs will be available as checkpoints.
 
@@ -753,7 +755,7 @@ This command isn't available in the 0.28.1 OpenAI Python library. Upgrade to the
     {
       "id": "ftchkpt-148ab69f0a404cf9ab55a73d51b152de",
       "created_at": 1715743077,
-      "fine_tuned_model_checkpoint": "gpt-35-turbo-0613.ft-372c72db22c34e6f9ccb62c26ee0fbd9",
+      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-372c72db22c34e6f9ccb62c26ee0fbd9",
       "fine_tuning_job_id": "ftjob-372c72db22c34e6f9ccb62c26ee0fbd9",
       "metrics": {
         "full_valid_loss": 1.8258173013035255,
@@ -770,7 +772,7 @@ This command isn't available in the 0.28.1 OpenAI Python library. Upgrade to the
     {
       "id": "ftchkpt-e559c011ecc04fc68eaa339d8227d02d",
       "created_at": 1715743013,
-      "fine_tuned_model_checkpoint": "gpt-35-turbo-0613.ft-372c72db22c34e6f9ccb62c26ee0fbd9:ckpt-step-90",
+      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-372c72db22c34e6f9ccb62c26ee0fbd9:ckpt-step-90",
       "fine_tuning_job_id": "ftjob-372c72db22c34e6f9ccb62c26ee0fbd9",
       "metrics": {
         "full_valid_loss": 1.7958603267428241,
@@ -787,7 +789,7 @@ This command isn't available in the 0.28.1 OpenAI Python library. Upgrade to the
     {
       "id": "ftchkpt-8ae8beef3dcd4dfbbe9212e79bb53265",
       "created_at": 1715742984,
-      "fine_tuned_model_checkpoint": "gpt-35-turbo-0613.ft-372c72db22c34e6f9ccb62c26ee0fbd9:ckpt-step-80",
+      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-372c72db22c34e6f9ccb62c26ee0fbd9:ckpt-step-80",
       "fine_tuning_job_id": "ftjob-372c72db22c34e6f9ccb62c26ee0fbd9",
       "metrics": {
         "full_valid_loss": 1.6909511662736725,
@@ -848,7 +850,7 @@ Alternatively, you can deploy your fine-tuned model using any of the other commo
 | resource_group | The resource group name for your Azure OpenAI resource |
 | resource_name | The Azure OpenAI resource name |
 | model_deployment_name | The custom name for your new fine-tuned model deployment. This is the name that will be referenced in your code when making chat completion calls. |
-| fine_tuned_model | Retrieve this value from your fine-tuning job results in the previous step. It will look like `gpt-35-turbo-0613.ft-b044a9d3cf9c4228b5d393567f693b83`. You'll need to add that value to the deploy_data json. |
+| fine_tuned_model | Retrieve this value from your fine-tuning job results in the previous step. It will look like `gpt-4o-mini-2024-07-18.ft-b044a9d3cf9c4228b5d393567f693b83`. You'll need to add that value to the deploy_data json. |
 
 [!INCLUDE [Fine-tuning deletion](../includes/fine-tune.md)]
 
@@ -862,7 +864,7 @@ token = os.getenv("TEMP_AUTH_TOKEN")
 subscription = "<YOUR_SUBSCRIPTION_ID>"
 resource_group = "<YOUR_RESOURCE_GROUP_NAME>"
 resource_name = "<YOUR_AZURE_OPENAI_RESOURCE_NAME>"
-model_deployment_name = "YOUR_CUSTOM_MODEL_DEPLOYMENT_NAME"
+model_deployment_name = "gpt-4o-mini-2024-07-18-ft" # Custom deployment name you chose for your fine-tuning model
 
 deploy_params = {'api-version': "2023-05-01"}
 deploy_headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/json'}
@@ -872,7 +874,7 @@ deploy_data = {
     "properties": {
         "model": {
             "format": "OpenAI",
-            "name": "<YOUR_FINE_TUNED_MODEL>", #retrieve this value from the previous call, it will look like gpt-35-turbo-0613.ft-b044a9d3cf9c4228b5d393567f693b83
+            "name": "<YOUR_FINE_TUNED_MODEL>", #retrieve this value from the previous call, it will look like gpt-4o-mini-2024-07-18.ft-b044a9d3cf9c4228b5d393567f693b83
             "version": "1"
         }
     }
@@ -911,11 +913,11 @@ from openai import AzureOpenAI
 client = AzureOpenAI(
   azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"),
   api_key = os.getenv("AZURE_OPENAI_API_KEY"),
-  api_version = "2024-02-01"
+  api_version = "2024-06-01"
 )
 
 response = client.chat.completions.create(
-    model = "gpt-35-turbo-ft", # model = "Custom deployment name you chose for your fine-tuning model"
+    model = "gpt-4o-mini-2024-07-18-ft", # model = "Custom deployment name you chose for your fine-tuning model"
     messages = [
         {"role": "system", "content": "You are a helpful assistant."},
         {"role": "user", "content": "Does Azure OpenAI support customer managed keys?"},
@@ -937,11 +939,11 @@ import openai
 
 openai.api_type = "azure"
 openai.api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
-openai.api_version = "2024-02-01"
+openai.api_version = "2024-06-01"
 openai.api_key = os.getenv("AZURE_OPENAI_API_KEY")
 
 response = openai.ChatCompletion.create(
-    engine = "gpt-35-turbo-ft", # engine = "Custom deployment name you chose for your fine-tuning model"
+    engine = "gpt-4o-mini-2024-07-18-ft", # engine = "Custom deployment name you chose for your fine-tuning model"
     messages = [
         {"role": "system", "content": "You are a helpful assistant."},
         {"role": "user", "content": "Does Azure OpenAI support customer managed keys?"},
