Azure.AI.Projects beta 10 (Azure#51096)

* Sramsey/datasets (Azure#50459)

* regenerate code based on commit 06308c1
* change return type of `UploadFile` and `UploadFolder` from Response to DatasetVersion to better align with python + js
* add connectionName input for `UploadFile` and `UploadFolder`
* update snippets + api

* add datasets regex filter (Azure#50649)

* add optional regex filter for UploadFolder
* update samples/api

* regenerate code / remove body from datasets (Azure#50767)

* regenerate code / remove body from datasets

* update changelog

* [Azure.AI.Projects] Remove Inference related code (Azure#50774)

* Remove Inference related code. Modify samples to just use the base Inference client, by modifying the project endpoint

Signed-off-by: trangevi <[email protected]>

* missed the package dependency

Signed-off-by: trangevi <[email protected]>

* Missed an import

Signed-off-by: trangevi <[email protected]>

* Update snippets. Update readme

Signed-off-by: trangevi <[email protected]>

* API updates

Signed-off-by: trangevi <[email protected]>

* snippet fixes

Signed-off-by: trangevi <[email protected]>

---------

Signed-off-by: trangevi <[email protected]>

* change string to uri (Azure#50831)

* change DataUri, BlobUri, and SasUri from type string to uri, update references
* update api to reflect changes

* replace GetClient() with properties (Azure#50893)

* add functionality to use `Connections`, `Datasets`, `Deployments`, and `Indexes` properties instead of `GetConnectionsClient()`, `GetDatasetsClient()`, `GetDeploymentsClient()`, and `GetIndexesClient()`
* update changelog, samples, and api
* still left to do: mark `GetConnectionsClient()`, `GetDatasetsClient()`, `GetDeploymentsClient()`, and `GetIndexesClient()` as internal

* change Get*Client() functions to internal (Azure#50943)

* update custom code for `GetConnectionsClient()`, `GetDatasetsClient()`, `GetDeploymentsClient()`, `GetEvaluationsClient()`, `GetIndexesClient()`, and `GetRedteamsClient()` to be internal
* update api and samples
* emitted the code with `generate-sample-project: false`, so that Generated/Docs and Generated/Samples would not be generated to use internal functions

* add GetModelDeployment (Azure#50964)

* rename connection, index, SasCredential, Sku (Azure#50997)

* rename connection, index, SasCredential, Sku

* update changelog

* rename deployment (Azure#51000)

* Add aoai embeddings (Azure#51095)

* add aoai text embeddings

* update model_deployment_name

* Additional fixes for dataset upload (Azure#51138)

* Fix authentication on data sets

* Allow entra ID auth if SAS is not available

* address comments (Azure#51222)

---------

Signed-off-by: trangevi <[email protected]>
Co-authored-by: trangevi <[email protected]>
Co-authored-by: Nikolay Rovinskiy <[email protected]>

Author: 3 people

sdk/ai/Azure.AI.Projects/CHANGELOG.md

@@ -3,11 +3,22 @@
 ## 1.0.0-beta.10 (Unreleased)
 
 ### Features Added
+* Support for generating embeddings through Azure OpenAI using `GetAzureOpenAIEmbeddingClient`.
 
 ### Breaking Changes
+* Name changes:
+  * In `Datasets` methods `PendingUpload` and `PendingUploadAsync`, argument `body` was replaced with `pendingUploadRequest`
+  * `Connection` has been renamed `ConnectionProperties`
+  * `Deployment` has been renamed `AIDeployment`
+  * `Index` has been renamed `DatasetIndex`
+  * `SasCredential` has been renamed `BlobReferenceSasCredential`
+  * `Sku` has been renamed `ModelDeploymentSku`
+* Removing `GetChatCompletionsClient`, `GetEmbeddingsClient`, and `GetImageEmbeddingsClient` methods from `AIProjectClient`. The Inference client should be used directly instead.
+* Replacing `GetConnectionsClient`, `GetDatasetsClient`, `GetDeploymentsClient`, and`GetIndexesClient` with `Connections`, `Datasets`, `Deployments`, and `Indexes` properties.
 
 ### Bugs Fixed
 * Fix getting model deployed on the Azure Open AI resource, if the resource is authenticated using Entra ID. [see GitHub issue 49064](https://github.com/Azure/azure-sdk-for-net/issues/49064)
+* Fix dataset uploading datasets.
 
 ### Other Changes
 

sdk/ai/Azure.AI.Projects/README.md

@@ -159,12 +159,15 @@ The code below assumes `ModelDeploymentName` (a string) is defined. It's the dep
 
 You can update the `connectionName` with one of the connections in your Foundry project, and you can update the `apiVersion` value with one found in the "Data plane - inference" row [in this table](https://learn.microsoft.com/azure/ai-services/openai/reference#api-specs).
 
-```C# Snippet:AI_Projects_AzureOpenAISync
+```C# Snippet:AI_Projects_AzureOpenAIChatSync
 var endpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
 var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
+var connectionName = System.Environment.GetEnvironmentVariable("CONNECTION_NAME");
+Console.WriteLine("Create the Azure OpenAI chat client");
 AIProjectClient projectClient = new AIProjectClient(new Uri(endpoint), new DefaultAzureCredential());
-ChatClient chatClient = projectClient.GetAzureOpenAIChatClient(deploymentName: modelDeploymentName, connectionName: null, apiVersion: null);
+ChatClient chatClient = projectClient.GetAzureOpenAIChatClient(deploymentName: modelDeploymentName, connectionName: connectionName, apiVersion: null);
 
+Console.WriteLine("Complete a chat");
 ChatCompletion result = chatClient.CompleteChat("List all the rainbow colors");
 Console.WriteLine(result.Content[0].Text);
 ```
@@ -176,10 +179,17 @@ Your Azure AI Foundry project may have one or more AI models deployed that suppo
 The code below assumes `ModelDeploymentName` (a string) is defined. It's the deployment name of an AI model in your Foundry Project, or a connected Azure OpenAI resource. As shown in the "Models + endpoints" tab, under the "Name" column.
 
 ```C# Snippet:AI_Projects_ChatClientSync
-var endpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
+var projectEndpoint = new Uri(System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT"));
 var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
-AIProjectClient client = new AIProjectClient(new Uri(endpoint), new DefaultAzureCredential());
-ChatCompletionsClient chatClient = client.GetChatCompletionsClient();
+var inferenceEndpoint = $"{projectEndpoint.GetLeftPart(UriPartial.Authority)}/models";
+
+AzureAIInferenceClientOptions clientOptions = new AzureAIInferenceClientOptions();
+
+var credential = new DefaultAzureCredential();
+BearerTokenAuthenticationPolicy tokenPolicy = new BearerTokenAuthenticationPolicy(credential, new string[] { "https://ai.azure.com/.default" });
+clientOptions.AddPolicy(tokenPolicy, HttpPipelinePosition.PerRetry);
+
+ChatCompletionsClient chatClient = new ChatCompletionsClient(new Uri(inferenceEndpoint), credential, clientOptions);
 
 var requestOptions = new ChatCompletionsOptions()
 {
@@ -203,22 +213,21 @@ var endpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
 var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
 var modelPublisher = System.Environment.GetEnvironmentVariable("MODEL_PUBLISHER");
 AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
-Deployments deployments = projectClient.GetDeploymentsClient();
 
 Console.WriteLine("List all deployments:");
-foreach (var deployment in deployments.GetDeployments())
+foreach (AIDeployment deployment in projectClient.Deployments.GetDeployments())
 {
     Console.WriteLine(deployment);
 }
 
 Console.WriteLine($"List all deployments by the model publisher `{modelPublisher}`:");
-foreach (var deployment in deployments.GetDeployments(modelPublisher: modelPublisher))
+foreach (AIDeployment deployment in projectClient.Deployments.GetDeployments(modelPublisher: modelPublisher))
 {
     Console.WriteLine(deployment);
 }
 
-Console.WriteLine($"Get a single deployment named `{modelDeploymentName}`:");
-var deploymentDetails = deployments.GetDeployment(modelDeploymentName);
+Console.WriteLine($"Get a single model deployment named `{modelDeploymentName}`:");
+ModelDeployment deploymentDetails = projectClient.Deployments.GetModelDeployment(modelDeploymentName);
 Console.WriteLine(deploymentDetails);
 ```
 
@@ -230,35 +239,34 @@ The code below shows some Connection operations, which allow you to enumerate th
 var endpoint = Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
 var connectionName = Environment.GetEnvironmentVariable("CONNECTION_NAME");
 AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
-Connections connectionsClient = projectClient.GetConnectionsClient();
 
 Console.WriteLine("List the properties of all connections:");
-foreach (var connection in connectionsClient.GetConnections())
+foreach (ConnectionProperties connection in projectClient.Connections.GetConnections())
 {
     Console.WriteLine(connection);
-    Console.Write(connection.Name);
+    Console.WriteLine(connection.Name);
 }
 
 Console.WriteLine("List the properties of all connections of a particular type (e.g., Azure OpenAI connections):");
-foreach (var connection in connectionsClient.GetConnections(connectionType: ConnectionType.AzureOpenAI))
+foreach (ConnectionProperties connection in projectClient.Connections.GetConnections(connectionType: ConnectionType.AzureOpenAI))
 {
     Console.WriteLine(connection);
 }
 
 Console.WriteLine($"Get the properties of a connection named `{connectionName}`:");
-var specificConnection = connectionsClient.Get(connectionName, includeCredentials: false);
+ConnectionProperties specificConnection = projectClient.Connections.Get(connectionName, includeCredentials: false);
 Console.WriteLine(specificConnection);
 
 Console.WriteLine("Get the properties of a connection with credentials:");
-var specificConnectionCredentials = connectionsClient.Get(connectionName, includeCredentials: true);
+ConnectionProperties specificConnectionCredentials = projectClient.Connections.Get(connectionName, includeCredentials: true);
 Console.WriteLine(specificConnectionCredentials);
 
 Console.WriteLine($"Get the properties of the default connection:");
-var defaultConnection = connectionsClient.GetDefault(includeCredentials: false);
+ConnectionProperties defaultConnection = projectClient.Connections.GetDefault(includeCredentials: false);
 Console.WriteLine(defaultConnection);
 
 Console.WriteLine($"Get the properties of the default connection with credentials:");
-var defaultConnectionCredentials = connectionsClient.GetDefault(includeCredentials: true);
+ConnectionProperties defaultConnectionCredentials = projectClient.Connections.GetDefault(includeCredentials: true);
 Console.WriteLine(defaultConnectionCredentials);
 ```
 
@@ -268,45 +276,57 @@ The code below shows some Dataset operations. Full samples can be found under th
 
 ```C# Snippet:AI_Projects_DatasetsExampleSync
 var endpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
+var connectionName = Environment.GetEnvironmentVariable("CONNECTION_NAME");
 var datasetName = System.Environment.GetEnvironmentVariable("DATASET_NAME");
+var datasetVersion1 = System.Environment.GetEnvironmentVariable("DATASET_VERSION_1") ?? "1.0";
+var datasetVersion2 = System.Environment.GetEnvironmentVariable("DATASET_VERSION_2") ?? "2.0";
+var filePath = System.Environment.GetEnvironmentVariable("SAMPLE_FILE_PATH") ?? "sample_folder/sample_file1.txt";
+var folderPath = System.Environment.GetEnvironmentVariable("SAMPLE_FOLDER_PATH") ?? "sample_folder";
 AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
-Datasets datasets = projectClient.GetDatasetsClient();
 
-Console.WriteLine("Uploading a single file to create Dataset version '1'...");
-var datasetResponse = datasets.UploadFile(
+Console.WriteLine($"Uploading a single file to create Dataset version {datasetVersion1}:");
+FileDatasetVersion fileDataset = projectClient.Datasets.UploadFile(
     name: datasetName,
-    version: "1",
-    filePath: "sample_folder/sample_file1.txt"
+    version: datasetVersion1,
+    filePath: filePath,
+    connectionName: connectionName
     );
-Console.WriteLine(datasetResponse);
+Console.WriteLine(fileDataset);
 
-Console.WriteLine("Uploading folder to create Dataset version '2'...");
-datasetResponse = datasets.UploadFolder(
+Console.WriteLine($"Uploading folder to create Dataset version {datasetVersion2}:");
+FolderDatasetVersion folderDataset = projectClient.Datasets.UploadFolder(
     name: datasetName,
-    version: "2",
-    folderPath: "sample_folder"
+    version: datasetVersion2,
+    folderPath: folderPath,
+    connectionName: connectionName,
+    filePattern: new Regex(".*\\.txt")
 );
-Console.WriteLine(datasetResponse);
+Console.WriteLine(folderDataset);
 
-Console.WriteLine("Retrieving Dataset version '1'...");
-DatasetVersion dataset = datasets.GetDataset(datasetName, "1");
+Console.WriteLine($"Retrieving Dataset version {datasetVersion1}:");
+DatasetVersion dataset = projectClient.Datasets.GetDataset(datasetName, datasetVersion1);
 Console.WriteLine(dataset);
 
+Console.WriteLine($"Retrieving credentials of Dataset {datasetName} version {datasetVersion1}:");
+AssetCredentialResponse credentials = projectClient.Datasets.GetCredentials(datasetName, datasetVersion1);
+Console.WriteLine(credentials);
+
 Console.WriteLine($"Listing all versions for Dataset '{datasetName}':");
-foreach (var ds in datasets.GetVersions(datasetName))
+foreach (DatasetVersion ds in projectClient.Datasets.GetVersions(datasetName))
 {
     Console.WriteLine(ds);
+    Console.WriteLine(ds.Version);
 }
 
 Console.WriteLine($"Listing latest versions for all datasets:");
-foreach (var ds in datasets.GetDatasetVersions())
+foreach (DatasetVersion ds in projectClient.Datasets.GetDatasetVersions())
 {
     Console.WriteLine(ds);
 }
 
-Console.WriteLine("Deleting Dataset versions '1' and '2'...");
-datasets.Delete(datasetName, "1");
-datasets.Delete(datasetName, "2");
+Console.WriteLine($"Deleting Dataset versions {datasetVersion1} and {datasetVersion2}:");
+projectClient.Datasets.Delete(datasetName, datasetVersion1);
+projectClient.Datasets.Delete(datasetName, datasetVersion2);
 ```
 
 ### Indexes operations
@@ -320,44 +340,42 @@ var indexVersion = Environment.GetEnvironmentVariable("INDEX_VERSION") ?? "1.0";
 var aiSearchConnectionName = Environment.GetEnvironmentVariable("AI_SEARCH_CONNECTION_NAME") ?? "my-ai-search-connection-name";
 var aiSearchIndexName = Environment.GetEnvironmentVariable("AI_SEARCH_INDEX_NAME") ?? "my-ai-search-index-name";
 AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
-Indexes indexesClient = projectClient.GetIndexesClient();
 
 RequestContent content = RequestContent.Create(new
 {
     connectionName = aiSearchConnectionName,
     indexName = aiSearchIndexName,
-    indexVersion = indexVersion,
     type = "AzureSearch",
     description = "Sample Index for testing",
     displayName = "Sample Index"
 });
 
 Console.WriteLine($"Create an Index named `{indexName}` referencing an existing AI Search resource:");
-var index = indexesClient.CreateOrUpdate(
+var index = projectClient.Indexes.CreateOrUpdate(
     name: indexName,
     version: indexVersion,
     content: content
 );
 Console.WriteLine(index);
 
 Console.WriteLine($"Get an existing Index named `{indexName}`, version `{indexVersion}`:");
-var retrievedIndex = indexesClient.GetIndex(name: indexName, version: indexVersion);
+DatasetIndex retrievedIndex = projectClient.Indexes.GetIndex(name: indexName, version: indexVersion);
 Console.WriteLine(retrievedIndex);
 
 Console.WriteLine($"Listing all versions of the Index named `{indexName}`:");
-foreach (var version in indexesClient.GetVersions(name: indexName))
+foreach (DatasetIndex version in projectClient.Indexes.GetVersions(name: indexName))
 {
     Console.WriteLine(version);
 }
 
 Console.WriteLine($"Listing all Indices:");
-foreach (var version in indexesClient.GetIndices())
+foreach (DatasetIndex version in projectClient.Indexes.GetIndices())
 {
     Console.WriteLine(version);
 }
 
 Console.WriteLine("Delete the Index version created above:");
-indexesClient.Delete(name: indexName, version: indexVersion);
+projectClient.Indexes.Delete(name: indexName, version: indexVersion);
 ```
 
 ## Troubleshooting
@@ -367,7 +385,7 @@ Any operation that fails will throw a [RequestFailedException][RequestFailedExce
 ```C# Snippet:AI_Projects_Readme_Troubleshooting
 try
 {
-    projectClient.GetDatasetsClient().GetDataset("non-existent-dataset-name", "non-existent-dataset-version");
+    projectClient.Datasets.GetDataset("non-existent-dataset-name", "non-existent-dataset-version");
 }
 catch (RequestFailedException ex) when (ex.Status == 404)
 {