DPG stater package script for cadl and doc (Azure#33113)

chunyu3 · web-flow · commit 633474503e68 · 2023-01-11T11:14:59.000+08:00
* update ci script

* support local swagger
diff --git a/doc/DataPlaneCodeGeneration/Autorest_DataPlane_Quickstart.md b/doc/DataPlaneCodeGeneration/Autorest_DataPlane_Quickstart.md
@@ -10,7 +10,7 @@ This tutorial has following sections:
 
 - [Azure SDK Code Generation Quickstart Tutorial (Data Plane)](#azure-sdk-code-generation-quickstart-tutorial-data-plane)
   - [Prerequisites](#prerequisites)
-  - [Setup your repo](#setup-your-repo)
+  - [Setup your repository](#setup-your-repository)
   - [Create starter package](#create-starter-package)
   - [Add package ship requirements](#add-package-ship-requirements)
     - [Tests](#tests)
@@ -31,9 +31,9 @@ This tutorial has following sections:
 
 ## Prerequisites
 
-- Install VS 2022 (Community or higher) and make sure you have the [latest updates](https://www.visualstudio.com/).
+- Install Visual Studio 2022 (Community or higher) and make sure you have the [latest updates](https://www.visualstudio.com/).
   - Need at least .NET Framework 4.6.1 and 4.7 development tools
-  - Install the **.NET Core cross-platform development** workloads in VisualStudio
+  - Install the **.NET Core cross-platform development** workloads in Visual Studio
 - Install **.NET 6.0 SDK** for your specific platform. (or a higher version)
 - Install the latest version of [git](https://git-scm.com/downloads)
 - Install [PowerShell](https://docs.microsoft.com/powershell/scripting/install/installing-powershell), version 7 or higher.
diff --git a/doc/DataPlaneCodeGeneration/AzureSDKCodeGeneration_DataPlane_Quickstart.md b/doc/DataPlaneCodeGeneration/AzureSDKCodeGeneration_DataPlane_Quickstart.md
@@ -10,14 +10,11 @@ This tutorial has following sections:
 
 - [Azure SDK Code Generation Quickstart Tutorial (Data Plane)](#azure-sdk-code-generation-quickstart-tutorial-data-plane)
   - [Prerequisites](#prerequisites)
-  - [Setup your repo](#setup-your-repo)
+  - [Setup your repository](#setup-your-repository)
   - [Create starter package](#create-starter-package)
     - [Use Cadl as Input](#use-cadl-as-input)
       - [Create Cadl project](#create-cadl-project)
-      - [Create a sdk project folder](#create-a-sdk-project-folder)
-      - [Generate the library source code](#generate-the-library-source-code)
-      - [Build the library project](#build-the-library-project)
-      - [Export the library's public API](#export-the-librarys-public-api)
+      - [Create sdk starter package](#create-sdk-starter-package)
     - [Use swagger as Input](#use-swagger-as-input)
   - [Add package ship requirements](#add-package-ship-requirements)
     - [Tests](#tests)
@@ -32,18 +29,17 @@ This tutorial has following sections:
 
 ## Prerequisites
 
-- Install VS 2020 (Community or higher) and make sure you have the [latest updates](https://www.visualstudio.com/).
+- Install Visual Studio 2022 (Community or higher) and make sure you have the [latest updates](https://www.visualstudio.com/).
   - Need at least .NET Framework 4.6.1 and 4.7 development tools
-  - Install the **.NET Core cross-platform development** workloads in VisualStudio
+  - Install the **.NET Core cross-platform development** workloads in Visual Studio
 - Install **.NET 6.0 SDK** for your specific platform. (or a higher version)
-- Install **.NET core 3.1** for your specific platform.
 - Install the latest version of [git](https://git-scm.com/downloads)
 - Install [PowerShell](https://docs.microsoft.com/powershell/scripting/install/installing-powershell), version 7 or higher.
 - Install [NodeJS](https://nodejs.org/) (16.x.x or above).
 
-## Setup your repo
+## Setup your repository
 
-- Fork and clone an [azure-sdk-for-net](https://github.com/Azure/azure-sdk-for-net) repo. Follow the instructions in the [.NET CONTRIBUTING.md](https://github.com/Azure/azure-sdk-for-net/issues/12903) to fork and clone the `azure-sdk-for-net` repo.
+- Fork and clone the [azure-sdk-for-net](https://github.com/Azure/azure-sdk-for-net) repository. Instructions for doing so can be found in the [.NET CONTRIBUTING.md](https://github.com/Azure/azure-sdk-for-net/blob/main/CONTRIBUTING.md).
 - Create a branch to work in.
 
 ## Create starter package  
@@ -123,60 +119,50 @@ emitters:
 - `clear-output-folder` indicate if you want to clear up the output folder.
 
 
-#### Create a sdk project folder
+#### Create sdk starter package
   
-You can manually create the project folder. Please refer to [Azure.Template](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template) as an example.
+We will use the Azure SDK template [Azure.Template](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template) to create the initial project skeleton.
 
-**Note**: 
+You can run `eng\scripts\automation\Invoke-CadlDataPlaneGenerateSDKPackage.ps1` to generate the starting SDK client library package directly as following:
 
-- Please refer to [sdk-directory-layout](https://github.com/Azure/azure-sdk/blob/main/docs/policies/repostructure.md#sdk-directory-layout) for detail information.
-- remove `autorest.md` from sdk\<service name>\<package name>\src
-- add `cadl-location.yaml` under sdk\<service name>\<package name>\src
-  
-  Modify `cadl-location.yaml` to add some project meta data if needed, Otherwise just add an empty file.
-
-  e.g. if you want specify cadl project folder in azure-rest-api-spec repo, you can define (directory, commit, repo) to specify the cadl project.
-
-  ```yml
-  directory: specification/cognitiveservices/data-plane/AnomalyDetector/stable/v1.1/cadl
-  commit: 8804b10cf61267b81e06ebccbd1dd46677a54425
-  repo: Azure/azure-rest-api-specs
-  cleanup: false
-  ```
-
-#### Generate the library source code
-
-Generate the library source code files to the directory `<sdkPath>/sdk/<service>/<namespace>/src/Generated`
+```powershell
+eng/scripts/automation/Invoke-CadlDataPlaneGenerateSDKPackage.ps1 -service <servicename> -namespace Azure.<group>.<service> -sdkPath <sdkrepoRootPath> -cadlRelativeFolder <relativeCadlProjectFolderPath> [-commit <commitId>] [-repo <specRepo>] [-specRoot <specRepoRootPath>] [-additionalSubDirectories <relativeFolders>]
+```
 
-  Enter `Cadl Project Folder`, run
+e.g. 
+Use git url
 
-  ```shell
-  npm install
-  npx cadl compile --emit @azure-tools/cadl-csharp --output-path <Path-to-source-code-folder> <path-to-cadl-file>
-  ```
-
-  e.g.
+```powershell
+pwsh /home/azure-sdk-for-net/eng/scripts/automation/Invoke-CadlDataPlaneGenerateSDKPackage.ps1 -service anomalydetector -namespace Azure.AI.AnomalyDetector -sdkPath /home/azure-sdk-for-net -cadlRelativeFolder specification/cognitiveservices/AnomalyDetector -commit ac8e06a2ed0fc1c54663c98f12c8a073f8026b90 -repo Azure/azure-rest-api-specs
+```
+or 
+Use local Cadl project
 
-  ```shell
-  npx cadl compile --emit @azure-tools/cadl-csharp --output-path /home/azure-sdk-for-net/sdk/deviceupdate/Azure.IoT.DeviceUpdate/src main.cadl
-  ```
+```powershell
+pwsh /home/azure-sdk-for-net/eng/scripts/automation/Invoke-CadlDataPlaneGenerateSDKPackage.ps1 -service anomalydetector -namespace Azure.AI.AnomalyDetector -sdkPath /home/azure-sdk-for-net -cadlRelativeFolder specification/cognitiveservices/AnomalyDetector -specRoot /home/azure-rest-api-specs
+```
 **Note**:
 
-- `<path-to-cadl-file>` - Should be the entrypoint cadl file of the cadl project. e.g. When you define `client.cadl` to customize the client, the `client.cadl` will be the entrypoint cadl file, and you should specify the path to the `client.cadl`.
-  
-#### Build the library project
+- `-service` takes Azure client service directory name. ie. purview. It equals to the name of the directory in the specification folder of the azure-rest-api-specs repo that contains the REST API definition file.
+- For `- namespace`, please use one of the pre-approved namespace groups on the [.NET Azure SDK Guidelines Approved Namespaces list](https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-namespaces-approved-list). This value will also provide the name for the shipped package, and should be of the form `Azure.<group>.<service>`.
+- `-sdkPath` takes the address of the root directory of sdk repo. e.g. /home/azure-sdk-for-net
+- `cadlRelativeFolder` takes the relative path of the cadl project folder in spec repo. e.g. specification/cognitiveservices/AnomalyDetector
+- `-additionalSubDirectories` takes the relative paths of the additional directories needed by the cadl project, such as share library folder, separated by semicolon if there is more than one folder.
+- `-commit` takes the git commit hash  (e.g. ac8e06a2ed0fc1c54663c98f12c8a073f8026b90)
+- `-repo` takes the `<owner>/<repo>` of the REST API specification repository. (e.g. Azure/azure-rest-api-specs)
+- `-specRoot` takes the file system path of the spec repo. e.g. /home/azure-rest-api-specs
+- You need to provide one of (`-commit`, `-repo`) pair to refer to an URL path of the cadl project and `-specRoot` parameters. If you provide both, `-specRoot` will be ignored.
 
-Run `dotnet build` under project folder ``<sdkPath>/sdk/<service>/<namespace>` to build the project to create the starter package binary.
+When you run `eng\scripts\automation\Invoke-CadlDataPlaneGenerateSDKPackage.ps1`, it will:
 
-#### Export the library's public API
+- Create a project folder, install template files from `sdk/template/Azure.Template`, and create `.csproj` and `.sln` files for your new library.
 
-You will need to run the `eng\scripts\Export-API.ps1` script to update the public API listing. This will generate a file in the library's directory `<sdkPath>/sdk/<service>/<namespace>/api` similar to the example found in [sdk\template\Azure.Template\api\Azure.Template.netstandard2.0.cs](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/api/Azure.Template.netstandard2.0.cs).
+    These files are created following the guidance for the [Azure SDK Repository Structure](https://github.com/Azure/azure-sdk/blob/master/docs/policies/repostructure.md).
 
-e.g. Running the script for a project in `sdk\deviceupdate` would look like this:
+- Generate the library source code files to the directory `<sdkPath>/sdk/<service>/<namespace>/src/Generated`
+- Build the library project to create the starter package binary.
+- Export the library's public API to the directory `<sdkPath>/sdk/<service>/<namespace>/api`
 
-```powershell
-eng\scripts\Export-API.ps1 deviceupdate
-```
 
 ### Use swagger as Input
 
@@ -237,7 +223,8 @@ Adding convenience APIs is not required for Azure SDK data plane generated libra
 
 You can add convienice APIs by adding a customization layer on top of the generated code.  Please see the [autorest.csharp README](https://github.com/Azure/autorest.csharp#setup) for the details of adding the customization layer.  This is the preferred method for adding convenience APIs to your generated client.
 
-If other modifications are needed to the generated API, you can consider making them directly to the Open API specification, which will have the benefit of making the changes to the library in all languages you generate the library in.  As a last resort, you can add modifications with swagger transforms in the `autorest.md` file.  [AnomalyDetector autorest.md](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/anomalydetector/Azure.AI.AnomalyDetector/src/autorest.md) shows and example of how this can be accomplished.
+
+If other modifications are needed to the generated API, you can consider making them directly to the Open API specification, which will have the benefit of making the changes to the library in all languages you generate the library in.  As a last resort, you can add modifications with swagger transforms in the `autorest.md` file.  Details for various transforms can be found in [Customizing the generated code](https://github.com/Azure/autorest.csharp#customizing-the-generated-code).
 
 Once you've made changes to the public API, you will need to run the `eng\scripts\Export-API.ps1` script to update the public API listing. This will generate a file in the library's directory similar to the example found in [sdk\template\Azure.Template\api\Azure.Template.netstandard2.0.cs](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/api/Azure.Template.netstandard2.0.cs).
 
diff --git a/eng/scripts/Cadl-Project-Generate.ps1 b/eng/scripts/Cadl-Project-Generate.ps1
@@ -42,20 +42,27 @@ $npmWorkingDir = Resolve-Path $tempFolder/$innerFolder
 $mainCadlFile = If (Test-Path "$npmWorkingDir/client.cadl") { Resolve-Path "$npmWorkingDir/client.cadl" } Else { Resolve-Path "$npmWorkingDir/main.cadl"}
 $resolvedProjectDirectory = Resolve-Path "$ProjectDirectory/src"
 
+$HasEmitterOutputDir = $false;
 if (Test-Path "$npmWorkingDir/cadl-project.yaml") {
     $cadlProjectYaml = Get-Content -Path "$npmWorkingDir/cadl-project.yaml" -Raw
     $yml = ConvertFrom-YAML $cadlProjectYaml
-    if ( $yml["emitters"]["@azure-tools/cadl-csharp"]["sdk-folder"]) {
-        $yml["emitters"]["@azure-tools/cadl-csharp"]["sdk-folder"] = ""
+    if ($yml -And $yml["options"] -And $yml["options"]["@azure-tools/cadl-csharp"] -And $yml["options"]["@azure-tools/cadl-csharp"]["emitter-output-dir"]) {
+        $HasEmitterOutputDir = $true
     }
-    (ConvertTo-Yaml $yml) | Out-File "$npmWorkingDir/cadl-project.yaml"
 }
 
 try {
     Push-Location $npmWorkingDir
     NpmInstallForProject $npmWorkingDir
-    Write-Host("npx cadl compile $mainCadlFile --emit `"`@azure-tools/cadl-csharp`" --output-path `"$resolvedProjectDirectory`"")
-    npx cadl compile $mainCadlFile --emit "@azure-tools/cadl-csharp" --output-path "$resolvedProjectDirectory"
+    if ($HasEmitterOutputDir) {
+        $csharpSdkFolder = (Join-Path $ProjectDirectory ".." ".." "..")
+        $csharpSdkFolder = Resolve-Path $csharpSdkFolder
+        Write-Host("npx cadl compile $mainCadlFile --emit `"`@azure-tools/cadl-csharp`" --arg `"csharp-sdk-folder=$csharpSdkFolder`"")
+        npx cadl compile $mainCadlFile --emit "@azure-tools/cadl-csharp" --arg "csharp-sdk-folder=$csharpSdkFolder"
+    } else {
+        Write-Host("npx cadl compile $mainCadlFile --emit `"`@azure-tools/cadl-csharp`" --option `"@azure-tools/cadl-csharp.emitter-output-dir=$resolvedProjectDirectory`"")
+        npx cadl compile $mainCadlFile --emit "@azure-tools/cadl-csharp" --option @azure-tools/cadl-csharp.emitter-output-dir=$resolvedProjectDirectory
+    }
 }
 finally {
     Pop-Location
diff --git a/eng/scripts/Invoke-GenerateAndBuildV2.ps1 b/eng/scripts/Invoke-GenerateAndBuildV2.ps1
@@ -119,16 +119,20 @@ if ($relatedCadlProjectFolder) {
 
         Install-ModuleIfNotInstalled "powershell-yaml" "0.4.1" | Import-Module
         $yml = ConvertFrom-YAML $cadlProjectYaml
-        $sdkFolder = $yml["emitters"]["@azure-tools/cadl-csharp"]["sdk-folder"]
-        $projectFolder = (Join-Path $sdkPath $sdkFolder)
-        # $projectFolder = $projectFolder -replace "\\", "/"
-        if ($projectFolder) {
-            $directories = $projectFolder -split "/|\\"
-            $count = $directories.Count
-            $projectFolder = $directories[0 .. ($count-2)] -join "/"
-            $service = $directories[-3];
-            $namespace = $directories[-2];
+        $service = ""
+        $namespace = ""
+        if ($yml) {
+            if ($yml["parameters"] -And $yml["parameters"]["service-directory-name"]) {
+                $service = $yml["parameters"]["service-directory-name"]["default"];
+            }
+            if ($yml["options"] -And $yml["options"]["@azure-tools/cadl-csharp"] -And $yml["options"]["@azure-tools/cadl-csharp"]["namespace"]) {
+                $namespace = $yml["options"]["@azure-tools/cadl-csharp"]["namespace"]
+            }
         }
+        if (!$service || !$namespace) {
+            throw "Not provide service name or namespace."
+        }
+        $projectFolder = (Join-Path $sdkPath "sdk" $service $namespace)
         New-CADLPackageFolder `
             -service $service `
             -namespace $namespace `
diff --git a/eng/scripts/automation/GenerateAndBuildLib.ps1 b/eng/scripts/automation/GenerateAndBuildLib.ps1
@@ -366,15 +366,18 @@ function CreateOrUpdateCadlConfigFile() {
         [string]$directory,
         [string]$commit = "",
         [string]$repo = "",
-        [string]$specRoot = ""
+        [string]$specRoot = "",
+        [string]$additionalSubDirectories="" #additional directories needed, separated by semicolon if more than one
+        
     )
     if (!(Test-Path -Path $cadlConfigurationFile)) {
         New-Item -Path $cadlConfigurationFile
     }
 
+    Install-ModuleIfNotInstalled "powershell-yaml" "0.4.1" | Import-Module
     $configuration = Get-Content -Path $cadlConfigurationFile -Raw | ConvertFrom-Yaml
     if ( !$configuration) {
-        $configuration = [System.Collections.Generic.Dictionary[string,string]](New-Object 'System.Collections.Generic.Dictionary[string,string]')
+        $configuration = @{}
     }
     $configuration["directory"] = $directory
     if ($commit) {
@@ -394,14 +397,14 @@ function CreateOrUpdateCadlConfigFile() {
         $configuration.Remove("spec-root-dir")
     }
 
-    $fileContent = ""
-    foreach ( $key in $configuration.keys) {
-        if ($configuration[$key]) {
-            $fileContent += ( $key + ": " + $configuration[$key] + [Environment]::NewLine)
-        }
+    if ($additionalSubDirectories) {
+        $directoryArray = [string[]]$additionalSubDirectories.Split(";")
+        $configuration["additionalDirectories"] = [Collections.Generic.List[string]]$directoryArray;
+    } else {
+        $configuration.Remove("additionalDirectories")
     }
 
-    $fileContent | Out-File $cadlConfigurationFile
+    $configuration |ConvertTo-Yaml | Out-File $cadlConfigurationFile
 }
 
 function New-CADLPackageFolder() {
@@ -413,6 +416,7 @@ function New-CADLPackageFolder() {
         [string]$commit = "",
         [string]$repo = "",
         [string]$specRoot = "",
+        [string]$additionalSubDirectories="", #additional directories needed, separated by semicolon if more than one
         [string]$outputJsonFile = "$PWD/output.json"
     )
     $serviceFolder = (Join-Path $sdkPath "sdk" $service)
@@ -435,7 +439,8 @@ function New-CADLPackageFolder() {
             -directory $relatedCadlProjectFolder `
             -commit $commit `
             -repo $repo `
-            -specRoot $specRoot
+            -specRoot $specRoot `
+            -additionalSubDirectories $additionalSubDirectories
         
         Update-CIYmlFile -ciFilePath $ciymlFilePath -artifact $namespace
     } else {
@@ -484,7 +489,8 @@ function New-CADLPackageFolder() {
             -directory $relatedCadlProjectFolder `
             -commit $commit `
             -repo $repo `
-            -specRoot $specRoot
+            -specRoot $specRoot `
+            -additionalSubDirectories $additionalSubDirectories
 
         dotnet sln remove src/$namespace.csproj
         dotnet sln add src/$namespace.csproj
diff --git a/eng/scripts/automation/Invoke-CadlDataPlaneGenerateSDKPackage.ps1 b/eng/scripts/automation/Invoke-CadlDataPlaneGenerateSDKPackage.ps1
diff --git a/eng/scripts/automation/Invoke-GenerateAndBuild.ps1 b/eng/scripts/automation/Invoke-GenerateAndBuild.ps1
