Dpg and ApiView integration doc (#19122)

Author: dw511214992

documentation/onboard-dpg-in-sdkautomation/.net/README.md

@@ -0,0 +1,22 @@
+# Add Autorest Configuration of .Net SDK
+
+## Parameter Description
+
+- `<ServiceName>`: The RP name, which is usually same as folder name in swagger.
+- `<NameSpace>`: Python package name.
+- `<Title>`: SDK Client Name. It's optional if there is a title defined in spec readme.md file.
+
+## Autorest Configuration
+Please copy the following configuration into spec PR comment.
+~~~
+# azure-sdk-for-net-track2
+``` yaml
+title: <Title>
+output-folder: sdk/<ServiceName>/<NameSpace>
+require:
+ - specification/<RPName>/data-plane/readme.md
+```
+~~~
+- `title`: If it's already defined in spec readme.md, you don't need to define it here again. (By default, there is no `title` defined in spec readme.md in multi client scenario.)
+- `output-folder`: The relative path of destination to generate SDK.
+- `require`: The item of the value is the relative path of spec readme.md file.

documentation/onboard-dpg-in-sdkautomation/Integrate-dpg-and-apiview-in-sdk-automation-pipeline.md

@@ -0,0 +1,32 @@
+# Integrate DPG and ApiView in SDK Automation Pipeline
+This document is targeting for everyone who wants to know all the implementation details of SDK automation pipeline.
+It describes the details of integrating DPG and ApiView in SDK Automation Pipeline.
+
+Before go through this document, please go through [Service Onboard DPG with Swagger CI Pipeline](README.md) to be familiar with the user experience in service team side. Also, you need to be familiar with [SDK Automation Pipeline Framework](../sdkautomation/README.md).
+
+# WorkFlow
+![integrate-dpg-and-apiview](imgs/integrate-dpg-and-apiview.png)
+
+__Description:__
+1. SDK Automation Pipeline is triggered in spec PR CI.
+2. SDK Automation Pipeline Framework checks whether there is a comment containing autorest configuration in spec PR. If found, generate sdk by the autorest configuration in comment.
+Otherwise, sdk automation pipeline will try to find if there is autorest configuration file in sdk repo. If found, generate sdk with the founded autorest configuration file. If not found, pipeline stops.
+   1. When there is a comment containing autorest configuration, sdk automation pipeline will extract the autorest configuration from the comment, and let automation script use it to generate a new autorest configuration file, and use the new generated autorest configuration file to generate SDK.
+      1. We need to make the comment as simple as possible. Currently, the common fields needed by all sdk are `output-folder` and `require` keyword.
+      2. When getting the comment, automation script needs to parse it to extract needed information, and then use the extracted information to generate autorest configuration file in sdk repository.
+      3. The value of `output-folder` is a relative path from sdk root folder. It is mainly used to extract information, such as service name, package name. In generating autorest configuration file in sdk repository, please replace it to the correct value.
+      4. The value of `require` is a relative path to readme.md from spec root folder. In generating autorest configuration file in sdk repository, please concat relative path from package folder to sdk repo and `specFolder` in `generateInput.json`. For example:
+         ```yaml
+         require:
+           - ../../../../../azure-rest-api-specs/specification/deviceupdate/data-plane/readme.md
+         ```
+         *You can also use the absolute path of spec repo, which can be resolved by yourself.*
+      5. If there is other metadata in autorest configuration of spec PR comment, please copy them to autorest configuration file directly. (Although we only list little metadata service team can use in spec PR comment, we still need to support to add more metadata because the metadata listed in document maybe not enough.)
+   2. When using the autorest configuration file found in sdk repository to generate SDK, SDK automation script need to modify the autorest configuration file in the SDK repo.
+      1. If `require` block is used, change the `require` block to include the latest swagger `readme.md` in the PR.
+      2. `input-file` will not be used. If the existing autorest configuration file uses `input-file`, we should change it to `require` before triggering the sdk automation pipeline, or add the autorest configuration in spec comment.
+3. After generating SDK, SDK automation pipeline generates ApiView and then add comments about results to the Swagger PR.
+
+# Future Work
+Currently, we ask service team must add a comment containing autorest configuration in spec PR when new service onboards DPG.
+In the future, we are going to integrate it with PowerAPP Workflow. Then service team can provide the necessary information in PowerAPP Workflow, and the information will be added to spec PR automatically. 

documentation/onboard-dpg-in-sdkautomation/README.md

@@ -0,0 +1,43 @@
+# Service Onboard DPG with Swagger CI Pipeline
+
+Service Onboarding DPG with Swagger CI pipeline can help you get generated SDK and ApiView during the procedure of creating swagger PR.
+The benefits are following:
+1. API signature could be reviewed at the same time when we submit a Swagger PR. This could help to detect API issues at the early stage.
+2. It enables service teams to access their SDKs even before their Swagger PR is merged.
+
+This Document describes the process of how to onboard DPG with SDK automation pipeline. __This document is targeting for service team, who wants to onboard DPG.__ 
+
+*If you want to go through details of pipeline implementation, please go to [Integrate DPG and ApiView in SDK Automation Pipeline](Integrate-dpg-and-apiview-in-sdk-automation-pipeline.md).*
+
+## WorkFlow
+
+![workflow-for-service-team](imgs/workflow-service-team.png)
+
+__Description:__
+1. Creates a swagger PR with [Guide to design and creation of Data Plane REST API and Client Libraries](https://aka.ms/azsdk/dpcodegen).
+2. If you have added a comment, which contains autorest configuration, and run `/azp run` in the spec PR, the autorest configuration in the comment will be used to generate SDK.
+Otherwise, pipeline will try to find autorest configuration file in sdk repository. If found, use the autorest configuration file to generate SDK. If not found, pipeline will stop.
+   - Please refer to [Add autorest configuration in spec comment](./add-autorest-configuration-for-dataplane-sdk.md) on how to add autorest configuration in spec comment.
+3. SDK and ApiView will be generated in spec CI pipeline, and you can find them in spec PR comment. Please refer to the [Guide of Reviewing the Api Design/Definition](https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/591/Guide-to-design-and-creation-of-Data-Plane-REST-API-and-Client-Libraries?anchor=ii.-review-the-api-design/definition) to review the API definition.
+4. If any change needed, please update the autorest configuration in spec PR comment and regenerate the SDK by adding a comment `/azp run` to swagger PR.
+
+## FAQ
+
+1. Where can I find the generated sdk and corresponding ApiView?
+
+   __Answer__: You can find them in spec PR comment. The generated SDK can be found in comment `Swagger Generation Artifacts`, and the corresponding ApiView Link can be found in comment `Generated ApiView`. Following screenshots comes from [example PR](https://github.com/Azure/azure-rest-api-specs/pull/20017)
+
+   ![](./imgs/example_generated_sdk.png)
+   ![](./imgs/example_generated_apiview.png)
+ 
+3. What should I do when I encounter error in generating SDK in swagger PR CI pipeline?
+   
+   __Answer__: Please check the comment [Swagger Generation Artifacts] and the error message of failed pipeline. If you find the error messages ask you to fix autorest configuration comment or swagger, please fix it. If you cannot understand error message, please ask sdk owners for help with mail: [email protected].
+
+4. What should I do when it creates ApiView Failed?
+
+   __Answer__: If the ApiView is created failed, you can find there is an error message: `Create ApiView failed. Please ensure your github account in Azure/Microsoft is public and re-trigger the CI. If issue still exists, please ask PR assignee for help`. If you want to get the ApiView, 
+please follow the action in error message, and make you github account's organization information public, then re-trigger the CI (You can re-trigger CI by comment `/azp run`, or close and reopen the PR). If it still cannot generate ApiView, please ask PR assignee for help. 
+
+5. Is there ApiView generated when there is no sdk generated in sdk automation pipeline?
+   __Answer__: No. Creating ApiView is based on the generated codes in sdk automation pipeline.

documentation/onboard-dpg-in-sdkautomation/add-autorest-configuration-for-dataplane-sdk.md

@@ -0,0 +1,16 @@
+# Add Autorest Configuration for Dataplane SDK
+
+## Prerequisites
+There should be a basic readme.md together with swagger, and you can refer to the [sample](../samplefiles-dp).
+
+## Add Autorest Configuration for .Net SDK
+Please refer to [Add Autorest Configuration for .Net SDK](.net/README.md).
+
+## Add Autorest Configuration for Java SDK
+Please refer to [Add Autorest Configuration for Java SDK](java/README.md).
+
+## Add Autorest Configuration for Python SDK
+Please refer to [Add Autorest Configuration for Python SDK](python/README.md).
+
+## Add Autorest Configuration for JS SDK
+Please refer to [Add Autorest Configuration for JS SDK](js/README.md).

documentation/onboard-dpg-in-sdkautomation/imgs/example_generated_apiview.png

@@ -0,0 +1,45 @@
+# Add Autorest Configuration of Java SDK 
+
+## Parameter Description
+
+- `<ServiceName>`: The RP name, which is usually same as rp name in swagger.
+- `<PackageName>`: Java package name.
+- `<Namespace>`: It should start with `com.`, and the remaining part can be got from `<PackageName>` by replacing `- with `.'. For example, if `<PackageName>` is `azure-analytics-purview-administration`, then `<Namespace>` is `com.azure.analytics.purview.administration`.
+
+
+## Single Client
+If you want to generate sdk with single client, please copy the following configuration into spec PR comment.
+~~~
+# azure-sdk-for-java
+``` yaml
+output-folder: sdk/<ServiceName>/<PackageName>
+namespace: <Namespace>
+data-plane: true
+require:
+ - specification/<RPName>/data-plane/readme.md
+```
+~~~
+- `namespace`: The namespace of generated sdk.
+- `output-folder`: The relative path of destination to generate SDK.
+- `require`: The item of the value is the relative path of spec readme.md file.
+
+## Multi Client
+If you want to generate sdk with multi client, please copy the following configuration into spec PR comment.
+~~~
+# azure-sdk-for-java
+``` yaml
+tag: false
+output-folder: sdk/<ServiceName>/<PackageName>
+namespace: <Namespace>
+data-plane: true
+require:
+ - specification/<ServiceName>/data-plane/readme.md
+batch:
+ - package-A: true
+ - package-B: true
+```
+~~~
+- `namespace`: The namespace of generated sdk.
+- `output-folder`: The relative path of destination to generate SDK.
+- `require`: The item of the value is the relative path of spec readme.md file.
+- `package-A` and `package-B` must be defined in swagger `readme.md` file. For samples, please refer to [dataplane samples for multi client](../../samplefiles-dp/samplefiles-dp-for-multi-client).

documentation/onboard-dpg-in-sdkautomation/imgs/example_generated_sdk.png

@@ -0,0 +1,45 @@
+# Add Autorest Configuration of JS SDK
+
+## Parameter Description
+
+- `<ServiceName>`: The RP name, which is usually same as folder name in swagger.
+- `<PackageFolder>`: Python package folder name, which must ends with `-rest`. For example: `purview-administration-rest`.
+- `<SubFolderName>`: The sub-folder name in the package. It's only used by multi-client scenario. 
+
+## Single Client
+If you want to generate sdk with single client, please copy the following configuration into spec PR comment.
+~~~
+# azure-sdk-for-js
+``` yaml
+output-folder: sdk/<ServiceName>/<PackageFolder>
+require:
+ - specification/<ServiceName>/data-plane/readme.md
+```
+~~~
+- `output-folder`: The relative path of destination to generate SDK.
+- `require`: The item of the value is the relative path of spec readme.md file.
+
+## Multi Client
+If you want to generate sdk with multi client, please copy the following configuration into spec PR comment.
+~~~
+# azure-sdk-for-js
+``` yaml $(multi-client)
+tag: false
+require:
+ - specification/<RPName>/data-plane/readme.md
+batch:
+ - package-A: true
+ - package-B: true
+```
+
+``` yaml $(package-A)
+output-folder: sdk/<ServiceName>/<PackageFolder>/src/<SubFolderName>
+```
+
+``` yaml $(package-B)
+output-folder: sdk/<ServiceName>/<PackageFolder>/src/<SubFolderName>
+```
+~~~
+- `output-folder`: The relative path of destination to generate SDK.
+- `require`: The item of the value is the relative path of spec readme.md file.
+- `package-A` and `package-B` must be defined in swagger `readme.md` file. For samples, please refer to [dataplane samples for multi client](../../samplefiles-dp/samplefiles-dp-for-multi-client).