Merge pull request #300087 from msftadam/patch-74

Update safe-upgrade-practices.md

Author: prmerger-automator[bot]

articles/operator-service-manager/safe-upgrade-practices.md

@@ -3,20 +3,20 @@ title: Get started with Azure Operator Service Manager Safe Upgrade Practices
 description: Safely execute complex upgrades of CNF workloads on Azure Operator Nexus
 author: msftadam
 ms.author: adamdor
-ms.date: 02/19/2024
+ms.date: 05/20/2024
 ms.topic: upgrade-and-migration-article
 ms.service: azure-operator-service-manager
 ---
 
 # Get started with safe upgrade practices
-This article introduces Azure Operator Service Manager (AOSM) safe upgrade practices (SUP). This feature set enables the safe execution of complex container network function (CNF) hosted on Azure Operator Nexus. These upgrades are structured in general compliance with partner In Service Software Upgrade (ISSU) requirements. Look for future articles to expand on advanced SUP features and capabilities.
+This article introduces Azure Operator Service Manager (AOSM) safe upgrade practices (SUP). This feature set enables upgrades to complex container network function (CNF) hosted on Azure Operator Nexus. These upgrades generally support partner In Service Software Upgrade (ISSU) methods and requirements. While this article introduces basic concepts, look for other articles which expand on advanced SUP features and capabilities.
 
 ## Introduction to safe upgrades
-A given network service supported by AOSM is composed of one to many CNFs which, over time, require software upgrades. For each upgrade, it's necessary to run one to many helm operations, updating dependent network function applications (nfApps), in a particular order, in a manner which least impacts the network service. AOSM SUP represents a set of features, which enables safe automation of these operations on Azure Operator Nexus.
+A given network service supported by AOSM, composed of one to many CNFs, includes components which, over time, require software and/or configuration changes. To make these component level changes it's necessary to run one to many helm operations, upgrading each network function application (nfApp) in a particular order and in a manner which least impacts the network service. AOSM safe upgrade practices apply the following high level capabilities to handle upgrade process and workflow requirements:
 
 * SNS Reput Support - Execute helm upgrade operation across all nfApps in network function design version (NFDV).
 * Nexus Platform - Support SNS reput operations on Nexus platform targets. 
-* Operation Time-outs - Ability to set operational time-outs for each nfApp operation.
+* Operation Timeouts - Ability to set operational timeouts for each nfApp operation.
 * Synchronous Operations - Ability to run one serial nfApp operation at a time.
 * Control Upgrade Order - Define different nfApp sequence for install and upgrade.
 * Pause On Failure - Default behavior pauses after an nfApp operation failure.
@@ -26,28 +26,30 @@ A given network service supported by AOSM is composed of one to many CNFs which,
 * Image Preloading - Ability to preload images to edge repository.
 
 ## Safe upgrade approach
-To update an existing Azure Operator Service Manager site network service (SNS), the Operator executes a reput update request against the deployed SNS resource. Where the SNS contains CNFs with multiple nfApps, the request is fanned out across all nfApps defined in the network function definition version (NFDV). By default, in the order, which they appear, or optionally in the order defined by `updateDependsOn` parameter.
+To update an existing AOSM site network service (SNS), the operator executes a reput request against the deployed SNS resource. Where the SNS contains CNFs with multiple nfApps, the request is fanned out across all nfApps defined in the network function definition version (NFDV). By default, in the order, which they appear, or optionally in the order defined by `updateDependsOn` parameter.
 
-For each nfApp, the reput update request supports increasing a helm chart version, adding/removing helm values and/or adding/removing any nfApps. Time-outs can be set per nfApp, based on known allowable runtimes, but nfApps can only be processed in serial order, one after the other. The reput update implements the following processing logic:
+For each nfApp, the reput request supports various changes including increasing a helm chart version, adding/removing helm values and/or adding/removing any nfApps. While timeouts can be set per nfApp, based on known allowable runtimes, nfApps can only be processed in serial order, one after the other. The reput update implements the following processing logic:
 
-* nfApps are processed following either updateDependsOn ordering, or in the sequential order they appear.
+* nfApps are processed following either `updateDependsOn` ordering, or in the sequential order they appear.
 * nfApps with parameter `applicationEnabled` set to disable are skipped.
 * nfApps with parameter `skipUpgrade` set to `enabled` are skipped if no changes detected.
-* nfApps which are common between old and new NFDV are upgraded.
-* nfApps which are only in the new NFDV are installed.
-* nfApps deployed, but not referenced by the new NFDV, are deleted.
+* nfApps which are common between old and new NFDV are upgraded using `helm upgrade`.
+* nfApps which are only in the new NFDV are installed using `helm install`.
+* nfApps deployed, but not referenced by the new NFDV, are deleted using `helm delete`.
 
-To ensure outcomes, nfApp testing is supported using helm, either helm upgrade pre/post tests, or standalone helm tests. For pre/post tests failures, the atomic parameter is honored. With atomic/true, the failed chart is rolled back. With atomic/false, no rollback is executed. For more information on standalone helm testing, see the following article: [Run tests after install or upgrade](safe-upgrades-helm-test.md)
+To ensure outcomes, nfApp testing is supported using helm methods, either tests triggered by helm pre or post hooks, or using the standalone helm test hook. For pre or post hook failure, the `atomic` parameter is honored. With atomic/true, the failed chart is rolled back. With atomic/false, no rollback is executed. For standalone helm test hook failure, the `rollbackOnTestFailure` is honored, following similar logic as atomic. For more information on standalone helm testing, see the following article: [Run tests after install or upgrade](safe-upgrades-helm-test.md)
+
+When an nfApp operation failure occurs, and after the failed nfApp is handled via `atomic` or `rollbackOnTestFailure` parameters, the operator can control behavior on how to handle any nfApps changed before the failed nfApp. With pause-on-failure the operator can force AOSM to break after addressing the failed nfApp, preserving the mixed version environment. With rollback-on-failure the operator can force AOSM to rollback any prior nfApp, restoring the original environment snapshot. For more information on controlling upgrade failure behavior, see the following article: [Control upgrade failure behavior](safe-upgrades-nf-level-rollback.md)
 
 ## Considerations for in-service upgrades
-Azure Operator Service Manager generally supports in service upgrades, an upgrade method which advances a deployment version without interrupting the running service. Some considerations are necessary to ensure the proper behavior of AOSM during ISSU operations. 
+Azure Operator Service Manager generally supports in service upgrades, an upgrade method which advances a deployment version without interrupting the running service. Some network function owner considerations are necessary to ensure the proper behavior of AOSM during ISSU operations. 
 * Where AOSM performs an upgrade against an ordered set of multiple nfApps, AOSM first upgrades or creates all new nfApps, then deletes all old nfApps. This approach ensures service isn't impacted until all new nfApps are ready but requires extra platform capacity for transient hosting of both old and new nfApps. 
 * Where AOSM upgrades an nfApp with multiple replicas, AOSM honors the deployment profile settings for either the rolling or recreate option. Where rolling is used, expose the values `maxUnavailable` and `maxSurge` as CGS parameters, which can then be set via operator CGV at run-time.
 
 Ultimately, the ability for a given service to be upgraded without interruption is a feature of the service itself. Consult further with the service publisher to understand the in-service upgrade capabilities and ensure they're aligned with the proper AOSM behavioral options.
 
 ## Safe upgrade prerequisites
-When planning for an upgrade using Azure Operator Service Manager, address the following requirements in advance of upgrade execution to optimize the time spent attempting the upgrade.
+When planning for an upgrade using AOSM, address the following requirements in advance of upgrade execution, to optimize time spent attempting and ensure success of the upgrade.
 
 - Onboard updated artifacts using publisher and/or designer workflows.
   - In most cases, use the existing publisher to host new version artifacts.
@@ -73,7 +75,7 @@ When planning for an upgrade using Azure Operator Service Manager, address the f
   - Settings used for production may suppress failures details, while settings used for debugging, or testing, may choose to expose these details.
 
 ## Safe upgrade procedure
-Follow the following process to trigger an upgrade with Azure Operator Service Manager.
+Follow the following process to trigger an upgrade with AOSM.
 
 * Create new NFDV resource
   * For new NFDV versions, it must be in a valid SemVer format. The new version can be an upgrade, a greater value versus the deployed version, or a downgrade, a lower value versus the deployed version. The new version can differ by major, minor, or patch values.
@@ -99,34 +101,50 @@ In cases where a reput update fails, the following process can be followed to re
   * By default, the reput retries nfApps in the declared update order, unless they're skipped using `applicationEnablement` flag.
 
 ## Control timeouts with installOptions and UpgradeOptions
-When an SNS operation starts either a helm install and helm upgrade, a 27-minute default timeout value. This value can be customized at the global NF, but we recommend to customize this value at the component NF levelby defining override values in the NF payload template. Further the values in the NF payload template and be exposed as operator values, allowing final customization at run-time. The following example demonstrates supported installOptions and upgradeOptions parameters applied to a single nfApp component;
+When an SNS operation starts either a `helm install` or a `helm upgrade`, a 27-minute default timeout value is used. While this value can be customized at the global network function (NF) level, we recommend customizing this value at the component NF level using `roleOverrideValues` in the NF payload template. Further exposing the `roleOverrideValues` in CGS/CGV allows control by the operator at run-time. The following example demonstrates supported installOptions and upgradeOptions parameters applied across two nfApp components;
 
-```
-"roleOverrideValues": ["{ 
-   "name": "hellotest",
-    "deployParametersMappingRuleProfile": {
-      "helmMappingRuleProfile": {
-       "options": {
-        "installOptions": {
-         "atomic": true,
-         "wait": true,
-         "timeout": "1" },
-        "upgradeOptions": {
-         "atomic": true,
-         "wait": true,
-         "timeout": "2" }
-        } } } }"
-    ]
+```json
+{
+  "roleOverrideValues": [
+    {
+      "name": "nfApplication1",
+      "deployParametersMappingRuleProfile": {
+        "helmMappingRuleProfile": {
+          "options": {
+            "installOptions": {
+              "atomic": "true",
+              "wait": "true",
+              "timeout": "1"
+            },
+            "upgradeOptions": {
+              "atomic": "true",
+              "wait": "true",
+              "timeout": "1"
+            } } } } },
+    {
+      "name": "nfApplication2",
+      "deployParametersMappingRuleProfile": {
+        "helmMappingRuleProfile": {
+          "options": {
+            "installOptions": {
+              "atomic": "true",
+              "wait": "true",
+              "timeout": "1"
+            },
+            "upgradeOptions": {
+              "atomic": "true",
+              "wait": "true",
+              "timeout": "1"
+            } } } } }
+  ]
+}
 ```
 
 ## Skip nfApps using applicationEnablement
-In the NFDV resource, under `deployParametersMappingRuleProfile` there's a supported property `applicationEnablement` of type enum, which takes values of Unknown, Enabled, or disabled. It can be used to manually exclude nfApp operations during network function (NF) deployment. The following example demonstrates a generic method to parameterize `applicationEnablement` as an included value in `roleOverrideValues` property.
-
-### Template changes
-While no NFDV changes are necessarily required, optionally the publisher can use the NFDV to set a default value for the `applicationEnablement` property. The default value is used, unless its changed via `roleOverrideValues`. 
+In the NFDV resource, under `deployParametersMappingRuleProfile` there's a supported property `applicationEnablement` of type enum, which takes values of Unknown, Enabled, or disabled. It can be used to manually exclude nfApp operations during network function deployment. The following example demonstrates a generic method to parameterize `applicationEnablement` as an included value in `roleOverrideValues` property.
 
-#### NFDV template
-Use the NFDV template to set a default value for `applicationEnablement`. The following example sets `enabled` state as the default value for `hellotest` networkfunctionApplication. 
+### NFDV template changes
+While no NFDV changes are necessarily required, optionally the publisher can use the NFDV to set a default value for the `applicationEnablement` property. The default value is used, unless its changed via `roleOverrideValues`.  Use the NFDV template to set a default value for `applicationEnablement`. The following example sets `enabled` state as the default value for `hellotest` networkfunctionApplication. 
 
 ```json
       "location":"<location>", 
@@ -145,7 +163,7 @@ Use the NFDV template to set a default value for `applicationEnablement`. The fo
 
 To manage the `applicationEnablement` value more dynamically, the Operator can pass a real-time value using the NF template `roleOverrideValues` property. While it's possible for the operator to manipulate the NF template directly, instead parameterize the `roleOverrideValues`, so that values can be passed via a CGV template at runtime. The following examples demonstrate the needed modifications to the CGS, NF templates, and finally the CGV.
 
-#### CGS template
+### CGS template changes
 The CGS template must be updated to include one variable declaration for each line to parameterize under `roleOverrideValues`. The following example demonstrates three override values.
 
 ```json
@@ -160,8 +178,8 @@ The CGS template must be updated to include one variable declaration for each li
         }
 ```
 
-#### NF payload template
-The NF template must be update three ways. First, the implicit config parameter must be defined as type object. Second, `roleOverrideValues0`, `roleOverrideValues1`, and `roleOverrideValues2` must be declared as variables mapped to config parameter. Third, `roleOverrideValues0`, `roleOverrideValues1` and `roleOverrideValues2` must be referenced for substitution under `roleOverrideValues` in proper order and following proper syntax.
+### NF payload template changes
+The NF template must be update three ways. First, the implicit config parameter must be defined as type object. Second, `roleOverrideValues0`, `roleOverrideValues1`, and `roleOverrideValues2` must be declared as variables mapped to config parameter. Third, `roleOverrideValues0`, `roleOverrideValues1`, and `roleOverrideValues2` must be referenced for substitution under `roleOverrideValues` in proper order and following proper syntax.
 
 ```json
   "parameters": {
@@ -186,7 +204,7 @@ The NF template must be update three ways. First, the implicit config parameter
    }
 ```
 
-#### CGV template
+### CGV template changes
 The CGV template can now be updated to include the content for each variable to be substituted into `roleOverrideValues` property at run-time. The following example sets `rollbackEnabled` to true, followed by override sets for `hellotest` and `hellotest1` nfApplications.
 
 ```json
@@ -238,10 +256,8 @@ To enable the SkipUpgrade feature via `roleOverrideValues`, refer to the followi
 - **nfApplication: `runnerTest`**
   - The `skipUpgrade` flag isn't specified. Therefore, `runnerTest` executes a traditional Helm upgrade at the cluster level, even if the precheck criteria are met.
 
-
-
 ## Complete roleOverrideValues option reference
-Bringing together all examples in this and other articles, the following reference demonstrates all presently supported install and upgrade options available through the `roleOverrideValues` mechanism.
+Bringing together all examples in this and other articles, the following reference demonstrates all presently supported options available through the `roleOverrideValues` mechanism.
 
 ```json
 {