Initial draft

Author: erikadoyle

articles/service-fabric/service-fabric-application-upgrade-advanced.md

@@ -3,15 +3,89 @@ title: Advanced Application Upgrade Topics
 description: This article covers some advanced topics pertaining to upgrading a Service Fabric application.
 
 ms.topic: conceptual
-ms.date: 2/23/2018
+ms.date: 1/28/2020
 ---
-# Service Fabric application upgrade: advanced topics
-## Adding or removing service types during an application upgrade
+# Service Fabric application upgrade: Advanced topics
+
+## Add or remove service types during an application upgrade
+
 If a new service type is added to a published application as part of an upgrade, then the new service type is added to the deployed application. Such an upgrade does not affect any of the service instances that were already part of the application, but an instance of the service type that was added must be created for the new service type to be active (see [New-ServiceFabricService](https://docs.microsoft.com/powershell/module/servicefabric/new-servicefabricservice?view=azureservicefabricps)).
 
 Similarly, service types can be removed from an application as part of an upgrade. However, all service instances of the to-be-removed service type must be removed before proceeding with the upgrade (see [Remove-ServiceFabricService](https://docs.microsoft.com/powershell/module/servicefabric/remove-servicefabricservice?view=azureservicefabricps)).
 
+## Avoid connection drops during planned downtime of stateless services
+
+For planned stateless instance downtime, like when the application/cluster is upgraded or nodes are getting deactivated, there are a few connections drops which are observed because the endpoint exposed by the instances is removed after it goes down.
+
+To avoid getting connections drops for planned downtime, the request drain feature has been enabled which can be configured by adding a replica close delay duration in the service configuration. This would be used to delay the close of the stateless Instance. The endpoint advertised by the stateless Instance is removed, before we start the delay timer prior to closing this Instance. This delay duration would give existing requests to drain gracefully before the instance actually goes down. This endpoint change is advertised to the clients, and they would have to resolve the endpoints again, so that they do not send new requests to the instance which is going down.
+
+### Service configuration
+
+The delay can be configured in the service in following ways:
+
+#### When creating a new service
+
+By setting an `-InstanceCloseDelayDuration` value:
+
+```powershell
+New-ServiceFabricService -Stateless [-ServiceName] <Uri> -InstanceCloseDelayDuration <TimeSpan>`
+```
+
+#### While defining the service in the defaults section in the application manifest
+
+With the `InstanceCloseDelayDurationSeconds` property:
+
+```xml
+      <StatelessService ServiceTypeName="Web1Type" InstanceCount="[Web1_InstanceCount]" InstanceCloseDelayDurationSeconds="15">
+          <SingletonPartition />
+      </StatelessService>
+```
+
+#### When updating an existing service
+
+By setting an `-InstanceCloseDelayDuration` value:
+
+```powershell
+Update-ServiceFabricService [-Stateless] [-ServiceName] <Uri> [-InstanceCloseDelayDuration <TimeSpan>]`
+```
+
+### Client configuration
+
+Since we remove the endpoint advertised by an instance and then start the delay timer before we close the Instance down, the clients will get a change notification for the endpoints updated for which they would need to register a callback (`ServiceManager_ServiceNotificationFilterMatched`) like this:
+
+```csharp
+    var filterDescription = new ServiceNotificationFilterDescription
+    {
+        Name = new Uri(serviceName),
+        MatchNamePrefix = true
+    };
+    fbClient.ServiceManager.ServiceNotificationFilterMatched += ServiceManager_ServiceNotificationFilterMatched;
+    await fbClient.ServiceManager.RegisterServiceNotificationFilterAsync(filterDescription);
+
+private static void ServiceManager_ServiceNotificationFilterMatched(object sender, EventArgs e)
+{
+      // Resolve service to get a new endpoint list
+}
+```
+
+The change notification is an indication that the endpoints have changed, the client should re-resolve the endpoints, and not use the endpoints which are not advertised anymore as they will go down soon.
+
+### Optional upgrade overrides
+
+Apart from the pre-configured delay duration, you also have the option to override the delay, at the time of initiating an application/cluster upgrade using the same (`InstanceCloseDelayDurationSec`) option:
+
+```powershell
+Start-ServiceFabricApplicationUpgrade [-ApplicationName] <Uri> [-ApplicationTypeVersion] <String> [-InstanceCloseDelayDurationSec <UInt32>]
+
+Start-ServiceFabricClusterUpgrade [-CodePackageVersion] <String> [-ClusterManifestVersion] <String> [-InstanceCloseDelayDurationSec <UInt32>]
+```
+
+This delay duration will only be valid for this instance of the upgrade and will not change the individual service delay configurations. For example, you can use this to give a 0 delay, to run through an upgrade and not wait for the pre-configured delay.
+
+
+
 ## Manual upgrade mode
+
 > [!NOTE]
 > The *Monitored* upgrade mode is recommended for all Service Fabric upgrades.
 > The *UnmonitoredManual* upgrade mode should only be considered for failed or suspended upgrades. 
@@ -25,6 +99,7 @@ In *UnmonitoredManual* mode, the application administrator has total control ove
 Finally, the *UnmonitoredAuto* mode is useful for performing fast upgrade iterations during service development or testing since no user input is required and no application health policies are evaluated.
 
 ## Upgrade with a diff package
+
 Instead of provisioning a complete application package, upgrades can also be performed by provisioning diff packages that contain only the updated code/config/data packages along with the complete application manifest and complete service manifests. Complete application packages are only required for the initial installation of an application to the cluster. Subsequent upgrades can either be from complete application packages or diff packages.  
 
 Any reference in the application manifest or service manifests of a diff package that can't be found in the application package is automatically replaced with the currently provisioned version.
@@ -108,7 +183,7 @@ HealthState            : Ok
 ApplicationParameters  : { "ImportantParameter" = "2"; "NewParameter" = "testAfter" }
 ```
 
-## Rolling back application upgrades
+## Roll back application upgrades
 
 While upgrades can be rolled forward in one of three modes (*Monitored*, *UnmonitoredAuto*, or *UnmonitoredManual*), they can only be rolled back in either *UnmonitoredAuto* or *UnmonitoredManual* mode. Rolling back in *UnmonitoredAuto* mode works the same way as rolling forward with the exception that the default value of *UpgradeReplicaSetCheckTimeout* is different - see [Application Upgrade Parameters](service-fabric-application-upgrade-parameters.md). Rolling back in *UnmonitoredManual* mode works the same way as rolling forward - the rollback will suspend itself after completing each UD and must be explicitly resumed using [Resume-ServiceFabricApplicationUpgrade](https://docs.microsoft.com/powershell/module/servicefabric/resume-servicefabricapplicationupgrade?view=azureservicefabricps) to continue with the rollback.