Load balancing backoffice documentation

17/umbraco-cms/SUMMARY.md

@@ -44,6 +44,7 @@
     * [Running Umbraco in Docker](fundamentals/setup/server-setup/running-umbraco-in-docker.md)
     * [Umbraco in Load Balanced Environments](fundamentals/setup/server-setup/load-balancing/README.md)
       * [Load Balancing Azure Web Apps](fundamentals/setup/server-setup/load-balancing/azure-web-apps.md)
+      * [Load Balancing the Backoffice](fundamentals/setup/server-setup/load-balancing/load-balancing-backoffice.md)
       * [Standalone File System](fundamentals/setup/server-setup/load-balancing/file-system-replication.md)
       * [Advanced Techniques With Flexible Load Balancing](fundamentals/setup/server-setup/load-balancing/flexible-advanced.md)
       * [Logging With Load Balancing](fundamentals/setup/server-setup/load-balancing/logging.md)

17/umbraco-cms/fundamentals/setup/server-setup/load-balancing/load-balancing-backoffice.md

@@ -0,0 +1,49 @@
+# Load Balancing the Backoffice
+
+This article contains specific information about load balancing the Umbraco backoffice, ensure you read the [Load Balancing Overview](./) and relevant articles about general load balancing principles before you begin.
+
+By default, the Umbraco load balancing setup assumes there are a single backoffice server, and multiple front-end servers. From version 17 it's possible to load balance the backoffice, meaning there's no need to differentiate from backoffice servers and front-end servers, however, this requires some additional configuration steps.
+
+## Server Role Accessor
+
+Umbraco has the concept of server roles, to differentiate between backoffice servers and front-end servers. Since all servers will be backoffice servers, we need to add a custom `IServerRoleAccessor` to specify this.
+
+Start by implementing a custom `IServerRoleAccessor` that pins the role as `SchedulingPublisher`:
+
+```csharp
+ public class StaticServerAccessor : IServerRoleAccessor
+{
+    public ServerRole CurrentServerRole => ServerRole.SchedulingPublisher;
+}
+```
+
+You can now register this accessor, either in `Program.cs` or via a Composer:
+
+```csharp
+umbracoBuilder.SetServerRegistrar(new StaticServerAccessor());
+```
+
+This will ensure that all servers are treated as backoffice servers.
+
+## Load balancing Isolated Caches
+
+One of the issues with load balancing the backoffice is that all servers will have their own isolated caches. This means that if you make a change on one server, it won't be reflected on the other servers until their cache expires.
+
+To solve this issue a cache versioning mechanism is used, this is similar to optimistic concurrency control, where each server has a version number for its cache. When a server makes a change, it changes the version identifier. The other servers can then check the version identifier before accessing the cache and invalidate their cache if it's out of date.
+
+This does mean that the server needs to check this version identifier before a cache lookup, so by default this behaviour is disabled as it's only required when load balancing the backoffice.
+
+You can enable this on the Umbraco builder, either in `Program.cs` or via a Composer:
+
+```csharp
+umbracoBuilder.LoadBalanceIsolatedCaches();
+```
+
+## SignalR
+
+The Umbraco Backoffice uses SignalR for multiple things, including real-time updates and notifications. When load balancing the backoffice, it's important to ensure that SignalR is configured correctly. See the [SignalR in a Backoffice Load Balanced Environment](./signalR-in-backoffice-load-balanced-environment.md) document for information regarding this.
+
+
+## Background Jobs
+
+If you have any custom recurring background jobs that should only run on a single server, you'll need to implement `IDistributedBackgroundJob`, see [Scheduling documentation](../../../../reference/scheduling.md#background-jobs-when-load-balancing-the-backoffice) for more information.

17/umbraco-cms/fundamentals/setup/server-setup/load-balancing/signalR-in-backoffice-load-balanced-environment.md

@@ -0,0 +1,59 @@
+# SignalR in a Backoffice Load Balanced Environment
+When load balancing the backoffice, we also need to take care of the client to server communication outside of web requests.
+Umbraco uses SignalR to abstract away these types of communication. This also allows us to support load balancing by replacing how the communication is done by introducing a backplane.
+
+## Introducing a SignalR Backplane
+A SignalR backplane is akin to a load balancer for direct client to server web traffic. It keeps track of which client is connected to which server. So that when a client sends a message, it arrives at the right server. It also allows any connected server to send a message to all clients, even those that are not directly connected to it.
+
+## Choosing the right backplane
+Choosing the right backplane comes down to a few things
+- Message throughput
+- Cost
+- What infrastructure you already have in place
+
+Microsoft has a good list of available backplanes in its [SignalR load balancing article](https://learn.microsoft.com/en-us/aspnet/core/signalr/scale?view=aspnetcore-10.0), including a list of well known [third party offerings](https://learn.microsoft.com/en-us/aspnet/core/signalr/scale?view=aspnetcore-9.0#third-party-signalr-backplane-providers).
+
+## Code examples
+The following code examples show you how you can activate SignalR load balancing using an Umbraco composer.
+Note: Both Umbraco Core and these composers use `.AddSignalR().` which is ok since the underlying code registers the required services as singletons.
+
+### Using existing infrastructure
+It is possible to use your existing database as a backplane. If this database is hosted in Azure it is not possible to enable Service Broker which will have an impact on message throughput. We do however feel that when you start out with load balancing, it might be enough to cover your needs.
+For more information, check out the [Github page](https://github.com/IntelliTect/IntelliTect.AspNetCore.SignalR.SqlServer).
+- Add a reference to the IntelliTect.AspNetCore.SignalR.SqlServer NuGet package
+- Add the following composer to your project
+```csharp
+using Umbraco.Cms.Core.Composing;
+
+namespace Umbraco.Cms.Web.UI.SignalRLoadBalancing;
+
+public class SignalRComposer : IComposer
+{
+    public void Compose(IUmbracoBuilder builder)
+    {
+        var connectionString = builder.Config.GetUmbracoConnectionString();
+        if (connectionString is null)
+        {
+            return;
+        }
+
+        builder.Services.AddSignalR().AddSqlServer(connectionString);
+    }
+}
+```
+
+### Azure SignalR Service
+- Setup a resource as described in the [Microsoft tutorial](https://learn.microsoft.com/en-us/azure/azure-signalr/signalr-quickstart-dotnet-core#create-an-azure-signalr-resource)
+- Make sure the connectionstring is setup under the following key: `Azure:SignalR:ConnectionString`
+- Add a reference to the Microsoft.Azure.SignalR NuGet package
+- Add the following composer to your project
+```csharp
+using Umbraco.Cms.Core.Composing;
+
+namespace Umbraco.Cms.Web.UI.SignalRLoadBalancing;
+
+public class SignalRComposer : IComposer
+{
+    public void Compose(IUmbracoBuilder builder) => builder.Services.AddSignalR().AddAzureSignalR();
+}
+```

17/umbraco-cms/reference/configuration/README.md

@@ -120,6 +120,7 @@ A complete list of all the configuration sections included in Umbraco, by defaul
 * [Connection strings settings](connectionstringssettings.md)
 * [Content settings](contentsettings.md)
 * [Debug settings](debugsettings.md)
+* [Distributed jobs settings](distributedjobssettings.md)
 * [Examine settings](examinesettings.md)
 * [Exception filter settings](exceptionfiltersettings.md)
 * [Global settings](globalsettings.md)

17/umbraco-cms/reference/configuration/distributedjobssettings.md

@@ -0,0 +1,31 @@
+# Distributed jobs settings
+
+The distributed jobs settings allow you to configure how Umbraco handles distributed background jobs in a load-balanced environment.
+
+## Configuration
+```json
+"Umbraco": {
+  "CMS": {
+    "DistributedJobs": {
+      "Period": "00:00:05",
+      "Delay": "00:01:00"
+    }
+  }
+}
+```
+
+## Settings
+
+### Period
+
+**Default:** `00:00:05` (5 seconds)
+
+Specifies how frequently each server checks for distributed background jobs that need to be run.
+
+A shorter period means jobs are picked up more quickly, but increases the frequency of database queries. A longer period reduces overhead but may introduce delays in job execution.
+
+### Delay
+
+**Default:** `00:01:00` (1 minute)
+
+Specifies how long the server should wait after initial startup before beginning to check for and run distributed background jobs. This startup delay ensures that the application is fully initialized and stable before participating in distributed job processing.

17/umbraco-cms/reference/scheduling.md

@@ -332,3 +332,58 @@
         return Task.CompletedTask; // We return Task.CompletedTask to try again as the server role may change!
 }
 ```
+
+## Background jobs when load balancing the backoffice
+
+When load balancing the backoffice, all servers will have the `SchedulingPublisher` role. 
+This means that the approach described above for restricting jobs to specific server roles will not work as intended, since all servers will match the `SchedulingPublisher` role.
+
+Instead, for jobs that should only run on a single server, you should implement an `IDistrutedBackgroundJob`.
+
+`IDistributedBackgroundJobs` is separate from `IRecurringBackgroundJob`, and are tracked in the database to ensure that only a single server runs the job at any given time.
+This also means that you are not guaranteed what server will run the job, but you are guaranteed that only one server will run it. 
+
+By default, distributed background jobs are checked every 5 seconds, with an initial delay of 1 minute after application startup. These settings can be changed in appsettings, see [Distributed jobs settings](./configuration/distributedjobssettings.md) for more information.
+
+### Implementing a custom distributed background job
+
+To implement a custom distributed background job, create a class that implements the `IDistributedBackgroundJob` interface. Just like with `IRecurringBackgroundJob` DI is available in the constructor.
+
+```csharp
+public class MyCustomBackgroundJob : IDistributedBackgroundJob
+{
+    private readonly ILogger<MyCustomBackgroundJob> _logger;
+    public string Name => "MyCustomBackgroundJob";
+
+    public TimeSpan Period { get; private set; }
+
+    public MyCustomBackgroundJob(ILogger<MyCustomBackgroundJob> logger)
+    {
+        _logger = logger;
+        Period = TimeSpan.FromSeconds(20);
+    }
+
+    public Task ExecuteAsync()
+    {
+        // Your custom background job logic here
+        _logger.LogInformation("MyCustomBackgroundJob is executing.");
+        return Task.CompletedTask;
+    }
+}
+```
+
+It's required to give your job a unique name via the `Name` property. This is used to track the job in the database.
+
+The period is specified via the `Period` property, which controls how often the job should be run, in this example it's every 20 seconds.
+
+It's not required to manually register the job in the database, however you must register it to dependency injection so Umbraco can find it. This can be done with a composer or in `Program.cs`
+
+```csharp
+public class MyComposer : IComposer
+{
+    public void Compose(IUmbracoBuilder builder)
+    {
+        builder.Services.AddSingleton<IDistributedBackgroundJob, MyCustomBackgroundJob>();
+    }
+}
+```