Feature Proposal: Two-Phase Application Startup with Deferred Migrations

[meyntony]

The Problem

When running large database migrations on IIS-hosted Umbraco sites, the application can be killed silently before migrations complete.

IIS enforces a startupTimeLimit (default: 120 seconds) via the ASP.NET Core Module (ANCM). If the application doesn't respond within that window, the process is terminated. Database migrations run synchronously during app startup, blocking the entire pipeline — the app doesn't respond to anything until every migration has finished.

Real-World Example

A project upgrading to v17.2.0 with a large content tree — a root node with 70-80k child nodes, each containing a numeric field and an RTE across 10 languages — consistently failed at the same point during the RTE migration. The migration ran fine locally, but on IIS the process was killed before it could complete.

Using diagnostic tooling, the root cause was confirmed: startup time exceeded the 120-second ANCM limit. Increasing startupTimeLimit to 600 seconds got the migration further, but it appears it could require up to the maximum value of 3600 seconds to fully complete.

This is not an edge case. Any Umbraco site with a large content tree doing a major version upgrade (especially ones involving content migrations like the TinyMCE-to-TipTap RTE migration) can hit this. Related issues going back years confirm this is a recurring problem:

• [#13051](NuCache & Examine taking a long time to build on large websites #13051) — A site with ~200,000 nodes taking 30+ minutes to start, hitting Azure App Service timeouts.
• [#6243](Custom visitors page when RuntimeLevel is RuntimeLevel.Upgrade #6243) — Feature request for a custom visitor-facing page during RuntimeLevel.Upgrade, labelled as a wishlist item but never implemented.

Why This Matters

• Large content migrations can easily exceed default timeout limits on IIS.
• The failure is silent — the process is killed with no meaningful error surfaced to the user.
• Developers spend significant time debugging what is ultimately an infrastructure constraint, not an application bug.
• Increasing startupTimeLimit is a workaround, not a solution — and many teams don't know this setting exists or that it's the cause of their failure.

Current Workaround

Add a startupTimeLimit override to web.config:

<aspNetCore processPath="..." startupTimeLimit="3600" ... />

This works, but:

• Most people don't know this setting exists or that it's the cause of their failure.
• It doesn't solve the UX problem — the site is completely unresponsive during migration.
• Some hosting environments may not allow direct control over web.config generation.

Proposed Approach: Two-Phase Startup

Split application startup into two phases:

Phase 1 — Lightweight Shell

• Boot the application quickly with minimal middleware.
• Return healthy responses to IIS/load balancers immediately (satisfies startupTimeLimit and health checks).
• Serve a maintenance page: "Database migration in progress, please wait..."

Phase 2 — Background Migration

• Run database migrations in a background IHostedService.
• Use SignalR (now native to the CMS) to broadcast real-time progress to connected clients (migration step, page count, percentage, etc.).
• Once migrations complete, switch the application to normal operation — routing, content delivery, backoffice, all of it.

Architectural Consideration

Today, Umbraco's entire runtime assumes a fully migrated database. Routing, content resolution, backoffice controllers — all of it expects the schema to be in its final state. This means the proposal requires a clear separation between:

• "App is running" — can respond to HTTP requests, serve a status page, pass health checks.
• "App is ready" — database is migrated, full CMS functionality is available.

This is a meaningful architectural change, but the pattern (sometimes called "startup probes" or "readiness vs. liveness") is well-established in container orchestration and could benefit Umbraco in multiple scenarios beyond migrations (plugin initialization, index rebuilding, cache warming, etc.).

Benefits

• Eliminates silent startup failures on IIS-hosted sites with large migrations.
• Improves the migration UX — users see progress instead of a hanging browser.
• Reduces support burden — no more debugging mysterious timeout-related crashes.
• Future-proofs the startup pipeline for other heavy initialization tasks.
• Aligns with modern hosting patterns — readiness/liveness probes in Kubernetes, Azure App Service health checks, etc.

Questions for Discussion

1. Is a full two-phase startup the right scope, or should we start smaller (e.g., just making migrations run in a background service with a blocking middleware)?
2. Are there other parts of the startup pipeline (beyond migrations) that would benefit from being deferred?
3. Should the maintenance page be configurable (custom HTML, similar to what was requested in [#6243](Custom visitors page when RuntimeLevel is RuntimeLevel.Upgrade #6243))?
4. What's the appetite for this as a core CMS change vs. something individual hosting setups need to handle?

---

References:

• ASP.NET Core Module startupTimeLimit documentation
• IIS web.config hosting configuration
• [#13051 — NuCache & Examine taking a long time to build on large websites](NuCache & Examine taking a long time to build on large websites #13051)
• [#6243 — Custom visitors page when RuntimeLevel is Upgrade](Custom visitors page when RuntimeLevel is RuntimeLevel.Upgrade #6243)
• [Stack Overflow: Process startup timeout in IIS](https://stackoverflow.com/questions/60694012/how-do-you-configure-the-process-startup-timeout-in-iis)

[meyntony]

Important scoping note: This proposal applies specifically to unattended upgrades — i.e. when Umbraco:CMS:Unattended:UpgradeUnattended is set to true and database migrations need to run at startup.

On a normal boot where the database schema is already current, the application starts exactly as it does today — no change in behavior. The two-phase approach only activates when the runtime detects pending migrations during an unattended upgrade, which is the scenario where the current startup pipeline is most fragile.

The existing documentation already hints at the problem:

"If your database experiences timeout issues after an upgrade, it might be due to ASP.NET Core Module's startupTimeLimit configuration."

But increasing startupTimeLimit is a workaround, not a solution — and many teams don't know this setting exists or that it's the cause of their 503s. Worse, startupTimeLimit has a hard maximum of 3600 seconds (1 hour). For sites with large content sets or complex migrations spanning multiple major versions, even the maximum may not be enough. At that point there is no workaround — the migration simply cannot complete within the ANCM startup window, and the app fails to boot in a loop.

A two-phase startup that decouples the migration from the ANCM startup pipeline removes this ceiling entirely. The app becomes "live" (serving a maintenance/upgrade-in-progress response) within seconds, and the migration runs to completion in the background regardless of how long it takes.

For reference:

• Upgrade Unattended — how to enable unattended upgrades
• Unattended Settings — full configuration surface including UpgradeUnattended and PackageMigrationsUnattended

[meyntony]

Seems like this fixes it.