Prevent container restart when application stops .StopApplication() is called on IHostApplicationLifetime

[Fazer01]

Hi there,

We have followed this tutorial to use a separate migration service to handle migrations when deployment is done through azd.
The migration service is used in the app host in the following way:

var builder = DistributedApplication.CreateBuilder(args);

var sql = builder.AddSqlServer("sql")
                 .AddDatabase("sqldata");

builder.AddProject<Projects.SupportTicketApi_Api>("api")
    .WithReference(sql);

builder.AddProject<Projects.SupportTicketApi_MigrationService>("migrations")
    .WithReference(sql);

builder.Build().Run();

Local
During local development, the migration service application is ran just once, and then is 'Finished' according to the Aspire Dash. So far so good. No repeated runs of the migration service are executed.

Azure deployment
When the lot is deployed to Azure, the migration service is doing it's job. E.g. the container is spinned up, migrations, if any, are applied and then the application is shut down. After about 5 minutes, the migration service container is spinned up, running migrations, but there are none to be applied. .This keeps going forever as it seems.

How can we prevent this from happening?

Regards,

Tom

[behdad088]

Hi @Fazer01
Just out of curiosity, why do you run the migration when deploying your project in Azure?

[behdad088]

Hey @Fazer01 ,

It sounds like the migration service container is being restarted repeatedly because the Azure Container Apps environment is treating it like a long-running service, rather than a "run-once-and-done" kind of job. This is probably because Azure Container Apps assumes that the container should be continuously running, but that’s not what you want for a migration service.

Here’s how you can fix it:

1. Use a "Job" Instead of a Container App

Azure Container Apps recently introduced job support that is specifically designed for tasks like running migrations, one-off scripts, etc. Instead of running your migration service as a standard container, you can configure it to run as a job. This will ensure it only runs once when deployed, and doesn’t keep restarting.

You can configure a scheduled job with a cron schedule or make it a run-once job with a manual trigger.

Here’s a quick guide on how to create a run-once job:

1. Create a Job:
   Use the Azure CLI to create a job.

   az containerapp job create \
     --name migration-job \
     --resource-group <your-resource-group> \
     --location <your-location> \
     --cpu 0.5 \
     --memory 1.0Gi \
     --image <your-migration-service-image> \
     --replica-timeout 3600 \
     --replica-retry-limit 0 \
     --parallelism 1 \
     --trigger-type Manual

2. Trigger the Job:
   You can then trigger the job manually when you need to run migrations (like after deployment):

   az containerapp job start --name migration-job --resource-group <your-resource-group>

This makes sure that the migration service runs just once when needed and doesn't automatically restart.

2. Shutting Down the Container Gracefully

If you want to stick with the current container setup, you could add logic to shut the container down after migrations are done. But the issue here is that the container infrastructure might still restart it after some time.

So, another option would be to tweak the container’s liveness and readiness probes (health checks) so that Azure knows the container should stop running after it has completed its task. If the container becomes "unhealthy" after finishing its migration task, Azure will stop it:

1. Liveness Probe: Configure a health check (like a specific endpoint or status) that fails once the migration task completes.
2. Exit Code: Ensure the migration service exits with a specific code (like 0 for success), and use a container restart policy that doesn’t restart the container on success.

3. Azure Logic:

You can also modify the logic of the container app by setting its replica settings to avoid constant restarts. For example, you can change the replica settings in Azure so that it only attempts to run a single instance, or set it up so that it doesn’t automatically restart unless there’s a failure.

4. Cron Trigger or Event-Driven Jobs:

You can also schedule the migration service as an event-driven job if you want to make it run only on certain conditions, like a deployment event or a specific cron schedule.

The best solution is to treat your migration service as a run-once job or configure Azure to properly recognize it as such. That will stop it from restarting and running migrations endlessly.

Hope this helps! Let me know if you have more questions.

Cheers

[Fazer01]

Hi @behdad088

Thanks for this extensive reply. Just wondering how we can implement this within .NET Aspire. Any suggestions?
After running azd infra synth, to generate the template for the migration service, used for provisioning / deployment, I was wondering if there is something like a property restartPolicy or something like that. So you can control how the container should behave in case when the appliation stops running, as the migrations are finished. In the bicep documentation of the ContainerApps resource I couldn't find any...
https://learn.microsoft.com/en-us/azure/templates/microsoft.app/containerapps?pivots=deployment-language-bicep

I know that .NET Aspire has no support yet for init containers but maybe will come to a later version of .NET Aspire. See #2866 for more details.

Maybe @davidfowl can elaborate on this?

Thanks in advance

[AndiRudi]

I have the same issue. I followed the Official Example and faced the issue. It is especially annoying because it is quite expensive. In fact I just realized the issue because the migration service has enormous cost compare to the other services.

I think - as a workaround - I just let it running instead of stopping it. That is cheaper ....

[Fazer01]

Can someone elaborate on this? :)

[davidfowl]

Letting it run is the right workaround for now. There's no native support for container job as yet.

[huberttrueselftrueme]

I have the same issue. I followed the Official Example and faced the issue. It is especially annoying because it is quite expensive. In fact I just realized the issue because the migration service has enormous cost compare to the other services.

I think - as a workaround - I just let it running instead of stopping it. That is cheaper ....

Can you help me with this ? I facing exactly same issue. Have no idea how with aspire setup this. Official Example Do not mention about this at all.

var migrations = builder.AddProject<MigrationWorker>(OrchestrationResources.Migrations)
    .PublishAsAzureContainerApp((config, app) =>
    {
        app.Template.Scale.MinReplicas = 0;
        app.Template.Scale.MaxReplicas = 1;
    })
    .WaitFor(postgresDb)
    .WaitFor(storage)
    .WithReference(serviceBus)
    .WithReference(postgresDb);

But still after migration worker is working on 1 instance. And generating costs.

[AndiRudi]

Sure, I can help. I have this to run my service

var migrator = builder.AddProject<Projects.MyMigrator>("migrator")
    .WithReference(myDb);

The migrator is a web project which has Worker class that extends BackgroundService

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
   // Do the migration
   var strategy = dbContext.Database.CreateExecutionStrategy();
   await strategy.ExecuteAsync(async () =>
   {
      // Create the database if it does not exist.
      // Do this first so there is then a database to start a transaction against.
      if (!await dbCreator.ExistsAsync(cancellationToken))
      {
          await dbCreator.CreateAsync(cancellationToken);
      }
   });

   // Exit the application
   //In production is cheaper to let the service run, but in dev
   //we need to stop it because the other services are waiting
   if (hostEnvironment.IsDevelopment())
   {
       Thread.Sleep(500);
       hostApplicationLifetime.StopApplication();
   }
}