WorkflowHost Completes Exit Before Steps Within Workload Can Attempt To Stop Gracefully

[DelineaSamGeorge]

Written on issue #645, but created a new issue because it may be unrelated.

I expect WorkflowHost not to complete its Stop lifecycle hooks until after its Steps have been given a chance to complete. WorkflowHost instead seems to exit as soon as the signal is sent, preventing steps from finishing their work / their cleanup.

The ability to do that cleanup is more important to me than the specifics of implementation, so if there is a better way to have the behavior that we can carry out some process once WorkflowHost receives a graceful stop signal, I'd appreciate that, too.

This appears to still be occurring on 3.17.0. I am completely prepared for a response that we are using WorkflowCore essentially incorrectly, but I think the behavior I'm seeing is unexpected. I want my step processes to complete or at least attempt to complete when the WorkflowHost is stopped. Instead, that stop seems to try exiting the steps completely without any lifecycle hooks running.

I install and run the following service in Windows. No matter when along the path I stop the service, I would expect MyInnerService to attempt its StopAsync. It does not. I only see the Exec and Start logs. I also tried replacing the hosted MyInnerService with a simple Task.Delay in the step. In that case, too, the Step does not let the delay complete and the workload simply exits out when I stop the service.

Is that expected behavior?

Directory.SetCurrentDirectory(SharedPaths.BaseDataDirectory);

var bootstrapBuilder = Host.CreateApplicationBuilder();

var configuration = new ConfigurationBuilder()
    .SetBasePath(AppDomain.CurrentDomain.BaseDirectory)
    .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true).Build();

bootstrapBuilder.Services.AddSingleton<IConfiguration>(configuration);

bootstrapBuilder.Services.AddWorkflow();

bootstrapBuilder.Services
    .AddTransient<MyStep>()
    .AddHostedService<WorkflowHost>();

bootstrapBuilder.Services
    .AddLogging(
        options =>
        {
            var logFile = string.Format("Bootstrap.log");
            var logPath = Path.Combine(SharedPaths.CommonLogDirectory, logFile);
            var logger = new LoggerConfiguration()
                .ReadFrom.Configuration(bootstrapBuilder.Configuration)
                .WriteTo.File(logPath, formatProvider: CultureInfo.InvariantCulture, rollingInterval: RollingInterval.Day, retainedFileCountLimit: 15, shared: true)
                .WriteTo.Console(formatProvider: CultureInfo.InvariantCulture)
                .CreateLogger();
            options
                .ClearProviders()
                .AddSerilog(logger);
        });


bootstrapBuilder.Services
    .AddWindowsService();

var bootstrapper = bootstrapBuilder.Build();

var host = bootstrapper.Services.GetRequiredService<IWorkflowHost>();
host.Options.UseMaxConcurrentWorkflows(1);
host.RegisterWorkflow<MyWorkflow, MyBootstrapState>();

await host.StartWorkflow("Workflow", 1, new MyBootstrapState { });

await bootstrapper.RunAsync();

public class MyWorkflow : IWorkflow<MyBootstrapState>
{
    /// <inheritdoc />
    public string Id => "Workflow";

    /// <inheritdoc />
    public int Version => 1;

    /// <inheritdoc />
    public void Build(IWorkflowBuilder<MyBootstrapState> builder)
    {
        builder
            .While(state => true)
            .Do(recur => recur.Saga(mainSaga =>
            {
                mainSaga
                    .StartWith<MyStep>();
            }));
    }
}

public class MyStep : StepBodyAsync
{
    public override async Task<ExecutionResult> RunAsync(IStepExecutionContext context)
    {
        var bootstrapState = (MyBootstrapState)context.Workflow.Data;

        var webBuilder = WebApplication.CreateBuilder();
        webBuilder.Services
            .AddLogging(configure =>
            {
                var logPath = Path.Combine(SharedPaths.CommonLogDirectory, GetLogFileName());
                var logger = new LoggerConfiguration()
                    .ReadFrom.Configuration(webBuilder.Configuration)
                    .WriteTo.File(logPath, formatProvider: CultureInfo.InvariantCulture, rollingInterval: RollingInterval.Day, retainedFileCountLimit: 15, shared: true)
                    .WriteTo.Console(formatProvider: CultureInfo.InvariantCulture)
                    .CreateLogger();
                configure
                    .ClearProviders()
                    .AddSerilog(logger);
            });

        webBuilder.Services.AddHostedService<MyInnerService>();

        await webBuilder.Build().RunAsync();

        return ExecutionResult.Next();
    }

    public static string GetLogFileName()
    {
        return string.Format("My_Step.log");
    }
}

public class MyInnerService : BackgroundService
{
    public ILogger<MyInnerService> _logger;
    public MyInnerService(ILogger<MyInnerService> logger) { _logger = logger; }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("STARTING EXEC INNER SERVICE");
        await Task.Delay(5000, stoppingToken);
        _logger.LogInformation("ENDING EXEC INNER SERVICE");
    }

    public override async Task StartAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("STARTING START INNER SERVICE");
        await Task.Delay(5000, stoppingToken);
        _logger.LogInformation("ENDING START INNER SERVICE");
    }

    public override async Task StopAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("STARTING STOP INNER SERVICE");
        await Task.Delay(5000, stoppingToken);
        _logger.LogInformation("ENDING STOP INNER SERVICE");
    }
}

public class MyBootstrapState
{
}

Originally posted by @DelineaSamGeorge in #645

[danielgerlag]

You're running a web server inside a workflow step?