Ability to import Modules during startup of PowerShell runspace

[FabienTschanz]

Hey there, awesome module you did! I'm currently checking for parallelization in the Microsoft365DSC module when exporting configuration, so that we can speed up the export by a couple of times.

Right now, the export might run faster than sequential, but because we're dynamically loading modules into the session, it might (on shorter runs) be the opposite way. We're still talking about a duration of 2-5 minutes, where sequentially executing the export can be almost as fast as the parallel run because of the module import.

After tracing the module import down, it seems that when the PowerShell runspace is open and commands are executed, loading modules across multiple runspaces at the same time slows down the import massively compared to running it in a single session. Is it possible to "share" a common module state or have the runspaces import the modules during creation? I don't know if that is actually faster or not, but I'm curious about it. Adding them to the session state is pretty easy using the ImportPSModule method of the InitialSessionState class.

Thank you!

[santisq]

@FabienTschanz hey! not sure how useful this will be to your use case, the reason why I don't think it's gonna help is because once a module gets added to the ISS, the module gets loaded only when we call Runspace.Open(), meaning that adding it to the ISS or calling Import-Module from the parallel scope would have an equivalent performance. I could technically use OpenAsync() in which case the module might get loaded in parallel, tho this method is a void task (who knows what the pwsh guys were thinking here...) and is event driven making it all a bit more tedious to implement. Will see how it goes, thanks for submitting the issue 👍

[FabienTschanz]

Thank you for the quick answer, really appreciated. If you think the effort is not worth the output, feel free to disregard it. Was simply thinking about the async approach and maybe have a better import speed if the runspace gets created compared to doing it while it's already open. After looking at the source code though, it seems that ImpotPSModule just leads to another Import-Module command in the runspace just after opening it. So maybe it doesn't help.

Still curious about what you will find - Thanks a lot for investigating, appreciate the effort.

Bit of context about the loading of the modules, if we open the default of 5 runspaces, then importing of the modules until they're ready is about 1.5 to 2 times slower compared to when we "only" use 3 runspaces in parallel (using -ThrottleLimit 3). Maybe that also has to do with file-locking and reading from disk if multiple runspaces are importing a module in quick succession or even concurrently. This means that an export which takes ~90 seconds with three runspaces, has upwards of 110 seconds when using 5 (although it should actually be faster because we're doing network requests and calculations in parallel). The benefit of the 2 more runspaces get swallowed by the longer module import duration sadly.

If you need any code on how we import the modules, just ping me and I'll gladly provide it.

[santisq]

@FabienTschanz Hi Fabien, would you be willing to test the branch: https://github.com/santisq/PSParallelPipeline/tree/48-ability-to-import-modules-during-startup-of-powershell-runspace ? If you need help building the module I'll help you. Let's see if this gives any improvement over importing manually. After some research, OpenAsync() is a disaster and unreliable, mainly because they open the runspace before binding ISS LocalConnection.cs#L687-L695, this means that even if I somehow await the runspace from Opening to Opened I'm risking errors because the binding (loading of modules, variables, functions, etc) happens after which ends with an exception:

"The pipeline was not run because a pipeline is already running. Pipelines cannot be run concurrently."

[FabienTschanz]

@santisq Thank you very much, will try it out and give feedback asap.

[santisq]

@FabienTschanz If you're already testing, please pull the branch again and re-build. I've made a few changes, now it is actually very noticeable:

This test was taking 15 seconds before. For reference, TestModule.psm1 is:

Start-Sleep 5
function Get-HelloWorld { "Hello world!" }

[FabienTschanz]

After testing it out, yes indeed, it works way better than previously. The modules are available at a blazingly fast speed pretty much immediately after the session is started. Now having a throttle limit of 3 vs 5 is not too different because I guess the workload I'm testing with is not big enough, but it's still somewhere between 5 and 15 seconds on a 70-80 second workload. So yes, indeed it's making a huge impact and benefits us greatly in the parallelization.

Now only to figure out which modules we need to load during startup without bloating the runspaces too much, but that's a problem we need to figure out ourselves 😆

Edit: Even the last change you just pushed makes a difference, now the runspaces are created asynchronously if I'm not mistaken. I can see that in the console as well, where validating the same modules now pops up way quicker than previously.

[santisq]

After testing it out, yes indeed, it works way better than previously. The modules are available at a blazingly fast speed pretty much immediately after the session is started. Now having a throttle limit of 3 vs 5 is not too different because I guess the workload I'm testing with is not big enough, but it's still somewhere between 5 and 15 seconds on a 70-80 second workload. So yes, indeed it's making a huge impact and benefits us greatly in the parallelization.

Now only to figure out which modules we need to load during startup without bloating the runspaces too much, but that's a problem we need to figure out ourselves 😆

Edit: Even the last change you just pushed makes a difference, now the runspaces are created asynchronously if I'm not mistaken. I can see that in the console as well, where validating the same modules now pops up way quicker than previously.

@FabienTschanz That is correct, before the last push when a PSTask requested a runspace via GetRunspaceAsync() it was blocking the worker task due to rs.Open():

    private Runspace CreateRunspace()
    {
        Runspace rs = RunspaceFactory.CreateRunspace(_settings.InitialSessionState);
        rs.Open();
        return rs;
    }

    internal async Task<Runspace> GetRunspaceAsync()
    {
        await _semaphore
            .WaitAsync(Token)
            .ConfigureAwait(false);

        if (_pool.TryDequeue(out Runspace runspace))
        {
            return runspace;
        }

        return CreateRunspace();
    }

After the change, the main difference is the wrapping of CreateRunspace() in a task:

    private Runspace CreateRunspace()
    {
        Runspace rs = RunspaceFactory.CreateRunspace(_settings.InitialSessionState);
        rs.Open();
        return rs;
    }

    private Task<Runspace> CreateRunspaceAsync() =>
        Task.Run(CreateRunspace, cancellationToken: Token);

    internal async Task<Runspace> GetRunspaceAsync()
    {
        await _semaphore
            .WaitAsync(Token)
            .ConfigureAwait(false);

        if (_pool.TryDequeue(out Runspace runspace))
        {
            return runspace;
        }

        return await CreateRunspaceAsync().ConfigureAwait(false);
    }

Will need to test a bit further, document changes and write a few Pester tests. If all goes well the new version should be published by tomorrow.

[FabienTschanz]

For what it concerns us, you do not have to put any time pressure on you, so if you want to enjoy the weekend outside of programming, please do so. Appreciate the very quick answer and implementation.