Resolving issue with Costura packed assemblies

[MaYoT27]

Hello!

I am using Fody Costura for packing/embedding assemblies into one DLL. When the method from e.g. a test is called, Costura unpacks the needed assemblies into the AppData\Local\Temp\Costura\{guid} folder. For example the assembly with test is called TestAssembly.dll and inside there is Costura packed assembly EmbeddedAssembly.dll that is not present in the directory of the TestAssembly.dll, but unpacked into that Costura folder on demand.

Now, the issue is, that when I'm trying to run the tests with NUnit.Engine, when the test contains something from the EmbeddedAssembly.dll, it's properly unpacked into the Costura folder, but the assembly cannot be resolved. The internal log file shows:

Debug [17] TestAssemblyLoadContext: Loading EmbeddedAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null assembly
Debug [17] TestAssemblyResolver: Best version dir for C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App is 6.0.31, but there is no C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App\6.0.31\EmbeddedAssembly.dll file
Debug [17] TestAssemblyResolver: Best version dir for C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App is 6.0.31, but there is no C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App\6.0.31\EmbeddedAssembly.dll file
Info  [17] TestAssemblyResolver: Cannot resolve assembly 'EmbeddedAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'

The TestAssembly.dll is .NET 8.0. My project that runs those tests is in .NET Standard 2.0 with NUnit.Engine version 3.18.3, but the behavior is the same when I build the project on .NET 8.0 and use the newest engine (3.20.1).

I have my own solution for that issue, but that requires changes in the NUnit.Engine itself. I cloned the repo and added new ResolutionStrategy for the TestAssemblyResolver.cs that gets all the DLLs from the Costura folder and also very important for my case, checks the version of the DLL, not only the name:

public class CosturaDirectoryStrategy : ResolutionStrategy
{
    private string _costuraDirectory;

    public CosturaDirectoryStrategy(string costuraDirectory)
    {
        _costuraDirectory = costuraDirectory;
    }

    public override bool TryToResolve(
        AssemblyLoadContext loadContext, AssemblyName assemblyName, out Assembly loadedAssembly)
    {
        loadedAssembly = null;
        if (assemblyName.Version == null)
            return false;

        try
        {
            var allDirs = Directory.GetDirectories(_costuraDirectory, "*", SearchOption.AllDirectories);

            var sortedDirs = allDirs
                .Select(d => new DirectoryInfo(d))
                .OrderByDescending(di => di.LastWriteTimeUtc)
                .ToList();

            foreach (var dir in sortedDirs)
            {
                string candidate = Path.Combine(dir.FullName, assemblyName.Name + ".dll");
                if (File.Exists(candidate))
                {
                    try
                    {
                        var candidateName = AssemblyName.GetAssemblyName(candidate);

                        if (candidateName.Version == assemblyName.Version)
                        {
                            loadedAssembly = loadContext.LoadFromAssemblyPath(candidate);

                            log.Info("'{0}' ({1}) assembly v{2} is loaded from CosturaDirectory {3}",
                                assemblyName.Name,
                                loadedAssembly.Location,
                                candidateName.Version,
                                dir.FullName);

                            return true;
                        }
                    }
                    catch (BadImageFormatException)
                    {
                        continue;
                    }
                }
            }

            log.Debug("No matching DLL for {0}, version {1} found in {2}",
                assemblyName.Name,
                assemblyName.Version,
                _costuraDirectory);

            return false;
        }
        catch (Exception ex)
        {
            log.Error("Error while resolving from Costura directories: '{0}'", ex.Message);
            return false;
        }
    }
}

With that and pointing to the Path.Combine(Path.GetTempPath(), "Costura") directory it works well.

Is there any possibility to get it done with current implementation of NUnit.Engine, without having to resort to cloning the repo and adding that ResolutionStrategy? I was looking at the TrustedPlatformAssembliesStrategy, but the problem is that the assembly might not be yet available before Costura unpacks it and this is already happening during the test execution.

Also worth to mention that test run from Rider or Visual Studio test explorers works well, without this issue.

[CharliePoole]

@MaYoT27 It's interesting that it works under the VS adapter, which uses the engine. How do you actually trigger the unpacking? I.E. does the test call something, is an attribute used, something else?

[MaYoT27]

@CharliePoole In my scenario it's trigger by simply invoking a method that is a part of the Costura packed assembly. The unpacking is triggered correctly, but none of the original ResolutionStrategy resolves that as it should. Maybe the VS adapter and the Visual Studio itself is doing something in the background that makes the engine to resolve it properly?

With debugging and looking at a possible options for the TestEngine, TestPackage and TestRunner I can't see any possibility that it would work without that additional ResolutionStrategy. Just an idea, but perhaps extending the engine with possibility to inject custom ResolutionStrategy would be a nice thing?

[CharliePoole]

@MaYoT27 Can you test using the latest from our myget feed, which is version 3.21.0-alpha.2?

[CharliePoole]

I think that allowing extensions that provide new resolution strategies is a reasonable idea. However, it's probably a bit more than we want to do in the remaining life of the 3.x series.

Let's first take another look at the issue as it appears (assuming it does) when you run with 3.21.0-alpha.2.

[CharliePoole]

@MaYoT27 Am I understanding correctly that the "Costura folder" is the same folder that holds your test assemblies and that the dependencies are therefore in the same directory as the test assembly on execution?

If that's so, I'm wondering why we don't find it with this code.

[MaYoT27]

@CharliePoole I tested that with 3.21.0-alpha.2 and it still doesn't work. The behavior is the same as previously.

The "Costura folder" is not the same as the folder that holds test assemblies, actually it will never be and that's why the part of the code you provided won't find the requested assemblies. The unpacked dependency assemblies are located always in the %TEMP%\Costura\{guid}.

I researched and tested a little bit more the behavior of that inside the Visual Studio and why it works outside of the tests and it seems that there is some resolving mechanism for the load contexts in the background.

My console app has only one line of code and a nuget package reference TestAssembly that also has Costura embedded assemblies inside. The method I am invoking is from the Costura packed EmbeddedAssembly:

var instance = TestAssembly.EmbeddedAssembly.Create();

Before execution of that line inside the AssemblyLoadContext.All there is just one context Default and with all the regular System stuff and my directly referenced TestAssembly. When the line is executing, additional context appears:

{"Assembly.LoadFile(C:\Users\{my_user}\AppData\Local\Temp\Costura\{some_random_guid}\EmbeddedAssembly.dll)" System.Runtime.Loader.IndividualAssemblyLoadContext #1}

It's done automatically and I don't have any additional resolving mechanism. Additionally, the folder with the guid and unpacked assembly appears in the %TEMP%\Costura\. Then the line is executed properly as the dependency is resolved most likely inside the Default context.

Edit:

Actually this stuff with additional context is also working when running with NUnit Engine!

I have a test that I start via NUnit Engine:

[Test]
public void EmbeddedTest()
{
    object instance = null;
    Assert.DoesNotThrow(() =>
    {
        instance = TestAssembly.EmbeddedAssembly.Create();
    });
    Assert.That(instance, Is.Not.Null);
}

When I check the AssemblyLoadContext.All before the execution there are 2 contexts: Default and NUnit.Engine.Internal.TestAssemblyLoadContext. When the Create() method throws exception that the method cannot be found, I can see that this System.Runtime.Loader.IndividualAssemblyLoadContext is also available, but it's already too late. I tried some quick approach with looking for the assembly in other contexts in TestAssemblyResolver.cs (cloned 3.21.0-alpha.2), but it's not available yet when it executes... Strange.

private Assembly OnResolving(AssemblyLoadContext loadContext, AssemblyName assemblyName)
{
    if (loadContext == null) throw new ArgumentNullException("context");

    foreach (var strategy in ResolutionStrategies)
        if (strategy.TryToResolve(loadContext, assemblyName, out Assembly loadedAssembly))
            return loadedAssembly;

    foreach (var alc in AssemblyLoadContext.All.Where(c => c != AssemblyLoadContext.Default && c != _loadContext))
    {
        foreach (var asm in alc.Assemblies)
        {
            if (AssemblyName.ReferenceMatchesDefinition(asm.GetName(), assemblyName))
                return asm;
        }
    }

    log.Info("Cannot resolve assembly '{0}'", assemblyName);
    return null;
}

[CharliePoole]

@MaYoT27
The fact that it passes under the Visual Studio is likely due to one of two factors:

1. Visual Studio already has created a process and loaded all assemblies when it creates the process that runs the test adapter. They may already recognize Costura assemblies.
2. Even if they don't, the NUnit engine is run multiple times under Visual Studio, once for discovery and again each time the tests are executed. Each of these executions takes place in a separate process. It's possible that the directory has already been created by the time we are executing.

Bear with me as I'm trying to debug this without installing Costura in my own development environment. Time is a bit short for when we need to produce a new release 3.21.0 and I'd still like to find an easy fix for this. If not, I'll "promote" this issue to V4, giving me more time to deal with it properly but not providing a fix for you right now.

I'm still unclear about where in your test code you normally invoke the Create method on your embedded assembly, causing the directory to be populated. Can you be specific? Do you specify the directory location or is it automatic? Is it possible to know that directory in advance and place it into an environment variable or use it on the command-line? It seems as if it should be unnecessary to add a new strategy for every location where dependencies may be found.

[MaYoT27]

@CharliePoole You're right, that might be why it passes under VS. When debugging the test run with NUnit Engine it also looks like it's kind of a race condition, where Costura unpacks the assembly, but adds the new context AFTER all ResolutionStrategies are used.

It's not a burning issue for me now, because I have that "workaround" with my own ResolutionStrategy, but that of course requires to use custom NUnit Engine rather than just a nuget package.

The Create method is actually just an example, there are more methods from other embedded assemblies and they can be invoked anywhere in the test, in the set up, test itself or tear down. I believe that is not a concern here as Costura handles it well, automatically unpacks them on demand wherever they're invoked.

For the Costura itself, the directory is automatically selected and created. The default is as I mentioned %TEMP%\Costura\. Then, the unpacking mechanism creates another directory inside with some random Guid, but it's also automatic. Inside that directory are located unpacked assemblies.

So for the test I'm currently using the full "flow" is:

1. The TestAssembly.dll has an explicit package dependency to TestComponents. That package is just one assembly, but has Costura packed EmbeddedAssembly with some version compatible to that TestComponents.
2. Some test is executed and a method is invoked from TestComponents that also invokes a method from EmbeddedAssembly. Costura properly unpacks it into the %TEMP%\Costura\, not directly into that directory, but creates a folder with a Guid name and unpacks it there. Costura also creates a new assembly load context (too late for NUnit Engine either way) with that unpacked assembly loaded.
3. In case of a VS test explorer, the dependency is resolved correctly from that new context (I think so?) and the test runs without any exception.

You're right about knowing the directory path and placing in advance some environmental variable. In my case I know it's 100% going to be %TEMP%\Costura\, BUT I don't know what kind of inner directory will be used with that Guid name. So for my CosturaDirectoryStrategy in the first post I search for all directories inside that one (ordered by modification date descending as a small optimization effort) and also if the version of the found DLL matches the one that's requested.

[CharliePoole]

@MaYoT27 Thanks, that was very clear!

For now, I'm postponing this issue to a future release. I'll mark it for V4 as it's easier to publish experimental releases there, without affecting production use of V3. Our current plan is that 3.21.0, which I'm about to release will be the last minor version of V3.

For V4, I'm hoping to eliminate some of the existing strategies - that's one reason I hesitated to add a new one. Your suggestion of allowing strategies provided by an extension makes sense. There are several ways this could be done.

I'd like to keep working with you on this in the V4 timeframe.

[MaYoT27]

@CharliePoole Great. I can provide any type of support that's needed for that feature for the V4 🙂