How to embed other "Unknown" digital assets on publish via GenerateBundle (Single File Publish) without self-extraction

[nietras]

I've been diving into how single file publish works as I had an idea of how to embed other digital assets (e.g. zip/7-zip files) into a published single file exe. This is possible by adding such files to FilesToBundle and defining metadata RelativePath for these items IF one uses <IncludeAllContentForSelfExtract>true</IncludeAllContentForSelfExtract>. This, however, has the unfortunate effect of causing all embedded content to be self-extracted which I do not want or need since quite large assets.

Looking at the code as listed below. All the parts are there to be able to use this for my purpose, but a bit frustratingly the way options are defined do not allow me to do this. All I needed was to be able to say include "unknown" content without self-extracting, unfortunately inclusion and self-extration are tied together it seems (?). My question then is how I can possibly circumvent this in some way?

I tried hacking this by changing the assets extension to .pdb and using IncludeSymbolsInSingleFile, but this for some reason isn't possible any more in .NET 5.0+?

Why not allow devs to use this feature to embed other "unknown" assets (at own risk/peril etc.) so one can avoid going down the path of having to embed such assets as content in a managed dll that has memory consumption issues. This is important for my use case due to 32-bit limits. And don't want to put GBs in temp directory.

Microsoft.NET.Publish.targets
https://github.com/dotnet/sdk/blob/cfa6ec2353784e22171bc4539181a1a935e066e9/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets#L959-L988
GenerateBundle.cs
https://github.com/dotnet/sdk/blob/f65053d80604949ed62c663e3e6975bc75273ba6/src/Tasks/Microsoft.NET.Build.Tasks/GenerateBundle.cs
Bundler.cs
https://github.com/dotnet/runtime/blob/0090de47c7f1d11250c2d650bf8892fa58acad33/src/installer/managed/Microsoft.NET.HostModel/Bundle/Bundler.cs
BundleProbeTester/Program.cs
https://github.com/dotnet/runtime/blob/main/src/installer/tests/Assets/TestProjects/BundleProbeTester/Program.cs

[vitek-karas]

Why not allow devs to use this feature to embed other "unknown" assets

The main reason is that we would have to define new APIs which would let you access these assets. .NET already has something similar - embedded assets in managed assemblies and the relevant APIs for that. Defining new APIs for basically a very similar purpose feels wrong. There are other discussions on allowing to access these through special formed paths (Windows allows this in some cases to access Win32 resources), but that comes with a lot of challenges - there are lot of APIs in .NET which access file path and it would confusing to support these new paths only in some of them (and doing them all is very expensive).

The other consideration is compatibility - is you use embedding into managed assemblies the code will work regardless if it's published as single-file or not (so debug builds will work the same as published app). Any APIs which only work in single-file mode would not have that property and testing the code using them would be much more difficult.

If you're on Windows you could see if embedding them as Win32 resources works - you can access those via PInvokes easily. But I don't know if the same 2Gb limit applies there as well.

[nietras]

See my answer for how I got this working. As far as I am concerned no new API is needed. There already is one, although perhaps unofficial, but the BundleProbeDelegate is fine for advanced usage.

Embedding assets into managed assemblies simply isn't usable. Consider ML and other cases where one has very large digital assets. This will only become more common. We need support for it. And all the pieces are there with the single file publishing and only a few things would be needed to allow using it for that e.g.

• Add optional SelfExtract metadata to FilesToBundle items to allow overriding whether a given specific file should self-extracted (overrides any defaults)
• Add optional Compress metadata to FilesToBundle items to allow overriding whether a given specific file should be compressed or not (overrides any defaults)
• Allow embedding any kind of file disregarding hard-coded rules for which files and IncludeAllContentForSelfExtract perhaps via some other option IncludeAllContent.

[nietras]

Filed issue on this #91202

[nietras]

I managed to get this working by going deep into the bowels of the bundling code. Here a short recap. In the project where we want to embed some extra files the following targets is added. Note how IncludeNativeLibrariesForSelfExtract is set to true, as discussed in question above.

  <PropertyGroup>
    <PublishSingleFile>true</PublishSingleFile>
    <EnableCompressionInSingleFile>true</EnableCompressionInSingleFile>
    <IncludeNativeLibrariesForSelfExtract>true</IncludeNativeLibrariesForSelfExtract>

    <IncludeAllContentForSelfExtract>true</IncludeAllContentForSelfExtract>
    <TraceSingleFileBundler>true</TraceSingleFileBundler>
    <FilesToBundleAddArchivesEnable>true</FilesToBundleAddArchivesEnable>
  </PropertyGroup>
  <Target Name="FilesToBundleAddArchives" BeforeTargets="GenerateSingleFileBundle" DependsOnTargets="PrepareForBundle"
          Condition="$(FilesToBundleAddArchivesEnable)">
    <Message Text="PublishDir '$(PublishDir)'" Importance="high" />
    <ItemGroup>
      <ArchivesToAddToFilesToBundle Include="$([System.IO.Path]::GetFullPath($(PublishDir)))\..\publish-archives\*.7z" />
      <!-- Add metadata for the relative path for GenerateBundle -->
      <ArchivesToAddToFilesToBundle RelativePath="%(ArchivesToAddToFilesToBundle.FileName)%(ArchivesToAddToFilesToBundle.Extension)" />
    </ItemGroup>
    <Message Text="RelativePath: %(ArchivesToAddToFilesToBundle.RelativePath) - Items: @(ArchivesToAddToFilesToBundle)" Importance="high" />
    <ItemGroup>
      <FilesToBundle Include="@(ArchivesToAddToFilesToBundle)" />
    </ItemGroup>
  </Target>

To then fix the bundle meta-data to avoid the * .7z files being self-extracted a custom MSBuild task modifies the bundle manifest after the single-file exe has been created.

  <!--
  IncludeAllContentForSelfExtract=true will cause all embedded content in
  single file to be self-extracted to
  `C:\Users\<USERNAME>\AppData\Local\Temp\.net\<EXE>\<HASH>` for our 
  archives of GB size this is unfortunate as takes up space but also since
  this happens on app start so that is sluggish. 

  To fix that a custom task modifies the bundle manifest to avoid this
  extraction for all but the native library files as normal, while still
  embedding the archives.
  -->
  <UsingTask TaskName="SingleFileBundleManifestModifyTask"
           TaskFactory="RoslynCodeTaskFactory"
           AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.Core.dll">
    <Task>
      <Code Type="Class" Language="cs" Source="$(MSBuildThisFileDirectory)../SingleFileBundleManifestTasks/SingleFileBundleManifestModifyTask.cs" />
    </Task>
  </UsingTask>
  <Target Name="SingleFileBundleManifestModifyTask-Installer"
          AfterTargets="GenerateSingleFileBundle">
    <Message Text="MSBuildToolsPath: $(MSBuildToolsPath)" Importance="high" />

    <SingleFileBundleManifestModifyTask BundlePath="$(PublishDir)\$(TargetName).exe"
                                        FileEntryRelativePathEndsWith=".7z"
                                        FileEntryFileTypeToChange="Unknown"
                                        FileEntryNewFileType="Assembly" />
  </Target>

The task C# code is based on bundler code e.g. see and follow code from:

runtime/src/installer/managed/Microsoft.NET.HostModel/Bundle/Bundler.cs

Line 245 in 3bda6e0

public string GenerateBundle(IReadOnlyList<FileSpec> fileSpecs)

)

Basically, the task HeaderFlags so no longer NetcoreApp3CompatMode but None and the changes FileType from Unknown to Assembly for the 7z files.

This trick can be used to embed any kind of file without it being self-extracted. The contents can then be accessed via the BundleProbeDelegate as shown in.

runtime/src/installer/tests/Assets/TestProjects/BundleProbeTester/Program.cs

Line 7 in e18e4aa

namespace BundleProbeTester

And using a PartialStream or similar to then access the segment of the single-file exe relevant to the embedded file.

[vitek-karas]

Please note that the existence of the BUNDLE_PROBE runtime property is an implementation detail which is actually going to change.
In .NET 8 the host/runtime already uses a different mechanism - Host runtime contract - specifically:

runtime/src/native/corehost/hostpolicy/hostpolicy_context.cpp

Line 374 in f470200

host_contract.bundle_probe = &bundle_probe;

The host still sets the BUNDLE_PROBE property, but we're likely going to remove that sometime soon from the host (there's no good reason to keep it there, just bloats the memory usage on startup).

@elinor-fung as FYI