-
Notifications
You must be signed in to change notification settings - Fork 799
Make aspire.exe a self-extracting binary for polyglot scenarios #14398
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…d, build integration - Add BundleTrailer (src/Shared/) with read/write/extract helpers for the 32-byte trailer appended to native AOT CLI binaries - Change IAppHostServerProjectFactory.Create() → CreateAsync() and update all 8 call sites (GuestAppHostProject, SdkGenerateCommand, SdkDumpCommand, ScaffoldingService, AppHostServerSession) - Add EnsureBundleAsync in AppHostServerProjectFactory that lazily extracts the embedded tar.gz payload on first polyglot command (run/publish/add) - Add 'aspire setup' command for explicit extraction with --install-path and --force options - Add --embed-in-cli option to CreateLayout tool that appends tar.gz payload + trailer to the native CLI binary - Update Bundle.proj to pass --embed-in-cli pointing to the layout CLI binary
- Version marker file (.aspire-bundle-version) written after extraction; EnsureBundleAsync and SetupCommand skip extraction when version matches - update --self proactively extracts embedded payload after replacing binary - 10 unit tests for BundleTrailer: roundtrip, payload slicing, version marker, and tar.gz extraction with strip-components - README documents self-extracting binary install path
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 14398Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 14398" |
dd89b88 to
6ae6a53
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR shifts Aspire’s “bundle” distribution to a self-extracting CLI binary model by introducing a shared trailer format (BundleTrailer) and a centralized extraction implementation (BundleService). It also simplifies the self-update/install flows and updates CI/artifacts/scripts/docs accordingly.
Changes:
- Add
BundleTrailer(shared) +IBundleService/BundleService(CLI) to support self-extracting binaries with version-marked, thread-safe extraction. - Simplify CLI update and install UX by adding
aspire setup, removing the bundle self-update path, and deduplicating archive extraction viaArchiveHelper. - Update bundle build tooling (
CreateLayout,Bundle.proj) and CI/scripts/docs to publish and consume the single self-extracting binary artifact.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/CreateLayout/Program.cs | Adds --embed-in-cli and embeds archive payload + trailer into the native CLI; CreateArchiveAsync now returns archive path. |
| tools/CreateLayout/CreateLayout.csproj | Source-links BundleTrailer.cs into CreateLayout for trailer writing. |
| tests/Aspire.Cli.Tests/BundleTrailerTests.cs | Adds unit tests covering trailer read/write, payload slicing, version marker, and extraction behavior. |
| src/Shared/BundleTrailer.cs | Implements trailer format read/write, payload substream, and version marker support. |
| src/Aspire.Cli/Utils/BundleDownloader.cs | Switches archive extraction to shared ArchiveHelper. |
| src/Aspire.Cli/Utils/ArchiveHelper.cs | Introduces a shared .zip/.tar.gz extraction helper. |
| src/Aspire.Cli/Scaffolding/ScaffoldingService.cs | Updates AppHost server factory usage to async creation. |
| src/Aspire.Cli/Projects/GuestAppHostProject.cs | Updates AppHost server factory usage to async creation in multiple flows. |
| src/Aspire.Cli/Projects/AppHostServerSession.cs | Updates AppHost server factory usage to async creation. |
| src/Aspire.Cli/Projects/AppHostServerProject.cs | Changes factory API to async and ensures bundle extraction occurs before bundle layout use. |
| src/Aspire.Cli/Program.cs | Registers IBundleService and wires SetupCommand. |
| src/Aspire.Cli/Commands/UpdateCommand.cs | Removes bundle-update path; updates self-update to optionally extract embedded payload after binary swap. |
| src/Aspire.Cli/Commands/SetupCommand.cs | Adds aspire setup to explicitly extract the embedded bundle payload. |
| src/Aspire.Cli/Commands/Sdk/SdkGenerateCommand.cs | Updates AppHost server factory usage to async creation. |
| src/Aspire.Cli/Commands/Sdk/SdkDumpCommand.cs | Updates AppHost server factory usage to async creation. |
| src/Aspire.Cli/Commands/RootCommand.cs | Adds setup subcommand to the CLI root. |
| src/Aspire.Cli/Bundles/IBundleService.cs | Defines bundle extraction contract and result enum. |
| src/Aspire.Cli/Bundles/BundleService.cs | Implements extraction logic (platform-aware tar handling, version marker, cleanup, thread safety). |
| src/Aspire.Cli/Aspire.Cli.csproj | Source-links BundleTrailer.cs into the CLI build. |
| eng/scripts/get-aspire-cli-bundle-pr.sh | Updates PR install script to place the binary in bin/ and rely on extraction. |
| eng/scripts/get-aspire-cli-bundle-pr.ps1 | Updates PR install script to place the binary in bin/ and rely on extraction. |
| eng/scripts/README.md | Documents self-extracting binary option and updated install steps. |
| eng/Bundle.proj | Passes --embed-in-cli to CreateLayout to produce the self-extracting binary. |
| docs/specs/bundle.md | Updates the bundle spec for self-extracting binary architecture, extraction flow, and update flow. |
| .github/workflows/build-bundle.yml | Uploads only the self-extracting CLI binary as the bundle artifact. |
Comments suppressed due to low confidence (1)
tools/CreateLayout/Program.cs:676
- When
--embed-in-cliis used on Windows RIDs,CreateArchiveAsyncproduces a.zippayload, but the self-extracting format andBundleServiceextraction logic assume an embeddedtar.gzpayload. This will make Windows self-extracting bundles fail to extract. Consider always generating a tar.gz payload when embedding (even on Windows), or teachBundleServiceto detect/extract a zip payload on Windows.
var archiveName = $"aspire-{_version}-{_rid}";
var isWindows = _rid.StartsWith("win", StringComparison.OrdinalIgnoreCase);
var archiveExt = isWindows ? ".zip" : ".tar.gz";
var archivePath = Path.Combine(Path.GetDirectoryName(_outputPath)!, archiveName + archiveExt);
9a16d53 to
333f9f7
Compare
🎬 CLI E2E Test RecordingsThe following terminal recordings are available for commit
📹 Recordings uploaded automatically from CI run #21875444504 |
333f9f7 to
94d0d48
Compare
…ate CI - Extract IBundleService/BundleService with thread-safe extraction - Create ArchiveHelper to deduplicate extraction utilities - Simplify UpdateCommand: remove ExecuteBundleSelfUpdateAsync path - CI: upload self-extracting binary instead of directory tree - PR scripts: simplified to place binary in bin/ (lazy extraction) - Drop --archive from Bundle.proj, remove archive artifact upload - Update bundle spec with self-extracting binary architecture
94d0d48 to
e799ac3
Compare
The DI factories for INuGetPackageCache and ICertificateToolRunner previously checked DiscoverLayout() to decide between bundle and SDK implementations. This failed on fresh installs where the bundle hadn't been extracted yet, causing the factory to permanently pick the SDK implementation (which requires dotnet). Now the factories check BundleTrailer.TryRead(ProcessPath) to detect if the running binary is a bundle. This works before extraction. The bundle implementations call EnsureExtractedAsync lazily on first use, triggering extraction only when needed. - Program.cs: Use BundleTrailer.TryRead instead of DiscoverLayout - BundleNuGetPackageCache: Add IBundleService, call EnsureExtractedAsync - BundleCertificateToolRunner: Take ILayoutDiscovery+IBundleService instead of LayoutConfiguration, resolve layout lazily
Replace in-process SemaphoreSlim with a named Mutex to prevent concurrent aspire processes from racing during bundle extraction.
Named Mutex with Global\ prefix doesn't work on Linux, causing 'Object synchronization method was called from an unsynchronized block of code' errors. Use a file lock (.aspire-bundle-lock) in the extraction directory instead — works on all platforms.
Document why we use a file-based lock instead of Mutex: cross-platform compatibility and async/await safety.
FileShare.None throws IOException immediately on Windows instead of blocking. Retry with 200ms delay up to a 2 minute timeout.
- EnsureExtractedAsync: log early exits and extraction target - ExtractAsync: log trailer details, version mismatch - ExtractCoreAsync: log clean, extraction timing, marker write, verification - FileLock: throw TimeoutException with context on timeout
- Move from Bundles/ to Utils/ for reusability - Change from IDisposable to ExecuteWithLock/ExecuteWithLockAsync API - Add UnauthorizedAccessException handling (transient during DeleteOnClose) - Use CancellationToken instead of fixed timeout - Use FileOptions.DeleteOnClose to auto-clean lock files - Use 10ms retry delay (matches NuGet) instead of 200ms - Add sync overload for non-async callers
| /// <summary> | ||
| /// Extracts the embedded bundle payload from a self-extracting Aspire CLI binary. | ||
| /// </summary> | ||
| internal sealed class SetupCommand : BaseCommand |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we'll need in the long run but I wanted to expose an explicit extract to see if we could use it from other places for a non lazy install.
- Use EmbeddedResource (conditional on BundlePayloadPath) instead of appended trailer - Add BundleService.IsBundle and OpenPayload() for detection and access - Use .NET TarReader for extraction on all platforms (remove system tar dependency) - Add path traversal guard, null symlink check, fix resource leak in CreateLayout - Reorder Bundle.proj: managed+DCP -> tar.gz -> AOT compile with embedded resource - Simplify build-bundle.yml (no concat step) - Remove BundleTrailer, SubStream, EmbedPayloadInCli
…undle.proj - Add ConfigureAwait(false) to async disposals in tar archive creation - Add BundleRuntimePath property to Bundle.proj for local builds (skips download)
|
Nice design — collapsing CLI + bundle into a single self-extracting binary is a big UX win. One thing I wanted to flag:
await TarFile.ExtractToDirectoryAsync(gzipStream, destinationPath, overwriteFiles: true, cancellationToken);Meanwhile Not a blocker for the current PR since the embedded payload path is well-protected, but worth hardening before the self-update flow ships to users. |
docs/specs/bundle.md
Outdated
| │ tar.gz payload (~100 MB compressed) │ | ||
| │ (runtime, dashboard, dcp, etc.) │ | ||
| ├─────────────────────────────────────────────────┤ | ||
| │ Trailer (32 bytes) │ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this still up-to-date?
|
|
||
| namespace Aspire.Cli.Tests; | ||
|
|
||
| public class BundleServiceTests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(nit) the file name doesn't match the class name
src/Aspire.Cli/Utils/FileLock.cs
Outdated
| // Sharing violation — another process holds the lock. On Windows the | ||
| // FileStream constructor throws immediately; on Unix it may also throw | ||
| // if the file is exclusively locked. Wait and retry. | ||
| await Task.Delay(s_defaultRetryDelay, cancellationToken).ConfigureAwait(false); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have a limit in case it's not transient, or this could wait indefinitely (nothing ensure the caller also defines a time limit)
| { | ||
| Directory.CreateDirectory(dir); | ||
| } | ||
| await entry.ExtractToFileAsync(fullPath, overwrite: true, cancellationToken); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not setting the file permissions (specifically for execute). They are in the tar but not applied by this method.
| <_CliBinlog Condition="'$(ContinuousIntegrationBuild)' == 'true'">-bl:$(ArtifactsLogDir)PublishCli.binlog</_CliBinlog> | ||
| <!-- The tar.gz archive produced by CreateLayout is embedded as a resource in the CLI binary --> | ||
| <_BundleArchivePath>$(ArtifactsDir)bundle\aspire-$(BundleVersion)-$(TargetRid).tar.gz</_BundleArchivePath> | ||
| <_VersionSuffixArg Condition="'$(VersionSuffix)' != ''">/p:VersionSuffix=$(VersionSuffix)</_VersionSuffixArg> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we setting VersionSuffix here? Shouldn't it come through the normal props when building the CLI project?
| <_CreateLayoutArgs>--output "$(_BundleOutputDirArg)" --artifacts "$(_ArtifactsDirArg)" --rid $(TargetRid) --bundle-version $(BundleVersion) --runtime-version $(BundleRuntimeVersion) --archive --verbose --download-runtime</_CreateLayoutArgs> | ||
|
|
||
| <!-- Use local runtime path if provided, otherwise download --> | ||
| <_RuntimeArgs Condition="'$(BundleRuntimePath)' != ''">--runtime "$(BundleRuntimePath)"</_RuntimeArgs> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When/where is this provided?
| /// Gets the assembly informational version of the current CLI binary. | ||
| /// Used as the version marker to detect when re-extraction is needed. | ||
| /// </summary> | ||
| internal static string GetCurrentVersion() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have this code in multiple places in the CLI. PackagingService and AspireCliTelemetry. This is now a 3rd place.
| /// </summary> | ||
| internal static async Task ExtractAsync(string archivePath, string destinationPath, CancellationToken cancellationToken) | ||
| { | ||
| if (archivePath.EndsWith(".zip", StringComparison.OrdinalIgnoreCase)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When do we use .zip files?
eerhardt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feel free to merge when you have the feedback addressed.
… code removal - FileLock: add timeout (5min default) to prevent indefinite waits - BundleService: set Unix file permissions from tar entry metadata - BundleService: move IsBundle to IBundleService instance property - BundleService: use VersionHelper to deduplicate version-reading code - IBundleService: add EnsureExtractedAndGetLayoutAsync combined method - ArchiveHelper: harden with path-traversal and symlink validation - Bundle.proj: add comments explaining VersionSuffix and BundleRuntimePath - Rename BundleTrailerTests.cs to BundleServiceTests.cs - Update bundle.md to describe embedded resource approach - Remove dead IBundleDownloader/BundleDownloader/FileAccessRetrier code
Description
This PR makes
aspire.exea self-extracting binary for polyglot scenarios (TypeScript, Python). The CLI binary contains an embedded tar.gz payload with the runtime, dashboard, DCP, and tools needed to run Aspire without the .NET SDK.Why?
Today the Aspire CLI is distributed as a separate binary alongside a bundle archive that gets extracted during install. This means:
aspire update --selfmust download and extract a bundle archive separatelyWith self-extracting binaries, the CLI is the bundle. On first run (or
aspire setup), it extracts its own payload to~/.aspire/. Theaspire update --selfflow downloads a single file and extraction happens automatically.What changed?
Embedded resource payload — The bundle payload (tar.gz) is embedded as a .NET embedded resource in the native AOT binary via
/p:BundlePayloadPath. Detection usesAssembly.GetManifestResourceInfo(metadata-only check, nanoseconds). Payload access usesAssembly.GetManifestResourceStream(demand-paged by OS, zero memory impact at startup). The payload is inside the PE/ELF signed envelope, so code signing covers it.BundleService— Centralized extraction with:IsBundleproperty andOpenPayload()methodTarReaderon all platforms (no systemtardependency)FileLock(file-based, modeled after NuGet'sConcurrencyUtilities)Build pipeline (
Bundle.proj) — Managed components publish → CreateLayout assembles layout + creates tar.gz → AOT compile CLI with tar.gz as embedded resource.CreateLayout— The CLI binary is no longer included in the bundle layout. The native AOT binary is the CLI. Always produces tar.gz on all platforms.UpdateCommand— Simplified: no longer does proactive extraction after self-update. The new binary extracts itself on first run viaEnsureExtractedAsync().Key files
src/Aspire.Cli/Bundles/BundleService.cssrc/Aspire.Cli/Aspire.Cli.csproj<EmbeddedResource>whenBundlePayloadPathis setsrc/Aspire.Cli/Utils/FileLock.cssrc/Aspire.Cli/Program.csBundleService.IsBundlefor detectioneng/Bundle.projtools/CreateLayout/Program.csChecklist