Merge prep-source-build into build script

[am11]

Having a separate environment preparation step is not great UX from an end-user perspective. It feels like an internal detail that leaked into the workflow. The standard process requires running prep followed by build:
https://github.com/dotnet/dotnet/blob/3a4520155cd43fe9bf469961e91159822ba67405/README.md#building

If there are use cases where build -sb needs to run without prep, we could add a --skip-prep flag or similar to the build script.

Expected:

# clone VMR
./build.sh --clean-while-building -sb --os linux --arch riscv64 

Actual (https://github.com/filipnavara/dotnet-riscv/blob/3e4eb9b126c8bad193d556922be925eb539d0653/.github/workflows/build.yml#L35-L36):

# clone VMR
./prep-source-build.sh
./build.sh --clean-while-building -sb --os linux --arch riscv64 -p:DisableDevBuildAsDefaultForSourceOnly=true

cc @mthalman, @ViktorHofer, @tmds, @jkoritzinsky

[jkoritzinsky]

My understanding is that the prep script exists in part for our distro partners to strip out files that violate their rules before they make the tarball they'll use to hold the sources they use to build .NET.

[tmds]

When .NET is built for a distro, it's important the bits used were "built from source".

To bootstrap a distro .NET, the maintainer calls prep-source-build to get the required prebuilt binaries.

Successive builds for that .NET version will no longer call prep-source-build but use assets that were "built from source" by a previous build.

cc @omajid

[am11]

In stateless mode, running bulid -sb without prep fails:

$ ./build.sh --clean-while-building -sb --os linux --arch riscv64 -p:DisableDevBuildAsDefaultForSourceOnly=true
...
NuGet packages cache: '/dotnet/.packages/'
Cannot find PackagesVersions.props.  Debugging info:
  Attempted custom PVP path: /PackageVersions.props
  Attempted previously-source-built path: /dotnet/prereqs/packages/previously-source-built/PackageVersions.props
  Attempted archive path: /dotnet/prereqs/packages/archive/
Error: Process completed with exit code 1.

build.sh, which is doing 99.99% of work anyway, can internally call prep too.

[tmds]

Yes, in for successive builds you need to give it the packages and sdk through the --with-... arguments.

I think things are the way they are because that makes it explicit when a build uses prebuilt binaries.

[am11]

Successive builds for that .NET version will no longer call prep-source-build, but instead reuse assets that were "built from source" by a previous build.

Now that the .NET10 runtime is built with --bootstrap during build.sh -sb in the VMR, the bootstrap binaries are automatically generated and placed in artifacts/bootstrap for use in the full runtime build; all behind the scenes, with no user involvement (or control over it).

Given this, we could revisit the current VMR build design as well. Some prep-source-build.sh steps can be inferred and performed automatically by the build script. Others could be exposed via gesture like --only-generate-layout (if we really need to split this step), keeping a unified build.sh interface. For example, when build.sh -sb runs in GitHub Actions without prebuilt artifacts, it could implicitly perform the necessary prep steps because there is no gesture which is preventing it to 'go ahead and do what makes sense'.

[tmds]

Now that the .NET10 runtime is built with --bootstrap during build.sh -sb in the VMR, the bootstrap binaries are automatically generated and placed in artifacts/bootstrap for use in the full runtime build; all behind the scenes, with no user involvement (or control over it).

It's not related to that bootstrap, which is a refactoring of the runtime repo build.

The prebuilt binaries for the initial distro bootstrap build are any nuget packages and the SDK itself needed to do an initial build. The outcome of that build will include these same packages and an SDK which are "built from source" and suitable to be used for successive builds by distro guidelines.

I'm not opposed to improving the UX for building the VMR.

I think the reason for the separate prep-source-build script is what I mentioned in the previous comment:

that makes it explicit when a build uses prebuilt binaries.

[am11]

It's not related to that bootstrap, which is a refactoring of the runtime repo build.

It's an analogy for implicit vs. explicit. The give away was Given this, we could revisit the current VMR build design as well in my previous comment.

--with-sdk and --with-packages are common args between prep-source-build and build script already, so it can also handle environment preparation.

[tmds]

I think things are the way they are because that makes it explicit when a build uses prebuilt binaries.

which is no longer the case with what you are proposing.

[am11]

When there is no previously built SDK, then I don't see the issue.

[tmds]

I think they made it explicit because it's important for distro maintainers to know when they use prebuilt binaries.

[am11]

Actual (filipnavara/dotnet-riscv@3e4eb9b/.github/workflows/build.yml#L35-L36):

In that workflow, we can differentiate that use-case from the distro maintainer one. That use case is:

• it's a cross build
• it is a community platform
  • with no MSFT build or preexisting artifacts whatsoever

The common thread between distro maintainer and this from scratch use-cases are -sb. If we can make build.sh infer the intent, we can simplify it.

[MichaelSimons]

There are scenarios in which distro maintainers have to perform select prep steps separate from the build. These are related to gathering prebuilts and prepping the source (removing binaries). The resulting src/prebuilts are then passed into their build systems which run in isolated/offline environments. That is not to say the UX could be refactored, it is a statement about the requirements.