Skip to content

Conversation

michaeltlombardi
Copy link
Collaborator

@michaeltlombardi michaeltlombardi commented Oct 6, 2025

PR Summary

Prior to this change, the project didn't use a cargo workspace. Instead, every crate independently managed its dependencies and we associated crates with Rust Analyzer in VS Code through the configuration settings.

This change:

  • Defines a virtual workspace at the repository root, including every crate as a workspace member, defining the release profile, and defining every crate dependency.
  • Updates the cargo manifest for every crate to inherit their dependencies from the workspace.
  • Replaces the per-crate cargo lock files with the workspace-level lock file. Now dependency management happens with a single manifest and lock file.
  • Makes minor changes to the build script to ensure functionality continues as previously defined.

Note that this change on its own doesn't drastically improve build times. For build times to improve substantially, we need to:

  1. Restructure the build to compile all crates with a single command. When we use cargo build at the repository root or the --workspace flag, cargo builds every crate and its dependencies together. Cargo uses the dependency graph to only compile each dependency crate once during the build, instead of recompiling every dependency when we compile on a crate-by-crate basis.
  2. Enable the use of caching, like with sccache. When we enable cached builds, only crates that have been modified since the last build will be recompiled. This substantially improves build times for local development and when we invoke tests after initial compilation.

However, both of these changes require more substantial modifications to the build script.

To do:

  • Verify CI passes for initial commit
  • Add alternate build script (need to ensure functionality before removing existing build script)
  • Reorganize repository and update build scripts to match

Other changes required as part of this PR:

  • Fix a bug in the Windows PowerShell adapter to remove the builtin PSModulePaths instead of blanket removing any module path with powershell as an ancestor folder.

PR Context

@michaeltlombardi michaeltlombardi force-pushed the gh-1152/main/rust-workspace branch 4 times, most recently from 099fbd5 to 31d1402 Compare October 6, 2025 21:25
@michaeltlombardi
Copy link
Collaborator Author

Initial commit passed CI:

@michaeltlombardi michaeltlombardi changed the title (GH-1152) Define cargo workspace (GH-1152, GH-1153) Define cargo workspace and reorganize projects Oct 6, 2025
Prior to this change, the project didn't use a cargo workspace.
Instead, every crate independently managed its dependencies and
we associated crates with Rust Analyzer in VS Code through the
configuration settings.

This change:

- Defines a virtual workspace at the repository root, including
  every crate as a workspace member, defining the release profile,
  and defining every crate dependency.
- Updates the cargo manifest for every crate to inherit their
  dependencies from the workspace.
- Replaces the per-crate cargo lock files with the workspace-level
  lock file. Now dependency management happens with a single manifest
  and lock file.
- Makes minor changes to the build script to ensure functionality
  continues as previously defined.

Note that this change on its own doesn't drastically improve build
times. For build times to improve substantially, we need to:

1. Restructure the build to compile all crates with a single command.
   When we use `cargo build` at the repository root or the `--workspace`
   flag, cargo builds every crate and its dependencies together. Cargo
   uses the dependency graph to only compile each dependency crate once
   during the build, instead of recompiling every dependency when we
   compile on a crate-by-crate basis.
1. Enable the use of caching, like with `sccache`. When we enable cached
   builds, only crates that have been modified since the last build will
   be recompiled. This _substantially_ improves build times for local
   development and when we invoke tests after initial compilation.

However, both of these changes require more substantial modifications to
the build script, so they aren't addressed in this change. Those changes
will be addressed in a future commit.
This change moves the library crates, which don't have an executable,
into the `lib` folder. This makes discovering shared libraries easier,
instead of relying on the `_lib` convention the crates use. This change
doesn't rename the libraries, it only moves them.

This change doesn't require modifying any code or cargo manifests,
except for pointing to the correct path in the virtual workspace.
This change moves the resource subprojects, which have an executable,
into the `resources` folder. This makes discovering which projects
provide DSC resources easier, instead of checking whether the folder
contains any resource manifests. This change doesn't rename the folders,
it only moves them.

This change doesn't require modifying any code or cargo manifests,
except for pointing to the correct path in the virtual workspace and
build script.
This change moves the adapter subprojects, which have an executable,
into the `adapters` folder. This makes discovering which projects
provide DSC resource adapters easier, instead of relying on the
`-adapter` suffix convention. This change does rename the folders,
dropping the suffix. The adapters aren't rust crates, so this doesn't
affect code compilation.

This change doesn't require modifying any code or cargo manifests,
except for pointing to the correct path in the build script.
This change moves the tree-sitter subprojects, which define grammars,
into the `grammars` folder. This makes discovering which projects
provide DSC grammars easier, instead of relying on the `tree-sitter-`
prefix convention. This change does not rename the folders, it only
moves them.

This change doesn't require modifying any code or cargo manifests,
except for pointing to the correct path in the build script and
virtual workspace manifest.

This change also updates the `.gitignore` to generically ignore
generated grammar files in the `grammars` folder.
This change renames `dsc_lib` to `dsc-lib`. This is done for
a few reasons:

1. The general naming convention for rust crates is `kebab-case` for
   crates, `snake_case` for modules. This aligns the naming convention.

   Note that kebab-case crate names are still referenced with underscores
   as `snake_case`, so no code needed to be changed in crates, only the
   cargo manifests.
1. When we eventually publish the library crates, we will need to use a
   coherent naming convention. Standardizing on `dsc-lib` as a prefix
   ensures that the other library crates can follow this convention and
   more readily be discovered and collected. The other libraries are only
   intended for use with DSC.

This change ensures that the manifests and build script are updated to use
the renamed crate. I decided to keep a 1:1 map of crate name and folder,
to limit confusion over having more than one `dsc` folder in the project.
Prior to this change, the discovery tests referenced the path
to test files for the PowerShell adapter by folder. This change
corrects those paths to their new location.
This change updates the WMI adapter test to point to the
correct location for the test configuration after the folder
move and rename.
… for WinPS

Prior to this change, the adapter munged the PSModulePath by
removing any path items containing the segment `\powershell\`
to ensure no PowerShell 7 modules are used for discovery when
invoked with the Windows PowerShell adapter.

This implementation was highlighted as a bug after moving and
renaming the `powershell-adapter` folder to `adapters/powershell`.
The tests showed that none of the test modules in the `Tests`
subfolder were discovered by the adapter.

The prior implementation removed not only the actual PowerShell 7
module paths, but _any_ module path with `powershell` anywhere in
the ancestor folder path. Not only does this cause the tests to
fail, but it also prevents users from discovering any modules
they might have added in custom locations like `~\dev\powershell`
or `D:\Tools\PowerShell`.

This change updates the implementation to specifically search for
and remove the default module paths for PowerShell _if_ PowerShell
is installed. It no longer uses a simple match for any path that
contains `powershell` as an ancester folder.
This change renames the `osinfo_lib` crate to `dsc-lib-osinfo`.
This is to support eventual publishing of the crates. This crate
is a dependency not only for the `osinfo` resource but also the
`dsc-lib` crate. When we implement the Rust RDK, we will need to
publish the library crates, and this ensures the naming is both
consistent and easily discoverable.

This change makes the requisite updates to cargo manifests, Rust
code, and the build script.
This change renames the `pal` crate to `dsc-lib-pal`. This is to
support eventual publishing of the crates. When we implement the
Rust RDK, we will need to publish the library crates, and this
ensures the naming is both consistent and easily discoverable.

This change makes the requisite updates to cargo manifests, Rust
code, and the build script.
This change renames the `registry_lib` crate to `dsc-lib-registry`.
This is to support eventual publishing of the crates. This crate
is a dependency not only for the `registry` resource but also the
`sshdconfig` resource. When we implement the Rust RDK, we will need
to publish the library crates, and this ensures the naming is both
consistent and easily discoverable.

This change makes the requisite updates to cargo manifests, Rust
code, and the build script.
…y_context`

This change renames the `security_context_lib` crate to
`dsc-lib-security_context`. This is to support eventual
publishing of the crates. This crate is a dependency not
only for the `registry` resource but also the `dsc-lib`
crate. When we implement the Rust RDK, we will need to
publish the library crates, and this ensures the naming
is both consistent and easily discoverable.

This change makes the requisite updates to cargo manifests, Rust
code, and the build script.
Prior to this change, the Rust crates for specific DSC
resources used the binary name as the crate name. This
causes a few problems:

- When using `cargo` commands with the `--package`/`-p`
  option to limit which workspace packages to use for
  an operation, some of the packages, like `registry`,
  conflict with dependencies of the same name.
- If we eventually publish any of these crates, they are
  not obviously related to DSC and might be mistaken as
  general purpose tools, which we don't currently plan to
  support.
- It isn't obvious in the cargo lock file whether an item
  is for a workspace package or an external package.

This change:

1. Renames the resource crates to `dsc-resource-<name>`.
1. Defines an entry in the `bin` table for the cargo manifest
   to ensure that the binary name remains the same as it was
   before the change. We _may_ want to consider renaming the
   binaries in a future, potentially breaking change, so it
   is always obvious when a resource is builtin / which binary
   to use.

This change has no effect on the build scripts or workspace
manifest, since we explicitly point to folder paths instead of
using crate names.
@michaeltlombardi michaeltlombardi force-pushed the gh-1152/main/rust-workspace branch 8 times, most recently from 7a1584d to 00a08cc Compare October 8, 2025 13:01
This change adds a reworked build script to the project and
a GHA workflow that uses it. Currently this duplicates both
the build script and the CI jobs. For now, this enables us
to ensure continuity of existing behavior while we iteratively
improve the new build script and CI configuration to speed up
builds and improve maintainability.

The new build script generally provides equivalent functionality
with the existing legacy script to the best of my ability.

In the process, this change also:

- Defines a helper module for build-and-maintenance related functions.
  This made the build script itself somewhat easier to reason about,
  and the functions in the module can be used outside of the script.
- Moves build data out of the script and into a separate datafile. For
  now, this puts build data at the root, instead of per-project. In the
  future, I would like to refactor things to retrieve build data from
  the individual projects instead.

Finally, this change adds a benchmarking comparison script to enable us
to programmatically compare the build scripts for local and CI workflows.
@michaeltlombardi michaeltlombardi force-pushed the gh-1152/main/rust-workspace branch from 00a08cc to d2afbc5 Compare October 8, 2025 13:08
@michaeltlombardi
Copy link
Collaborator Author

Final builds passing, here is comparison data:

Linux:
  Comparison:
    Test:
      DifferenceMultiplier: 1.55x
      DifferenceDuration: 2m 22s
    Build:
      DifferenceDMultiplier: 9.7x
      DifferenceDuration: 0m -7s
    Total:
      DifferenceMultiplier: 1.48x
      DifferenceDuration: 3m 45s
  New:
    Prereq: 1m 31s
    Tests: 4m 19s
    Build: 1m 58s
    Total: 7m 53s
  Legacy:
    Tests: 6m 41s
    Build: 3m 22s
    Total: 11m 38s
Windows:
  Comparison:
    Test:
      DifferenceMultiplier: 1.15x
      DifferenceDuration: 5m 28s
    Build:
      DifferenceDMultiplier: 1.19x
      DifferenceDuration: 1m 37s
    Total:
      DifferenceMultiplier: 1.16x
      DifferenceDuration: 7m 8s
  New:
    Prereq: 3m 26s
    Tests: 35m 23s
    Build: 5m 6s
    Total: 44m 8s
  Legacy:
    Tests: 40m 51s
    Build: 10m 9s
    Total: 51m 16s
macOS:
  Comparison:
    Test:
      DifferenceMultiplier: 1.87x
      DifferenceDuration: 4m 24s
    Build:
      DifferenceDMultiplier: 1.1x
      DifferenceDuration: 0m 3s
    Total:
      DifferenceMultiplier: 1.46x
      DifferenceDuration: 4m 25s
  New:
    Prereq: 1m 43s
    Tests: 5m 3s
    Build: 2m 34s
    Total: 9m 30s
  Legacy:
    Tests: 9m 27s
    Build: 4m 20s
    Total: 13m 55s

@michaeltlombardi michaeltlombardi marked this pull request as ready for review October 8, 2025 14:59
@SteveL-MSFT SteveL-MSFT added this pull request to the merge queue Oct 8, 2025
Merged via the queue into PowerShell:main with commit 6458502 Oct 8, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants