|
| 1 | +# Prototype a new set of Cargo "plumbing" commands |
| 2 | + |
| 3 | +| Metadata | | |
| 4 | +| :-- | :-- | |
| 5 | +| Point of contact | @epage | |
| 6 | +| Teams | <!-- TEAMS WITH ASKS --> | |
| 7 | +| Task owners | <!-- TASK OWNERS --> | |
| 8 | +| Status | Proposed for mentorship | |
| 9 | +| Zulip channel | N/A | |
| 10 | +| Tracking issue | [rust-lang/rust-project-goals#264] | |
| 11 | + |
| 12 | +## Summary |
| 13 | + |
| 14 | +Create a third-party cargo subcommand that has "plumbing" (programmatic) |
| 15 | +subcommands for different phases of Cargo operations to experiment with what |
| 16 | +Cargo should integrate. |
| 17 | + |
| 18 | +## Motivation |
| 19 | + |
| 20 | +Cargo is a "porcelain" (UX) focused command and is highly opinionated which can work well for common cases. |
| 21 | +However, as Cargo scales into larger applications, users need the ability to adapt Cargo to their specific processes and needs. |
| 22 | + |
| 23 | +### The status quo |
| 24 | + |
| 25 | +While most Cargo commands can be used programmatically, they still only operate at the porcelain level. |
| 26 | +Currently, Cargo's plumbing commands are |
| 27 | +- `cargo read-manifest`: |
| 28 | + - works off of a `Cargo.toml` file on disk |
| 29 | + - uses a custom json schema |
| 30 | + - deprecated |
| 31 | +- `cargo locate-project`: |
| 32 | + - works off of a `Cargo.toml` file on disk |
| 33 | + - text or json output, undocumented json schema |
| 34 | + - uses a pre-1.0 term for package |
| 35 | +- `cargo metadata`: |
| 36 | + - works off of `Cargo.toml`, `Cargo.lock` files on disk |
| 37 | + - uses a custom json schema |
| 38 | + - can include dependency resolution but excludes feature resolution |
| 39 | + - some users want this faster |
| 40 | + - some users want this to report more information |
| 41 | + - See also [open issues](https://github.com/rust-lang/cargo/issues?q=is%3Aissue%20state%3Aopen%20label%3ACommand-metadata) |
| 42 | +- `cargo pkgid`: |
| 43 | + - works off of `Cargo.toml`, `Cargo.lock` files on disk |
| 44 | + - text output |
| 45 | +- `cargo verify-project`: |
| 46 | + - works off of a `Cargo.toml` file on disk |
| 47 | + - uses a custom json schema |
| 48 | + - uses a pre-1.0 term for package |
| 49 | + - deprecated |
| 50 | + |
| 51 | +There have been experiments for a plumbing for builds |
| 52 | +- [`--build-plan`](https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-plan) attempts to report what commands will be run so external build tools can manage them. |
| 53 | + - The actual commands to be run is dynamic, based on the output of build scripts from build graph dependencies |
| 54 | + - Difficulty in supporting build pipelining |
| 55 | +- [`--unit-graph`](https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#unit-graph) reports the graph the build operates off of which corresponds to calls to the compiler and build scripts |
| 56 | + - Also provides a way to get the results of feature resolution |
| 57 | + |
| 58 | +### The next 6 months |
| 59 | + |
| 60 | +Continue on the third-party subcommand to experiment with plumbing commands ([source](https://github.com/crate-ci/cargo-plumbing)). |
| 61 | + |
| 62 | +A build in Cargo can roughly be split into |
| 63 | +1. Locate project |
| 64 | +2. Read the manifests for a workspace |
| 65 | +3. Read lockfile |
| 66 | +4. Lock dependencies |
| 67 | +5. Write lockfile |
| 68 | +6. Resolve features |
| 69 | +7. Plan a build, including reading manifests for transitive dependencies |
| 70 | +8. Execute a build |
| 71 | +9. Stage final artifacts |
| 72 | + |
| 73 | +These could serve as starting points for experimenting with plumbing commands. |
| 74 | +Staging of final artifacts may not be worth having a dedicated command for. |
| 75 | +This is exclusively focused on build while other operations may be of interest to users. |
| 76 | +We can evaluate those commands in the future as they tend to still build off of these same core primitives. |
| 77 | + |
| 78 | +At minimum, later commands in the process would accept output from earlier commands, |
| 79 | +allowing the caller to either replace commands (e.g. custom dependency resolver) |
| 80 | +or customize the output (e.g. remove `dev-dependencies` from manifests). |
| 81 | + |
| 82 | +Encapsulating stabilized file formats can serve as a starting point for output |
| 83 | +schemas as we already output those and have to deal with stability guarantees |
| 84 | +around these. |
| 85 | + |
| 86 | +Between planning a build and executing a build is likely to look like |
| 87 | +`--unit-graph` and a plan will need to be put forward for how to work through |
| 88 | +the open issues. |
| 89 | +There will likely be similar issues for any other output that can't leverage existing formats. |
| 90 | + |
| 91 | +Cargo's APIs may not be able to expose each of these stages and work may need to be done to adapt it to support these divisions. |
| 92 | + |
| 93 | +The performance of piping output between these commands may be sub-par, coming from a combination of at least |
| 94 | +- Cargo's APIs may require doing more work than is needed for these stages |
| 95 | +- Cargo focuses on json for programmatic output which may prove sub-par (see also [zulip](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/.60cargo.20metadata.60.20performance/near/476523460)) |
| 96 | +- Cargo's serde structures may not be optimized |
| 97 | +- If customizing only a single step in this process, |
| 98 | + requiring serializing and deserializing through all of the other stages may be superfluous |
| 99 | + |
| 100 | +Low hanging or egregious bottlenecks may need to be addressed. |
| 101 | +Otherwise, performance should wait on user feedback. |
| 102 | + |
| 103 | +A schema evolution plan will need to be considered with the design of the schema. |
| 104 | +How Cargo deals with evolution of existing output could serve as potential starting points: |
| 105 | +- `Cargo.toml` (generated by `cargo package`) should still be readable by `cargo` versions within the specified `package.rust-version` |
| 106 | + - In the absence of a `package.rust-version`, `Cargo.toml` should only represent features the user explicitly used or optional features that were always allowed on stable `cargo` |
| 107 | +- `Cargo.lock` (generated by most commands) is strictly versioned: all versions of Cargo should output a lockfile that works in all other versions of Cargo for that given version and changing Cargo versions should not cause the output to change |
| 108 | + - Cargo bumps the default format version after it has been stabilized for a "sufficient period of time" |
| 109 | + - The default is capped by what is supported by the lowest `package.rust-version` in the workspace |
| 110 | +- `cargo metadata --format-version`: defaults to "latest" with a warning |
| 111 | + - We attempt to follow the same practice as `Cargo.toml` |
| 112 | +- `--message-format`: no versioning currently |
| 113 | + - We attempt to follow the same practice as `Cargo.toml` |
| 114 | + |
| 115 | +### The "shiny future" we are working towards |
| 116 | + |
| 117 | +- Collect user feedback on these commands and iterate on them for eventual inclusion into Cargo |
| 118 | +- Evaluate refactoring Cargo to better align with these plumbing commands to have better boundaries between subsystems |
| 119 | +- Evaluate splitting the `cargo` `[lib]` into crates for each of these plumbing commands as smaller, more approachable, more "blessed" Rust APIs for users to call into |
| 120 | + |
| 121 | +## Design axioms |
| 122 | + |
| 123 | +- The changes to Cargo should not impede the development of Cargo |
| 124 | +- The schemas and planned evolution should not impede the development of Cargo |
| 125 | +- The plumbing commands should be focused on solving expected or known needs, avoiding speculation. |
| 126 | + |
| 127 | +## Ownership and team asks |
| 128 | + |
| 129 | +**Owner:** *Identify a specific person or small group of people if possible, else the group that will provide the owner. Github user names are commonly used to remove ambiguity.* |
| 130 | + |
| 131 | +*This section defines the specific work items that are planned and who is expected to do them. It should also include what will be needed from Rust teams. The table below shows some common sets of asks and work, but feel free to adjust it as needed. Every row in the table should either correspond to something done by a contributor or something asked of a team. For items done by a contributor, list the contributor, or ![Heap wanted][] if you don't yet know who will do it. For things asked of teams, list ![Team][] and the name of the team. The things typically asked of teams are defined in the [Definitions](#definitions) section below.* |
| 132 | + |
| 133 | +| Task | Owner(s) or team(s) | Notes | |
| 134 | +|-----------------------------------------|--------------------------|-------| |
| 135 | +| Discussion and moral support | ![Team][] [cargo] | | |
| 136 | +| Implementation | ![Help wanted][] | | |
| 137 | +| Optimizing Cargo | ![Help wanted][], @epage | | |
| 138 | +| Inside Rust blog post inviting feedback | @epage | | |
| 139 | + |
| 140 | +### Definitions |
| 141 | + |
| 142 | +Definitions for terms used above: |
| 143 | + |
| 144 | +* *Discussion and moral support* is the lowest level offering, basically committing the team to nothing but good vibes and general support for this endeavor. |
| 145 | +* *Author RFC* and *Implementation* means actually writing the code, document, whatever. |
| 146 | +* *Design meeting* means holding a synchronous meeting to review a proposal and provide feedback (no decision expected). |
| 147 | +* *RFC decisions* means reviewing an RFC and deciding whether to accept. |
| 148 | +* *Org decisions* means reaching a decision on an organizational or policy matter. |
| 149 | +* *Secondary review* of an RFC means that the team is "tangentially" involved in the RFC and should be expected to briefly review. |
| 150 | +* *Stabilizations* means reviewing a stabilization and report and deciding whether to stabilize. |
| 151 | +* *Standard reviews* refers to reviews for PRs against the repository; these PRs are not expected to be unduly large or complicated. |
| 152 | +* *Prioritized nominations* refers to prioritized lang-team response to nominated issues, with the expectation that there will be *some* response from the next weekly triage meeting. |
| 153 | +* *Dedicated review* means identifying an individual (or group of individuals) who will review the changes, as they're expected to require significant context. |
| 154 | +* Other kinds of decisions: |
| 155 | + * [Lang team experiments](https://lang-team.rust-lang.org/how_to/experiment.html) are used to add nightly features that do not yet have an RFC. They are limited to trusted contributors and are used to resolve design details such that an RFC can be written. |
| 156 | + * Compiler [Major Change Proposal (MCP)](https://forge.rust-lang.org/compiler/mcp.html) is used to propose a 'larger than average' change and get feedback from the compiler team. |
| 157 | + * Library [API Change Proposal (ACP)](https://std-dev-guide.rust-lang.org/development/feature-lifecycle.html) describes a change to the standard library. |
| 158 | + |
| 159 | +## Frequently asked questions |
0 commit comments