perf(package): Always reuse the workspace's target-dir

[ranger-ross]

What does this PR try to resolve?

While preparing to stabilize build-dir, it was discovered that cargo package will sometimes use the build cache for dependency crates if CARGO_TARGET_DIR is explicitly set.

If target-dir is not set, the build will be preformed in an "inner target dir" at target/package/{name}-{version}/target and rebuild everything.

This PR changes the default behavior to always use the build cache,
matching the behavior of having target-dir set.

How to test and review this PR?

Added a dedicated test to verify the previous behavior (cargo recompiled non-workspace crates) in the first commit.
Then updated it to verify the new behavior.

Related links/notes

• I did some Git archaeology and I think this behavior was introduced in Move target_dir to Workspace and fix #2848 #2912
  • While searching through the commit/pr history was never able to find any explicit reason why the behavior is the way it is.
  • Even if it was intentional, it has been 9 years since and Cargo's fingerprinting/rebuild detection is much more mature now so I believe its worth revisiting if this is still the desired behavior.
• Would help mitigate cargo package (and hence cargo publish) verify step is slow from doing a full build #14941

[rustbot]

r? @ehuss

rustbot has assigned @ehuss.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[ranger-ross]

Linking to the Cargo meeting discussion about this PR.

It appears no consensus was made as many members were out that week.

[ehuss]

I think I'm fine with making this change. I don't recall any reasoning for keeping it separate, but I suspect if it was intentional that the intent was to avoid bad fingerprinting from properly validating the package. However, I'm comfortable now with proper fingerprinting from catching anything that needs to be rebuilt.

@epage or @weihanglo Do you have any other context on this? Otherwise I think we should merge.

[epage]

Build scripts and proc macros can mess with things.

However, if the packaged packages (which is what we care about verifying) are getting a unique package id (which it should) it should then have a unique fingerprint, avoiding those problems.

Unless people are digging around outside of whats fingerprinted. I assume the current-dir is the package root and I'm assuming that is in the packaging directory, so that interferes with mucking around. I don't think we officially expose target-dir but people try to approximate it through what cargo metadata returns (dir for config lookup will be different). Unsure what problems people will have, if any, and if its worth holding this up.

[epage]

We discussed this in today's meeting. There were no objections and we can always revert this later.