pants package failing due to no roots even where they exist

[kdowney-talos]

When I run pants package :: at the top level of our monorepo, I get

NoSourceRootError: No source root found for `serenity.analytics`. See https://www.pantsbuild.org/2.23/docs/using-pants/key-concepts/source-roots for how to define source roots.

But if I run pants roots:

serenity.analytics/src/python
serenity.analytics/tests

as expected based on this setting:

[source]
root_patterns = [
  "src/protos",
  "src/python",
  "tests",
  "/serenity.risk"
]

Here is serenity.analytics/BUILD:

python_requirements(name="deps", source="pyproject.toml")

resource(name="pyproject", source="pyproject.toml")
resource(name="version", source="VERSION.txt")

python_distribution(
    name="dist",
    dependencies=["//serenity.analytics/src/python:lib", ":pyproject", ":version"],
    provides=python_artifact(name="serenity-analytics"),
    generate_setup=False,
)

and the corresponding src/python/BUILD is just this one line:

python_sources(name="lib", sources=["**/*.py"]])

What am I missing?

This is with Pants 2.23, Python 3.12.

[benjyw]

Ah, yes, the issue is that there is no source root for those resources directly in serenity.analytics/ that are not under one of those lower source roots. You can fix this by adding serenity.analytics/ itself as a source root.

[kdowney-talos]

In case it helps, it appears the issue is broader. I am hitting it with export-codegen too.

(.venv) serenitydev@adb319d55c1a:/workspaces/pms$ pants export-codegen ::
16:11:29.08 [INFO] Writing generated files to dist/codegen
16:11:29.09 [ERROR] 1 Exception encountered:

Engine traceback:
  in `export-codegen` goal

IntrinsicError: Error copying bytes from /home/serenitydev/.cache/pants/lmdb_store/immutable/files/2d/2d522219da490246fd3842165d3ebb7362314e218a510c16daeb09bc8d5fc5a6 to /workspaces/pms/dist/codegen/serenity.risk/experiments/dar-wilshire/DAR_prices_data.csv: Permission denied (os error 13)

There's only one module that has Protobuf files to code generate, so I don't know why it's even looking at this CSV file, but when it does, it trips that same filesystem race condition as the SQL files.

If I restrict to the single module it's fine, so I can work around this one too, more just an additional data point ....

[benjyw]

Ah, well, that second case, is about writing into the workspace. I recently fixed a known race condition related to writing into the workspace under certain circumstances where such writes cannot be properly serialized. But I don't see how it would trigger in the export-codegen case, where there is just one write at the very end of the goal rule (and goal rules are serialized).

Still, worth trying with that fix, just in case. When 2.25.0.dev1 is released (hopefully in the next few hours) can you give that a try?

(The first error, though, is a race on writing into lmdb_store itself, and I doubt that fix will have an effect.)

[benjyw]

(watch #announce on Slack or the pants-devel mailing list to find out about the release.)

[kdowney-talos]

Thanks @benjyw! I will gladly try it out once released. Pants has been a great upgrade for our monorepo's build already. :-)

[kdowney-talos]

Hi @benjyw you're right, export-codegen appears to be a different race condition or issue, because it's still there post-upgrade:

(.venv) serenitydev@c774de47c447:/workspaces/pms$ pants export-codegen ::
00:27:09.91 [INFO] Writing generated files to dist/codegen
00:27:09.98 [ERROR] 1 Exception encountered:

Engine traceback:
  in `export-codegen` goal

IntrinsicError: Error setting permissions on /home/serenitydev/.cache/pants/lmdb_store/immutable/files/2d/2d522219da490246fd3842165d3ebb7362314e218a510c16daeb09bc8d5fc5a6: Permission denied (os error 13)

(.venv) serenitydev@c774de47c447:/workspaces/pms$ pants --version
2.25.0.dev1

However, good news: the other issue flagged above with pants package appears to be fixed with this patch!

[benjyw]

(resources are technically source files, in that they are loaded by a relpath relative to a package)

[benjyw]

Can you file an issue with this separate bug (and link to it here)? The content here is drifting ever further from the title of the discussion...

[kdowney-talos]

Done; I also put in some additional diagnostic information in case it helps. Thanks, @benjyw.

#21771