Skip to content

Commit d8966b8

Browse files
rickeylevrickeylev
authored
refactor(pystar): load (but don't use) Starlark implementation. (#1428)
Always loading the code provides several benefits: * It's easier to reason about what code paths are taken. * Iteratively working on them is simply changing an environment variable instead of editing several files. * Ensures the files are loadable on older versions of Bazel. Usage of the Starlark implemenation is controlled by an environment variable, `RULES_PYTHON_ENABLE_PYSTAR=1`. An environment variable must be used because the decision about which implementation to use must be made before regular build flags are able to run (loading phase logic is affected). The Starlark implementation is almost entirely compatible with pre-Bazel 7, except for the `py_internal` symbol. This symbol is special in a couple ways: * It only exists within the `@rules_python` repo * It does not exist prior to Bazel 7. This requires using a repo rule, `@rules_python_internal`, to do some feature/version detection to generate a shim bzl file so that the `py_internal` symbol is always loadable. Regular rules_python code then loads the shim and can act accordingly. Also fixes some other loading-time issues (beyond simply py_internal being None): * `configuration_field()` args are validated at time of call, so those must be guarded so Bazel 5.4 doesn't fail on them. * The `init` arg of `provider()` isn't supported under Bazel 5.4; change them to no-op stubs behind a guard. * The `|` operator for dicts isn't supported under Bazel 5.4; change to use skylib's `dicts.add` Work towards #1069
1 parent 0d0e183 commit d8966b8

33 files changed

+471
-47
lines changed

‎.bazelignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ bazel-rules_python
66
bazel-bin
77
bazel-out
88
bazel-testlogs
9+
# Prevent the convenience symlinks within the examples from being
10+
# treated as directories with valid BUILD files for the main repo.
911
examples/bzlmod/bazel-bzlmod
12+
examples/bzlmod/other_module/bazel-other_module
1013
examples/bzlmod_build_file_generation/bazel-bzlmod_build_file_generation
14+
examples/pip_parse/bazel-pip_parse
1115
examples/py_proto_library/bazel-py_proto_library

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,11 @@ A brief description of the categories of changes:
3232
* (multi-version) The `distribs` attribute is no longer propagated. This
3333
attribute has been long deprecated by Bazel and shouldn't be used.
3434

35+
* Calling `//python:repositories.bzl#py_repositories()` is required. It has
36+
always been documented as necessary, but it was possible to omit it in certain
37+
cases. An error about `@rules_python_internal` means the `py_repositories()`
38+
call is missing in `WORKSPACE`.
39+
3540

3641
### Added
3742

‎MODULE.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ internal_deps = use_extension("@rules_python//python/extensions/private:internal
1515
internal_deps.install()
1616
use_repo(
1717
internal_deps,
18+
"rules_python_internal",
1819
# START: maintained by 'bazel run //tools/private:update_pip_deps'
1920
"pypi__build",
2021
"pypi__click",

‎examples/build_file_generation/WORKSPACE‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,11 @@ local_repository(
7171
path = "../../gazelle",
7272
)
7373

74-
# Next we load the toolchain from rules_python.
75-
load("@rules_python//python:repositories.bzl", "python_register_toolchains")
74+
# Next we load the setup and toolchain from rules_python.
75+
load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")
76+
77+
# Perform general setup
78+
py_repositories()
7679

7780
# We now register a hermetic Python interpreter rather than relying on a system-installed interpreter.
7881
# This toolchain will allow bazel to download a specific python version, and use that version

‎gazelle/WORKSPACE‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,9 @@ local_repository(
3434
path = "..",
3535
)
3636

37-
load("@rules_python//python:repositories.bzl", "python_register_toolchains")
37+
load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")
38+
39+
py_repositories()
3840

3941
python_register_toolchains(
4042
name = "python39",

‎internal_setup.bzl‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,13 @@ load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
2020
load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
2121
load("//:version.bzl", "SUPPORTED_BAZEL_VERSIONS")
2222
load("//python/pip_install:repositories.bzl", "pip_install_dependencies")
23+
load("//python/private:internal_config_repo.bzl", "internal_config_repo") # buildifier: disable=bzl-visibility
2324

2425
def rules_python_internal_setup():
2526
"""Setup for rules_python tests and tools."""
2627

28+
internal_config_repo(name = "rules_python_internal")
29+
2730
# Because we don't use the pip_install rule, we have to call this to fetch its deps
2831
pip_install_dependencies()
2932

‎python/BUILD.bazel‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,11 @@ bzl_library(
8484
bzl_library(
8585
name = "py_binary_bzl",
8686
srcs = ["py_binary.bzl"],
87-
deps = ["//python/private:util_bzl"],
87+
deps = [
88+
"//python/private:util_bzl",
89+
"//python/private/common:py_binary_macro_bazel_bzl",
90+
"@rules_python_internal//:rules_python_config_bzl",
91+
],
8892
)
8993

9094
bzl_library(
@@ -101,19 +105,31 @@ bzl_library(
101105
bzl_library(
102106
name = "py_info_bzl",
103107
srcs = ["py_info.bzl"],
104-
deps = ["//python/private:reexports_bzl"],
108+
deps = [
109+
"//python/private:reexports_bzl",
110+
"//python/private/common:providers_bzl",
111+
"@rules_python_internal//:rules_python_config_bzl",
112+
],
105113
)
106114

107115
bzl_library(
108116
name = "py_library_bzl",
109117
srcs = ["py_library.bzl"],
110-
deps = ["//python/private:util_bzl"],
118+
deps = [
119+
"//python/private:util_bzl",
120+
"//python/private/common:py_library_macro_bazel_bzl",
121+
"@rules_python_internal//:rules_python_config_bzl",
122+
],
111123
)
112124

113125
bzl_library(
114126
name = "py_runtime_bzl",
115127
srcs = ["py_runtime.bzl"],
116-
deps = ["//python/private:util_bzl"],
128+
deps = [
129+
"//python/private:util_bzl",
130+
"//python/private/common:py_runtime_macro_bzl",
131+
"@rules_python_internal//:rules_python_config_bzl",
132+
],
117133
)
118134

119135
bzl_library(
@@ -125,13 +141,21 @@ bzl_library(
125141
bzl_library(
126142
name = "py_runtime_info_bzl",
127143
srcs = ["py_runtime_info.bzl"],
128-
deps = ["//python/private:reexports_bzl"],
144+
deps = [
145+
"//python/private:reexports_bzl",
146+
"//python/private/common:providers_bzl",
147+
"@rules_python_internal//:rules_python_config_bzl",
148+
],
129149
)
130150

131151
bzl_library(
132152
name = "py_test_bzl",
133153
srcs = ["py_test.bzl"],
134-
deps = ["//python/private:util_bzl"],
154+
deps = [
155+
"//python/private:util_bzl",
156+
"//python/private/common:py_test_macro_bazel_bzl",
157+
"@rules_python_internal//:rules_python_config_bzl",
158+
],
135159
)
136160

137161
# NOTE: Remember to add bzl_library targets to //tests:bzl_libraries

‎python/extensions/private/internal_deps.bzl‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,11 @@
99
"Python toolchain module extension for internal rule use"
1010

1111
load("//python/pip_install:repositories.bzl", "pip_install_dependencies")
12+
load("//python/private:internal_config_repo.bzl", "internal_config_repo")
1213

1314
# buildifier: disable=unused-variable
1415
def _internal_deps_impl(module_ctx):
16+
internal_config_repo(name = "rules_python_internal")
1517
pip_install_dependencies()
1618

1719
internal_deps = module_extension(

‎python/private/BUILD.bazel‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,11 @@ licenses(["notice"])
2222

2323
filegroup(
2424
name = "distribution",
25-
srcs = glob(["**"]) + ["//python/private/proto:distribution"],
25+
srcs = glob(["**"]) + [
26+
"//python/private/common:distribution",
27+
"//python/private/proto:distribution",
28+
"//tools/build_defs/python/private:distribution",
29+
],
2630
visibility = ["//python:__pkg__"],
2731
)
2832

‎python/private/common/BUILD.bazel‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,194 @@
1111
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
14+
15+
load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
16+
17+
package(
18+
default_visibility = ["//python:__subpackages__"],
19+
)
20+
21+
bzl_library(
22+
name = "attributes_bazel_bzl",
23+
srcs = ["attributes_bazel.bzl"],
24+
)
25+
26+
bzl_library(
27+
name = "attributes_bzl",
28+
srcs = ["attributes.bzl"],
29+
deps = [
30+
":common_bzl",
31+
":providers_bzl",
32+
":py_internal_bzl",
33+
":semantics_bzl",
34+
],
35+
)
36+
37+
bzl_library(
38+
name = "cc_helper_bzl",
39+
srcs = ["cc_helper.bzl"],
40+
deps = [":py_internal_bzl"],
41+
)
42+
43+
bzl_library(
44+
name = "common_bazel_bzl",
45+
srcs = ["common_bazel.bzl"],
46+
deps = [
47+
":common_bzl",
48+
":providers_bzl",
49+
":py_internal_bzl",
50+
"@bazel_skylib//lib:paths",
51+
],
52+
)
53+
54+
bzl_library(
55+
name = "common_bzl",
56+
srcs = ["common.bzl"],
57+
deps = [
58+
":cc_helper_bzl",
59+
":providers_bzl",
60+
":py_internal_bzl",
61+
":semantics_bzl",
62+
],
63+
)
64+
65+
filegroup(
66+
name = "distribution",
67+
srcs = glob(["**"]),
68+
)
69+
70+
bzl_library(
71+
name = "providers_bzl",
72+
srcs = ["providers.bzl"],
73+
deps = [
74+
":semantics_bzl",
75+
"@rules_python_internal//:rules_python_config_bzl",
76+
],
77+
)
78+
79+
bzl_library(
80+
name = "py_binary_macro_bazel_bzl",
81+
srcs = ["py_binary_macro_bazel.bzl"],
82+
deps = [
83+
":common_bzl",
84+
":py_binary_rule_bazel_bzl",
85+
],
86+
)
87+
88+
bzl_library(
89+
name = "py_binary_rule_bazel_bzl",
90+
srcs = ["py_binary_rule_bazel.bzl"],
91+
deps = [
92+
":attributes_bzl",
93+
":py_executable_bazel_bzl",
94+
":semantics_bzl",
95+
"@bazel_skylib//lib:dicts",
96+
],
97+
)
98+
99+
bzl_library(
100+
name = "py_executable_bazel_bzl",
101+
srcs = ["py_executable_bazel.bzl"],
102+
deps = [
103+
":attributes_bazel_bzl",
104+
":common_bazel_bzl",
105+
":common_bzl",
106+
":providers_bzl",
107+
":py_executable_bzl",
108+
":py_internal_bzl",
109+
":semantics_bzl",
110+
],
111+
)
112+
113+
bzl_library(
114+
name = "py_executable_bzl",
115+
srcs = ["py_executable.bzl"],
116+
deps = [
117+
":attributes_bzl",
118+
":cc_helper_bzl",
119+
":common_bzl",
120+
":providers_bzl",
121+
":py_internal_bzl",
122+
"@bazel_skylib//lib:dicts",
123+
],
124+
)
125+
126+
bzl_library(
127+
name = "py_internal_bzl",
128+
srcs = ["py_internal.bzl"],
129+
deps = ["@rules_python_internal//:py_internal_bzl"],
130+
)
131+
132+
bzl_library(
133+
name = "py_library_bzl",
134+
srcs = ["py_library.bzl"],
135+
deps = [
136+
":attributes_bzl",
137+
":common_bzl",
138+
":providers_bzl",
139+
":py_internal_bzl",
140+
"@bazel_skylib//lib:dicts",
141+
],
142+
)
143+
144+
bzl_library(
145+
name = "py_library_macro_bazel_bzl",
146+
srcs = ["py_library_macro_bazel.bzl"],
147+
deps = [":py_library_rule_bazel_bzl"],
148+
)
149+
150+
bzl_library(
151+
name = "py_library_rule_bazel_bzl",
152+
srcs = ["py_library_rule_bazel.bzl"],
153+
deps = [
154+
":attributes_bazel_bzl",
155+
":common_bazel_bzl",
156+
":common_bzl",
157+
":py_library_bzl",
158+
],
159+
)
160+
161+
bzl_library(
162+
name = "py_runtime_macro_bzl",
163+
srcs = ["py_runtime_macro.bzl"],
164+
deps = [":py_runtime_rule_bzl"],
165+
)
166+
167+
bzl_library(
168+
name = "py_runtime_rule_bzl",
169+
srcs = ["py_runtime_rule.bzl"],
170+
deps = [
171+
":attributes_bzl",
172+
":common_bzl",
173+
":providers_bzl",
174+
":py_internal_bzl",
175+
"@bazel_skylib//lib:dicts",
176+
"@bazel_skylib//lib:paths",
177+
],
178+
)
179+
180+
bzl_library(
181+
name = "py_test_macro_bazel_bzl",
182+
srcs = ["py_test_macro_bazel.bzl"],
183+
deps = [
184+
":common_bazel_bzl",
185+
":py_test_rule_bazel_bzl",
186+
],
187+
)
188+
189+
bzl_library(
190+
name = "py_test_rule_bazel_bzl",
191+
srcs = ["py_test_rule_bazel.bzl"],
192+
deps = [
193+
":attributes_bzl",
194+
":common_bzl",
195+
":py_executable_bazel_bzl",
196+
":semantics_bzl",
197+
"@bazel_skylib//lib:dicts",
198+
],
199+
)
200+
201+
bzl_library(
202+
name = "semantics_bzl",
203+
srcs = ["semantics.bzl"],
204+
)

0 commit comments

Comments
 (0)