Skip to content

Comments

Replace ctx.files.srcs with ctx.attr.srcs in run_binary#615

Open
hauserx wants to merge 2 commits intobazelbuild:mainfrom
hauserx:avoid-ctx-files
Open

Replace ctx.files.srcs with ctx.attr.srcs in run_binary#615
hauserx wants to merge 2 commits intobazelbuild:mainfrom
hauserx:avoid-ctx-files

Conversation

@hauserx
Copy link

@hauserx hauserx commented Feb 23, 2026

The ctx.files.srcs maybe memory heavy in case of many input sources as it flattens depset, see bazelbuild/bazel#28753

In below reproduction that creates 100k actions, each having 10k inputs this fix allows the used-heap-size-after-gc to drop from 8GB to 300M.

Repo

MODULE.bazel

bazel_dep(name = "bazel_skylib", version = "1.9.0")

myecho

echo "$1" > $2

BUILD.bazel

load("@bazel_skylib//rules:run_binary.bzl", "run_binary")

GENERATED_INPUTS=10000
ACTIONS = 100000

[
    run_binary(
        name="gen_file_" + str(idx),
        args = ["content_gen" + str(idx), "$(location :file_" + str(idx) + ")"],
        tool = ":myecho",
        outs = ["file_" + str(idx)],
    )
    for idx in range(GENERATED_INPUTS)
]

filegroup(
     name="genfiles",
     srcs = ["gen_file_" + str(idx) for idx in range(GENERATED_INPUTS)]
)

[
    run_binary(
        name = "action_" + str(idx),
        args = ["content_action" + str(idx), "$(location :out_" + str(idx) + ")"],
        tool = ":myecho",
        outs = ["out_" + str(idx)],
        srcs = [":genfiles"],
    )
    for idx in range(ACTIONS)
]

run_binary(
    name = "result",
    args = ["content_result", "$(location :final_out)"],
    tool = ":myecho",
    srcs = [":out_" + str(idx) for idx in range(ACTIONS)],
    outs = ["final_out"],
)

Before:

$ USE_BAZEL_VERSION=9.0.0 bazelisk --host_jvm_args=-Xmx64g build :result --nobuild
...
$ USE_BAZEL_VERSION=9.0.0 bazelisk --host_jvm_args=-Xmx64g info used-heap-size-after-gc
8461MB

After:

$ USE_BAZEL_VERSION=9.0.0 bazelisk --host_jvm_args=-Xmx64g info used-heap-size-after-gc
305MB

@hauserx hauserx changed the title Replace ctx.files.srcs with ctx.attrs in run_binary Replace ctx.files.srcs with ctx.attr.srcs in run_binary Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant