Let jobs retweak easyconfigs themselves by passing down --try-* options

[bartoldeman]

This can be accomplished by tweak() optionally also returning a dict which maps the tweaked easyconfig to the original version. Then the job can run eb ... <original_easyconfig.eb> --try-* and that original easyconfig will be retweaked in the job itself.

If the easyconfig passed to the job is not tweaked, then --try-* is not passed down (so, with --robot, some jobs will have --try-* and some don't).

This removes the requirement of a shared tmpdir with --job --try-*.

Fixes #1355

[bartoldeman]

Putting to draft because this needs a proper test combining --job --robot --try-*.

[boegel]

@bartoldeman Seems like test was implemented, so this shouldn't be a draft PR anymore?

[bartoldeman]

Justed wanted to test locally using eb HPL-2.3-foss-2023a.eb --try-toolchain=foss,2024a --job. This works fine.

Note that a shared temporary directory is still needed with --job --from-pr, but you can use a shared temporary directory in the submitting easybuild and NOT on the worker node by setting $TMPDIR instead of using --tmpdir or (equivalently) $EASYBUILD_TMPDIR, as$TMPDIR is not passed down to the job; a shared TMPDIR on a worker node can cause quite a performance degredation since even GCC temporary .s (asm) files will be stored on that, causing a lot of expensive networked IOPS.

[bartoldeman]

@bartoldeman Seems like test was implemented, so this shouldn't be a draft PR anymore?

@boegel undrafted now

[Micket]

Sorry for my delay on this, was really busy with a deploy of a new cluster, then i got really sick for the past week. Feel free to steal this from me, i only intended to test it out on the cluster to see that it worked as intended.

[boegel]

reviewed & tested, lgtm