[CI] PR Authors have no idea their PR is tested by rebasing onto main

[DavidSpickett]

In #154814, the author got a test failure from CI that only reproduced when they rebased their changes onto the main of the current time. I see the reason to do it this way, it will catch surprise failures exactly like the one in this case.

Except, I might spend a few hours trying to compare more obvious things like host OS, cmake configurations, etc. before attempting to rebase. Especially as we advise against rebasing during review and who knows what changes I'd end up bringing in.

Can we leave clues somewhere in the build or testing reports to make authors aware of the situation?

Perhaps we could end the log with Building PR #whatever when rebased onto main (at <hash>) failed.") ? Then it will be the first thing the author sees after clicking the results link from GitHub.

@boomanaiden154 What do you think?

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: David Spickett (DavidSpickett)

In https://github.com//pull/154814, the author got a test failure from CI that only reproduced when they rebased their changes onto the `main` of the current time. I see the reason to do it this way, it will catch surprise failures exactly like the one in this case.

Except, I might spend a few hours trying to compare more obvious things like host OS, cmake configurations, etc. before attempting to rebase. Especially as we advise against rebasing during review and who knows what changes I'd end up bringing in.

Can we leave clues somewhere in the build or testing reports to make authors aware of the situation?

Perhaps we could end the log with Building PR #whatever when rebased onto main (at <hash>) failed.") ? Then it will be the first thing the author sees after clicking the results link from GitHub.

@boomanaiden154 What do you think?

[boomanaiden154]

We've been meaning to add a system to write out reproduction instructions on failures but got hung up on spurious local LLDB test failures in certain distro default container configurations. We can look at moving that up the priority list, because that would address this issue.

Especially as we advise against rebasing during review and who knows what changes I'd end up bringing in.

We advise against rebasing, but merging main into your branch has the same effect and doesn't mess up review comments. It's what the update branch button on a PR does by default.

[DavidSpickett]

Yes that sounds good.

We advise against rebasing, but merging main into your branch has the same effect and doesn't mess up review comments. It's what the update branch button on a PR does by default.

Ok partly my lacking git skills then. Still, it would be the last thing I'd check, so instructions would be a great help.