Have OpenAI's Codex suggest commit messages by Stephan202 · Pull Request #1984 · PicnicSupermarket/error-prone-support

Stephan202 · 2025-11-15T21:54:52Z

Suggested commit message:

Have OpenAI's Codex suggest commit messages (#1984)

This new GitHub Actions workflow upserts a pull request comment any time
the PR is updated, to suggest an appropriate commit message for the 
squash-and-merge workflow.

The workflow can also be triggered manually to simplify testing.

I tested part of this PR by temporarily changing the default branch (necessary because the new workflow doesn't exist on master yet) and running e.g.:

gh workflow run suggest-commit-message.yml --ref 'sschroevers/automate-commit-message-suggestions' -f pr_number=1963

Examples of generated commit messages:

Upgrade Google Java Format
Upgrade JUnit (this one isn't always generated to my liking.)
Upgrade Checkstyle
Upgrade NullAway
Upgrade Error Prone
Upgrade jOOQ
Upgrade Spring (I think we can still improve here.)
Upgrade maven-jar-plugin
This PR

This PR was largely created using Windsurf with the GPT-5-Codex model, with tweaks from my side.

github-actions · 2025-11-15T21:55:44Z

Suggested commit message:

Add workflow to suggest PR commit messages (#1984)

Introduce `.github/workflows/suggest-commit-message.yml` to resolve PR
metadata, run the Codex action, and post the formatted suggestion as a
pull request comment.

Automate commit message suggestions to reduce manual editing and keep
messaging consistent across pull requests.

Edit by Stephan: This comment was posted before I updated the marker string. Hence the second bot-generated comment further down. As you can see, that one has been edited by the bot quite a number of times.

github-actions · 2025-11-15T21:57:25Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T11:08:44Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T11:12:16Z

Suggested commit message:

Have OpenAI's Codex suggest commit messages (#1984)

Introduce a workflow that upserts a Codex-suggested commit message
comment when a pull request changes. It also allows manual runs to ease
testing for new changes.

github-actions · 2025-11-16T11:13:52Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T11:23:25Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T11:36:43Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T11:59:12Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:08:56Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:20:54Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:24:49Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:31:34Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:36:06Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T12:46:11Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T20:28:04Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T20:55:30Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T21:17:17Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T21:19:08Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-16T21:23:01Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

mohamedsamehsalah

🤖 🤖 🤖 🤖 🤖

Left some suggestions 🙌

(Am I a prompt engineer now? 👀)

mohamedsamehsalah · 2025-11-17T08:08:56Z

.github/workflows/suggest-commit-message.yml

+            6. For dependency upgrades in particular, *very precisely* follow the pattern of past commit messages: reuse the summary wording (only adjust version numbers) and list updated changelog, release note, and diff URLs in the body.
+            7. Don't hallucinate URLs, version numbers, or other factual information.
+            8. Never split URLs across multiple lines, even if they exceed 72 characters.
+            9. If the pull request description already contains a suitable commit message, prefer using that as-is.


Suggested change

9. If the pull request description already contains a suitable commit message, prefer using that as-is.

9. If the pull request description already contains a suitable commit message, consider using that as-is or as an inspiration for your generated message.

mohamedsamehsalah · 2025-11-17T08:15:25Z

.github/workflows/suggest-commit-message.yml

+            7. Don't hallucinate URLs, version numbers, or other factual information.
+            8. Never split URLs across multiple lines, even if they exceed 72 characters.
+            9. If the pull request description already contains a suitable commit message, prefer using that as-is.
+


Suggested change

11. Finally, consider adding a small description of work that is not relevant to the main PR changes, on a separate line as part of the commit message body, starting with "While here, " phrase. (i.e, "While here, fixed a typo in variable name.")

I fear that this will yield very low signal to noise; already I found the non-upgrade commit summaries "okay, not great". I'll think of a different way of phrasing this, but may leave it out; TBD.

mohamedsamehsalah · 2025-11-17T08:16:27Z

.github/workflows/suggest-commit-message.yml

+            Requirements:
+            1. Write the summary line in the imperative mood. Try not to exceed 80 characters.
+            2. End the summary line with the PR number in parentheses, i.e., " (#${prNumber})".
+            2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.


Suggested change

2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.

2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details. Unlike summary lines, message bodies are complete sentences that should end with a full stop.

mohamedsamehsalah · 2025-11-17T08:17:04Z

.github/workflows/suggest-commit-message.yml

+            2. End the summary line with the PR number in parentheses, i.e., " (#${prNumber})".
+            2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.


😄

Suggested change

2. End the summary line with the PR number in parentheses, i.e., " (#${prNumber})".

2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.

2. End the summary line with the PR number in parentheses, i.e., " (#${prNumber})".

3. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.

Stephan202

Tnx for the review @mohamedsamehsalah! I'll circle back to this later, when I combine your suggestions and do some more tests.

Stephan202 · 2025-11-17T08:31:29Z

.github/workflows/suggest-commit-message.yml

+            7. Don't hallucinate URLs, version numbers, or other factual information.
+            8. Never split URLs across multiple lines, even if they exceed 72 characters.
+            9. If the pull request description already contains a suitable commit message, prefer using that as-is.
+


I fear that this will yield very low signal to noise; already I found the non-upgrade commit summaries "okay, not great". I'll think of a different way of phrasing this, but may leave it out; TBD.

rickie

Already saw some nice feedback from @mohamedsamehsalah . This looks really nice. Amazing way to use AI to automate some of the things we have to do!

rickie · 2025-11-17T08:24:28Z

.github/workflows/suggest-commit-message.yml

+            Some further guidelines to help you craft good upgrade commit messages:
+            - Unless highly salient, don't summarize code changes made as part of the upgrade.
+            - Don't bother linking to anchors within changelogs or release notes; just link to the main page.
+            - For GitHub-hosted projects, always link to all relevant GitHub release pages, including those for intermediate versions.


Suggested change

- For GitHub-hosted projects, always link to all relevant GitHub release pages, including those for intermediate versions.

- For GitHub-hosted projects, always link to all relevant GitHub release pages, including those for intermediate versions, like milestones.

I know we mean that with intermediate versions, but noticed that the milestones from Spring were not added 🤔. So maybe we nudge it slightly more?

See the sentence below... 🙃. This doesn't help, apparently.

Lol, my bad 😂.

larslockefeer · 2025-11-17T23:47:35Z

.github/workflows/suggest-commit-message.yml

+        id: prompt
+        uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
+        env:
+          PR_NUMBER: ${{ steps.pr-details.outputs.number }}


Suggested change

PR_NUMBER: ${{ steps.pr-details.outputs.number }}

REPOSITORY: ${{ github.repository }}

PR_NUMBER: ${{ steps.pr-details.outputs.number }}

larslockefeer · 2025-11-17T23:48:28Z

.github/workflows/suggest-commit-message.yml

+              return `${truncated}\n... (${limit} of ${lines.length} lines shown)`;
+            };
+
+            const env = process.env;


Suggested change

const env = process.env;

const env = process.env;

const repository = env.REPOSITORY;

larslockefeer · 2025-11-17T23:49:02Z

.github/workflows/suggest-commit-message.yml

+            const cleanedBody = (body || '').trim() || '<no pull request description>';
+
+            const instructions = `
+            You are an experienced maintainer helping to craft the squash commit message for PR #${prNumber} in the PicnicSupermarket/error-prone-support repository.


Suggested change

You are an experienced maintainer helping to craft the squash commit message for PR #${prNumber} in the PicnicSupermarket/error-prone-support repository.

You are an experienced maintainer helping to craft the squash commit message for PR #${prNumber} in the ${repository} repository.

This new GitHub Actions workflow upserts a pull request comment any time the PR is updated, to suggest an appropriate commit message for the squash-and-merge workflow. The workflow can also be triggered manually to simplify testing.

github-actions · 2025-11-22T12:54:29Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-22T13:56:37Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-22T14:30:02Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-22T14:49:17Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2025-11-22T14:57:09Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

sonarqubecloud · 2025-11-22T14:57:34Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Stephan202 · 2025-11-22T15:11:02Z

I tweaked a few things. Some aspects are still hit-or-miss. But will merge now to officially wrap this up.

Stephan202 added the chore A task not related to code (build, formatting, process, ...) label Nov 15, 2025

Stephan202 temporarily deployed to codex November 15, 2025 21:54 — with GitHub Actions Inactive

Stephan202 marked this pull request as draft November 15, 2025 21:55

Stephan202 had a problem deploying to codex November 16, 2025 11:06 — with GitHub Actions Failure

Stephan202 temporarily deployed to codex November 16, 2025 11:11 — with GitHub Actions Inactive

Stephan202 temporarily deployed to codex November 16, 2025 11:21 — with GitHub Actions Inactive

Stephan202 temporarily deployed to codex November 16, 2025 11:34 — with GitHub Actions Inactive

Stephan202 temporarily deployed to codex November 16, 2025 11:56 — with GitHub Actions Inactive

Stephan202 had a problem deploying to codex November 16, 2025 12:06 — with GitHub Actions Failure

Stephan202 had a problem deploying to codex November 16, 2025 12:18 — with GitHub Actions Failure

Stephan202 had a problem deploying to codex November 16, 2025 12:22 — with GitHub Actions Failure

Stephan202 temporarily deployed to codex November 16, 2025 12:29 — with GitHub Actions Inactive

Stephan202 temporarily deployed to codex November 16, 2025 12:33 — with GitHub Actions Inactive

Stephan202 had a problem deploying to codex November 16, 2025 12:43 — with GitHub Actions Failure

Stephan202 force-pushed the sschroevers/automate-commit-message-suggestions branch from 9183931 to 68d7834 Compare November 16, 2025 13:02

Stephan202 temporarily deployed to codex November 16, 2025 13:02 — with GitHub Actions Inactive

mohamedsamehsalah approved these changes Nov 17, 2025

View reviewed changes

Stephan202 commented Nov 17, 2025

View reviewed changes

rickie approved these changes Nov 17, 2025

View reviewed changes

larslockefeer reviewed Nov 17, 2025

View reviewed changes

Stephan202 added 10 commits November 22, 2025 13:42

Have OpenAI's Codex suggest commit messages

162081c

This new GitHub Actions workflow upserts a pull request comment any time the PR is updated, to suggest an appropriate commit message for the squash-and-merge workflow. The workflow can also be triggered manually to simplify testing.

More tweaks

632f5d0

Don't overwrite edits

0e20d3c

Debug it

d9b7e93

Another attempt

b24a944

Tweaks

92c41fe

Try more nudging

a9deb07

Try another way

0a4f6ff

Some other way

dd7924c

Apply tweaks

d2408d1

Try to get those wiki links

b8157ca

Some more nudging

2fdac31

Try to reduce the input size

f3a9d59

More nudging may be better

d9613c5

	9. If the pull request description already contains a suitable commit message, prefer using that as-is.
	9. If the pull request description already contains a suitable commit message, consider using that as-is or as an inspiration for your generated message.


	11. Finally, consider adding a small description of work that is not relevant to the main PR changes, on a separate line as part of the commit message body, starting with "While here, " phrase. (i.e, "While here, fixed a typo in variable name.")

	2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.
	2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details. Unlike summary lines, message bodies are complete sentences that should end with a full stop.

		2. End the summary line with the PR number in parentheses, i.e., " (#${prNumber})".
		2. Wrap each body paragraph at 72 characters. Focus on the "what" and "why" rather than implementation details.

	- For GitHub-hosted projects, always link to all relevant GitHub release pages, including those for intermediate versions.
	- For GitHub-hosted projects, always link to all relevant GitHub release pages, including those for intermediate versions, like milestones.

	PR_NUMBER: ${{ steps.pr-details.outputs.number }}
	REPOSITORY: ${{ github.repository }}
	PR_NUMBER: ${{ steps.pr-details.outputs.number }}

	const env = process.env;
	const env = process.env;
	const repository = env.REPOSITORY;

	You are an experienced maintainer helping to craft the squash commit message for PR #${prNumber} in the PicnicSupermarket/error-prone-support repository.
	You are an experienced maintainer helping to craft the squash commit message for PR #${prNumber} in the ${repository} repository.

Conversation

Stephan202 commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 15, 2025 • edited by Stephan202 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 15, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025 • edited by Stephan202 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

github-actions bot commented Nov 16, 2025

Uh oh!

mohamedsamehsalah left a comment

Choose a reason for hiding this comment

Uh oh!

mohamedsamehsalah Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohamedsamehsalah Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stephan202 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Stephan202 commented Nov 15, 2025 •

edited

Loading

github-actions bot commented Nov 15, 2025 •

edited by Stephan202

Loading

github-actions bot commented Nov 16, 2025 •

edited by Stephan202

Loading

mohamedsamehsalah Nov 17, 2025 •

edited

Loading

mohamedsamehsalah Nov 17, 2025 •

edited

Loading