AI development tools trial

[thibaudcolas]

👋 just noting this repository is configured to trial AI development tools if anyone is interested in this. You can use this issue to share any feedback, concerns, or things you think are worth trying. This is strictly for trial purposes. There is no implication that AI dev tools are suitable / appropriate / provide good results for this type of development.

Why this project

In short, because it constitutes a good small-scale trial. This project has established long-term contributors and occasional new contributors. A lot of tech debt but with clearly-identified opportunities for improvements. Opportunities for both front-end and back-end contributions, and docs, and broader improvements. And it’s not under super active development so the trial won’t get in the way of many people’s contributions.

What is set up

AI development agents

• GitHub Copilot agent. In GitHub, assign an issue to Copilot and it will create a draft PR. For example Upgrade css-loader library #270 was assigned and it created Upgrade css-loader to v6 and migrate to webpack asset modules #312.
• Cursor agent. For example Fix toggle image functionality issue #311.

AI code review

• Cursor BugBot. On all PRs. See for example Remove inline scripts and styles from domainindex.html template. Fix #305 #310
• GitHub Copilot code review. On all PRs. See for example Remove inline scripts and styles from domainindex.html template. Fix #305 #310

llms.txt

• llms.txt generation with sphinx-llms-txt: https://sphinx-wagtail-theme.readthedocs.io/en/latest/llms.txt
• llms-full.txt with the same library: https://sphinx-wagtail-theme.readthedocs.io/en/latest/llms-full.txt

[thibaudcolas]

Noting other uses of AI for development which are relevant to our work

• Generate GitHub issue descriptions (with Copilot)
• Generate release notes (select a tag, it diffs, mentions relevant contributors) (with Copilot)
• And another option for code review: CodeRabbit

[thibaudcolas]

Sharing a few notes about my testing of different code review tools, via this repo and others:

• I’m stopping the trial of Cursor BugBot. Has potential but not good enough currently, and it’s GitHub-only. Might try again if they moved it inside the IDE.
• I think I’m going to give up on Warp (general-purpose AI coding with custom prompts for reviews). It’s promising but I just don’t have time to write my own code review prompts
• Same for OpenAI Codex. Same reasons.
• Finally got to try GitHub Copilot reviews within VS Code, rather than via GitHub only. It’s pretty good, but currently only works when your changes are uncommitted which is a bit of a workflow faff (have to git reset --soft constantly)
• CodeRabbit is pretty good, good enough that I’ll probs spend time configuring it properly.
• Might try Greptile if I get the chance.

Other R&D:

• Started drafting an estimate of the environmental footprint of AI coding, including code review. It’s not great, just because AI coding involves a large number of tokens processed for input and output. AI code review might be better, I haven’t found a figure for the number of tokens used up by code review yet
• Because of this, I think I’ll spend some time looking at open source coding models, where you could control that they’re in a data centre with low-carbon electricity. Seems like Qwen3-Coder is the largest open source coding model and it’s available on Cerebras. There’s also Codestral which is now available in VS Code.
• Had a look at whether there are benchmarks specifically on code review performance of the models. The answer is "yes", but they seem pretty early days / early-stage research. Didn’t seem super relevant or now

[thibaudcolas]

This project is now set up with two uses of AI in CI for trial purposes:

• Bug reproduction pass / fail check
• Automated changelog generation

They’re set up but I’ve not tried them yet, hope to be an interesting trial.