Fix release automation to work with md instead of asciidoc

[carsonip]

From 9.0, docs are markdown files, not asciidoc files. See #16082. Release automation around changelog needs to be updated accordingly.

Note that 8.x still relies on asciidoc.

[endorama]

Testing the updated automation without triggering a release is problematic. Going forward we will have 2 automations:

• one in main for 9.x releases
• one in 8.x for 8.x releases (which will be the release process before ci: update changelog release automation #16358 is merged)

We are pausing work on this until the next feature freeze on the 8.x track (next will be 8.17.5 release, expected soon).

[endorama]

I've been working on this for the 8.17.5 release.

Unfortunately the 8.17.5 automation was already run ~2 weeks ago #16416, so I've only been able to partially test this, as the CI run ended with nothing to do, see https://github.com/elastic/apm-server/actions/runs/14353602051

But I was able to review the workflow for 8.x patch releases:

• running the workflow from the 8.x branch works
• we need to update the docs to mention the 8.x release process, PR is up docs: update release docs for v9 #16585
• we need to update the workflow for the 9.x releases. The PR is up, ci: update changelog release automation #16358 and ready for review. Testing these changes is not trivial, I tried on a fork but there are so many details that didn't work out. I'll engage with our eng prod team to see how this can be achieved.

[endorama]

The current situation is more convoluted than expected.

TL;DR: the current automation is not working. This will impact the next minor releases.

Next minors

Next minors in main (9.x) line cannot benefit of the current automation, not only due to the changelog automation not been there but also because the automation is currently locked to release minors from 8.x branch. This will be address soon by @v1v, reverting the changes introduced to work on the new 9.x release in September 2024.

Next patches

These versions should not be affected and be working as expected (didn't test it). Note that changelog should be handled manually.

Automation

Once the minor automation is back in place, we will have main (9.x) and 8.x.
8.x will become the last 8 minor (like 7.17 has been for 7.x).

Our changelog automation will need to work on 2 different scenarios:

1. main/9.x: release notes are on main for all versions and don't need to be backported or be present in any release branch
2. 8.x: the automation will target 8.x branch (or latest 8 minor) and there is no need to backport to other 8.x release branches

This is addressed in #16358
9.x has been tested by Victor, functioning 8.x is still to assess.

[endorama]

9.x release automation is now working as expected.

I'll proceed testing that 8.x still works as expected.

[endorama]

I confirmed 8.x automation works: the automation did not work already for changelogs on patch releases, requiring manual action, so we are covered there. As minors in 8.x are expected to be few, if necessary changelog can be manually added/edited, fixing the automation for so few releases is not a valuable trade off at this moment.

[endorama]

The PR with 9.x automation is up and waiting for team review: #16358

[endorama]

I created the PR to update the documentation and close this issue, it's now ready for review.