diff --git a/docs/dev/github-workflows.rst b/docs/dev/github-workflows.rst new file mode 100644 index 0000000000..fa1c36b66c --- /dev/null +++ b/docs/dev/github-workflows.rst @@ -0,0 +1,204 @@ +GitHub workflows +================ + +We run our continuous integration (CI) validation and our release automation +via GitHub workflows. This allows us to merge PRs with confidence that they +won't catastrophically break DFHack functionality. GitHub workflows also allow +us to quickly produce stable release builds with fewer manual steps. Reducing +manual steps for releases is important since it is easy for a person to forget +a small but impactful step and therefore produce a bad release that causes +trouble for our users. + +Background +---------- + +`GitHub workflows `_ run +on provisioned VMs in the cloud with stable environments that we specify. They +are free to use since DFHack is an open source project. They have proven to be +reliably available within a few seconds when our workflows are triggered. The +logic for the workflows is written in yaml, and the files that control our +workflows are stored in the :file:`.github/workflows/` directory in each of our +repos. Example: :source:`.github/workflows`. + +Each workflow contains metadata that specifies: +- when it `triggers `_ +- what `base environment `_ it uses (OS, pre-installed dependencies, etc.) +- what additional dependencies should be installed (if any) +- custom business logic + +Workflows run in the context of a single repo, but workflows defined in one +repo can inherit logic from workflows in other repos. All our common CI logic +is in the main DFHack/dfhack repo, but our submodules, like our ``scripts`` and +``df-structures`` repos, have CI workflows defined that inherit from the logic +in DFHack/dfhack. That way we can fix bugs and extend functionality in one +place and have it benefit the entire org tree. + +Caches +~~~~~~ + +GitHub also provides 10GB per repository for `caches `_. +We utilize the cache system to keep state between workflow runs, cache +downloads, and keep compiler output to speed up subsequent builds. Efficient +use of the cache system is a critical part of our workflow design. It allows us +to iterate on test failures in PRs in one minute instead of 20. It allows us to +put out an entire emergency release build in 5 minutes instead of 45. We have +tuned our build and test workflows to minimize spurious cache misses and keep +the fast path fast. + +Caches are namespaced by key prefixes, and we have one key prefix per build +context. For example, release builds on gcc-10 are kept in one cache namespace, +whereas test builds on gcc-10 are kept separate. MSVC release and test builds +similarly have their own namespaces. Each cache has a maximum size that is +enforced by the business logic that writes the cache data. + +In order to maintain consistency in a distributed environment, caches are +versioned. A workflow will read the latest version of the cache with its key +prefix, maybe modify the cache with new data, then write back a new version. +Caches that are not used for 2 weeks are purged from GitHub storage, but if a +repo goes over the 10GB limit, caches are deleted in LRU order until the repo +is under the storage limit again. + +CI workflows +------------ + +Build +~~~~~ + +The Build workflow is the main CI workflow. It runs on every PR and push to a +branch. The ``build.yml`` file is essentially an orchestration layer for the +logic in several other .yml files: + +- ``test.yml`` builds DFHack with the test suite enabled (but stonesense and + windows pdb files disabled) and runs the test suite. It is optimized for + speed and is intended to give PR authors quick feedback on their changes. + The test suite is executed in a real running DF game on both Linux and + Windows. The ``test`` job populates the ``test`` cache, which is used by many + other workflows for non-distributed builds. +- ``package.yml`` builds DFHack as it would be released: test suite disabled + but stonesense and windows pdb files enabled. The ``package`` job populates the ``release`` cache, which is used to build all distributed binaries. +- The ``docs`` target does a docs-only build of DFHack and reports any errors. + Doc errors would show up in the ``test`` and ``package`` builds anyway, but + the ``docs`` target runs very fast and can identify doc errors in less than + 1m. +- ``lint.yml`` runs the verification scripts in the ``ci`` directory. These + scripts check for common errors in the codebase that are not caught by the + compiler. The lint scripts are written in Python and shell script and are + intended to be run quickly and catch common errors. + +Check type sizes +~~~~~~~~~~~~~~~~ + +``check-type-sizes.yml`` is a df-structures-only workflow that checks for +changes in the sizes of types in the xml structures. It builds the +``xml-dump-type-sizes`` binary on both Linux and Windows for both the +structures in this PR and for the structures in the target merge branch. It +then runs the built binary on its native OS and compares the output. If any +type sizes have changed, the workflow generates a PR comment (via the +``comment-pr.yml`` workflow) with details. + +.. _workflows-release-automation: + +Release automation workflows +---------------------------- + +Watch DF Releases +~~~~~~~~~~~~~~~~~ + +This workflow runs every 8 minutes and checks the Steam metadata, the Itch +website, and the Bay 12 website for evidence of new releases. If a new release +is found, it generates an announcement in a private channel on the DFHack +Discord server. + +Inside the ``watch-df-releases.yml`` workflow, there are separate jobs for +watching Steam branches and watching the websites. For the Steam watcher, it +takes configuration for: + +- which branches to watch +- whether to kick off the Generate symbols workflow when a new release is found +- whether to autodeploy to Steam when the Generate symbols workflow completes + +The workflow has protections against concurrent runs, so if you suspect a new +release is out, you can manually trigger the workflow to check. If the cron +trigger happens to run the workflow at the same time, the second run will be +paused while the first run completes. + +Generate symbols +~~~~~~~~~~~~~~~~ + +This workflow can be triggered manually or by the Watch DF Releases workflow. +It downloads the specified DF version for the selected distribution platform(s) +and OS target(s), then updates the ``symbol-table`` entries in ``symbols.xml``. +If the distribution platform is Steam, it can also autodetect the DF version by +extracting the version string from the DF title screen data. + +For Linux, it always builds DFHack -- just the core library (no plugins) -- and +generates symbols via the `devel/dump-offsets` and `devel/scan-vtables` scripts. + +For Windows, we extract symbol data via static analysis, so the workflow only +builds DFHack if it needs to autodetect the DF version. + +Once the symbols.xml file is updated, the workflow commits the changes to the +specified df-structures branch and updates the xml submodule ref in the +specified DFHack/dfhack branch. If a deploy Steam branch is specified, it also +launches the Deploy to Steam workflow. + +Deploy to GitHub +~~~~~~~~~~~~~~~~ + +`github-release.yml `_ +can be triggered manually or automatically by creating a new release version +tag in git. It builds DFHack with the release configuration, packages the +aritifacts for GitHub, creates a new GitHub release, and uploads the packages +to the GitHub release page. + +It uses text in :source:`.github/release_template.md` to generate the release +notes, and appends the changelog contents for the tagged version. + +If you need to re-tag the release to fix a mistake, it will automatically run +again and replace the binaries attached to the GitHub release for the tagged +version. It will not overwrite the release notes, though, to preserve any edits +you may have made in the GitHub UI. If you *want* it to completely regenerate +the release notes, you can delete the release before you re-tag the version. + +GitHub releases end up here: https://github.com/DFHack/dfhack/releases. + +Deploy to Steam +~~~~~~~~~~~~~~~ + +`steam-deploy.yml `_ +can be triggered manually or automatically by creating a new release version +tag in git. It builds DFHack with the release configuration, packages the +aritifacts for Steam, and uploads them to the specified Steam branch. + +The workflow caches steamcmd to speed the deployment up by 30s or so. +Otherwise, steamcmd would have to be downloaded and updated every time the +workflow runs. + +Steam releases end up here: +https://partner.steamgames.com/apps/builds/2346660. The "version" you +specified for the workflow is used as the "description" for the build. + +Maintenance workflows +--------------------- + +Update submodules +~~~~~~~~~~~~~~~~~ + +`update-submodules.yml `_ +runs daily, or can be run manually as needed. It checks DFHack submodules for +new commits on the main branches and updates the submodule refs in the DFHack +develop branch. + +You generally should not run this workflow for anything other than the develop +branch, as it will overwrite any changes you have made to the submodule refs in +other branches. + +Clean up PR caches +~~~~~~~~~~~~~~~~~~ + +This workflow runs automatically whenever a PR is closed or merged. It removes +caches created for the PR so they don't take up quota. + +Note that if you merge a PR before all the workflows have completed, the caches +may be created after this workflow runs. In that case, the caches will be +orphaned and will be purged by GitHub's cache eviction policy after 2 weeks. diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 4992015d86..931b9bb006 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -18,6 +18,8 @@ These are pages relevant to people developing for DFHack. /docs/dev/overlay-dev-guide /docs/dev/Structures-intro /docs/dev/data-identity + /docs/dev/github-workflows + /docs/dev/release-process /docs/dev/Memory-research /docs/dev/Binpatches /docs/dev/Remote diff --git a/docs/dev/release-process.rst b/docs/dev/release-process.rst new file mode 100644 index 0000000000..bef09be412 --- /dev/null +++ b/docs/dev/release-process.rst @@ -0,0 +1,209 @@ +Release process +=============== + +This page details the process we follow for beta and stable releases. + +For documentation on the related GitHub workflows, see +`workflows-release-automation`. + +Beta release +------------ + +This process pushes a pre-release build to GitHub and Steam. It is intended to +be lower-toil than the stable release process and allows us to facilitate +frequent public testing and feedback without compromising the stability of our +"stable" releases. + +1. Run the `Update submodules `_ GitHub action on the ``develop`` branch to ensure that all submodules are up to date. + +2. Update version strings in :source:`CMakeLists.txt` as appropriate. Set ``DFHACK_RELEASE`` to the *next* stable release version with an "rc#" suffix. For example, if the last stable release was "r1" then set the string to "r2rc1". If we do a second beta release before the final stable "r2" then the string would be "r2rc2". + + - Ensure the ``DFHACK_PRERELEASE`` flag is set to ``TRUE``. + - Commit and push to ``develop`` + - Set ``RELEASE`` in your environment for the commands below (e.g. ``RELEASE=51.07-r2rc1`` for bash) + +3. Tag ``develop`` (no need to tag the submodules) and push: ``git tag -a $RELEASE -m "Bump to $RELEASE"; git push --tags origin`` + + - This will automatically trigger `Deploy to Steam `_ and `Deploy to GitHub `_ build workflows. + +4. Write release notes highlights and any requests for feedback in the `draft GitHub release `_ and publish the draft. + +5. Associate release notes with the build on Steam + + - Go to the `announcement creation page `_ + - Select "A game update" + - Select "Small update / Patch notes" + - Set the "Event title" to the release version (e.g. "DFHack 51.07-r2rc1") + - Set the "Subtitle" to "DFHack pre-release (beta channel)" + - Mention release highlights in the "Summary" field + - Transcode release notes from the GitHub release into the "Event description" field + - patch notes must be in BBcode; see `converting-markdown-to-bbcode` below for how to convert our release notes to BBcode + - Click "Link to build" and select the staging branch (be sure the build has been deployed to the branch first. By the time you're done with the patch notes the build will likely be ready for you) + - Go to the Artwork tab, select "Previously uploaded images", and search for and double-click on dfhack_logo.png. Click "Upload" (even though it has already been uploaded). + - Switch to the "Publish" tab and publish! + - `Promote `_ the build to the "beta" branch (and the "testing" branch if it's newer than what is on the "testing" branch) + +6. Monitor for beta channel subscriber feedback on the Steam `community page `_ + +7. *Maybe* also post to Reddit and other announcement channels if we feel like we need to recruit more beta testers into the pool, but we should avoid posting so often that it is annoying for those who don't use Steam or just want announcements for stable releases. + +Stable release +-------------- + +This process creates a stable DFHack release meant for widespread distribution. +Stable releases come in two forms: straight from ``develop`` or from a point +release branch. + +During "normal" times, we will test out new features in beta releases until we +reach a point of stability. Then, after the ``develop`` branch is feature frozen +while we polish and fix bugs, we tag a release directly from ``develop`` +``HEAD``. + +However, if we have already started committing beta features to ``develop`` and +it becomes necessary to put out a bugfix release for a problem in an +already-released stable release, then we will create a new branch from the +stable tag, cherry-pick fixes from ``develop`` onto that branch, and spin a +release from there. After the point release is published, we'll merge the +branch back into ``develop`` and remove the release branch to clean up. + +1. Triage remaining issues/PRs in the `release project `_ + + - Don't feel pressure to merge anything risky just before a stable release. That's what beta releases are for. + +2. In your local clone of the ``DFHack/develop`` branch, make sure your checkout and all submodules (listed in :source:`.gitmodules`) are up to date with their latest public commits and have no uncommitted/unpushed local changes. + +3. Ensure that CI has not failed unexpectedly on the latest online changes: + + - https://github.com/DFHack/dfhack/commits/develop + - https://github.com/DFHack/scripts/commits/master + - https://github.com/DFHack/df-structures/commits/master + +4. Update version strings in :source:`CMakeLists.txt` as appropriate + + - Ensure the ``DFHACK_PRERELEASE`` flag is set to ``FALSE``. + - Set ``RELEASE`` in your environment for the commands below (e.g. ``RELEASE=51.07-r1``) + +5. Replace "Future" with the version number and clean up changelog entries; add new "Future" section (with headers pre-populated from the template at the top of the file): + + - ``docs/changelog.txt`` + - ``scripts/changelog.txt`` + - ``library/xml/changelog.txt`` + - ``plugins/stonesense/docs/changelog.txt`` + +6. Do a top-level build to ensure the docs build cleanly + +7. Commit/push changes to submodules and tag (``git tag -a $RELEASE -m "Bump to $RELEASE"; git push --tags origin master``) + + - ``scripts`` + - ``library/xml`` + - ``plugins/stonesense`` + +8. Commit and push changes to ``develop`` + + - Ensure that any updates you pushed to submodules are tracked in the commit to ``DFHack/develop`` + +9. Tag ``dfhack``: ``git tag -a $RELEASE -m "Bump to $RELEASE"; git push --tags origin develop`` + + - This will automatically trigger a `Deploy to Steam `_ GitHub action to the "staging" Steam branch and a `Deploy to GitHub `_ GitHub action to create a draft `release `_ from a template and attach the built artifacts. + +10. Switch to the Steam ``staging`` release channel in the Steam client (password: ``stagingstagingstaging``) and download/test the update. + + - Ensure DFHack starts DF when run from the Steam client + - Ensure the DFHack version string is accurate on the title page (should just be the release number, e.g. ``DFHack 51.07-r1``, with no git hash or warnings) + - Run `devel/check-release` + - If something goes wrong with this step, fix it, delete the tag (both from `GitHub `_ and locally (``git tag -d $RELEASE``)), re-tag, re-push, and re-test. Note that you do *not* need to remove the GitHub draft release -- the existing one will just get updated with the new tag and binaries. You *can* remove the draft release, though, if you want the release notes to get regenerated. + +11. Prep release on GitHub + + - Go to the draft `release `_ on GitHub + - Add announcements, highlights (with demo videos), etc. to the description + +12. Push develop to master (``git push origin develop:master``) + + - This will start the documentation build process and update the published "stable" docs + - Note that if this is a -r1 release, you won't be able to complete this step until a classic build is available on the Bay 12 website so the DFHack Test workflow can pass, which is a prerequisite for being able to push to ``master``. + +13. Post release notes on Steam + + - Go to the `announcement creation page `_ + - Select "A game update" + - Select "Regular update" + - Set the "Event title" to the release version (e.g. "DFHack 51.07-r1") + - Set the "Subtitle" to "DFHack stable release" + - Add list of highlights (and maybe announcements, if significant) to the "Summary" field + - Upload screenshots and demo videos via the button at the bottom of the "Previously uploaded videos" area + - Add release notes to the "Event description" field (must be in BBcode; see `converting-markdown-to-bbcode` below for how to convert our release notes to BBcode) + - Drag uploaded images/videos into their appropriate places in the announcement text (replace the GitHub URL tags, which won't work from Steam) + - If the generated release notes exceed the announcement length limits, add a link to the GitHub release page at the bottom of the announcement instead + - Click "Link to build" and select the staging branch (be sure the build has been deployed to the branch first. by the time you're done with the patch notes the build will likely be ready for you) + - the release notes will travel with the build when we promote it to other branches + - Go to the Artwork tab, select "Previously uploaded images", and search for and double-click on STABLEannouncement6.png. Click "Upload" (even though it has already been uploaded). + - Switch to the "Publish" tab and publish! + +14. Go to the `Steam builds page `_ and promote the build to the "default" branch + + - For the build that you just pushed to "staging", click the "-- Select an app branch --" drop-down and select "default" + - Click on "Preview Change" + - Commit the change (you may need to verify with 2FA) + - If the release is newer than what's on the ``beta`` and/or ``testing`` branches, set it live on those branches as well + +15. Publish the prepped GitHub release + +16. Send out release announcements + + - Announce new version in r/dwarffortress. Example: https://www.reddit.com/r/dwarffortress/comments/1i3l5xl/dfhack_5015r2_released_highlights_stonesense/ + - Create the post in the Reddit web interface; the mobile app is extremely painful to use for posting + - Do an "Images & Video" post, sample title: "DFHack 51.07-r1 released! Hilights: Open legends mode directly from an active fort, Dig through warm or damp tiles without interruption, Unlink buildings from levers" + - Add the animated gifs to the post (with appropriate captions naming the relevant tool and what is being demonstrated) + - Add the "DFHack Official" flair to the post. If you're not a r/dwarffortress mod, ask Myk to do this after posting. + - After posting, add each section of the release notes as its own comment, splitting out individual announcements and highlights. This gives people the opportunity to respond directly to the portion of the release notes that interests them; it also helps us avoid size limits for comments. You can include a single still shot (.png file) per comment, but you have to switch to "Fancy Pants Editor" to do it. You can only switch editors once, or the image will get messed up (that is, the image will turn into a hyperlink to an image). Suggested procedure is to prepare the comment in markdown, switch to Fancy Pants Editor, and add images just before submitting the comment. + - Announce new version in forum thread. Example: http://www.bay12forums.com/smf/index.php?topic=164123.msg8567134#msg8567134 + - Update latest version text and link in `first post `_ (if you are not Lethosor, ping Lethosor for this) + - Announce in `#announcements `_ on DFHack Discord + - Announce in `#mod-releases `_ on Kitfox Discord + - Change the name of the release thread on Kitfox Discord to match the release version (if you are not Myk, ping Myk for this) + +17. Monitor all announcement channels for feedback and respond to questions/complaints + +18. Create a `project `_ on GitHub in the DFHack org for the next release + + - Open the `project template `_ + - Click "Use this template" + - Name the project according to the version, e.g. "51.07-r2" and click "Use template" + - In the new project, select settings and set the visibility to Public + - Move any remaining To Do or In Progress items from last release project to next release project + - Close project for last release + +19. If this is a -r2 release or later, go to https://readthedocs.org/projects/dfhack/versions/ and "Edit" previous DFHack releases for the same DF version and mark them "Hidden" (keep the "Active" flag set) so they no longer appear on the docs version selector. + +.. _converting-markdown-to-bbcode: + +Converting Markdown to BBcode +----------------------------- + +Hopefully we can `automate `_ this in the future, but for now, here is the procedure: + +1. Get the markdown that you want to convert into some field on GitHub (can be a temporary text field that you then preview without saving) + +2. View the rendered release notes in your browser (these instructions are for Chrome, but other browsers probably have similar capabilities) + +3. Right click on the rendered text and inspect the DOM + +4. Copy the HTML element that contains the release notes + +5. Click on the "Import HTML" button on the Steam announcement form; paste in the HTML and click "Overwrite" + +6. Copy the generated BBCode out from the description field and into a text editor + +7. Fix it up: + + - Remove the "How do I download DFHack?" section -- people on Steam don't need it + - Some ``

`` elements aren't converted properly and need to be rewritten with square brackets + - Any monospaced text gets HTML tags instead of BBCode ``[code]`` tags, but you can't use them either since they force newlines. ``[tt]`` isn't supported. Any ```` tags just need to be removed entirely. + - Any ``
`` and ```` tags need to be removed + +8. Copy it all back into the description field for the announcement + +9. Click on "Preview event" to double check that it renders sanely + +10. You're done.