Refactor GitHub Actions workflows for build, documentation, and linting #2260

Odyno · 2025-12-27T13:59:08Z

Description:

UPDATE_NOTE_START
Since I don’t like providing untested PRs, I decided to add a new commit to this PR that includes a series of scripts which allowed me to be CI-agnostic and test my proposal. I’m about 90% confident that everything works (I can’t test anything else).
UPDATE_NOTE_END

Based on the discussion on https://community.openmqttgateway.com/t/holiday-questions-d/4345, this PR provides a comprehensive refactoring of the CI/CD infrastructure.

This Pull Request maintains feature parity with the existing CI/CD pipeline, with no new actions/workflow introduced and no regressions, while significantly improving maintainability and local development experience.

Full documentation is available in:

Key Improvements

GitHub Actions Workflow Reorganization

Consolidated build logic into reusable workflows (task-build.yml, task-docs.yml, task-lint.yml) to eliminate duplication
Introduced environments.json to centralize PlatformIO build environments, improving maintainability
Updated main workflows (build.yml, build_and_docs_to_dev.yml, release.yml) to leverage reusable components
Streamlined release process with integrated documentation generation

Platform-Agnostic CI/CD Scripts

Created unified CI orchestration through ci.sh dispatcher script
Implemented modular, composable scripts:
- ci_build.sh - Firmware build pipeline orchestration
- ci_site.sh - Documentation build with version management
- ci_qa.sh - Code linting and formatting checks
- ci_set_version.sh - Automated version tagging
- ci_build_firmware.sh - PlatformIO build execution
- ci_prepare_artifacts.sh - Artifact packaging and organization
Local development friendly: All CI scripts can be executed locally with same behavior as GitHub Actions
Centralized configuration via ci_00_config.sh for DRY principles
Enhanced error handling, logging, and dependency validation

Comprehensive Documentation

Detailed workflow architecture documentation with data flow diagrams
Complete CI scripts reference with examples and troubleshooting guides
Improved code comments across Python helper scripts

Benefits

Consistency: Same scripts run locally and in CI/CD
Testability: Developers can validate builds/docs before pushing
Portability: Easy migration to other CI platforms (GitLab CI, Jenkins, etc.)
Maintainability: Centralized logic reduces duplication
Debugging: Easier to troubleshoot issues locally

Future Enhancements

SEMPLIFICATION
Security improvements are planned for a follow-up PR:
- SBOM (Software Bill of Materials) generation
- Trivy security scans
- Dependency vulnerability reporting

Testing

All 85 GitHub Actions checks passing
Build tested on 83 PlatformIO environments
Documentation generation validated
Local execution tested on Linux/macOS

Checklist

Pull request is done against the latest development branch
Only one feature/fix was added per PR and the code change compiles without warnings
I accept the DCO

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows. - Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity. - Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions. - Enhanced `release.yml` to streamline the release process and integrate documentation generation. - Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository. - Simplified manual documentation workflow by leveraging the new reusable documentation workflow. - Improved artifact management and retention policies across workflows. - Updated dependencies and versions in workflows to ensure compatibility and performance.

…ctions - Implemented ci.sh for orchestrating the complete build pipeline. - Created ci_00_config.sh for centralized configuration of build scripts. - Created ci_build_firmware.sh for building firmware for specified PlatformIO environments. - Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment. - Created ci_set_version.sh for updating version tags in firmware configuration files. - Created ci_build.sh to orchestrate the complete build pipeline. - Created ci_qa.sh for code linting and formatting checks using clang-format. - Created ci_site.sh for building and deploying VuePress documentation with version management. - Implemented checks for required tools and dependencies in the new scripts. - Updated common_wu.py, compressFirmware.py, gen_wu.py, generate_board_docs.py, and prepare_deploy.sh with descriptive comments. Refactor CI/CD scripts for improved modularity and clarity - Consolidated build steps in task-build.yml to utilize ci.sh for version tagging, building, and artifact preparation. - Updated task-lint.yml to use ci.sh for code formatting checks instead of ci_qa.sh. - Enhanced CI_SCRIPTS.md documentation to reflect changes in script usage, command structure, and output organization. - Improved internal scripts for better error handling and logging. - Streamlined the output structure for build artifacts and documentation.

1technophile · 2026-01-03T11:29:01Z

.github/workflows/environments.json

+            "sonoff-basic",
+            "sonoff-basic-rfr3"
+        ],
+        "metadata": {


If it is not used by the code I would remove the metadata, they are currently out of sync (the count does not match the qty of environments (83 in metadata 82 in the list)

Yes, I agree, it was one experiment not cleaned.

1technophile · 2026-01-03T11:35:36Z

scripts/ci_build.sh

+}
+
+# Function to list available environments
+list_environments() {


environment listing should come from environments.json and not the ini I suspect

Let me create a double option... one from environment.json and the other from real platform.ini... so we can use it in all cases and not only in CI

1technophile · 2026-01-03T11:43:21Z

scripts/ci_set_version.sh

+    log_info "Setting PRODUCTION version: $version"
+
+    replace_version "$USER_CONFIG" "$version" || return 1
+    replace_version "$VERSION_JSON" "$version" || return 1


VERSION_JSON does not look defined somewhere

I deleted this file in the end because I realized it was only useful for the website, and it made a small change to the code… a change that cannot be reverted and works only one time. This way of doing things is an anti‑pattern, because it stops local development of this pipeline and makes the code messy. I changed this part so the functionality still works, but without modifying files that are tracked by Git. I tried to work on this and in latest_version.json, latest_version_dev.json, board.md, web-install.md, and config.js. Please check out the proposal

1technophile · 2026-01-03T11:43:37Z

scripts/ci_set_version.sh

+    log_info "Setting DEVELOPMENT version: $version"
+
+    replace_version "$USER_CONFIG" "$version" || return 1
+    replace_version "$VERSION_DEV_JSON" "$version" || return 1


VERSION_DEV_JSON does not look defined somewhere

1technophile · 2026-01-03T11:45:40Z

scripts/ci_00_config.sh

+# Used by: All build scripts for centralized configuration
+
+# Python Configuration
+PYTHON_VERSION="3.13"


do we really need all those variables ? If they are not used I would suggest to remove them.
They can mislead the developer in thinking that we have a centralized place for them but it is not the case

1technophile · 2026-01-03T11:49:11Z

.github/workflows/lint.yml


 on: [push, pull_request]

 jobs:


can we centralize this in task lint and remove this file ?

I agree! I left it just because it was present

Odyno · 2026-01-03T20:04:25Z

I'm very happy that you reviewed this MR!

Yes, in those 3 days I have done a small review of all the code of this MR (for this reason it is in draft) because I also tested the site and I discovered some errors.

Really happy to include the comment on the next commit we're I would like to provide a more bit steps to test the site locally ( do not worry always feature parity + add the ability to run all in localhost, also the generation of manifest of local develop version ...)

Hope you have another time to follow the update! Time to test

Odyno force-pushed the feature/workflow-reorganizzation branch from d34290f to f8779e1 Compare December 27, 2025 15:40

Odyno mentioned this pull request Dec 27, 2025

Workflow improvement + Security Check + SBOM generation (based on #2260 ) #2263

Draft

3 tasks

Odyno marked this pull request as draft December 29, 2025 18:08

Odyno force-pushed the feature/workflow-reorganizzation branch from 1134284 to 27c998b Compare December 29, 2025 19:36

Odyno marked this pull request as ready for review December 29, 2025 19:37

Odyno marked this pull request as draft December 29, 2025 22:47

1technophile reviewed Jan 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor GitHub Actions workflows for build, documentation, and linting #2260

Refactor GitHub Actions workflows for build, documentation, and linting #2260

Uh oh!

Odyno commented Dec 27, 2025 •

edited

Loading

Uh oh!

1technophile Jan 3, 2026

Uh oh!

Odyno Jan 3, 2026

Uh oh!

1technophile Jan 3, 2026

Uh oh!

Odyno Jan 3, 2026

Uh oh!

1technophile Jan 3, 2026

Uh oh!

Odyno Jan 3, 2026

Uh oh!

1technophile Jan 3, 2026

Uh oh!

1technophile Jan 3, 2026 •

edited

Loading

Uh oh!

1technophile Jan 3, 2026 •

edited

Loading

Uh oh!

Odyno Jan 3, 2026

Uh oh!

Odyno commented Jan 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Refactor GitHub Actions workflows for build, documentation, and linting #2260

Are you sure you want to change the base?

Refactor GitHub Actions workflows for build, documentation, and linting #2260

Uh oh!

Conversation

Odyno commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Key Improvements

GitHub Actions Workflow Reorganization

Platform-Agnostic CI/CD Scripts

Comprehensive Documentation

Benefits

Future Enhancements

Testing

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1technophile Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1technophile Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Odyno commented Jan 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Odyno commented Dec 27, 2025 •

edited

Loading

1technophile Jan 3, 2026 •

edited

Loading

1technophile Jan 3, 2026 •

edited

Loading