Skip to content

Introduce rolling docker config / secret update concept 🎉 🚀 #952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jan 30, 2025
Merged

Introduce rolling docker config / secret update concept 🎉 🚀 #952

merged 2 commits into from
Jan 30, 2025

Conversation

@YuryHrytsuk
Copy link
Collaborator

@YuryHrytsuk YuryHrytsuk commented Jan 29, 2025 •
edited
Loading

What do these changes do?

  • Add workflow to enable rolling (on-the-fly) docker config / secret updates.
  • Provide an example on how to use this workflow via appmotion gateway api env config

Details

  • In order to guarantee rolling updates, add a hash suffix to the config name.
  • This can also apply to external config / secrets via the same name-change mechanism.
  • With this mechanism in place, it is important to clean up old docker config versions. For this reason, special makefile targets were introduced
  • Find many useful articles, docs and resources in the description of Add automated sustainable workflow to update docker configs / secrets #949

Related issue/s

Related PR/s

Checklist

  • I tested and it works

@YuryHrytsuk YuryHrytsuk changed the title fixes Introduce rolling docker configs / secrets updates Jan 29, 2025
@YuryHrytsuk YuryHrytsuk requested a review from sanderegg January 29, 2025 13:48
@YuryHrytsuk YuryHrytsuk self-assigned this Jan 29, 2025
@YuryHrytsuk YuryHrytsuk added the t:enhancement New feature or request label Jan 29, 2025
@YuryHrytsuk YuryHrytsuk added this to the Singularity milestone Jan 29, 2025
@YuryHrytsuk YuryHrytsuk marked this pull request as ready for review January 29, 2025 13:49
@YuryHrytsuk YuryHrytsuk changed the title Introduce rolling docker configs / secrets updates Introduce rolling docker configs / secrets updates 🎉 🚀 Jan 29, 2025
@YuryHrytsuk YuryHrytsuk changed the title Introduce rolling docker configs / secrets updates 🎉 🚀 Introduce rolling docker configs / secrets updates concept 🎉 🚀 Jan 29, 2025
@YuryHrytsuk YuryHrytsuk changed the title Introduce rolling docker configs / secrets updates concept 🎉 🚀 Introduce rolling docker config / secret update concept 🎉 🚀 Jan 29, 2025
mrnicegyu11
mrnicegyu11 approved these changes Jan 29, 2025
View reviewed changes
Copy link
Member

@mrnicegyu11 mrnicegyu11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sieht gut aus, very surgical but this could work I see no issues with the code design. Good luck!

sanderegg
sanderegg approved these changes Jan 29, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you will update them on every release?

@YuryHrytsuk
Copy link
Collaborator Author

YuryHrytsuk commented Jan 30, 2025

so you will update them on every release?

only on file content change

@YuryHrytsuk YuryHrytsuk merged commit dc8e56e into ITISFoundation:main Jan 30, 2025
3 checks passed
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-ops-environments that referenced this pull request Feb 3, 2025
@YuryHrytsuk @mrnicegyu11
Introduce rolling docker config / secret update concept 🎉 🚀 (ITISFoun…
8728f6c 
…dation#952)

* fixes

* update comment
@YuryHrytsuk YuryHrytsuk mentioned this pull request Mar 11, 2025
Closed
18 tasks
mrnicegyu11 added a commit that referenced this pull request Mar 13, 2025
@mrnicegyu11 @YuryHrytsuk @sanderegg
✨ Add: Manage grafana content/assets via terraform (#913)
4f61106 
* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Arch Linux Certificates Customization

* Add grafana terrform tooling

* Make osparc-config dotenv-precommit pass: Use all caps env-vars

* Refactoring: jinja2 takes .env file path as explicit argument (like in osparc-config)

* Make CI_ENV_FILE vailable in makefile

* Refactor makefile targets

* Add grafana terraform gitignore

* Rename envvar: TF_STATE_S3_GRAFANAKEY

* Remove old scripts, makefile targets

* Remove unused files

* undue arch style commit

* Remove references to Tempo

* CHange request YH: Stop trying tor ecah grafana eventually

* Change request YH: Move tf scripts to terraform folder

* Change request YH: stricter check

* Add files remove typo

* Add terraform fmt pre-commit hook

* Use ansible.env file in lieu of ci.env if available

* Rename and refactor

* wip

* wip

* remove line

* Makefile repo base dir without git

* Grafana terraform ceph fixes

* Fix indentation

* Add manual to traefik redirect capture all rule (#933)

* Introduce rolling docker config / secret update concept 🎉 🚀  (#952)

* fixes

* update comment

* Update traefik router hardcoded priorities (#953)

* Update traefik router hardcoded priorities

* remove hardcoded priority from adminpanels

* Configure redis replicas via ENV (#957)

* Filestash: remove special docker node label (#959)

* rabbit: configurable replicas (#964)

* rabbit: configurable replicas

* clean up

* 💄 minor: Change DNS Server to Quad9 (#967)

* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Arch Linux Certificates Customization

* Change DNS server for aws to swiss privacy focused one

* revert wrong commit

---------

Co-authored-by: Dustin Kaiser <[email protected]>

* single replica (#968)

* Remove docker api proxy from validate simcore settings (#972)

* Add appmotiongateway add dalco

* Add appmotiongateway add dalco - 2

* Add appmotiongateway add dalco - 3

* Seperate dalco-staging: disable redis special handling (#976)

* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Arch Linux Certificates Customization

* Remove dalco special staging handling

* remove accidental commit

* remove accidental commit

* Remove dalco staging special handling

---------

Co-authored-by: Dustin Kaiser <[email protected]>

* Fix deploy ops failure

* Make curl in ensure_grafana_online_ timeout after 10s

* Timeout in wait_graylog_is_online

* Fix osparc.local pydantic validation failure director-v0

---------

Co-authored-by: Dustin Kaiser <[email protected]>
Co-authored-by: Yury Hrytsuk <[email protected]>
Co-authored-by: Sylvain <[email protected]>
This was referenced Apr 4, 2025
Merged
Merged
Merged
mrnicegyu11 added a commit that referenced this pull request Apr 7, 2025
@mrnicegyu11 @YuryHrytsuk @sanderegg
✨ Add: Grafana tempo (#910)
0459075 
* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Fix wrong filename

* Fix registry local deploy

* Traefik local deployment fixes

* Fix local deployment graylog provisioning

* Fix j2, double venv

* Add python version

* Idempotency for admin-panels

* Remove faulty command

* Local deploy fixes

* Clean Up Local Minio

* init work

* Remove unused code

* Update Minio

* Arch Linux Certificates Customization

* Add grafana terrform tooling

* Make osparc-config dotenv-precommit pass: Use all caps env-vars

* Refactoring: jinja2 takes .env file path as explicit argument (like in osparc-config)

* Make CI_ENV_FILE vailable in makefile

* Refactor makefile targets

* Add grafana terraform gitignore

* Rename envvar: TF_STATE_S3_GRAFANAKEY

* Remove old scripts, makefile targets

* Remove unused files

* undue arch style commit

* Remove references to Tempo

* CHange request YH: Stop trying tor ecah grafana eventually

* Change request YH: Move tf scripts to terraform folder

* Change request YH: stricter check

* Add files remove typo

* Add terraform fmt pre-commit hook

* Use ansible.env file in lieu of ci.env if available

* Rename and refactor

* wip

* wip

* remove line

* Makefile repo base dir without git

* Grafana terraform ceph fixes

* Fix indentation

* Add manual to traefik redirect capture all rule (#933)

* Introduce rolling docker config / secret update concept 🎉 🚀  (#952)

* fixes

* update comment

* Update traefik router hardcoded priorities (#953)

* Update traefik router hardcoded priorities

* remove hardcoded priority from adminpanels

* Configure redis replicas via ENV (#957)

* Filestash: remove special docker node label (#959)

* rabbit: configurable replicas (#964)

* rabbit: configurable replicas

* clean up

* 💄 minor: Change DNS Server to Quad9 (#967)

* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Arch Linux Certificates Customization

* Change DNS server for aws to swiss privacy focused one

* revert wrong commit

---------

Co-authored-by: Dustin Kaiser <[email protected]>

* single replica (#968)

* Remove docker api proxy from validate simcore settings (#972)

* Add appmotiongateway add dalco

* Add appmotiongateway add dalco - 2

* Add appmotiongateway add dalco - 3

* Seperate dalco-staging: disable redis special handling (#976)

* wip

* Add csi-s3 and have portainer use it

* Change request @Hrytsuk 1GB max portainer volume size

* Arch Linux Certificates Customization

* Remove dalco special staging handling

* remove accidental commit

* remove accidental commit

* Remove dalco staging special handling

---------

Co-authored-by: Dustin Kaiser <[email protected]>

* Fix deploy ops failure

* Make curl in ensure_grafana_online_ timeout after 10s

* Timeout in wait_graylog_is_online

* Fix osparc.local pydantic validation failure director-v0

* Move create tempo bucket function to monitoring stack makefile

* wip

* fix faulty commit

* Add tempo as exporter target to otlp collector

---------

Co-authored-by: Dustin Kaiser <[email protected]>
Co-authored-by: Yury Hrytsuk <[email protected]>
Co-authored-by: Sylvain <[email protected]>
@YuryHrytsuk YuryHrytsuk deleted the update-docker-configs-workflow branch April 16, 2025 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrnicegyu11 mrnicegyu11 mrnicegyu11 approved these changes

@sanderegg sanderegg sanderegg approved these changes

Assignees

@YuryHrytsuk YuryHrytsuk

Labels

t:enhancement New feature or request

Projects

None yet

Milestone

Singularity

Development

Successfully merging this pull request may close these issues.

Add automated sustainable workflow to update docker configs / secrets

3 participants

@YuryHrytsuk @mrnicegyu11 @sanderegg