Feature:4005 Docker settings usage warnings #4011

Json-Andriopoulos · 2025-09-29T09:26:53Z

Describe changes

I implemented/fixed _ to achieve _.

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
IMPORTANT: I made sure that my changes are reflected properly in the following resources:
- ZenML Docs
- Dashboard: Needs to be communicated to the frontend team.
- Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
- Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

strickvl · 2025-09-30T14:26:01Z

It's a somewhat common thing to switch between a local orchestrator and a cloud-based one, esp during development. I'm wondering if this warning might get a bit annoying?

htahir1 · 2025-09-30T14:35:27Z

100% i wouldnt raise usage warnings for this due to what alex said

bcdurak · 2025-09-30T14:40:01Z

@strickvl @htahir1, I think I disagree with this a little bit. If you are working with a pipeline where you define some DockerSettings and the orchestrator is currently ignoring it, I would want to see the warning.

bcdurak · 2025-09-30T14:41:12Z

It is an indicator of how to do it properly. If I am in the development setting, I would not be bothered by this if I switch to a local orchestrator.

htahir1 · 2025-10-01T14:42:04Z

I think we have talked a few times how overloading warnings for use-cases that dont matter causes people to just ignore them @bcdurak ... I will guarantee this will cauase one of two things a) Most people will ignore this warning b) People will start having if conditions in their code to not apply dockersettings on certain stacks, which is ugly AF

I am quite strongly opposed to this PR tbh

schustmi · 2025-10-06T07:54:00Z

@htahir1 @Json-Andriopoulos "a) Most people will ignore this warning": This is not an argument against having a warning at all. I think ZenML should warn users about stuff that is worthy of a warning. If they decide to ignore them and run into issues because of it, that's on them and warning them is the best we could do.

Regarding what we should do in this case:

When running a pipeline with an "invalid settings key" (e.g. settings={"not_docker": DockerSettings(...)} we currently log an info message:

zenml/src/zenml/config/compiler.py

Lines 400 to 403 in e025293

    
           logger.info( 
        
               "Not including stack component settings with key `%s`.", 
        
               key, 
        
           )

This case includes when you specify settings for a component or flavor that is not in your active stack when running.

When providing some keys for a settings class which don't exist (e.g. (e.g. settings={"docker": DockerSettings(invalid_key=...)} we currently log a warning message:

zenml/src/zenml/config/compiler.py

Lines 413 to 417 in e025293

    
           logger.warning( 
        
               "Ignoring invalid setting attributes `%s` defined for key `%s`.", 
        
               list(settings_instance.model_extra), 
        
               key, 
        
           )

The case we're trying to catch as part of this PR is probably more similar to the first bullet point, so maybe changing it to an info log would be the most inline with how we treat it currently? But in my opinion, both the first bullet point here and the message as part of this PR logically are warning messages.

htahir1 · 2025-10-20T08:27:21Z

@schustmi @Json-Andriopoulos

Overall: the intent (alerting when DockerSettings are defined but won’t be honored) is good, but the UX needs to avoid becoming noisy—especially when devs bounce between local and remote orchestrators. Here are some suggestions:

Warn only when it’s actionable
- Trigger only if DockerSettings are present and the current orchestrator is non-containerized (or otherwise ignores them).
- If DockerSettings are empty/default → no warning.
Make the message precise + helpful
- Example: “DockerSettings detected, but <LocalOrchestrator> does not build/use containers. Switch to <ContainerizedOrchestrator> or remove DockerSettings. See: .”
- Include the exact pipeline/run and the settings keys detected.
Right severity in dev vs. prod
- Default to INFO for known “local/dev” orchestrators.
- WARNING when running on remote/cloud where users are more likely to expect containerization.
De-dupe & throttle
- One warning per process/run (or per unique pipeline+orchestrator combo), with “suppressed N similar messages” thereafter.
Give users control
- Add a project/user config toggle or an env var to silence.
- Consider --strict mode to escalate to error/exit for CI.
Surface it where it matters
- CLI log + dashboard run details (a small banner with “settings ignored”).
- Optionally emit a distinct log code (e.g., ZNL-DS001) so teams can grep/alert.
Test the matrix
- Cases: (a) DockerSettings + local orchestrator, (b) DockerSettings + containerized orchestrator, (c) no DockerSettings, (d) partial support (some fields ignored), (e) multi-stack projects switching orchestrators.

This balances the concerns in the thread: folks worried about annoyance when switching locally (valid) and those wanting a clear heads-up when a real misconfiguration happens (also valid).

I'm happy if at least first three of the above are taken into account

schustmi · 2025-10-24T07:17:29Z

@htahir1 @Json-Andriopoulos IMO we could simply add a warning log message in the compiler class (in the place I linked above) if there are non-empty docker settings defined when running with a local orchestrator?

github-actions · 2025-10-30T13:24:44Z

ZenML CLI Performance Comparison (Threshold: 1.0s, Timeout: 60s, Slow: 5s)

❌ Failed Commands on Current Branch (feature/4005-docker-settings-usage-warnings)

zenml stack list: Command failed on run 1 (exit code: 1)
zenml pipeline list: Command failed on run 1 (exit code: 1)
zenml model list: Command failed on run 1 (exit code: 1)

🚨 New Failures Introduced

The following commands fail on your branch but worked on the target branch:

zenml stack list
zenml pipeline list
zenml model list

Performance Comparison

Command	develop Time (s)	feature/4005-docker-settings-usage-warnings Time (s)	Difference	Status
`zenml --help`	1.545380 ± 0.027893	1.434846 ± 0.010392	-0.111s	✓ No significant change
`zenml model list`	Not tested	Failed	N/A	❌ Broken in current branch
`zenml pipeline list`	Not tested	Failed	N/A	❌ Broken in current branch
`zenml stack --help`	1.539063 ± 0.026777	1.449837 ± 0.014832	-0.089s	✓ No significant change
`zenml stack list`	Not tested	Failed	N/A	❌ Broken in current branch

Summary

Total commands analyzed: 5
Commands compared for timing: 2
Commands improved: 0 (0.0% of compared)
Commands degraded: 0 (0.0% of compared)
Commands unchanged: 2 (100.0% of compared)
Failed commands: 3 (NEW FAILURES INTRODUCED)
Timed out commands: 0
Slow commands: 0

Environment Info

Target branch: Linux 6.11.0-1018-azure
Current branch: Linux 6.11.0-1018-azure
Test timestamp: 2025-10-30T13:23:55Z
Timeout: 60 seconds
Slow threshold: 5 seconds

schustmi

Love the warning registry! We should definitely add a way to ignore certain warning codes, which should be persisted locally and probably also forwarded to the execution environment where pipelines/steps get executed?

schustmi · 2025-10-31T08:36:44Z

src/zenml/utils/warnings/registry.py

+)
+
+
+class WarningCodes(str, Enum):


We have a StrEnum base class that we usually use in those cases, has some additional properties that you might/might not need :)

schustmi · 2025-10-31T08:41:24Z

src/zenml/utils/warnings/controller.py

+        """
+        self._log(warning_code, message, LoggingLevels.WARNING, **kwargs)
+
+    def info(self, *, warning_code: str, message: str, **kwargs: Any) -> None:


Suggested change

def info(self, *, warning_code: str, message: str, **kwargs: Any) -> None:

def info(self, *, warning_code: WarningCodes, message: str, **kwargs: Any) -> None:

We can use the enum here, no? Same in other places

Good point!

schustmi · 2025-10-31T08:44:03Z

src/zenml/utils/warnings/controller.py

+    def __init__(self) -> None:
+        """WarningController constructor."""
+        self._warning_configs: dict[str, WarningConfig] = {}
+        self._warning_statistics: dict[str, int] = defaultdict(int)


There's also from collections import Counter which we could use here, I'll leave it up to you.

Has some helper methods to e.g. count the total amount of warnings, in case we ever want to log some statistics.

Yeap, good idea. I mainly want to avoid constructing values if the key is missing, Counter supports that as well.

src/zenml/utils/warnings/base.py

schustmi · 2025-10-31T08:51:23Z

src/zenml/config/compiler.py

+        ):
+            return
+
+        warning_message = (


We do actually have a few orchestrators that are not containerized but still use some DockerSettings attributes to configure their behaviour (WheeledOrchestrator, LightningOrchestrator). So we either need to get rid of this warning here or also include those two

Good point. Do we have any production orchestrators that don't make use of the docker settings? Seems like we just need to catch the LocalOrchestrator here and info the message 🤔

Yeah I think just the local one ignores them entirely

- Warn when used without containerized orchestrator in place

github-actions bot added the enhancement New feature or request label Sep 29, 2025

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 2 times, most recently from 535f94f to 617ea24 Compare September 30, 2025 08:26

bcdurak linked an issue Oct 1, 2025 that may be closed by this pull request

Add a warning when no DockerSettings are in use #4005

Open

1 task

zenml-io deleted a comment from github-actions bot Oct 1, 2025

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 3 times, most recently from c4c25e8 to 7295416 Compare October 1, 2025 12:21

bcdurak requested a review from schustmi October 2, 2025 07:17

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch from 7295416 to 875e625 Compare October 30, 2025 13:21

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 2 times, most recently from f18f13b to b9e7252 Compare October 30, 2025 14:34

Json-Andriopoulos requested a review from bcdurak October 30, 2025 14:34

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 4 times, most recently from acc1871 to db9a427 Compare October 30, 2025 14:54

schustmi requested changes Oct 31, 2025

View reviewed changes

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 3 times, most recently from 8d88486 to 2b5bd20 Compare October 31, 2025 10:05

Json-Andriopoulos added the run-slow-ci label Oct 31, 2025

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch 2 times, most recently from a7af632 to b4e18c7 Compare October 31, 2025 10:19

Json-Andriopoulos requested a review from schustmi October 31, 2025 10:31

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch from b4e18c7 to f24d34e Compare October 31, 2025 10:37

schustmi approved these changes Nov 3, 2025

View reviewed changes

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch from f24d34e to 6c61de7 Compare November 3, 2025 10:06

Feature:4005 Docker settings usage warnings

58a9abd

- Warn when used without containerized orchestrator in place

Json-Andriopoulos force-pushed the feature/4005-docker-settings-usage-warnings branch from 6c61de7 to 58a9abd Compare November 3, 2025 10:08

Json-Andriopoulos merged commit abe1f12 into develop Nov 3, 2025
68 of 74 checks passed

Json-Andriopoulos deleted the feature/4005-docker-settings-usage-warnings branch November 3, 2025 12:31

	def info(self, , warning_code: str, message: str, *kwargs: Any) -> None:
	def info(self, , warning_code: WarningCodes, message: str, *kwargs: Any) -> None:

Feature:4005 Docker settings usage warnings #4011

Feature:4005 Docker settings usage warnings #4011

Uh oh!

Conversation

Json-Andriopoulos commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe changes

Pre-requisites

Types of changes

Uh oh!

strickvl commented Sep 30, 2025

Uh oh!

htahir1 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bcdurak commented Sep 30, 2025

Uh oh!

bcdurak commented Sep 30, 2025

Uh oh!

htahir1 commented Oct 1, 2025

Uh oh!

schustmi commented Oct 6, 2025

Uh oh!

htahir1 commented Oct 20, 2025

Uh oh!

schustmi commented Oct 24, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

ZenML CLI Performance Comparison (Threshold: 1.0s, Timeout: 60s, Slow: 5s)

❌ Failed Commands on Current Branch (feature/4005-docker-settings-usage-warnings)

🚨 New Failures Introduced

Performance Comparison

Summary

Environment Info

Uh oh!

schustmi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Json-Andriopoulos commented Sep 29, 2025 •

edited

Loading

htahir1 commented Sep 30, 2025 •

edited

Loading