Skip to content

Impoving container initialize logic #5280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

MantavyaDh
Copy link
Contributor

@MantavyaDh MantavyaDh commented Jul 29, 2025

Context

The PR aims to improve the container initialize logic, before retrying to start a container we have added a check to see if its already running as we've had cases wherein the container starts but not reported by docker daemon.


Description

Added logic to check if container already running before retrying.


Risk Assessment (Low / Medium / High)

Low, only introduces a check before retrying


Unit Tests Added or Updated (Yes / No)

Yes, Unit tests added


Additional Testing Performed

Tested in personal org


Change Behind Feature Flag (Yes / No)

Yes, FF- DistributedTask.Agent.CheckBeforeRetryDockerStart


Documentation Changes Required (Yes/No)

No


Logging Added/Updated (Yes/No)

Logs added.


Rollback Scenario and Process (Yes/No)

Turn off the FF


Dependency Impact Assessed and Regression Tested (Yes/No)

  • All impacted internal modules, APIs, services, and third-party libraries are analyzed.
  • Results are reviewed and confirmed to not break existing functionality.

@MantavyaDh MantavyaDh requested review from a team as code owners July 29, 2025 14:00
@MantavyaDh MantavyaDh added the Containers Issues related to Docker, containerd, etc. label Jul 29, 2025
@MantavyaDh
Copy link
Contributor Author

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@MantavyaDh MantavyaDh changed the title Impoving container initialize logic and adding unit tests Impoving container initialize logic Jul 29, 2025
@MantavyaDh
Copy link
Contributor Author

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

context.Output($"Container {containerId} is already running after {retryCount} retries. but exit code was {exitCode}.");
exitCode = 0; // Indicate success
}
// If the container is still not running after retries, log a warning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning inside this block already shows container not running after retires exhausted. No need for comment here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should log a warning that the container didn't start after all the retries, and the last exit code.
I agree that we are logging a warning for every failed attempt, but doesn't tell the overall outcome rather step-wise outcome.

{
context.Output($"Container {containerId} started successfully after {retryCount} retries.");
}
//return the exit code
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment necessary. This seems redundant so can be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It marks the end of the function Docker Start, with the exit code.

@@ -374,7 +379,7 @@ public async Task<List<PortMapping>> DockerPort(IExecutionContext context, strin
/// <returns
/// <c>true</c>, if specified container is running, <c>false</c> otherwise.
/// </returns>
public async Task<bool> IsContainerRunning(IExecutionContext context, string containerId)
public virtual async Task<bool> IsContainerRunning(IExecutionContext context, string containerId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason why we have made this function virtual?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the function virtual to override it in the unit test class DockerCommandManagerL0Tests.cs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of overriding a function in unit tests, a better practice is to mock the function in test with given input/output.

/// Executes 'docker start' with retries, checking if the container is already running before each retry.
/// Returns 0 if the container is running or started successfully, otherwise returns the last exit code.
/// </summary>
protected virtual async Task<int> ExecuteDockerStartWithRetriesAndCheck(IExecutionContext context, string containerId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason why we have made this function virtual?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of overriding a function in unit tests, a better practice is to mock the function in test with given input/output.

@rishabhmalikMS
Copy link
Contributor

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Containers Issues related to Docker, containerd, etc. enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants