[GOBBLIN-2189] Implement ContainerCompletion callback in DynamicScalingYarnService

[Blazer-007]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-2189

Description

• Here are some details about my PR, including screenshots (if applicable):
  Implemented handleContainerCompletion(...) which launches replacement container based on exit status
  Handled Scaling down of containers based on negative delta
  Refactored YarnService to remove use of helix related tags and names

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  Updated one existing test
  Manually triggered DummyDynamicScalingYarnServiceManager to check if scale down is happening or not, the stripped logs from that are

2025-01-21 09:09:59 PST [,,] INFO  [DynamicScalingYarnService STARTING] org.apache.gobblin.temporal.yarn.YarnService  - Requesting initial containers
2025-01-21 09:09:59 PST [,,] INFO  [DynamicScalingYarnService STARTING] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 2 new containers for profile <<BASELINE>> having currently 0 containers
2025-01-21 09:09:59 PST [,,] INFO  [DynamicScalingYarnService STARTING] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource = <memory:8192, vCores:2> and allocation request id = Optional.of(0)
...
2025-01-21 09:10:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 2 new containers for profile secondProfile having currently 0 containers
2025-01-21 09:10:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(1)
2025-01-21 09:10:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 3 new containers for profile firstProfile having currently 0 containers
2025-01-21 09:10:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 3 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(2)
...
2025-01-21 09:11:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 1 new containers for profile secondProfile having currently 2 containers
2025-01-21 09:11:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 1 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(3)
2025-01-21 09:11:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 2 new containers for profile firstProfile having currently 3 containers
2025-01-21 09:11:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(4)
...
2025-01-21 09:13:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Releasing 3 containers for profile secondProfile having currently 3 containers
2025-01-21 09:13:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Releasing 5 containers for profile firstProfile having currently 5 containers
...
2025-01-21 09:14:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 5 new containers for profile secondProfile having currently 0 containers
2025-01-21 09:14:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 5 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(5)
2025-01-21 09:14:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 5 new containers for profile firstProfile having currently 0 containers
2025-01-21 09:14:17 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 5 containers with resource = <memory:2048, vCores:2> and allocation request id = Optional.of(6)

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.75%. Comparing base (a0cef28) to head (7a80eb7).
Report is 2 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (a0cef28) and HEAD (7a80eb7). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (a0cef28)	HEAD (7a80eb7)
	3	1

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #4092       +/-   ##
=============================================
- Coverage     49.03%   38.75%   -10.29%     
+ Complexity    10008     1599     -8409     
=============================================
  Files          1895      388     -1507     
  Lines         73612    16016    -57596     
  Branches       8188     1588     -6600     
=============================================
- Hits          36097     6207    -29890     
+ Misses        34280     9311    -24969     
+ Partials       3235      498     -2737     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.