Skip to content

♻️Dask sidecar: Glob error tests + autoscaling monitoring fix #7650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
May 12, 2025
Merged

♻️Dask sidecar: Glob error tests + autoscaling monitoring fix #7650

merged 13 commits into from
May 12, 2025

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented May 8, 2025 •
edited
Loading

What do these changes do?

Refactored tests, tested against AWS when the glob error appears in fsspec.
This is actually the standard error when a file is not found.
if the file expired it is actually also the same.

Bonus:

  • do not fail autoscaling monitoring when dask-certificates are missing @YuryHrytsuk

Related issue/s

How to test

Dev-ops

@codecov
Copy link

codecov bot commented May 8, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.74%. Comparing base (2d0ae68) to head (5c2358f).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7650      +/-   ##
==========================================
+ Coverage   87.72%   87.74%   +0.02%     
==========================================
  Files        1791     1784       -7     
  Lines       69088    68898     -190     
  Branches     1134     1134              
==========================================
- Hits        60609    60457     -152     
+ Misses       8168     8130      -38     
  Partials      311      311
Flag Coverage Δ
integrationtests 64.93% <ø> (-0.03%) ⬇️
unittests 86.96% <100.00%> (+0.02%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library 93.92% <ø> (ø)
pkg_dask_task_models_library 98.48% <ø> (ø)
pkg_models_library 92.80% <ø> (ø)
pkg_notifications_library 85.26% <ø> (ø)
pkg_postgres_database 88.41% <ø> (ø)
pkg_service_integration 69.92% <ø> (ø)
pkg_service_library 73.02% <ø> (ø)
pkg_settings_library 90.90% <ø> (ø)
pkg_simcore_sdk 85.66% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 92.50% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.64% <ø> (ø)
clusters_keeper 99.25% <ø> (ø)
dask_sidecar 89.88% <100.00%> (ø)
datcore_adapter 98.12% <ø> (ø)
director 76.80% <ø> (ø)
director_v2 91.07% <ø> (-0.10%) ⬇️
dynamic_scheduler 96.76% <ø> (ø)
dynamic_sidecar 90.15% <ø> (ø)
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
payments 92.63% <ø> (ø)
resource_usage_tracker 89.12% <ø> (ø)
storage 87.63% <ø> (ø)
webclient ∅ <ø> (∅)
webserver 86.03% <ø> (+<0.01%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d0ae68...5c2358f. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg changed the title Dask sidecar/bugfix/glob error ♻️Dask sidecar: Glob error tests + autoscaling monitoring fix May 8, 2025
@sanderegg sanderegg marked this pull request as ready for review May 8, 2025 17:34
@sanderegg sanderegg requested a review from pcrespov as a code owner May 8, 2025 17:34
@sanderegg sanderegg self-assigned this May 8, 2025
@sanderegg sanderegg added the t:maintenance Some planned maintenance work label May 8, 2025
@sanderegg sanderegg added this to the Bazinga! milestone May 8, 2025
@YuryHrytsuk
Copy link
Contributor

YuryHrytsuk commented May 9, 2025

@sanderegg what are the implications? Will I get less information about comp. clusters in summary?

@sanderegg
Copy link
Member Author

sanderegg commented May 9, 2025

@sanderegg what are the implications? Will I get less information about comp. clusters in summary?

yes. no information about dask related data such as # jobs, dask resources, etc. but you would still see the machines.

@YuryHrytsuk
Copy link
Contributor

YuryHrytsuk commented May 9, 2025 •
edited
Loading

@sanderegg what are the implications? Will I get less information about comp. clusters in summary?

yes. no information about dask related data such as # jobs, dask resources, etc. but you would still see the machines.

jq -r ".computational_clusters[].primary.user_id" summary.json
jq -r ".dynamic_instances[].ec2_instance_id"  summary.json

Does CLI still preserve fields mentioned above? @sanderegg

@YuryHrytsuk YuryHrytsuk self-requested a review May 9, 2025 07:03
@sanderegg
Copy link
Member Author

sanderegg commented May 9, 2025

@sanderegg what are the implications? Will I get less information about comp. clusters in summary?

yes. no information about dask related data such as # jobs, dask resources, etc. but you would still see the machines.

jq -r ".computational_clusters[].primary.user_id" summary.json
jq -r ".dynamic_instances[].ec2_instance_id"  summary.json

Does CLI still preserve fields mentioned above? @sanderegg

yep @YuryHrytsuk

@sanderegg sanderegg force-pushed the dask-sidecar/bugfix/glob-error branch from 596d647 to 45390aa Compare May 9, 2025 08:05
@sonarqubecloud
Copy link

sonarqubecloud bot commented May 12, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@sanderegg sanderegg merged commit a194300 into ITISFoundation:master May 12, 2025
92 of 95 checks passed
@sanderegg sanderegg deleted the dask-sidecar/bugfix/glob-error branch May 12, 2025 08:31
@sanderegg sanderegg mentioned this pull request May 28, 2025
Open
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Jun 6, 2025
Closed
92 tasks
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GitHK GitHK GitHK approved these changes

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@pcrespov pcrespov Awaiting requested review from pcrespov pcrespov is a code owner

Assignees

@sanderegg sanderegg

Labels

t:maintenance Some planned maintenance work

Projects

None yet

Milestone

Bazinga!

Development

Successfully merging this pull request may close these issues.

4 participants

@sanderegg @YuryHrytsuk @GitHK @matusdrobuliak66