🎨Computational backend: Fail fast in case of malformed input syntax and improve unzipping #7804
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sanderegg
added
t:maintenance
sanderegg
requested review from
JavierGOrdonnez,
Copilot,
matusdrobuliak66,
odeimaiz,
pcrespov and
wvangeit
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the robustness of the dask-sidecar by introducing a fast fail when a file-to-key mapping is used with zip data and by improving the unzipping process using a temporary directory. Key changes include:
- Adding a new error (ServiceInputsUseFileToKeyMapButReceivesZipDataError) to trigger a fail in case of mismatched file mapping and zip input.
- Enhancing the unzipping process with a dedicated function (check_need_unzipping) and temporary directory usage.
- Adjusting log levels and updating test fixtures to cover these scenarios.
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/dask-sidecar/tests/unit/test_computational_sidecar_tasks.py | Updated tests and fixtures to validate the new error behavior with file-to-key mapping and zip input data. |
| services/dask-sidecar/src/simcore_service_dask_sidecar/utils/files.py | Introduced check_need_unzipping and adjusted the extraction logic using a temporary directory. |
| services/dask-sidecar/src/simcore_service_dask_sidecar/computational_sidecar/task_shared_volume.py | Modified log level from debug to info when creating the input folder. |
| services/dask-sidecar/src/simcore_service_dask_sidecar/computational_sidecar/core.py | Added logic to raise the new error when a file mapping is present along with zip data that needs extraction. |
| packages/dask-task-models-library/src/dask_task_models_library/container_tasks/errors.py | Defined a new error class for the file-to-key mapping and zip data mismatch scenario. |
services/dask-sidecar/src/simcore_service_dask_sidecar/utils/files.py
Outdated
Show resolved
Hide resolved
sanderegg
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #7804 +/- ##
==========================================
+ Coverage 86.73% 88.05% +1.31%
==========================================
Files 1850 1269 -581
Lines 71875 54189 -17686
Branches 1215 189 -1026
==========================================
- Hits 62338 47714 -14624
+ Misses 9196 6416 -2780
+ Partials 341 59 -282
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
pcrespov
approved these changes
Jun 3, 2025
services/dask-sidecar/src/simcore_service_dask_sidecar/computational_sidecar/core.py
Outdated
Show resolved
Hide resolved
sanderegg
force-pushed
the
improvements/download-zip-file-to-temporary-folder
branch
from
1951113 to
f2da9ef
Compare
Member
Author
|
@mergify queue |
Contributor
✅ The pull request has been merged automaticallyThe pull request has been merged automatically at dc35a5e |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
As part of the issues found by @wvangeit where a service would define file_to_key mapping for input and pass in a zip file.
Having a service input port defining a
file_to_keymapping and at the same time passing azipfile at runtime produces confusing results:values.json)values.jsonfile which overrides the originalvalues.jsonzip filevalues.jsonfile--> the jsonifier service would actually run with an empty input data and will not complain as it can run with no inputs.
This PR improves the dask-sidecar to raise an exception if it detects:
application/zipOn the other hand the unzipping process is enhanced by:
Related issue/s
How to test
Dev-ops