refactor(ais): Add backward compatibility and improve error handling in AISBatchLoader#1542
Conversation
gaikwadabhishek
force-pushed
the
get-batch-fallback
branch
from
fdc76e9 to
f519498
Compare
gaikwadabhishek
changed the title
pzelasko
left a comment
pzelasko
left a comment
There was a problem hiding this comment.
Looks good to me, but maybe it would be easier to just require an updated version of AIStore to simplify the logic here? Can you add to setup.py in a new item extras_require={..., "aistore": "aistore>=min-ver"} ?
…in AISBatchLoader - Add backward compatibility for older AIStore SDK versions (Colocation, ArchiveConfig) - Move all aistore imports to method level (remove top-level imports) - Replace module-level logging with logger instance for better configuration - Fix return type annotation for _collect_manifest_urls (None -> bool) - Add safe attribute access with getattr() in error messages - Simplify ValueError handling with early return Tests: - Add version compatibility tests for Colocation fallback This improves SDK version compatibility and code robustness while maintaining full backward compatibility with older AIStore deployments. Signed-off-by: Abhishek Gaikwad <gaikwadabhishek1997@gmail.com>
gaikwadabhishek
force-pushed
the
get-batch-fallback
branch
from
f519498 to
7bb64ac
Compare
|
added |
| f"returned empty content. Retrying with direct AIStore API call." | ||
| ) | ||
| try: | ||
| content = self._get_object_from_moss_in(info) |
There was a problem hiding this comment.
Why run individual GET every time content is empty? Can we read the returned status to determine if there was an issue, or it was OK (in which case we don't send GET again)?
There was a problem hiding this comment.
When we call batch.get(), if there are any errors (e.g., missing files, timeouts, connection resets), we place those entries under the __404__/ directory. Since we’re running GetBatch with cont_on_err, those files are created as empty placeholders. I’m using this as a backup strategy to preserve positional alignment and handle failures gracefully.
pzelasko
left a comment
pzelasko
left a comment
There was a problem hiding this comment.
LGTM with one comment (can be addressed later)
2 participants
Tests:
This improves SDK version compatibility and code robustness while maintaining
full backward compatibility with older AIStore deployments.
Signed-off-by: Abhishek Gaikwad gaikwadabhishek1997@gmail.com