Skip to content

Enable parallel data loading#17

Merged
AKKI0511 merged 1 commit intomainfrom
codex/refactor-fetch_data-for-parallel-download
Jul 1, 2025
Merged

Enable parallel data loading#17
AKKI0511 merged 1 commit intomainfrom
codex/refactor-fetch_data-for-parallel-download

Conversation

@AKKI0511
Copy link
Owner

@AKKI0511 AKKI0511 commented Jul 1, 2025

Summary

  • support parallel downloads via ThreadPoolExecutor
  • allow configuring thread count with max_workers
  • test new parallel mode

Testing

  • pre-commit run --files src/data/loader.py src/utils/config_schemas.py config/model_config.yaml tests/data/test_loader_extra.py
  • pytest -q

https://chatgpt.com/codex/tasks/task_e_686142f5fab8832aa80f15e5f25efd7e

@AKKI0511 AKKI0511 merged commit b205de8 into main Jul 1, 2025
1 check passed
@AKKI0511 AKKI0511 deleted the codex/refactor-fetch_data-for-parallel-download branch July 1, 2025 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant