Skip to content

Conversation

@ssubhanjali
Copy link

  • Added process_qa_chunk() function to handle parallel processing on individual GPUs
  • Modified make_dataset() to distribute QA pairs across available GPUs using multiprocessing
  • Each GPU process creates its own embedding model and backend directory
  • Added per-GPU progress bars for real-time monitoring

@ssubhanjali ssubhanjali changed the base branch from master to latest-txt2kg September 29, 2025 18:19
@ssubhanjali ssubhanjali changed the base branch from latest-txt2kg to master October 1, 2025 19:14
@ssubhanjali ssubhanjali changed the base branch from master to latest-txt2kg October 1, 2025 19:30
@puririshi98
Copy link
Contributor

closing in lieu of #10474

@puririshi98 puririshi98 closed this Oct 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants