This pull request introduces several improvements and optimizations to the generation_service.py module, focusing on processing efficiency, resource control, and enhanced data extraction. The most significant changes include increasing concurrency limits, introducing randomization to QA generation, extracting image URLs from documents, and adjusting batch sizes for chunk processing.

Performance and Resource Management:

• Increased the concurrency limit for question processing from 10 to 20 and the batch size for chunk processing from 20 to 100, allowing more tasks to be processed in parallel and improving throughput. [1] [2]

Feature Enhancements:

• Added a new function, extract_img_urls, that extracts image URLs from document content using a regular expression, and integrated this extraction into the question processing workflow to store found image URLs in the data object. [1] [2]

Quality Control and Randomization:

• Introduced a randomization step in the QA generation process: for each chunk, QA generation is now probabilistically skipped based on the temperature parameter in question_cfg, which can help diversify output and control resource usage.