About Dynamic Scheduling

[Zhangmj0621]

Hi, QReward developers. I highly appreciate your fantastic work. I just wonder about how designing an appropriate scheduling strategy to dynamically and efficiently distribute tasks works, can you provide more details?

[sunhailin-Leo]

@Zhangmj0621
That's a good question for designing a scheduling strategy. In practical engineering projects, design decisions must be made based on various scenarios.

For example, the core issue our project aims to solve is the long latency and high retry rate when calling the LLM service.
Of course, in addition to solving the issue of the LLM service invocation, our project will also incorporate multi-vendor scheduling capabilities in the future. This feature will enable dynamic adjustments based on load metrics, invocation failure rates, or latency statistics buckets, etc.

In summary, specific analysis for specific problems is the best practice, and the key is to return to what problem needs to be solved.

[Zhangmj0621]

@sunhailin-Leo Thanks for your detailed answer!