How to solve the queue problem after a long run?

[willjay97]

Bug Report

When I run a strategy and subscribe quote tick, after a period of strategy running, The strategy will report the following errors.

I have checked the document and found that the introduction of Queue Maxsize is simple. How should I solve this problem after a long run? Cloud I clear the data in the Queue manually?

Expected Behavior

How to solve the queue problem after a long run～～

Actual Behavior

The strategy will report a long log as follows:
Exception in callback <bound method Queue.put_nowait of <Queue at 0x7f31323c7110 maxsize=100000 _queue=[QuoteTick(ETHUSDT-PERP.BINANCE,3828.83,3828.84,42.978,2.146,1734202518160000000), QuoteTick(ETHUSDT-PERP.BINANCE,3828.83,3828.84,42.978,1.819,1734202518160000000), QuoteTick(ETHUSDT-PERP.BINANCE,3828.83,3828.84,49.378,1.819,1734202518160999936), QuoteTick(ETHUSDT-PERP.BINANCE,3828.83,3828.84,49.690,1.819,1734202518160999936), QuoteTick(ETHUSDT-PERP.BINANCE,3828.83,3828.84,49.690,1.507,1734202518164999936), ... ...]

Steps to Reproduce the Problem

1. subscribe an instrument quote_tick
2. run a long time, maybe 8 hours or more

Specifications

• OS platform: ubuntu 22.04
• Python version: 3.12
• nautilus_trader version: 1.207.0

[sunlei]

You can try:

• TradingNodeConfig.data_engine.qsize
• TradingNodeConfig.exec_engine.qsize

[willjay97]

You can try:

• TradingNodeConfig.data_engine.qsize
• TradingNodeConfig.exec_engine.qsize

Thank you for your reply so quickly!

If I should set a larger value? Setting a larger value can avoid reporting errors, but will it affect performance?

Or should I set a smaller value, and the NautilusTrader will automatically clean up the queue after reaching the small queue qsize?

[willjay97]

You can try:

• TradingNodeConfig.data_engine.qsize
• TradingNodeConfig.exec_engine.qsize

I have tried, need a larger qsize value.

[cjdsellers]

Hi @willjay97

You can change the qsize here https://github.com/nautechsystems/nautilus_trader/blob/develop/nautilus_trader/live/config.py#L49.
It sounds like something is up though, the queue shouldn't "fill up" after a long period of operation. It sounds like there's tasks taking a long time which start causing back-pressure onto the queue.

[willjay97]

Yes, you're right, really thank you a lot. I change my code and the problem has been solved! 😆😆

The logic of the previous code:

def on_quote_tick(tick):
  if tick match:
    (need too long time to process)

Changed code:

def __init__:
  self.in_processing = False # ✅ add flag
  
def on_quote_tick(tick):
  if tick match and not self.in_processing:
    self.in_processing = True
    (need too long time to process)
  else: # ✅ skip ticks to avoid blocking queue
    return

[cjdsellers]

Cool, also check out the ActorExecutor which can allow you to offload these tasks onto a thread pool.