intermittent KeyError

[sashakames]

When running integrated with the upgraded publisher, we see these errors on occasion (eg. 3 out of 20000 requests):

  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/asynchronous/rabbitthread.py", line 193, in run
    self.__builder.first_connection()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/asynchronous/thread_builder.py", line 105, in first_connection
    self.__trigger_connection_to_rabbit_etc()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/asynchronous/thread_builder.py", line 201, in __trigger_connection_to_rabbit_etc
    self.__please_open_connection()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/asynchronous/thread_builder.py", line 205, in __please_open_connection
    params = self.__node_manager.get_connection_parameters()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/nodemanager.py", line 339, in get_connection_parameters
    self.set_next_host()
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/nodemanager.py", line 400, in set_next_host
    self.__current_node = self.__get_highest_priority_node(self.__trusted_nodes)
  File "/export/witham3/anaconda2/envs/replica-pub/lib/python3.8/site-packages/esgfpid/rabbit/nodemanager.py", line 421, in __get_highest_priority_node
    list_of_priority_nodes = dict_of_nodes[current_priority]
KeyError: '1'

Using v0.8.1 with Python3

[merretbuurman]

Hi Sasha! Interesting. I'll have a look into that. The stacktrace indicates that it finds no RabbitMQ instance in its internal list that has the priority 1. Can you let me know how many instances you have in your config?

[sashakames]

We just have 1.

[sashakames]

I think we may have had an egress issue with the port so relying on the LLNL server, but could try adding DKRZ to the list.