Not sure what was happend but my Full Node sometimes stop syncing

[cccat6]

Hi all, I am facing some strange problem now.
My full_node sometimes stops syncing and obviously unable to receive new challenges.
When it stopped, the 'Status' still shows 'Synced' which looks fine, but the 'Peak Time', 'Peak Height', etc. will just stop there. The connection with other full_nodes is shown fine as usual.
All the Chia related processes will stop using CPU but they didn't crash. They just stay there idling.
The easiest way to solve it is simply Reload it or click the Keys and reenter the wallet.
Nothing related in the debug log (Log level is 'WARNING')

My environment is Windows Server 2022, 5950X, 128GB RAM, 2TB NVMe SSD.

Could anyone help me with that? It is really strange and I have never seen that before. It just happed randomly.

[keliew]

Are you plotting at the same time?

What does debug.log say?

[cccat6]

No, and the debug.log said literally nothing.

[cccat6]

I tried to monitor the processes, and I find that when this incident happened. There is one start_full_node process closed. Not sure about the reason and nothing is shown in the log file. I am not sure what does daemon did and why it was not been restarted.

[keliew]

Change log level to INFO and see if you can spot something.

I guess if you'd to restart the daemon (chia run_daemon), it works fine?

[cccat6]

It happens again. I just set the log_level to INFO and see if I can catch anything.
Restart always works fine for a couple of hours, it just randomly crashes.

[cccat6]

I caught something. Here is the log.
Nothing more than this. It just... crashed.

2021-08-17T07:38:05.053 daemon __main__                   : INFO     Websocket exception. Closing websocket with chia_full_node code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "asyncio\windows_events.py", line 457, in finish_recv
OSError: [WinError 64] 指定的网络名不再可用。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "websockets\protocol.py", line 827, in transfer_data
  File "websockets\protocol.py", line 895, in read_message
  File "websockets\protocol.py", line 971, in read_data_frame
  File "websockets\protocol.py", line 1047, in read_frame
  File "websockets\framing.py", line 105, in read
  File "asyncio\streams.py", line 723, in readexactly
  File "asyncio\streams.py", line 517, in _wait_for_data
  File "asyncio\proactor_events.py", line 280, in _loop_reading
  File "asyncio\windows_events.py", line 812, in _poll
  File "asyncio\windows_events.py", line 461, in finish_recv
ConnectionResetError: [WinError 64] 指定的网络名不再可用。

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\a\chia-blockchain\chia-blockchain\venv\Lib\site-packages\chia\daemon\server.py", line 172, in safe_handle
  File "websockets\protocol.py", line 439, in __aiter__
  File "websockets\protocol.py", line 509, in recv
  File "websockets\protocol.py", line 803, in ensure_open
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-17T07:38:08.934 wallet wallet_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x0000016C86EBDD40> [远程计算机拒绝网络连接。]
2021-08-17T07:38:09.022 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001441152CE40> [远程计算机拒绝网络连接。]
2021-08-17T07:38:11.935 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-08-17T07:38:12.026 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-08-17T07:38:13.939 wallet wallet_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x0000016C8A832AC0> [远程计算机拒绝网络连接。]
2021-08-17T07:38:14.029 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001441152CBC0> [远程计算机拒绝网络连接。]
2021-08-17T07:38:16.941 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}

[cccat6]

I am running farmer-no-wallet in CLI and the log_level is DEBUG now. Let's see if it will show something different.
This is a server so I didn't run anything else other than the full_node and some VMs for harvesters.

[cccat6]

Nothing in debug either.

2021-08-19T06:02:46.871 full_node full_node_server        : DEBUG    <- new_transaction from peer a9cef450717338690fdb4d2aba157d52a9923bc533c290b64a33fe7f296a85ab 82.136.100.51
2021-08-19T06:02:46.871 full_node full_node_server        : DEBUG    Time taken to process new_transaction from a9cef450717338690fdb4d2aba157d52a9923bc533c290b64a33fe7f296a85ab is 0.0 seconds
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    <- new_transaction from peer c3e5b0c8c851c9a27291412b477769abbdf4e3fc14d53153c1395e88016cc7bf 75.70.88.76
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    Time taken to process new_transaction from c3e5b0c8c851c9a27291412b477769abbdf4e3fc14d53153c1395e88016cc7bf is 0.0 seconds
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    <- new_transaction from peer 1f676077624422808cc7c8eac75092eaa528830a290443656d07edb954d95789 192.200.122.235
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 1f676077624422808cc7c8eac75092eaa528830a290443656d07edb954d95789 is 0.0 seconds
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    <- new_transaction from peer dce3aeff22bd60a120e6587aa4e6985bdf04d3eed894b20f099ae60bd5ffae93 79.90.204.71
2021-08-19T06:02:46.887 full_node full_node_server        : DEBUG    Time taken to process new_transaction from dce3aeff22bd60a120e6587aa4e6985bdf04d3eed894b20f099ae60bd5ffae93 is 0.0 seconds
2021-08-19T06:02:46.903 full_node full_node_server        : DEBUG    <- respond_transaction from peer c6c55644f6c1d728b7db60fab7543fbf2e3ee7c336411f04beb7d30f3386fc25 143.178.157.91
2021-08-19T06:02:46.903 full_node chia.full_node.full_node: DEBUG    Processing transaction: d6a2bdbb6a239325a16aec15f437892c23be8346f571ca751d7522cc7234ee9d
2021-08-19T06:02:46.903 full_node chia.full_node.mempool_manager: INFO     It took 0.0 to pre validate transaction
2021-08-19T06:02:46.903 full_node chia.full_node.mempool_manager: DEBUG    Cost: 10928864
2021-08-19T06:02:47.481 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:48.418 farmer farmer_server              : DEBUG    Inbound_handler task cancelled
2021-08-19T06:02:48.418 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with chia_full_node code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "asyncio\windows_events.py", line 457, in finish_recv
OSError: [WinError 64] 指定的网络名不再可用。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "websockets\protocol.py", line 827, in transfer_data
  File "websockets\protocol.py", line 895, in read_message
  File "websockets\protocol.py", line 971, in read_data_frame
  File "websockets\protocol.py", line 1047, in read_frame
  File "websockets\framing.py", line 105, in read
  File "asyncio\streams.py", line 723, in readexactly
  File "asyncio\streams.py", line 517, in _wait_for_data
  File "asyncio\proactor_events.py", line 280, in _loop_reading
  File "asyncio\windows_events.py", line 812, in _poll
  File "asyncio\windows_events.py", line 461, in finish_recv
ConnectionResetError: [WinError 64] 指定的网络名不再可用。

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "chia\daemon\server.py", line 172, in safe_handle
  File "websockets\protocol.py", line 439, in __aiter__
  File "websockets\protocol.py", line 509, in recv
  File "websockets\protocol.py", line 803, in ensure_open
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-08-19T06:02:48.418 farmer farmer_server              : INFO     Connection closed: 127.0.0.1, node id: 11372a89a6ec3d931cbc8d0a194f35803050d028e1565dba75aceff2f516010e
2021-08-19T06:02:49.434 farmer chia.farmer.farmer         : INFO     peer disconnected None
2021-08-19T06:02:49.434 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-08-19T06:02:49.434 farmer farmer_server              : DEBUG    Connecting: wss://127.0.0.1:8444/ws, Peer info: {'host': '127.0.0.1', 'port': 8444}
2021-08-19T06:02:49.434 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:50.434 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:51.231 farmer farmer_server              : DEBUG    <- farming_info from peer c7e6acd556bd4aae5bf0e08879d6564207c5ef97c64a31b54878790f3522de0b 169.254.2.1
2021-08-19T06:02:51.231 farmer farmer_server              : DEBUG    Time taken to process farming_info from c7e6acd556bd4aae5bf0e08879d6564207c5ef97c64a31b54878790f3522de0b is 0.0 seconds
2021-08-19T06:02:51.450 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:51.450 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001CC39743DC0> [远程计算机拒绝网络连接。]
2021-08-19T06:02:52.466 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:53.466 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:54.450 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-08-19T06:02:54.450 farmer farmer_server              : DEBUG    Connecting: wss://127.0.0.1:8444/ws, Peer info: {'host': '127.0.0.1', 'port': 8444}
2021-08-19T06:02:54.481 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:55.497 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:55.606 farmer farmer_server              : DEBUG    <- farming_info from peer fbef0c601878335af65c28e0269a82ad8d7ad2a3935ba2780abb9d68b7744891 169.254.2.2
2021-08-19T06:02:55.606 farmer farmer_server              : DEBUG    Time taken to process farming_info from fbef0c601878335af65c28e0269a82ad8d7ad2a3935ba2780abb9d68b7744891 is 0.0 seconds
2021-08-19T06:02:55.700 farmer farmer_server              : DEBUG    <- farming_info from peer fbef0c601878335af65c28e0269a82ad8d7ad2a3935ba2780abb9d68b7744891 169.254.2.2
2021-08-19T06:02:55.700 farmer farmer_server              : DEBUG    Time taken to process farming_info from fbef0c601878335af65c28e0269a82ad8d7ad2a3935ba2780abb9d68b7744891 is 0.0 seconds
2021-08-19T06:02:56.497 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001CC39743F40> [远程计算机拒绝网络连接。]
2021-08-19T06:02:56.497 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3
2021-08-19T06:02:57.497 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 3

[cccat6]

I found some similar issues in the other threads. #6305 #2810 #5136 #269
There is no solution yet.
The thing is, the harvester isn't 'crash', it just stops working without any log or exception.

[keliew]

ok, just start only the node..../chai start node does it crash over time?

in my lower end machine, i have to individually start each service, if not, it crashes. once it's running, then it's fine.

which Python version? I think for Windows, you'd need Python 3.9.

[cccat6]

Thank you for your help.
My machine is 5950X so it is not likely the problem of that.
In Windows we do not need to install any Python, the installer is the executable binary.
It should be the problem in exception disposal. Unfortunately, we didn't catch where is the place that causes this problem. I have some guesses but will need some time to check. This problem is annoying because it just happens randomly. So hard to catch.

[BrandtH22]

@cccat6 , the newest version(s) of chia has fixes in place to resolve this issue. Please download it from here: https://www.chia.net/downloads/

Since this issue has been resolved with the latest version(s) of chia we will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions, comments, or if the issue persists after an update.

Please note that upgrading major versions of chia generally requires the config.yaml to be recreated using these steps:
1. Stop chia
2. Rename the config or move it to a different directory
3. In CLI run chia init
4. Copy any custom settings from the old config to the new (plot drive directories, pool info, trusted node info, etc) - be careful here as yaml is very picky about formatting including those leading spaces
5. Start Chia

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.