if i run commands in a script daemon always fails with 1006 : #7710
Replies: 5 comments 6 replies
-
|
i also noticed that if all you do is just tail the log and ctrl-c the tail (not the daemon process) the daemon sometimes stops with this in the log : 2021-07-22T06:32:32.010 full_node chia.full_node.mempool_manager: INFO It took 0.003361225128173828 to pre validate transaction |
Beta Was this translation helpful? Give feedback.
-
|
so i wonder if some of the issues people report on closed sockets is related to log handling ? |
Beta Was this translation helpful? Give feedback.
-
|
"its repeatable 100% of the time. if i run all commands by hand it works". Why: What's "echo ./start.sh" ? Why sleep 2 before killing? The daemon told you he's waiting 5 secs so I'd opt for sleeping at least 5. |
Beta Was this translation helpful? Give feedback.
-
|
I tried with and without nohub , makes no difference. I added the nohub as i suspected there maybe an issue with output handling. Even if you remove all the stop logic of the script i get 100% failures by starting it in a script. |
Beta Was this translation helpful? Give feedback.
-
|
Chia processes are already demonized, you won't need nohup. The stop is async but 2 secs seems too short, looks like you kill processes while daemon is stil pending on their sockets and thus produce the i/o exceptions. Don't sleep after the start (no nohup), try sleep 5 before kill. |
Beta Was this translation helpful? Give feedback.
-
|
Also no need for sudo in the kill since current uid owns the processes. If you enhance your system with more than 1 harvester running under separate uids, as when you farm chia forks, with sudo you'll kill all of them. |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean tail -f .chia/mainnet/log/debug.log ? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, i mean ctrl-c the tail. I know this sounds like a red herring but i can provoke a shutdown with this all the time if the tail starts at the same time i start chia like : chia start all ; tail -F logfile If i ctrl-c the tail a few sec later the daemon goes down |
Beta Was this translation helpful? Give feedback.
-
|
If you ctrl-c before chia start returns, you're actually killing chia start, not tail. If you wait for the chia start to finish, then you kill the tail alone. Wait until you see tail output lines before signaling. |
Beta Was this translation helpful? Give feedback.
-
|
I run multiple instances of chia each in a separate container , i will change the sleep before the kill to 10 but as i stated the main issue is not the stop it's the start that's causing issues |
Beta Was this translation helpful? Give feedback.
-
|
here is an example of the output and the error i am seeing when i ctrl-c the tail : 2021-07-26T15:46:45.310 timelord chia.timelord.timelord_state: INFO Updated timelord peak to 472963b85141c98772529c04c23d824c82e4926e53e9fbc70f2c615299ff1c5b, total iters: 2159866414213 ^Croot@chia-W18-local-e:~# --- Logging error --- this farmer was running for over a day with a tail following its log, as soon as i ctrl-c the tail the daemon stops with |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
can't automate start of chia, if i run commands in a script daemon always fails with 1006 :
cd
source chia-blockchain/activate
chia stop all -d
sleep 2
sudo killall -9 chia_harvester chia_timelord_l chia_timelord chia_farmer chia_wallet chia_full_node
cd chia-blockchain
deactivate
git fetch
git checkout latest
git reset --hard FETCH_HEAD --recurse-submodule
sh install.sh
cd
source ~/chia-blockchain/activate
nohup chia start all < /dev/null > /dev/null 2>&1 &
sleep 10
cd ~/chiadog
echo ./start.sh
2021-07-21T15:32:38.755 daemon chia.daemon.server : INFO Daemon WebSocketServer closed
2021-07-21T15:35:39.616 daemon chia.daemon.server : INFO Starting Daemon Server
2021-07-21T15:35:39.619 daemon chia.daemon.server : INFO Waiting Daemon WebSocketServer closure
2021-07-21T15:35:42.143 daemon chia.daemon.server : INFO Register service {'service': 'chia_harvester'}
2021-07-21T15:35:42.144 daemon chia.daemon.server : INFO registered for service chia_harvester
2021-07-21T15:35:42.144 daemon chia.daemon.server : INFO {'success': True}
2021-07-21T15:35:44.849 daemon chia.daemon.server : INFO Register service {'service': 'chia_full_node'}
2021-07-21T15:35:44.849 daemon chia.daemon.server : INFO registered for service chia_full_node
2021-07-21T15:35:44.849 daemon chia.daemon.server : INFO {'success': True}
2021-07-21T15:35:44.959 daemon chia.daemon.server : INFO Register service {'service': 'chia_farmer'}
2021-07-21T15:35:44.959 daemon chia.daemon.server : INFO registered for service chia_farmer
2021-07-21T15:35:44.959 daemon chia.daemon.server : INFO {'success': True}
2021-07-21T15:35:46.481 daemon chia.daemon.server : INFO Register service {'service': 'chia_wallet'}
2021-07-21T15:35:46.583 daemon chia.daemon.server : INFO registered for service chia_wallet
2021-07-21T15:35:46.584 daemon chia.daemon.server : INFO {'success': True}
2021-07-21T15:39:43.759 daemon chia.daemon.server : INFO sending term signal to chia_timelord_launcher
2021-07-21T15:39:43.859 daemon chia.daemon.server : INFO process chia_timelord_launcher returned 1
2021-07-21T15:39:43.874 daemon chia.daemon.server : INFO sending term signal to chia_wallet
2021-07-21T15:39:43.874 daemon chia.daemon.server : INFO sending term signal to chia_full_node
2021-07-21T15:39:43.874 daemon chia.daemon.server : INFO sending term signal to chia_timelord
2021-07-21T15:39:43.875 daemon chia.daemon.server : INFO sending term signal to chia_farmer
2021-07-21T15:39:43.875 daemon chia.daemon.server : INFO sending term signal to chia_harvester
2021-07-21T15:39:44.376 daemon chia.daemon.server : INFO process chia_timelord returned 0
2021-07-21T15:39:46.378 daemon chia.daemon.server : INFO process chia_farmer returned 0
2021-07-21T15:39:46.880 daemon chia.daemon.server : INFO process chia_wallet returned 0
2021-07-21T15:39:46.961 daemon chia.daemon.server : INFO process chia_full_node returned 0
2021-07-21T15:39:59.075 daemon chia.daemon.server : INFO sending kill signal to chia_harvester
2021-07-21T15:39:59.092 daemon chia.daemon.server : INFO process chia_harvester returned -9
2021-07-21T15:39:59.092 daemon chia.daemon.server : INFO chia daemon exiting in 5 seconds
2021-07-21T15:39:59.095 daemon chia.daemon.server : INFO Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
2021-07-21T15:39:59.095 daemon chia.daemon.server : INFO Daemon WebSocketServer closed
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
message = await self.read_message()
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
frame = await self.read_frame(max_size)
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
frame = await Frame.read(
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
data = await reader(2)
File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/chia/daemon/server.py", line 172, in safe_handle
async for message in websocket:
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in aiter
yield await self.recv()
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
await self.ensure_open()
File "/root/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
2021-07-21T15:39:59.095 daemon chia.daemon.server : INFO Daemon WebSocketServer closed
its repeatable 100% of the time. if i run all commands by hand it works.
Beta Was this translation helpful? Give feedback.
All reactions