EngineClient: properly handle abort flag on close #591
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
StreamWorker.abort() calls Engine.remove() with abort set to True but this flag was never actually used. The problem is that EngineClient._close_stream() is then called and flushes any read buffers, which can lead to unexpected read events and messing with the user's code logic (StreamWorker is used by the Gateway and this issue was discovered there). With this patch, we ensure that worker.abort() actually aborts all engine events. Just a last note that we keep the current behavior when both abort and timeout are set to True in Engine.remove()/EngineClient._close(). In that case, timeout=True prevails as it is Engine-induced and we consider we can flush any read buffers (so potentially generating a few ev_read events).
|
For the record, discovered at the Gateway level when troubleshooting gateway abort/termination. In the trace below we can see that upon receiving bad XML data, the Traceback (most recent call last):
File "/usr/lib64/python3.9/xml/sax/expatreader.py", line 217, in feed
self._parser.Parse(data, isFinal)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 6, column 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/ClusterShell/Communication.py", line 234, in ev_read
self._parser.feed(msg + b'\n')
File "/usr/lib64/python3.9/xml/sax/expatreader.py", line 221, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python3.9/xml/sax/handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: <unknown>:6:1: not well-formed (invalid token)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/ClusterShell/Gateway.py", line 187, in recv
self.close()
File "/usr/lib/python3.9/site-packages/ClusterShell/Gateway.py", line 179, in close
self._close()
File "/usr/lib/python3.9/site-packages/ClusterShell/Communication.py", line 212, in _close
self.worker.abort()
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 581, in abort <<<
self.clients[0].abort()
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/EngineClient.py", line 438, in abort
engine.remove(self, abort=True)
File "/usr/lib/python3.9/site-packages/ClusterShell/Engine/Engine.py", line 495, in remove
self._remove(client, abort, did_timeout)
File "/usr/lib/python3.9/site-packages/ClusterShell/Engine/Engine.py", line 483, in _remove
client._close(abort=abort, timeout=did_timeout)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 405, in _close
EngineClient._close(self, abort, timeout)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/EngineClient.py", line 267, in _close
self._close_stream(sname)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/EngineClient.py", line 276, in _close_stream
self._flush_read(sname)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 438, in _flush_read
self.worker._on_msgline(self.key, stream.rbuf, sname)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 563, in _on_msgline
self.eh.ev_read(self, key, sname, msg)
File "/usr/lib/python3.9/site-packages/ClusterShell/Communication.py", line 244, in ev_read <<<
self.send(ErrorMessage('Parse error: %s' % ex.getMessage()))
File "/usr/lib/python3.9/site-packages/ClusterShell/Communication.py", line 270, in send
self.worker.write(msg.xml() + b'\n', sname=self.SNAME_WRITER)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 601, in write
self.clients[0].write(buf, sname)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/Worker.py", line 443, in write
self._write(sname, buf)
File "/usr/lib/python3.9/site-packages/ClusterShell/Worker/EngineClient.py", line 408, in _write
wfile = self.streams[sname]
KeyError: 'ch-writer' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
StreamWorker.abort()callsEngine.remove()withabortset toTruebut this flag was never actually used. The problem is thatEngineClient._close_stream()is then called and flushes any read buffers, which can lead to unexpected read events and messing with the user's code logic (StreamWorkeris used by theGatewayand this issue was discovered there).With this patch, we ensure that
worker.abort()actually aborts all engine events.Just a last note that we keep the current behavior when both abort and timeout are set to True in
Engine.remove()/EngineClient._close(). In that case,timeout=Trueprevails as it is Engine-induced and we consider we can flush any read buffers (so potentially generating a fewev_readevents).