Conpot stops functioning after nmap scans

[bestrocker221]

Conpot does not work anymore after nmap scanning. Both through the container and not.
The process keeps running, but the ports do not react anymore and the logging is stopped.

This unfortunately makes the honeypot pretty much useless since it cannot withstand a basic nmap scan.

To Reproduce

1. docker-compose up
2. nmap -sV -v 127.0.0.1
3. see few logs and maybe the scan terminates.
4. scan can also not terminate, and conpot is unresponsive, all future port probes will return no answer. No further logs.

Expected behavior
Conpot keeps running, with the logs saved of each connection attempt.

Any help?

Thank you!

[Vingaard]

Interesting - a quick question. what happens if you scan the Conpot from anything else than 127.0.0..1 (localhost)
like nmap scan:
nmap -sV -v {inset conpot_ip}

[bestrocker221]

Same exact behavior.

…

On Tue, Jan 4, 2022, 10:59 Vingaard ***@***.***> wrote: Interesting - a quick question. what happens if you scan the Conpot from anything else than 127.0.0..1 (localhost) like nmap scan: nmap -sV -v {inset conpot_ip} — Reply to this email directly, view it on GitHub <#564 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXQXIGDMOPMELFVU2S4AILUULAI3ANCNFSM5LGB75IA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[bestrocker221]

Anybody confirming the same behavior or found an alternative way to solve it?

[glaslos]

I've just checked out the latest code from GitHub, run docker-compose up and then nmap -sV -v 127.0.0.1 twice without issues. CPU usage was pretty much at 100% during the nmap scan. Scan took about 150 seconds.

[bestrocker221]

Strange, I tried both the docker-compose and through the virtualenv, on fresh new Debian and Arch systems and I got the explained issue.

I found many places in the code where the issue makes sense, the lack of sockets closing and error handling during loops. Gonna make a PR when finished. Now it seems to deterministically work

[t3chn0m4g3]

Maybe this is unrelated to the issues title, however I have been noticing conpot guardian_ast and iec104 to keep running at 100% CPU usage after some connection attempts being made.

Example guardian_ast:
{"timestamp": "2022-03-02T04:45:06.688727", "sensorid": "conpot", "id": "ce4b97df-bc04-416f-b5af-c7654d1d7d2b", "src_ip": "23.106.58.171", "src_port": 58678, "dst_ip": "172.23.0.2", "dst_port": 10001, "public_ip": "xx.xx.46.201", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "NEW_CONNECTION"}

Example iec104:
{"timestamp": "2022-03-02T09:49:38.909285", "sensorid": "conpot", "id": "efb38210-1ddc-4ff9-8468-0eaffed92b4b", "src_ip": "164.52.24.169", "src_port": 53032, "dst_ip": "172.22.0.2", "dst_port": 2404, "public_ip": "xx.xx.46.201", "data_type": "IEC104", "request": null, "response": null, "event_type": "NEW_CONNECTION"}

Conpot behavior changes afterwards. While connections still can be made there is no logging and no functionality.

This is based on git commit 1c2382e. Still have to test latest master.

Update: I can confirm to reproduce this behavior with nmap -sV -v 127.0.0.1, still have to test with latest master. Since @glaslos confirmed it works fine I am guessing that some library was updated, and I found one other honeypot showing the same behavior.

[t3chn0m4g3]

@glaslos

Built conpot from latest master and can now confirm that problems with iec104 are fixed, while guardian_ast repeatedly shows 100% CPU usage after nmap and leaves it without function.

It is reproducible with nmap -p 10001 -sV -v 127.0.0.1.

Console log:

2022-03-09 14:02:37,741 Config file found!
2022-03-09 14:02:37,743 Starting Conpot using template: /usr/lib/python3.9/site-packages/conpot/templates/guardian_ast
2022-03-09 14:02:37,743 Starting Conpot using configuration found in: /etc/conpot/conpot.cfg
2022-03-09 14:02:37,750 Serving tar:///usr/lib/python3.9/site-packages/conpot/data.tar as file system. File uploads will be kept at : /tmp/conpot
2022-03-09 14:02:37,750 Opening path /tmp/conpot for persistent storage of files.
2022-03-09 14:02:37,751 Initializing Virtual File System at /tmp/conpot/__conpot__hi0s02wj. Source specified : tar:///usr/lib/python3.9/site-packages/conpot/data.tar
 Please wait while the system copies all specified files
2022-03-09 14:02:37,818 Fetched 1.2.3.4 as external ip.
2022-03-09 14:02:37,819 Conpot GuardianAST initialized
2022-03-09 14:02:37,819 Found and enabled guardian_ast protocol.
2022-03-09 14:02:37,820 No proxy template found. Service will remain unconfigured/stopped.
2022-03-09 14:02:37,820 GuardianAST server started on: ('0.0.0.0', 10001)
2022-03-09 14:04:14,067 New guardian_ast session from 192.168.208.1 (7f0c9159-271d-432c-9e01-2c5eb1d75110)
2022-03-09 14:04:14,067 New GuardianAST connection from 192.168.208.1:60070. (7f0c9159-271d-432c-9e01-2c5eb1d75110)
2022-03-09 14:04:20,071 Non ^A command attempt 192.168.208.1:60070. (7f0c9159-271d-432c-9e01-2c5eb1d75110)
2022-03-09 14:04:20,071 GuardianAST client disconnected 192.168.208.1:60070. (7f0c9159-271d-432c-9e01-2c5eb1d75110)
2022-03-09 14:04:20,073 New GuardianAST connection from 192.168.208.1:60074. (7f0c9159-271d-432c-9e01-2c5eb1d75110)

JSON log:

{"timestamp": "2022-03-09T12:03:49.935986", "sensorid": "conpot", "id": "c1d7e4be-0a27-46fa-88f6-18d65f2deb1b", "src_ip": "192.168.128.1", "src_port": 52392, "dst_ip": "192.168.128.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "NEW_CONNECTION"}
{"timestamp": "2022-03-09T12:03:49.935986", "sensorid": "conpot", "id": "c1d7e4be-0a27-46fa-88f6-18d65f2deb1b", "src_ip": "192.168.128.1", "src_port": 52392, "dst_ip": "192.168.128.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "CONNECTION_LOST"}
{"timestamp": "2022-03-09T12:03:49.935986", "sensorid": "conpot", "id": "c1d7e4be-0a27-46fa-88f6-18d65f2deb1b", "src_ip": "192.168.128.1", "src_port": 52392, "dst_ip": "192.168.128.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "NEW_CONNECTION"}
{"timestamp": "2022-03-09T14:04:14.067252", "sensorid": "conpot", "id": "7f0c9159-271d-432c-9e01-2c5eb1d75110", "src_ip": "192.168.208.1", "src_port": 60070, "dst_ip": "192.168.208.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "NEW_CONNECTION"}
{"timestamp": "2022-03-09T14:04:14.067252", "sensorid": "conpot", "id": "7f0c9159-271d-432c-9e01-2c5eb1d75110", "src_ip": "192.168.208.1", "src_port": 60070, "dst_ip": "192.168.208.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "CONNECTION_LOST"}
{"timestamp": "2022-03-09T14:04:14.067252", "sensorid": "conpot", "id": "7f0c9159-271d-432c-9e01-2c5eb1d75110", "src_ip": "192.168.208.1", "src_port": 60070, "dst_ip": "192.168.208.2", "dst_port": 10001, "public_ip": "1.2.3.4", "data_type": "guardian_ast", "request": null, "response": null, "event_type": "NEW_CONNECTION"}

Happy to provide debug logs, just let me know what you need.

The only thing I noticed is, that every time I stop conpot_guardian_ast after it keeps running with 100% CPU I get the following error ...
Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/usr/lib/python3.9/site-packages/gevent/baseserver.py", line 34, in _handle_and_close_when_done return handle(*args_tuple) File "/usr/lib/python3.9/site-packages/conpot/protocols/guardian_ast/guardian_ast_server.py", line 327, in handle request += sock.recv(4096)
... which is not happening when being stopped otherwise.

[bestrocker221]

In my latest local version all the protocols in the default template works correctly after multiple -sV scans. I just need to find a proper way to make a PR since I am making modifications directly in my virtualenv dev environment.

[t3chn0m4g3]

@bestrocker221 Interesting, default shows no sign of 100% CPU usage (after nmap is finished) on my end with latest master.

[t3chn0m4g3]

Behavior is the same with Py 3.7/3.8/3.9 with Alpine and Ubuntu as Docker images.

Dirty, but works fine if docker-compose uses restart=always:

STOPSIGNAL SIGINT
HEALTHCHECK CMD if [ $(ps -C mpv -p 1 -o %cpu | tail -n 1 | cut -f 1 -d ".") -gt 75 ]; then kill -2 1; else exit 0; fi

• -p 1 Monitors PID 1 only and kills process if process CPU usage is > 75%.

Keeping 🤞 for a solution, though 😄

[bestrocker221]

@t3chn0m4g3 Hi, the solution might work but is just a workaround that does not actually fix the code behind it.

I left conpot with default template running for a week and it eventually stopped functioning, now I checked and it was using 100% CPU, but since it is one single process I can't distinguish without a debugger what is going on.. need more testing.

Restarting the process as you say can work, but is indeed "dirty".

[t3chn0m4g3]

@glaslos @xandfury Can you reproduce? Can I provide logs to help solving this?

[Kevalz95]

Hey @t3chn0m4g3 @bestrocker221 @glaslos , I know that is quite an old issue but I've just found a solution for the Modbus protocol handler freezing with NMAP aggressive scans (nmap -A and / or nmap -sV).

Basically, after some debugging, I've found out that the problem relies in the handle() method of conpot/protocols/modbus/modbus_server.py.

The main problem relies in the following while loop that downloads the Modbus TCP request from the connecting socket

_, _, length = struct.unpack(">HHH", request[:6])
while len(request) < (length + 6):
    try:
        new_byte = sock.recv(1)
        request += new_byte
    except Exception:
        break

It seems that, for some NMAP modbus requests, such while loop results into an infinite loop freezing the whole server and spiking the CPU usage to 100%!

More deeply, the new_byte = sock.recv(1) line is not considering the case where the sock.recv() doesn't receive any NEW byte and results with the while condition len(request) < (length + 6) never been achieved.

I've implemented and tested a fix for such issue by introducing a new variable called previous_request that, as the name say, it just stores the N-1 byte parsed by the socket and, if the bytes is the same for the following N+1000 received bytes, it just break the loop.

previous_request = None
stall_counter = 0
while len(request) < (length + 6):
    try:
        if previous_request and len(previous_request) == len(request):
            stall_counter += 1
            if stall_counter >= 1000:
                logger.info("Modbus client provided data {} but invalid.".format(session.id))
                session.add_event({"type": "CONNECTION_TERMINATED"})
                break
        new_byte = sock.recv(1)
        request += new_byte
        previous_request = request
    except Exception:
        break

I've tested such implementation against a NMAP Modbus scan (that was previously freezing the whole server) and it seems to work pretty well.

Hope it helps :)

[t3chn0m4g3]

Sounds great! Can you make it a PR?