zabbix-agent 2 issue: failed to process an incoming connection from ...: EOF

[LovingFox]

Connection between server and agent constantly become failed.
Zabbix-Server 7.0.0 is setup without docker, zabbix-agent 2 in the remote host works in docker.

There are not any errors if I use zabbix-agent 1 in docker.

Logs
1.2.3.4 is a server IP
my_host is a name of the agent at the remote host

Agent logs:

docker logs -f zabbix-agent
2024/06/23 06:40:09.166575 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.173052 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.177321 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.184596 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.188862 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.192093 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.195868 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.200036 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.204127 failed to process an incoming connection from 1.2.3.4: EOF
2024/06/23 06:40:09.208074 failed to process an incoming connection from 1.2.3.4: EOF

Server logs the same time:

tail -f /var/log/zabbix/zabbix_server.log
110655:20240623:063913.001 resuming Zabbix agent checks on host "my_host": connection restored
110655:20240623:063921.045 Zabbix agent item "system.cpu.util[,nice]" on host "my_host" failed: first network error, wait for 15 seconds
110655:20240623:063937.147 resuming Zabbix agent checks on host "my_host": connection restored
110655:20240623:063945.018 Zabbix agent item "system.cpu.util[,guest]" on host "my_host" failed: first network error, wait for 15 seconds
110655:20240623:064001.004 resuming Zabbix agent checks on host "my_host": connection restored
110655:20240623:064009.009 Zabbix agent item "vfs.fs.size[/etc/zabbix/agent.pass,pfree]" on host "my_host" failed: first network error, wait for 15 seconds
110655:20240623:064024.973 resuming Zabbix agent checks on host "my_host": connection restored
110655:20240623:064028.247 Zabbix agent item "system.cpu.util[,idle]" on host "my_host" failed: first network error, wait for 15 seconds

---

Versions

Server:

$ /usr/sbin/zabbix_server --version
zabbix_server (Zabbix) 7.0.0
Revision 49955f1fb5c 3 June 2024, compilation time: Jun  3 2024 05:55:33

Copyright (C) 2024 Zabbix SIA
License AGPLv3: GNU Affero General Public License version 3 <https://www.gnu.org/licenses/>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 3.0.9 30 May 2023
Running with OpenSSL 3.0.11 19 Sep 2023

Agent:

sudo docker pull zabbix/zabbix-agent2:ubuntu-7.0.0

---

Configs:

Server:

$ sudo cat /etc/zabbix/zabbix_server.conf | grep -v -E "^#|^$"
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/run/zabbix/zabbix_server.pid
SocketDir=/run/zabbix
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=......
DBSocket=/run/mysqld/mysqld.sock
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
Timeout=4
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
LogSlowQueries=3000
StatsAllowedIP=127.0.0.1
EnableGlobalScripts=0

Agent:

sudo docker run --name zabbix-agent \
  --restart unless-stopped \
  --net=host --pid=host \
  -v /etc/localtime:/etc/localtime:ro \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc/zabbix/agent.pass:/etc/zabbix/agent.pass \
  --privileged --restart=always --init \
  --env ZBX_SERVER_HOST=1.2.3.4 \
  --env ZBX_TLSCONNECT=psk \
  --env ZBX_TLSACCEPT=psk \
  --env ZBX_TLSPSKIDENTITY=my_host \
  --env ZBX_TLSPSKFILE=/etc/zabbix/agent.pass \
  --env ZBX_ACTIVE_ALLOW=false \
  -d zabbix/zabbix-agent2:ubuntu-7.0.0

[dotneft]

Incorrect usage ZBX_TLSPSKFILE. Please check documentation: https://github.com/zabbix/zabbix-docker/tree/7.0/Dockerfiles/agent2#varlibzabbixenc

[LovingFox]

Incorrect usage ZBX_TLSPSKFILE. Please check documentation: https://github.com/zabbix/zabbix-docker/tree/7.0/Dockerfiles/agent2#varlibzabbixenc

As you can see in my config (see below) that I mount the right one file as ZBX_TLSPSKFILE. A TLS-connection is established correctly that means doesn't matter where ZBX_TLSPSKFILE is located. It's wrong with something another...

  -v /etc/zabbix/agent.pass:/etc/zabbix/agent.pass \
...
  --env ZBX_TLSPSKFILE=/etc/zabbix/agent.pass \

[dotneft]

Does not matter, no reason to check something, while it is not properly configured. Specify only file name in the variable and mount the file to /var/lib/zabbix/enc. Then check again and provide full logs since Zabbix agent start with enabled debug mode = 4.

[dotneft]

Also, I recommend you check the problem without TLS connection options.
Do you have direct connection between server and agent? Without any additional services (NAT, load balancers and etc)?

[LovingFox]

Also, I recommend you check the problem without TLS connection options.

Yes, I did some tests and if I switch off TLS all working good, no errors.

Do you have direct connection between server and agent? Without any additional services (NAT, load balancers and etc)?

The host with agent is without any NAT/balancers. The host working as zabbix-server is hosted in AWS VM so that means the 1:1 NAT exists at AWS cloud network side.

Does not matter, no reason to check something, while it is not properly configured. Specify only file name in the variable and mount the file to /var/lib/zabbix/enc. Then check again and provide full logs since Zabbix agent start with enabled debug mode = 4.

Done. The log file is attached: zabbix-agent.log
It's seen that errors logged immediately.

agent run config:

sudo docker run --name zabbix-agent \
  --restart unless-stopped \
  --net=host --pid=host \
  --privileged --restart=always --init \
  -v /etc/localtime:/etc/localtime:ro \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc/zabbix/enc:/var/lib/zabbix/enc \
  --env ZBX_TLSCONNECT=psk \
  --env ZBX_TLSACCEPT=psk \
  --env ZBX_TLSPSKIDENTITY=my_host \
  --env ZBX_TLSPSKFILE=agent.pass \
  --env ZBX_SERVER_HOST=1.2.3.4 \
  --env ZBX_ACTIVE_ALLOW=false \
  --env ZBX_DEBUGLEVEL=4 \
  -d zabbix/zabbix-agent2:ubuntu-7.0.0

[dotneft]

Thank you for the explanation. Currently I see the following steps to check the problem:

1. Collect tcpdump on agent side (filter by 10050 port) and send it to my mailbox (you can find in each Dockerfile the address)
2. Try to install Zabbix agent 2 directly on the host and check without docker.

[justinclift]

As a data point, the same problem (many failed to process an incoming connection entries in the zabbix agent log) is happening in non-docker setups too:

https://support.zabbix.com/browse/ZBX-22351

Saying that as it's happening with my systems (not in docker) currently. Both on host servers (physical hardware) and VMs running on those servers. 😦

[justinclift]

Checking one of the host servers presently, the zabbix agent logs are 99.9% these messages:

# wc -l zabbix_agent2.log
540778 zabbix_agent2.log
# grep 'failed to process an incoming connection from' zabbix_agent2.log | wc -l
529908

So of the 540k lines in the log file so far today, 530k are this message being repeated over and over.

[dotneft]

do you monitor "net.tcp.service[tcp,,10050]"?

[justinclift]

No, I have no idea what that is. If that's some custom setting, then there's definitely no chance it was present.

In the meantime, I've rebuilt the entire setup to use Zabbix 6.0 LTS instead which isn't exhibiting the problem.

[matzmz]

I'm still having the same issue:

docker.io/zabbix/zabbix-agent2:alpine-6.4.13

Also other people using zabbix container (monitoring k8s cluster) have similar issue.
Any idea ?

[justinclift]

Any idea ?

Is downgrading to Zabbix 6.0 feasible?

[matzmz]

@justinclift Unfortunately, I am unable to downgrade to version 6.0.

[dotneft]

what image was working fine?

[rzemykers]

I tried many of agents (7.0.X, 6.0.X) on my k8s cluster and no results.. every time i get failed to process an incoming connection from...
I have 7.0.2 server at debian 12.