Crowdsec Helm chart unwanted behaviour

[aleksasiriski]

Moved the discussion from #705 (comment)

I've setup helper ingress to test with crowdsec, without affecting my production services. Will report the logs in case it starts restarting the agents.

[aleksasiriski]

time="09-04-2023 16:00:33" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="09-04-2023 16:00:33" level=info msg="Force add watch on /var/log/containers" type=file
time="09-04-2023 16:00:33" level=warning msg="No matching files for pattern /var/log/containers/traefik-*_traefik_*.log" type=file
time="09-04-2023 16:00:33" level=info msg="Starting processing data"
time="09-04-2023 21:19:48" level=error msg="heartbeat error : API error: signature is invalid"
time="09-04-2023 21:28:03" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": performing jwt auth: dial tcp 10.43.124.153:8080: i/o timeout"```

[mysticaltech]

The outgoing IP seems wrong, this is a local IP, weird huh.

[aleksasiriski]

The outgoing IP seems wrong, this is a local IP, weird huh.

That's correct, agents talk to Crowdsec LAPI whicn is local to the cluster

[aleksasiriski]

@Sneer It stopped working for me now with these logs:

time="09-04-2023 21:27:13" level=warning msg="Loaded 39 scenarios"
time="09-04-2023 21:27:13" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="09-04-2023 21:27:13" level=info msg="Force add watch on /var/log/containers" type=file
time="09-04-2023 21:27:13" level=warning msg="No matching files for pattern /var/log/containers/traefik-*_traefik_*.log" type=file
time="09-04-2023 21:27:13" level=info msg="Starting processing data"
time="09-04-2023 21:28:13" level=error msg="heartbeat error : API error: signature is invalid"
time="10-04-2023 17:18:59" level=error msg="heartbeat error : API error: updating machine last_heartbeat: database is locked: unable to update"
time="10-04-2023 18:37:58" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": performing jwt auth: dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:38:28" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:38:58" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
2023/04/10 18:39:11 Re-opening moved/deleted file /var/log/containers/traefik-566fd9cf6-4nx7d_traefik_traefik-db71f38624c6f40368d2962922a792de6f7cafcab0f4db7af3e3fb3aacce399b.log ...
2023/04/10 18:39:11 Waiting for /var/log/containers/traefik-566fd9cf6-4nx7d_traefik_traefik-db71f38624c6f40368d2962922a792de6f7cafcab0f4db7af3e3fb3aacce399b.log to appear...
time="10-04-2023 18:39:44" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:40:44" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:41:44" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:42:44" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
time="10-04-2023 18:43:45" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"
2023/04/10 18:44:29 Re-opening moved/deleted file /var/log/containers/traefik-566fd9cf6-4nx7d_traefik_traefik-5ec796ec3269fb537e3d7838a3639faac83898f00d23f92a77261cbda9de9f20.log ...
2023/04/10 18:44:29 Waiting for /var/log/containers/traefik-566fd9cf6-4nx7d_traefik_traefik-5ec796ec3269fb537e3d7838a3639faac83898f00d23f92a77261cbda9de9f20.log to appear...
time="10-04-2023 18:44:43" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": could not get jwt token: Post \"http://crowdsec-service.crowdsec:8080/v1/watchers/login\": dial tcp 10.43.124.153:8080: i/o timeout"

crowdsec-traefik-bouncer also has trouble connecting to crowdsec-lapi

2023-04-10T19:23:13Z WRN An error occurred while checking IP "" error="Get \"http://crowdsec-service.crowdsec.svc.cluster.local:8080/v1/decisions?type=ban&ip=REDACTEDIP\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
{"level":"warn","status":403,"method":"GET","path":"/api/v1/forwardAuth","ip":"REDACTEDIP","latency":5004.109095,"user_agent":"Mozilla/5.0 (Windows NT 10.0; rv:100.0) Gecko/20100101 Firefox/100.0","time":"2023-04-10T19:23:13Z","message":"Request"}

[Sneer]

@aleksasiriski How many replicas of lapi do you have?
Default database is SQLite: https://docs.crowdsec.net/docs/local_api/database/ so if you want to have more than one replica of lapi, then you should use another database. It's only idea I have for now.

[aleksasiriski]

I just used https://crowdsecurity.github.io/helm-charts with this config:

   lapi:
      persistentVolume:
        data:
          storageClassName: "longhorn"
        config:
          storageClassName: "longhorn"
    agent:
      acquisition:
        - namespace: traefik
          podName: traefik-*
          program: traefik
      env:
        - name: PARSERS
          value: "crowdsecurity/cri-logs"
        - name: COLLECTIONS
          value: "crowdsecurity/traefik"
        - name: DISABLE_PARSERS
          value: "crowdsecurity/whitelists"

It's a single replice of lapi, so it must be using sqlite. I would love to switch to 3 replices for HA and postgres, will look into how to do it.

[Sneer]

How about free space?
/var/lib/crowdsec/data is a directory with database and if it is running out of space it can couse problems. Mine is 1 GB size and database has 12.2M (testing cluster).

[Sneer]

Now I can see similar error in agent:

time="11-04-2023 01:11:31" level=warning msg="Loaded 40 scenarios"
time="11-04-2023 01:11:31" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="11-04-2023 01:11:31" level=info msg="Force add watch on /var/log/containers" type=file
time="11-04-2023 01:11:31" level=warning msg="No matching files for pattern /var/log/containers/traefik-*_traefik_*.log" type=file
time="11-04-2023 01:11:31" level=info msg="Starting processing data"
time="11-04-2023 01:52:02" level=error msg="heartbeat error : Get \"http://crowdsec-service.crowdsec:8080/v1/heartbeat\": performing jwt auth: dial tcp 10.43.179.103:8080: i/o timeout"

But it looks that it is because of nodes rolling update. Deployment restarted the pod and after that no more errors.

[aleksasiriski]

How about free space? /var/lib/crowdsec/data is a directory with database and if it is running out of space it can couse problems. Mine is 1 GB size and database has 12.2M (testing cluster).

Freshly spun up crowdsec deployment asks for 1GB volume and uses 50MB straight away for me. Will monitor the usage.

But it looks that it is because of nodes rolling update. Deployment restarted the pod and after that no more errors.

So a solution would be to use Postgresql for DB and have more replicas for HA