Goroutine leak

[g00g1]

What happened?

I am running a crowdsec setup with decent amount of decisions (100k simultaneously) with Prometheus metrics enabled and #4080 patch applied. Goroutine leak happens both before and after mentioned patch was applied.

Prometheus metrics

What did you expect to happen?

No growth in goroutines count and low stack memory usage

How can we reproduce it (as minimally and precisely as possible)?

No idea. If you need specific additional debugging - please let me know.

Anything else we need to know?

No response

Crowdsec version

$ cscli version
version: v1.7.0-rpm-pragmatic-amd64-c3036e21
Codename: alphaga
BuildDate: 2025-09-02_12:39:37
GoVersion: 1.24.6
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.7.0-rpm-pragmatic-amd64-c3036e21-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog

OS version

$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.7 (Moss Jungle Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.7"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.7 (Moss Jungle Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.7"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.7"
SUPPORT_END=2032-06-01
$ uname -a
Linux [REDACTED] 5.14.0-570.28.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jul 22 07:56:01 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
Loaded: 153 parsers, 11 postoverflows, 768 scenarios, 9 contexts, 5 appsec-configs, 137 appsec-rules, 145 collections
Unmanaged items: 3 local, 1 tainted
name,status,version,description,type
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/endlessh-logs,"enabled,local",,,parsers
crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/public-dns-allowlist,enabled,0.1,Allow events from public DNS servers,parsers
crowdsecurity/sshd-logs,enabled,3.1,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.9,,parsers
crowdsecurity/whitelists,"enabled,tainted",?,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/whitelists-allowed-scanners,"enabled,local",,,parsers
crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows
crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows
crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios
crowdsecurity/ssh-generic-test,enabled,0.2,Crowdsec Generic Test Scenario: SSH brute force trigger,scenarios
crowdsecurity/ssh-refused-conn,enabled,0.1,Detect sshd refused connections,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
skhron/endlessh-distinctconn,"enabled,local",,,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/linux,enabled,0.3,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.7,sshd support : parser and brute-force detection,collections
crowdsecurity/whitelist-good-actors,enabled,0.2,Good actors whitelists,collections

Acquisition config

$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
source: journalctl
journalctl_filter:
 - "_SYSTEMD_UNIT=endlessh@22.service"
labels:
  type: syslog
---

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Notification Folder    : /etc/crowdsec/notifications
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : error
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://[::1]:8080/
  - Login                   : e9f7f4e827034fd59271a062e2d8a2d3AK3KwObeypixPPUb
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : [::1]:8080
  - Listen Socket           : 
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : pgx
      - Host                : 
      - Port                : 0
      - User                : crowdsec
      - DB Name             : crowdsec
      - Max Open Conns      : 10
      - Decision Bulk Size  : 2000
      - Flush age           : 2160h0m0s
      - Flush size          : 3000000

Prometheus metrics

$ cscli metrics
{
 "acquisition": {
  "journalctl:journalctl-%s_SYSTEMD_UNIT=endlessh@22.service": {
   "parsed": 3164602,
   "pour": 856476,
   "reads": 6325986,
   "unparsed": 3161384
  }
 },
 "alerts": {
  "[REDACTED]": 36,
  "[REDACTED]": 1,
  "[REDACTED]": 348,
  "skhron/endlessh-distinctconn": 408058,
  "[REDACTED]": 11,
  "ti-as12332": 121,
  "ti-as12389": 20518,
  "ti-as131414": 38,
  "ti-as132203": 14163,
  "ti-as13335": 1476,
  "ti-as133692": 86,
  "ti-as134196": 1,
  "ti-as135089": 601,
  "ti-as135377": 542508,
  "ti-as135752": 6,
  "ti-as136052": 846,
  "ti-as137718": 3640,
  "ti-as138915": 95,
  "ti-as138968": 6,
  "ti-as14061": 1123578,
  "ti-as140787": 20,
  "ti-as142002": 15,
  "ti-as14956": 639,
  "ti-as152194": 78,
  "ti-as197414": 1212,
  "ti-as197450": 17,
  "ti-as198584": 11,
  "ti-as199785": 175,
  "ti-as200373": 744,
  "ti-as202302": 4,
  "ti-as202425": 8345,
  "ti-as203999": 1,
  "ti-as204490": 4,
  "ti-as204552": 2,
  "ti-as205809": 2,
  "ti-as206728": 19,
  "ti-as207159": 2,
  "ti-as207957": 23,
  "ti-as208137": 25,
  "ti-as208843": 4402,
  "ti-as209605": 10,
  "ti-as209847": 956,
  "ti-as210240": 1,
  "ti-as210558": 1,
  "ti-as210644": 448,
  "ti-as210705": 2,
  "ti-as211138": 15,
  "ti-as211522": 196,
  "ti-as211659": 353,
  "ti-as211720": 7,
  "ti-as211955": 3,
  "ti-as212283": 4,
  "ti-as212701": 5,
  "ti-as213035": 5,
  "ti-as213373": 2,
  "ti-as213389": 10,
  "ti-as213438": 1449,
  "ti-as213702": 6,
  "ti-as214196": 2,
  "ti-as214927": 4,
  "ti-as214967": 13,
  "ti-as215117": 9,
  "ti-as215311": 10,
  "ti-as215730": 48,
  "ti-as215826": 22,
  "ti-as215925": 12,
  "ti-as216127": 4,
  "ti-as216246": 9,
  "ti-as23679": 102,
  "ti-as23724": 901,
  "ti-as24544": 12,
  "ti-as266702": 1,
  "ti-as269926": 1,
  "ti-as33993": 12,
  "ti-as35346": 8,
  "ti-as35478": 2,
  "ti-as35830": 24,
  "ti-as36352": 4019,
  "ti-as37963": 8418,
  "ti-as396982": 19,
  "ti-as399471": 1,
  "ti-as400992": 44,
  "ti-as401696": 273,
  "ti-as401701": 20,
  "ti-as4134": 155626,
  "ti-as42237": 48,
  "ti-as42397": 704,
  "ti-as43444": 32,
  "ti-as43668": 5,
  "ti-as44208": 101,
  "ti-as44477": 307,
  "ti-as44559": 3,
  "ti-as45102": 14232,
  "ti-as45899": 16922,
  "ti-as4766": 13437,
  "ti-as4788": 18301,
  "ti-as47890": 2659,
  "ti-as4808": 2766,
  "ti-as4811": 1483,
  "ti-as4837": 109429,
  "ti-as48693": 11,
  "ti-as5065": 12,
  "ti-as51124": 1,
  "ti-as51167": 17088,
  "ti-as51396": 165,
  "ti-as51447": 1,
  "ti-as53958": 1,
  "ti-as54098": 8535,
  "ti-as55933": 275,
  "ti-as57509": 2,
  "ti-as58955": 13,
  "ti-as59651": 161,
  "ti-as60223": 107,
  "ti-as62281": 3,
  "ti-as62864": 1,
  "ti-as7713": 28445,
  "ti-as8048": 2442,
  "ti-as8075": 14372,
  "ti-as8452": 11130,
  "ti-as9340": 30,
  "ti-as9829": 19821,
  "ti-binaryedge.ninja": 1576,
  "ti-cyberresilience.io": 103,
  "ti-dtu.dk": 1,
  "ti-googleusercontent.com": 98329,
  "ti-internet-census.org": 15908,
  "ti-internet-measurement.com": 1331,
  "ti-kaspersky-labs.com": 3,
  "ti-leakix.org": 14,
  "ti-modat.io": 214,
  "ti-onyphe.net": 30,
  "ti-ruhr-uni-bochum.de": 1,
  "ti-shodan.io": 32,
  "ti-spamhaus_dropv4": 264873,
  "ti-stretchoid.com": 29510,
  "ti-tu-dresden.de": 5
 },
 "appsec-engine": {},
 "appsec-rule": {},
 "bouncers": {
  "[REDACTED]": {
   "": {
    "processed": {
     "byte": 0,
     "packet": 0
    }
   },
   "crowdsec-blacklists": {
    "active_decisions": {
     "ip": 109253
    }
   },
   "crowdsec6-blacklists": {
    "active_decisions": {
     "ip": 39
    }
   }
  }
 },
 "decisions": {
  "[REDACTED]": {
   "cscli": {
    "ban": 1
   }
  },
  "[REDACTED]": {
   "cscli": {
    "ban": 21
   }
  },
  "skhron/endlessh-distinctconn": {
   "crowdsec": {
    "ban": 34710
   }
  },
  "ti-as12332": {
   "cscli": {
    "ban": 8
   }
  },
  "ti-as12389": {
   "cscli": {
    "ban": 1831
   }
  },
  "ti-as131414": {
   "cscli": {
    "ban": 6
   }
  },
  "ti-as132203": {
   "cscli": {
    "ban": 498
   }
  },
  "ti-as13335": {
   "cscli": {
    "ban": 158
   }
  },
  "ti-as133692": {
   "cscli": {
    "ban": 18
   }
  },
  "ti-as135377": {
   "cscli": {
    "ban": 2180
   }
  },
  "ti-as135752": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as136052": {
   "cscli": {
    "ban": 141
   }
  },
  "ti-as137718": {
   "cscli": {
    "ban": 772
   }
  },
  "ti-as138915": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as138968": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as14061": {
   "cscli": {
    "ban": 14573
   }
  },
  "ti-as140787": {
   "cscli": {
    "ban": 9
   }
  },
  "ti-as142002": {
   "cscli": {
    "ban": 5
   }
  },
  "ti-as14956": {
   "cscli": {
    "ban": 33
   }
  },
  "ti-as152194": {
   "cscli": {
    "ban": 12
   }
  },
  "ti-as197450": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as198584": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as199785": {
   "cscli": {
    "ban": 12
   }
  },
  "ti-as200373": {
   "cscli": {
    "ban": 556
   }
  },
  "ti-as202425": {
   "cscli": {
    "ban": 200
   }
  },
  "ti-as204552": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as205809": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as206728": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as207957": {
   "cscli": {
    "ban": 7
   }
  },
  "ti-as208137": {
   "cscli": {
    "ban": 24
   }
  },
  "ti-as208843": {
   "cscli": {
    "ban": 340
   }
  },
  "ti-as209847": {
   "cscli": {
    "ban": 39
   }
  },
  "ti-as210644": {
   "cscli": {
    "ban": 20
   }
  },
  "ti-as210705": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as211138": {
   "cscli": {
    "ban": 5
   }
  },
  "ti-as211522": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as212283": {
   "cscli": {
    "ban": 3
   }
  },
  "ti-as213035": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as213373": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as213389": {
   "cscli": {
    "ban": 3
   }
  },
  "ti-as213438": {
   "cscli": {
    "ban": 6
   }
  },
  "ti-as213702": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as215730": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as215925": {
   "cscli": {
    "ban": 3
   }
  },
  "ti-as216127": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as23679": {
   "cscli": {
    "ban": 6
   }
  },
  "ti-as23724": {
   "cscli": {
    "ban": 93
   }
  },
  "ti-as24544": {
   "cscli": {
    "ban": 3
   }
  },
  "ti-as266702": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as35346": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as35830": {
   "cscli": {
    "ban": 20
   }
  },
  "ti-as36352": {
   "cscli": {
    "ban": 214
   }
  },
  "ti-as37963": {
   "cscli": {
    "ban": 695
   }
  },
  "ti-as396982": {
   "cscli": {
    "ban": 4
   }
  },
  "ti-as399471": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as401696": {
   "cscli": {
    "ban": 21
   }
  },
  "ti-as401701": {
   "cscli": {
    "ban": 10
   }
  },
  "ti-as4134": {
   "cscli": {
    "ban": 31618
   }
  },
  "ti-as42237": {
   "cscli": {
    "ban": 9
   }
  },
  "ti-as44208": {
   "cscli": {
    "ban": 18
   }
  },
  "ti-as45102": {
   "cscli": {
    "ban": 1653
   }
  },
  "ti-as45899": {
   "cscli": {
    "ban": 778
   }
  },
  "ti-as4766": {
   "cscli": {
    "ban": 3051
   }
  },
  "ti-as4788": {
   "cscli": {
    "ban": 667
   }
  },
  "ti-as47890": {
   "cscli": {
    "ban": 8
   }
  },
  "ti-as4808": {
   "cscli": {
    "ban": 221
   }
  },
  "ti-as4811": {
   "cscli": {
    "ban": 111
   }
  },
  "ti-as4837": {
   "cscli": {
    "ban": 20707
   }
  },
  "ti-as5065": {
   "cscli": {
    "ban": 4
   }
  },
  "ti-as51167": {
   "cscli": {
    "ban": 205
   }
  },
  "ti-as53958": {
   "cscli": {
    "ban": 1
   }
  },
  "ti-as54098": {
   "cscli": {
    "ban": 1519
   }
  },
  "ti-as55933": {
   "cscli": {
    "ban": 31
   }
  },
  "ti-as59651": {
   "cscli": {
    "ban": 41
   }
  },
  "ti-as60223": {
   "cscli": {
    "ban": 30
   }
  },
  "ti-as62281": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-as7713": {
   "cscli": {
    "ban": 1784
   }
  },
  "ti-as8048": {
   "cscli": {
    "ban": 279
   }
  },
  "ti-as8075": {
   "cscli": {
    "ban": 613
   }
  },
  "ti-as8452": {
   "cscli": {
    "ban": 1575
   }
  },
  "ti-as9340": {
   "cscli": {
    "ban": 7
   }
  },
  "ti-as9829": {
   "cscli": {
    "ban": 4230
   }
  },
  "ti-binaryedge.ninja": {
   "cscli": {
    "ban": 11
   }
  },
  "ti-cyberresilience.io": {
   "cscli": {
    "ban": 21
   }
  },
  "ti-googleusercontent.com": {
   "cscli": {
    "ban": 8121
   }
  },
  "ti-internet-census.org": {
   "cscli": {
    "ban": 898
   }
  },
  "ti-internet-measurement.com": {
   "cscli": {
    "ban": 195
   }
  },
  "ti-leakix.org": {
   "cscli": {
    "ban": 2
   }
  },
  "ti-shodan.io": {
   "cscli": {
    "ban": 3
   }
  },
  "ti-spamhaus_dropv4": {
   "cscli": {
    "ban": 1428
   }
  },
  "ti-stretchoid.com": {
   "cscli": {
    "ban": 2713
   }
  },
  "ti-tu-dresden.de": {
   "cscli": {
    "ban": 2
   }
  }
 },
 "lapi": {
  "/v1/alerts": {
   "GET": 2,
   "POST": 178516
  },
  "/v1/allowlists/check/:ip_or_range": {
   "GET": 140148
  },
  "/v1/decisions": {
   "DELETE": 14
  },
  "/v1/decisions/stream": {
   "GET": 332398
  },
  "/v1/heartbeat": {
   "GET": 5824
  },
  "/v1/usage-metrics": {
   "POST": 930
  },
  "/v1/watchers/login": {
   "POST": 140266
  }
 },
 "lapi-bouncer": {
  "cs-firewall-bouncer-1688295990": {
   "/v1/decisions/stream": {
    "GET": 228896
   }
  },
  "[REDACTED]": {
   "/v1/decisions/stream": {
    "GET": 11706
   }
  },
  "cs-firewall-bouncer-remote-01": {
   "/v1/decisions/stream": {
    "GET": 11648
   }
  },
  "cs-firewall-bouncer-remote-02": {
   "/v1/decisions/stream": {
    "GET": 11649
   }
  },
  "cs-firewall-bouncer-remote-05": {
   "/v1/decisions/stream": {
    "GET": 21906
   }
  },
  "[REDACTED]": {
   "/v1/decisions/stream": {
    "GET": 11650
   }
  },
  "[REDACTED]": {
   "/v1/decisions/stream": {
    "GET": 34943
   }
  }
 },
 "lapi-decisions": {},
 "lapi-machine": {
  "e9f7f4e827034fd59271a062e2d8a2d3AK3KwObeypixPPUb": {
   "/v1/alerts": {
    "GET": 2,
    "POST": 178516
   },
   "/v1/allowlists/check/:ip_or_range": {
    "GET": 140148
   },
   "/v1/decisions": {
    "DELETE": 14
   },
   "/v1/heartbeat": {
    "GET": 5824
   }
  }
 },
 "parsers": {
  "child-crowdsecurity/endlessh-logs": {
   "hits": 12655836,
   "parsed": 3164602,
   "unparsed": 9491234
  },
  "child-crowdsecurity/syslog-logs": {
   "hits": 6325986,
   "parsed": 6325986
  },
  "crowdsecurity/dateparse-enrich": {
   "hits": 3164602,
   "parsed": 3164602
  },
  "crowdsecurity/endlessh-logs": {
   "hits": 6325986,
   "parsed": 3164602,
   "unparsed": 3161384
  },
  "crowdsecurity/geoip-enrich": {
   "hits": 3164602,
   "parsed": 3164602
  },
  "crowdsecurity/public-dns-allowlist": {
   "hits": 3164602,
   "parsed": 3164602
  },
  "crowdsecurity/rdns": {
   "hits": 43362,
   "parsed": 43362
  },
  "crowdsecurity/syslog-logs": {
   "hits": 6325986,
   "parsed": 6325986
  }
 },
 "scenarios": {
  "skhron/endlessh-distinctconn": {
   "curr_count": 118838,
   "instantiation": 323509,
   "overflow": 152845,
   "pour": 856476
  }
 },
 "stash": {},
 "whitelists": {
  },
 }
}

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions]

@g00g1: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

You can get a dump of the existing goroutines by querying <crowdsec_ip>:6060/debug/pprof/goroutine with curl:

curl <crowdsec_ip>:6060/debug/pprof/goroutine -O goroutines.gz

Can you get a dump a few minutes after crowdsec has started, and another one when the count starts to increase significantly and join both files to the issue ?
This would help a lot to find the leak.

[g00g1]

@blotus, see below

goroutines-500.gz
goroutines-100k.gz

[blotus]

It looks like almost all your goroutines in the 100k file are from leaky buckets: after a log line has been parsed, crowdsec will pass it to each scenario, and if the filter matches, a new bucket will created if the groupby directive does not match an existing bucket (or added to the existing bucket if it matches).

Internally, each bucket is represented by a goroutine. Those goroutines are meant to be more or less short-lived (the exact duration will depend on the leakspeed attribute, a routine will stay alive for leakspeed at a minimum).

Unless you are reading logs coming from tens of thousands of different IPs in a very short span, I'd say the issue is likely coming from your skhron/endlessh-distinctconn scenario. Can you provide it?

It probably does not have a groupby directive (or it uses a key different for each log line) + maybe a long leakspeed: this will make crowdsec create a new bucket for each log line that matches the scenario filter, and the bucket will stay alive for a long time, and new buckets are created faster than the old ones are dying, leading to the increase in goroutines.

[g00g1]

Unless you are reading logs coming from tens of thousands of different IPs in a very short span

I run a TCP tarpit listening for 0-65535 ports at about 600 IPv4 addresses, so there could be a lot of pending buckets for sure. Didn't knew they are goroutines.

Details

$ cat /etc/crowdsec/scenarios/endlessh-distinctconn.yaml
type: leaky
leakspeed: "24h"
capacity: 3
name: skhron/endlessh-distinctconn
description: "Reacts on 4 distinct dstip:dstport pairs per srcip, default behaviour"
filter: "evt.Meta.log_type == 'endlessh_accept'"
groupby: evt.Meta.source_ip
distinct: evt.Meta.destination_ip + ":" + evt.Meta.destination_port
blackhole: 24h
labels:
  service: endlessh
  type: scan
  remediation: true

[blotus]

The issue is because of the 24h leakspeed: if an IP makes less than four requests to the same IP/port, the bucket will never overflow, and will stay in memory for at least 24h (it will be number of events poured * 24h) until it underflows and gets destroyed automatically.

The easiest fix is to reduce the value of leakspeed (I don't know what you are trying to achieve exactly, but if an IP were to make four requests on a specific IP/port combination, it will most likely be in a few minutes at most, not an entire day)

[g00g1]

Sorry, this is not the answer I would like to hear :D

Could it be considered to store information about buckets in persistent database or KV storage like Redis? This will both solve problems of persistence (as I can see buckets are not persisted across Crowdsec restarts) and my scaling problem (many observed scanning IP addresses are scanning network slowly on purpose, therefore 24 hour leak speed was configured).

[blotus]

This is actually something we have started to look at (in the context of being able to share buckets between multiple servers), but:

• we are very unsure of the performance impact of storing the buckets' data in an external service
• It's not trivial to implement without risking breaking custom scenarios that access the internal queue inside the buckets (in a conditional bucket for example)
• I'm not sure if we could do away with having one goroutine per bucket, as I don't think it is possible to reimplement the current bucket behaviour entirely inside redis (or another KV or database).

Anyway, it would be a significant change, and we are currently focusing on other areas (WAF and remote crowdsec configuration management), so I don't expect progress on this anytime soon.