Hunt/Dashboard really slow #9679

giveen · 2023-01-31T13:20:58Z

giveen
Jan 31, 2023

I could use some help improving the speed of the hunt and dashboard. Basically I feel like queries are really slow.

My server is beefy
80 cores
756Gb of RAM

giveen · 2023-01-31T17:50:41Z

giveen
Jan 31, 2023
Author

So I made these changes for far:

in the minion file for the manager

logstash_settings:
  ls_input_threads: 6
  batch: true
  ls_pipeline_batch_size: 250
  ls_pipeline_workers: 80
  lsheap: 151g

in the global.sls

redis_settings:
  redis_maxmemory: 24288

I also increased the amount of shards in the index, all of them are now at 3 except zeek i put to 14

0 replies

cm-ops · 2023-02-02T17:01:31Z

cm-ops
Feb 2, 2023
Maintainer

How many open indices/shards are you searching? What are the size of your Zeek shards?

1 reply

giveen Feb 2, 2023
Author

195 open indices

496 shards

I did notice a lot of my zeek shards were way over size, so I got my zeek shard setting so that it was belong 50Gb

global.txt

so-zeek-2023.01.24                  0  p STARTED 1
16932849    161gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.24                  1  p STARTED 116937719    161gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.24                  2  p STARTED 116931108  160.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.24                  3  p STARTED 116924479    161gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.24                  4  p STARTED 116949281  161.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.31              0  p STARTED  58427054   62.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.31              1  p STARTED  58420569   62.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.31              2  p STARTED  58417036   62.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.31              3  p STARTED  58408078   62.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.31              4  p STARTED  58397403   62.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.25                  0  p STARTED 115044473  155.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.25                  1  p STARTED 115057801  155.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.25                  2  p STARTED 115053841  155.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.25                  3  p STARTED 115062373  155.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.25                  4  p STARTED 115072313  155.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.20              0  p STARTED  64202972   66.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.20              1  p STARTED  64198610   66.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.20              2  p STARTED  64193995   66.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.20              3  p STARTED  64189260   66.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.20              4  p STARTED  64185527   66.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.28              0  p STARTED  61275637   61.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.28              1  p STARTED  61286067   61.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.28              2  p STARTED  61268407   61.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.28              3  p STARTED  61278388   61.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.28              4  p STARTED  61265796   61.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.16                  0  p STARTED        21  371.8kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.16                  1  p STARTED        23    447kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.16                  2  p STARTED        19  474.9kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.16                  3  p STARTED        18  166.4kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.16                  4  p STARTED        19  524.5kb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.26              0  p STARTED  59678097   62.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.26              1  p STARTED  59680777   62.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.26              2  p STARTED  59671578   62.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.26              3  p STARTED  59685594   62.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.26              4  p STARTED  59671191     62gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.14                  0  p STARTED  94109828  126.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.14                  1  p STARTED  94095929  126.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.14                  2  p STARTED  94106000  126.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.14                  3  p STARTED  94088770  126.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.14                  4  p STARTED  94095687  126.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.30              0  p STARTED  58055771   60.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.30              1  p STARTED  58050185   60.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.30              2  p STARTED  58043820   60.3gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.30              3  p STARTED  58043465   60.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.30              4  p STARTED  58051307   60.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.19              0  p STARTED  49200039   51.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.19              1  p STARTED  49197465   51.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.19              2  p STARTED  49204659   51.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.19              3  p STARTED  49193317   51.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.19              4  p STARTED  49199411   51.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.28                  0  p STARTED  89367374    119gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.28                  1  p STARTED  89372617  118.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.28                  2  p STARTED  89397131  118.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.28                  3  p STARTED  89384948    119gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.28                  4  p STARTED  89387692    119gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.26                  0  p STARTED 104814517  141.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.26                  1  p STARTED 104793354  141.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.26                  2  p STARTED 104826527  141.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.26                  3  p STARTED 104802169  141.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.26                  4  p STARTED 104818723  141.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.25              0  p STARTED  65435124   67.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.25              1  p STARTED  65429686   67.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.25              2  p STARTED  65428161   67.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.25              3  p STARTED  65425135   67.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.25              4  p STARTED  65423951   67.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.29                  0  p STARTED  85734109  114.4gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.29                  1  p STARTED  85734633  114.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.29                  2  p STARTED  85723117  114.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.29                  3  p STARTED  85753073  114.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.29                  4  p STARTED  85739377  114.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.23              0  p STARTED  64568895   67.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.23              1  p STARTED  64563476   67.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.23              2  p STARTED  64572027   67.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.23              3  p STARTED  64575758   67.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.23              4  p STARTED  64568260   67.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.22                  0  p STARTED 105773629  144.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.22                  1  p STARTED 105782005  144.4gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.22                  2  p STARTED 105779569  144.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.22                  3  p STARTED 105753639  144.4gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.22                  4  p STARTED 105766402  144.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              0  p STARTED  15205430   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              1  p STARTED  15208315   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              2  p STARTED  15216726   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              3  p STARTED  15203714   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              4  p STARTED  15214737   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              5  p STARTED  15217681   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              6  p STARTED  15204043   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              7  p STARTED  15205455   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              8  p STARTED  15210756   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              9  p STARTED  15212609   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              10 p STARTED  15211470   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              11 p STARTED  15205412   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              12 p STARTED  15208090   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              13 p STARTED  15205114   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              14 p STARTED  15215357   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              15 p STARTED  15213287   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              16 p STARTED  15212590   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              17 p STARTED  15212416   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              18 p STARTED  15209043   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              19 p STARTED  15208053   16.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              20 p STARTED  15208615   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              21 p STARTED  15204219   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              22 p STARTED  15207993   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.01              23 p STARTED  15207479   16.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.19                  0  p STARTED 112453605  153.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.19                  1  p STARTED 112437815  153.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.19                  2  p STARTED 112431619  153.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.19                  3  p STARTED 112427682  153.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.19                  4  p STARTED 112451743  153.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.17                  0  p STARTED  32647303     46gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.17                  1  p STARTED  32658027     46gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.17                  2  p STARTED  32659741     46gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.17                  3  p STARTED  32648837   45.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.17                  4  p STARTED  32654836     46gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.23                  0  p STARTED 113517909  158.7gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.23                  1  p STARTED 113525971  158.7gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.23                  2  p STARTED 113525261  158.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.23                  3  p STARTED 113522776  158.7gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.23                  4  p STARTED 113512661  158.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.17              0  p STARTED   9329659   10.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.17              1  p STARTED   9325159   10.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.17              2  p STARTED   9322981   10.3gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.17              3  p STARTED   9329562   10.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.17              4  p STARTED   9332995   10.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.21              0  p STARTED  71695278   73.3gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.21              1  p STARTED  71690760   73.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.21              2  p STARTED  71697327   73.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.21              3  p STARTED  71688211   73.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.21              4  p STARTED  71708741   73.4gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.22              0  p STARTED  72697232   74.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.22              1  p STARTED  72684301   74.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.22              2  p STARTED  72684668   74.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.22              3  p STARTED  72689620   74.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.22              4  p STARTED  72673101   74.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.15                  0  p STARTED        10  347.1kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.15                  1  p STARTED        12    331kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.15                  2  p STARTED         6  195.6kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.15                  3  p STARTED         5  179.3kb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.15                  4  p STARTED         7  219.9kb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              0  p STARTED  42256707   44.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              1  p STARTED  42255812   44.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              2  p STARTED  42257883   44.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              3  p STARTED  42272180   44.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              4  p STARTED  42258414   44.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              5  p STARTED  42266414   44.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.27              6  p STARTED  42265466   44.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  0  p STARTED  21239183   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  1  p STARTED  21247420   31.4gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  2  p STARTED  21240456   31.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  3  p STARTED  21242031   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  4  p STARTED  21257500   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  5  p STARTED  21241412   29.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  6  p STARTED  21243164   30.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  7  p STARTED  21243468   29.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  8  p STARTED  21230818   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  9  p STARTED  21230915   31.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  10 p STARTED  21225181   31.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  11 p STARTED  21214665   29.4gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  12 p STARTED  21224484   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  13 p STARTED  21222079   30.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  14 p STARTED  21265063   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  15 p STARTED  21264533   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  16 p STARTED  21263270   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  17 p STARTED  21257510   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  18 p STARTED  21267242   30.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  19 p STARTED  21243824   29.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  20 p STARTED  21258571   29.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  21 p STARTED  21251464   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  22 p STARTED  21240368   29.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.02                  23 p STARTED  21247698   29.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.21                  0  p STARTED 108490888  149.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.21                  1  p STARTED 108475236  149.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.21                  2  p STARTED 108480977  149.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.21                  3  p STARTED 108467404  149.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.21                  4  p STARTED 108461519  149.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              0  p STARTED  12423982   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              1  p STARTED  12425431   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              2  p STARTED  12426309   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              3  p STARTED  12426192   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              4  p STARTED  12427254   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              5  p STARTED  12419986   13.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              6  p STARTED  12426282   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              7  p STARTED  12432306   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              8  p STARTED  12431776   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              9  p STARTED  12433316   14.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              10 p STARTED  12437643     14gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              11 p STARTED  12433536   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              12 p STARTED  12430124   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              13 p STARTED  12431689   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              14 p STARTED  12436276   13.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              15 p STARTED  12436678   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              16 p STARTED  12432957   13.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              17 p STARTED  12430150   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              18 p STARTED  12433953   13.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              19 p STARTED  12436590   13.6gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              20 p STARTED  12426763   13.8gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              21 p STARTED  12434490   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              22 p STARTED  12429759   13.7gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.02.02              23 p STARTED  12425572   13.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.30                  0  p STARTED 101126458    138gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.30                  1  p STARTED 101130350    138gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.30                  2  p STARTED 101125606  138.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.30                  3  p STARTED 101138655  138.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.30                  4  p STARTED 101120338    138gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  0  p STARTED  69777754   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  1  p STARTED  69776196   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  2  p STARTED  69758036   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  3  p STARTED  69783460   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  4  p STARTED  69760461   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  5  p STARTED  69775570   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.27                  6  p STARTED  69763819   94.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.18              0  p STARTED  57016222   61.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.18              1  p STARTED  57019460     61gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.18              2  p STARTED  57014408   61.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.18              3  p STARTED  57012099     61gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.18              4  p STARTED  57015128   61.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.31                  0  p STARTED 131529874  179.9gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.31                  1  p STARTED 131528136  179.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.31                  2  p STARTED 131517924  179.8gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.31                  3  p STARTED 131526099    180gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.31                  4  p STARTED 131528600  179.9gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.29              0  p STARTED  61018734   61.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.29              1  p STARTED  61018579   61.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.29              2  p STARTED  61007718   61.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.29              3  p STARTED  61008588   61.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.29              4  p STARTED  61004756   61.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.14              0  p STARTED  57140201   59.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.14              1  p STARTED  57147733   59.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.14              2  p STARTED  57147808   59.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.14              3  p STARTED  57142497   59.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.14              4  p STARTED  57141742   59.5gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.24              0  p STARTED  60666186   63.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.24              1  p STARTED  60661826   63.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.24              2  p STARTED  60672648   63.2gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.24              3  p STARTED  60666724   63.1gb 10.21.132.25 somgr-lxp-01
so-zeek_dns-2023.01.24              4  p STARTED  60675077   63.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.20                  0  p STARTED 117493393  162.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.20                  1  p STARTED 117489708  162.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.20                  2  p STARTED 117484310  162.5gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.20                  3  p STARTED 117492293  162.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.20                  4  p STARTED 117488904  162.6gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.18                  0  p STARTED 126630637  175.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.18                  1  p STARTED 126635823  175.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.18                  2  p STARTED 126622333  175.2gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.18                  3  p STARTED 126644844  175.3gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.01.18                  4  p STARTED 126621585  175.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  0  p STARTED  29330988   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  1  p STARTED  29326757   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  2  p STARTED  29327237   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  3  p STARTED  29322511   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  4  p STARTED  29323317   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  5  p STARTED  29318494     41gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  6  p STARTED  29325499   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  7  p STARTED  29322116     41gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  8  p STARTED  29330222   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  9  p STARTED  29338550   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  10 p STARTED  29330483   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  11 p STARTED  29326822   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  12 p STARTED  29329920   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  13 p STARTED  29329482   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  14 p STARTED  29322583   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  15 p STARTED  29326265   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  16 p STARTED  29320173   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  17 p STARTED  29317364     41gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  18 p STARTED  29317105   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  19 p STARTED  29328150   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  20 p STARTED  29335131   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  21 p STARTED  29326592   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  22 p STARTED  29321045   41.1gb 10.21.132.25 somgr-lxp-01
so-zeek-2023.02.01                  23 p STARTED  29315678   41.1gb 10.21.132.25 somgr-lxp-01

cm-ops · 2023-02-08T20:34:48Z

cm-ops
Feb 8, 2023
Maintainer

How far back are you searching? What is the esheap setting in the manager's minion pillar? (this will be under manager)

0 replies

giveen · 2023-02-08T20:41:05Z

giveen
Feb 8, 2023
Author

@cm-ops

So I'm only searching the past 24 hours on the default dashboard load.

from manager pillar

manager:
  domainstats: 0
  elastalert: 1
  es_port: 9200
  esclustername: '{{ grains.host }}'
  esheap: 25000m
  freq: 0
  grafana: 1
  mainint: eno3
  mainip: 10.21.132.25
  mtu: null
  no_proxy: ''
  osquery: 1
  playbook: 1
  proxy: ''
  thehive: 0

I also improved shards, see attached text file
shard_count.txt

This is the dashboard that loads when I click on Dashboard, it takes anywhere between 3 minutes to Timeout

10 replies

giveen Feb 21, 2023
Author

What is your RAM utilization on the node after that change? Also, what type of disk does the node have? Are you able to check disk stats as well?

This is the manager server

giveen Feb 21, 2023
Author

cm-ops Feb 24, 2023
Maintainer

Is that normal iowait? If you have atop, can you check the percentages for sda with that while you perform a search?

giveen Feb 24, 2023
Author

Is that normal iowait? If you have atop, can you check the percentages for sda with that while you perform a search?

cm-ops Feb 28, 2023
Maintainer

That looks like the disk is your cause of the slowness.

Hunt/Dashboard really slow #9679

Uh oh!

giveen Jan 31, 2023

Replies: 4 comments · 11 replies

Uh oh!

giveen Jan 31, 2023 Author

Uh oh!

cm-ops Feb 2, 2023 Maintainer

Uh oh!

Uh oh!

giveen Feb 2, 2023 Author

Uh oh!

cm-ops Feb 8, 2023 Maintainer

Uh oh!

Uh oh!

giveen Feb 8, 2023 Author

Uh oh!

giveen Feb 21, 2023 Author

Uh oh!

giveen Feb 21, 2023 Author

Uh oh!

cm-ops Feb 24, 2023 Maintainer

Uh oh!

giveen Feb 24, 2023 Author

Uh oh!

cm-ops Feb 28, 2023 Maintainer

giveen
Jan 31, 2023

Replies: 4 comments 11 replies

giveen
Jan 31, 2023
Author

cm-ops
Feb 2, 2023
Maintainer

giveen Feb 2, 2023
Author

cm-ops
Feb 8, 2023
Maintainer

giveen
Feb 8, 2023
Author

giveen Feb 21, 2023
Author

giveen Feb 21, 2023
Author

cm-ops Feb 24, 2023
Maintainer

giveen Feb 24, 2023
Author

cm-ops Feb 28, 2023
Maintainer