Pipeline with id [$pipeline] does not exist - for all pipelines #1687

chuglo · 2020-10-27T20:36:23Z

chuglo
Oct 27, 2020

SO v2.3.2

1 Manager
1 Search Node
1 Sensor

On a fresh distributed deployment, I have 0 logs being ingested. It looks like an issue with Elasticsearch and/or Logstash.

Here is a snippet from /opt/so/log/logstash/logstash.log from the Search Node

[2020-10-27T20:24:50,415][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"so-zeek", :routing=>nil, :_type=>"_doc", :pipeline=>"zeek.files"}, #<LogStash::Event:0x5b7a8a3c>], :response=>{"index"=>{"_index"=>"so-zeek", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [zeek.files] does not exist"}}}}

Getting the same error for all pipelines.

Looking on the Manager, it actually looks like the pipelines truly don't exist? (If I am looking in the right spot)

[root@dit-so-master config]# pwd
/opt/so/saltstack/local/salt/logstash/pipelines/config
[root@dit-so-master config]# ls -lash
total 0
0 drwxr-xr-x. 4 socore socore 30 Oct 26 18:51 .
0 drwxr-xr-x. 3 socore socore 20 Oct 26 18:51 ..
0 drwxr-xr-x. 2 socore socore  6 Oct 26 18:51 custom
0 drwxr-xr-x. 2 socore socore  6 Oct 26 18:51 so
[root@dit-so-master config]# ls -lash custom/
total 0
0 drwxr-xr-x. 2 socore socore  6 Oct 26 18:51 .
0 drwxr-xr-x. 4 socore socore 30 Oct 26 18:51 ..
[root@dit-so-master config]# ls -lash so/
total 0
0 drwxr-xr-x. 2 socore socore  6 Oct 26 18:51 .
0 drwxr-xr-x. 4 socore socore 30 Oct 26 18:51 ..
[root@dit-so-master config]#

Any way to remedy this without reinstalling?

Answered by chuglo

Oct 30, 2020

I believe the root issue was that Salt did not properly get configured from setup as indicated by the setup log.

----
8% - GENERATING PATCH PILLAR
----
./so-functions: line 1354: /root/installtmp/pillar/minions/node1_searchnode.sls: No such file or directory
----

and that's because I had to switch to another TTY during setup, because I wanted to see what IF was currently up/down to ensure that I select the right one for mgmt (there are multiple) - and was prompted again to start the setup to which I selected No - thus "canceling" the setup, even though it still actually completed and was a reported success.

User cancelled setup.
/root/installtmp exists
Installer removing the following fi…

View full answer

TOoSmOotH · 2020-10-27T22:27:36Z

TOoSmOotH
Oct 27, 2020
Maintainer

On the search node try sudo so-elasticsearch-restart

0 replies

chuglo · 2020-10-28T12:11:42Z

chuglo
Oct 28, 2020
Author

Hi Mike,

Thank you - I've just done that but getting the same errors in logstash log on the search node. I'm not sure why the pipelines would be missing. Anything else I could try?

0 replies

weslambert · 2020-10-28T12:28:12Z

weslambert
Oct 28, 2020

What is the output of the following? sudo so-elasticsearch-pipelines-list Also, what is in /opt/so/saltstack/default/salt/elasticsearch/files/ingest, and /opt/so/conf/elasticsearch/ingest? Thanks, Wes

…

On Wed, Oct 28, 2020 at 8:11 AM chuglo ***@***.***> wrote: Hi Mike, Thank you - I've just done that but getting the same errors in logstash log on the search node. I'm not sure why the pipelines would be missing. Anything else I could try? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAM3KA72EF4REPUZXFTPS3SNADA3ANCNFSM4TBLP2YQ> .

-- https://twitter.com/therealwlambert https://securityonion.net/

1 reply

chuglo Oct 28, 2020
Author

Hi Wes,

Not sure which node you want me to grab that from so here are the results from both:

Manager

[user@dit-so-master config]# sudo so-elasticsearch-pipelines-list
[
  "beats.common",
  "common",
  "common.nids",
  "filterlog",
  "import.wel",
  "osquery.query_result",
  "ossec",
  "strelka.file",
  "suricata.alert",
  "suricata.common",
  "suricata.dhcp",
  "suricata.dnp3",
  "suricata.dns",
  "suricata.fileinfo",
  "suricata.flow",
  "suricata.ftp",
  "suricata.http",
  "suricata.ikev2",
  "suricata.krb5",
  "suricata.nfs",
  "suricata.rdp",
  "suricata.sip",
  "suricata.smb",
  "suricata.smtp",
  "suricata.snmp",
  "suricata.ssh",
  "suricata.tftp",
  "suricata.tls",
  "syslog",
  "sysmon",
  "win.eventlogs",
  "zeek.common",
  "zeek.common_ssl",
  "zeek.conn",
  "zeek.dce_rpc",
  "zeek.dhcp",
  "zeek.dnp3",
  "zeek.dns",
  "zeek.dns.tld",
  "zeek.dpd",
  "zeek.files",
  "zeek.ftp",
  "zeek.http",
  "zeek.intel",
  "zeek.irc",
  "zeek.kerberos",
  "zeek.modbus",
  "zeek.mysql",
  "zeek.notice",
  "zeek.ntlm",
  "zeek.pe",
  "zeek.radius",
  "zeek.rdp",
  "zeek.rfb",
  "zeek.signatures",
  "zeek.sip",
  "zeek.smb_files",
  "zeek.smb_mapping",
  "zeek.smtp",
  "zeek.snmp",
  "zeek.socks",
  "zeek.software",
  "zeek.ssh",
  "zeek.ssl",
  "zeek.syslog",
  "zeek.tunnel",
  "zeek.tunnels",
  "zeek.weird",
  "zeek.x509"
]

[user@dit-so-master config]# sudo ls /opt/so/saltstack/default/salt/elasticsearch/files/ingest
beats.common  osquery.query_result  suricata.dhcp      suricata.ftp    suricata.rdp   suricata.ssh   win.eventlogs    zeek.dhcp     zeek.files  zeek.kerberos  zeek.pe          zeek.sip          zeek.socks     zeek.tunnel
common        ossec                 suricata.dnp3      suricata.http   suricata.sip   suricata.tftp  zeek.common      zeek.dnp3     zeek.ftp    zeek.modbus    zeek.radius      zeek.smb_files    zeek.software  zeek.tunnels
common.nids   strelka.file          suricata.dns       suricata.ikev2  suricata.smb   suricata.tls   zeek.common_ssl  zeek.dns      zeek.http   zeek.mysql     zeek.rdp         zeek.smb_mapping  zeek.ssh       zeek.weird
filterlog     suricata.alert        suricata.fileinfo  suricata.krb5   suricata.smtp  syslog         zeek.conn        zeek.dns.tld  zeek.intel  zeek.notice    zeek.rfb         zeek.smtp         zeek.ssl       zeek.x509
import.wel    suricata.common       suricata.flow      suricata.nfs    suricata.snmp  sysmon         zeek.dce_rpc     zeek.dpd      zeek.irc    zeek.ntlm      zeek.signatures  zeek.snmp         zeek.syslog
[user@dit-so-master config]#

[user@dit-so-master config]# sudo ls /opt/so/conf/elasticsearch/ingest
beats.common  osquery.query_result  suricata.dhcp      suricata.ftp    suricata.rdp   suricata.ssh   win.eventlogs    zeek.dhcp     zeek.files  zeek.kerberos  zeek.pe          zeek.sip          zeek.socks     zeek.tunnel
common        ossec                 suricata.dnp3      suricata.http   suricata.sip   suricata.tftp  zeek.common      zeek.dnp3     zeek.ftp    zeek.modbus    zeek.radius      zeek.smb_files    zeek.software  zeek.tunnels
common.nids   strelka.file          suricata.dns       suricata.ikev2  suricata.smb   suricata.tls   zeek.common_ssl  zeek.dns      zeek.http   zeek.mysql     zeek.rdp         zeek.smb_mapping  zeek.ssh       zeek.weird
filterlog     suricata.alert        suricata.fileinfo  suricata.krb5   suricata.smtp  syslog         zeek.conn        zeek.dns.tld  zeek.intel  zeek.notice    zeek.rfb         zeek.smtp         zeek.ssl       zeek.x509
import.wel    suricata.common       suricata.flow      suricata.nfs    suricata.snmp  sysmon         zeek.dce_rpc     zeek.dpd      zeek.irc    zeek.ntlm      zeek.signatures  zeek.snmp         zeek.syslog
[user@dit-so-master config]#

Search Node

[user@dit-so-node1 ~]$ sudo so-elasticsearch-pipelines-list
[]
[user@dit-so-node1 ~]$

[user@dit-so-node1 ~]$ sudo ls /opt/so/saltstack/default/salt/elasticsearch/files/ingest
ls: cannot access /opt/so/saltstack/default/salt/elasticsearch/files/ingest: No such file or directory
[user@dit-so-node1 ~]$ sudo ls /opt/so/saltstack/default/
[user@dit-so-node1 ~]$

Is this supposed to be empty on a search node? ^

[user@dit-so-node1 ~]$ sudo ls /opt/so/conf/elasticsearch/ingest
beats.common  osquery.query_result  suricata.dhcp      suricata.ftp    suricata.rdp   suricata.ssh   win.eventlogs    zeek.dhcp     zeek.files  zeek.kerberos  zeek.pe          zeek.sip          zeek.socks     zeek.tunnel
common        ossec                 suricata.dnp3      suricata.http   suricata.sip   suricata.tftp  zeek.common      zeek.dnp3     zeek.ftp    zeek.modbus    zeek.radius      zeek.smb_files    zeek.software  zeek.tunnels
common.nids   strelka.file          suricata.dns       suricata.ikev2  suricata.smb   suricata.tls   zeek.common_ssl  zeek.dns      zeek.http   zeek.mysql     zeek.rdp         zeek.smb_mapping  zeek.ssh       zeek.weird
filterlog     suricata.alert        suricata.fileinfo  suricata.krb5   suricata.smtp  syslog         zeek.conn        zeek.dns.tld  zeek.intel  zeek.notice    zeek.rfb         zeek.smtp         zeek.ssl       zeek.x509
import.wel    suricata.common       suricata.flow      suricata.nfs    suricata.snmp  sysmon         zeek.dce_rpc     zeek.dpd      zeek.irc    zeek.ntlm      zeek.signatures  zeek.snmp         zeek.syslog
[user@dit-so-node1 ~]$

chuglo · 2020-10-30T14:32:30Z

chuglo
Oct 30, 2020
Author

I believe the root issue was that Salt did not properly get configured from setup as indicated by the setup log.

----
8% - GENERATING PATCH PILLAR
----
./so-functions: line 1354: /root/installtmp/pillar/minions/node1_searchnode.sls: No such file or directory
----

and that's because I had to switch to another TTY during setup, because I wanted to see what IF was currently up/down to ensure that I select the right one for mgmt (there are multiple) - and was prompted again to start the setup to which I selected No - thus "canceling" the setup, even though it still actually completed and was a reported success.

User cancelled setup.
/root/installtmp exists
Installer removing the following files:
/root/installtmp:
total 0
drwxr-xr-x. 3 root root 21 Oct 27 15:41 pillar

/root/installtmp/pillar:
total 0
drwxr-xr-x. 2 root root 6 Oct 27 15:41 minions

/root/installtmp/pillar/minions:
total 0
./so-functions: line 1054: /usr/sbin/so-mysql-stop: No such file or directory
/root/installtmp removed

At the end of sosetup.log

Successfully completed setup! Continuing with post-installation steps

If I could make a suggestion, it would be neat if the setup displayed whether or not IFs are up/down when you are selecting which one to use for mgmt. Especially since the naming of NICs/IFs can vary wildly between hardware in my experience.

Consider this one closed. Thanks!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pipeline with id [$pipeline] does not exist - for all pipelines #1687

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Pipeline with id [$pipeline] does not exist - for all pipelines #1687

Uh oh!

chuglo Oct 27, 2020

Replies: 4 comments · 1 reply

Uh oh!

TOoSmOotH Oct 27, 2020 Maintainer

Uh oh!

chuglo Oct 28, 2020 Author

Uh oh!

weslambert Oct 28, 2020

Uh oh!

chuglo Oct 28, 2020 Author

Manager

Search Node

Uh oh!

Uh oh!

chuglo Oct 30, 2020 Author

chuglo
Oct 27, 2020

Replies: 4 comments 1 reply

TOoSmOotH
Oct 27, 2020
Maintainer

chuglo
Oct 28, 2020
Author

weslambert
Oct 28, 2020

chuglo Oct 28, 2020
Author

chuglo
Oct 30, 2020
Author