Very Quick and Very Dirty HowTo: Parsing and Templates

[gebhard73]

I've put together a quick and dirty how-to for editing parsing and templates for a small, syslog-only environment in the initial installation and configuration phase. It may be useful as a starting point but should not be taken as a best practice for distributed or big environments.
Yes there are better ways to do stuff (e.g. creating additional pipelines, ...), but it works (at least for me) and keeps the effort and complexity low.
Hope this helps - comments, additions, and improvements are welcomed.

Preparation

create working directory:
mkdir /opt/so/saltstack/local/salt/elasticsearch/work

Example: modify syslog parsing

:preparation

• place copy of original syslog parsing file into work directory

cp /opt/so/saltstack/default/salt/elasticsearch/files/ingest/syslog /opt/so/saltstack/local/salt/elasticsearch/work

:start

• make new backup copy of /opt/so/saltstack/local/salt/elasticsearch/work/syslog

• edit /opt/so/saltstack/local/salt/elasticsearch/work/syslog

• copy new version from work to salt config directory
  cp /opt/so/saltstack/local/salt/elasticsearch/work/syslog /opt/so/saltstack/local/salt/elasticsearch/files/ingest/syslog

• restart elasticsearch
  so-elasticsearch-restart && date

• if error:

  • copy last working backup to /opt/so/saltstack/local/salt/elasticsearch/files/ingest/syslog
  • restart elasticsearch to keep it operational:
    so-elasticsearch-restart && date
  • goto :start

Example: modify field definitions ("index mapping")

• place copy of original file into work directory
  cp /opt/so/saltstack/default/salt/elasticsearch/templates/so/so-common-template.json /opt/so/saltstack/local/salt/elasticsearch/work

:start

• make new backup copy of /opt/so/saltstack/local/salt/elasticsearch/work/so-common-template.json
• edit /opt/so/saltstack/local/salt/elasticsearch/work/so-common-template.json
• copy new version from work to salt config directory
  cp /opt/so/saltstack/local/salt/elasticsearch/work/so-common-template.json /opt/so/saltstack/local/salt/elasticsearch/templates/so/so-common-template.json

:refresh_index

• delete all data (yes, we're still in the installation and configuration phase, so what ...):

so-elastic-clear

• wait till some new records have arrived
• reload templates:

so-elasticsearch-templates-load

• go into Kibana web UI -> hamburger menu -> "Stack Management" -> "Index Patterns" -> "*:so-*" -> "Refresh field list" (upper right corner) -> "Refresh"

• if error somewhere in the process above:

  • copy last working backup to /opt/so/saltstack/local/salt/elasticsearch/templates/so/so-common-template.json
  • goto :refresh_index

• goto :start

Hints

• work in work directory only
• make backup copies in work directory, and make them often
• only make small changes and then test
• only change files below /opt/so/saltstack/local/
• no empty lines in config files
• comments: /* COMMENT */, multiline possible
• because "if" uses painless json stuff:
  • "if" statements must be in one line (unless you do fancy """ stuff)
  • in the condition part of "if": prefix fields with "ctx." (blah gets ctx.blah)
• after restart of elasticsearch: make sure to notice the time so you know when the change will be visible in the web UI (tip: add "&& date" to the command)
• delete all data after modifying field definition to make it work is safest way and will save you headaches
• elastic is very picky about config files, esp. brackets, empty lines, comments, ...
• config files are fragile, get used to it