Location of the path.repo

[esbat-max]

Hello,

SO standard 2.7.50 version.

I am trying to back up the indices in the elasticsearch database to an external storage. When I change the config of the elasticsearch.yml file and insert path.repo: /arch the configuration is being overwritten when I restart elasticsearch.

Here below the error I am receiving from kibana GUI:

[abesinger]

I worked through the same issue, I've currently got a slightly clunky solution that requires some manual configuration, and I've got a pretty good idea of how to do it cleanly, but it will take a bit of work (so this is a good exercise in documenting what needs to get done :-).

To answer your question directly, your changes gets overwritten because elasticsearch.yml is generated from /opt/so/saltstack/default/salt/elasticsearch/files/elasticsearch.yml. While that file uses jinja2 templating to include settings from the minion pillar files, it doesn't have any logic for including repo settings directly. Fortunately, as of 2.3.80, you can add custom Elastic configuration settings in a config section in the pillar file. So that part is pretty easy to address (shown below).

But wait, there's more. Elastic runs in a container, and only sees the paths bound to the container. You have to bind the mount point for the repo to the Elasticsearch container, and then specify that path as the location when you register the repo. For that you have to edit one of the salt files and maintain your own copy (unless I've missed something).

Then you can register the repo with Elastic.

Here's a summary of what I did (on a Distributed install)

• Mount the repo filesystem on all the search nodes
• Add an entry for the repo in /opt/so/saltstack/local/pillar/minions/*searchnode.sls
• Made a local copy of/opt/so/saltstack/local/salt/elasticsearch/init.sls and modified it (to support the container binds).
• Let salt do its thing.
• Register the repo on each of the search nodes.

And here's the long version:

• Mounting the filesystem

I set up an NFSv4 server and exported a directory for each of the search nodes, and mounted them on their respective nodes.
Note that each node has to mount a separate remote directory; if you have two search nodes mounting the same remote directory, they'll clobber each other's snapshots. That's due to SecurityOnion using cross-cluster configuration -- each search node is it's own "cluster."

I'm not going to delve into the details of configuring fileservers/exports/mount points here. But if someone wants, I can share the specifics of what I did with NFS.

• Add an entry for the repo in /opt/so/saltstack/local/pillar/minions/*searchnode.sls

In each of the searchnode minion files, under the elasticsearch section, I added the following:

  config:
    path:
      repo:
        - "<path to mounted directory>"

Note that the the config section is a new feature in 2.3.80. See the Elastic customization section of the Security Onion documentation.

Note also that while I specified the mount path, technically it's actually the bind point on the container (see next bullet). It works here because I'm going to use the same path for both.

• Modified /opt/so/saltstack/default/salt/elasticsearch/init.sls to add the bind(s)

I made a copy of /opt/so/saltstack/default/salt/elasticsearch/init.sls in /opt/so/saltstack/local/salt/elasticsearch. That in turn will override the one in /opt/so/saltstack/default/....

Under so-elasticsearch:docker_container.running:extra_hosts:binds I added the following:

      {%- if grains.role == 'so-node' %}
        {%-   for REPO in salt['pillar.get']('elasticsearch:config:path:repo') %}
      - {{ REPO }}:{{ REPO }}:rw
        {%-   endfor %}
      {%- endif %}

That creates entries for a bind in the container with the same path as the actual directory. As I mentioned above, the location you use when registering the repo has to match the bind path; the container doesn't see the real path.

Note that every time I run soup I have to check whether the original init.sls was modified, and if so, update my local copy accordingly. This is sub-optimal, but it gets my repo working.

Also note that for a Standalone install, one might have to use a different role name.

• And then I registered the repo by hand, using the path specified in the pillar file.

---

For a more permanent, and comprehensive solution, I'm looking at doing the following, which will allow one to specify all the options necessary for creating repos, and generate all the configs including registering the repos.

• Have the pillar include settings for the repo in addition to the path. E.g. something like this:

elastic:
  repo:
    fs:
      "<repo name>":
        location: "<path to mount point>"
        bind: "<path to bind container to location>"
        settings:
           "<name> : <value>"

• Add some code to /opt/so/saltstack/default/salt/elasticsearch/files/elasticsearch.yml add repo:path entries using the bind value from the pillar file.
• Extend the code I did above to use those settings above to generate the bind line(s). Allows for having a bind path different than the actual path.
• Automagically generate Elastic queries to register the repos, using the entries in the pillar file for the repo name and settings. This will also require a bash script to load them, and some startup code to execute the load, similar to how the ingest pipelines get loaded.

The first three bullet items will be pretty straightforward, the last one will take some effort; I'll need to up my Salt game.

[esbat-max]

Hello abesinger,

The above threat is very difficult to understand. Especially, that there are discrepancies between the description and my current deployment.

1. I am using standalone mode.

2. The storage is mounted as shown in the df -h output:
   /dev/mapper/vg_arch-lv_arch 3.0T 33M 3.0T 1% /arch

3. config section under elasticsearch section does not exist, however, when I try to add it, Elastic database crashed.

sudo cat /opt/so/saltstack/local/pillar/minions/so-touch-std_standalone.sls
elasticsearch:
mainip: '172.16.241.76'
mainint: 'ens192'
esheap: '8045m'
esclustername: '{{ grains.host }}'
node_type: ''
es_port: 9200
log_size_limit: 2100
node_route_type: 'hot'

4. init.sls file does not exist:

sudo ls -l /opt/so/saltstack/local/salt/elasticsearch/

-rw-------. 1 socore socore 92 Jan 14 07:31 curl.config
drwxr-xr-x. 5 socore socore 89 Jan 14 11:07 files
drwxr-xr-x. 4 socore socore 30 Sep 20 10:40 templates

Hence, I cannot continue with the above steps due to the differences I faces.

can you please elaborate more and be straight forward.

Thanks,
E

[harleytomlinsonOMB]

Can anyone please let me know if there is a more succinct process for this somewhere? We have a distributed install running 2.4 and are attempting to send our Elastic snapshots to Azure.

[TOoSmOotH]

This will be supported out of the box in .100

[abesinger]

@TOoSmOotH , If I'm reading that PR correctly, it will use the the path from elasticsearch:extramount grain, and binds it to the (hardwired) path /snapshots in the elasticsearch container. So that addresses binding to the container, but it doesn't address the required path:repo entry in elasticsearch.yml, and I don't see any changes to elasticsearch/config.map.jinja nor elasticsearch/defaults.yaml.

So the pillar would have to contain:

elasticsearch:
  extramount: "/some/local/path"
  config:
    path:
      repo:
        - "/snapshots"

It seems a little clunky to have to do that (spelling out something that's hardwired in), and a bit rigid to be limited to one fixed mount point (and bind path).

Would you like me to submit my solution as a PR? It's a bit cleaner, and allows for multiple mounts/repos. I could easily change it to be outside of the elasticsearch:config section, e.g. under elasticsearch:repo, and accomodate specifying both the actual path and the bind path. For example, something like this:

elasticsearch:
  repo:
    "repo name" :
      mount: "/local/path/to/repo>"
      bind: "/container/bind/path"

That leaves the door open to later extending it to include the repo settings and automatically registering the repo with Elasticsearch.

[m0duspwnens]

@abesinger This feature has been slightly reworked PR here. The repo(s) will need to be defined under elasticsearch:config:path:repo and a symlink with the same repo name(s) will need to be created by the user, with the proper permissions, under /nsm/elasticsearch/repo/.

[abesinger]

@m0duspwnens @TOoSmOotH That looks a little better, but is there a reason to force the path to be under /nsm/elasticsearch/repo and then require the user to manually maintain a symlink, as opposed to just binding the Elastic container to the actual path provided by the user, or letting the user specify the bind path? (I'm currently doing exactly that and it works fine) After all, isn't the point of using Salt that one doesn't rely on having to do things manually?

Having to maintain a symlink is just one more thing to break if mistyped, etc;, one more thing to have to recreate on new search nodes/reinstalls, etc. Yes, the user (currently) has to manually create the mount point, but that's one thing instead of two.

[m0duspwnens]

just binding the Elastic container to the actual path provided by the user

Having to maintain a symlink is just one more thing to break if mistyped, etc;, one more thing to have to recreate on new search nodes/reinstalls, etc.

These are both good points. A new PR has been merged that removes the requirement for the symlink and will allow the user to define elasticsearch:config:path:repo to be the location on the host that will then be mounted as the same within the so-elasticsearch container. The requirement is still in place that the directory must be present on the host prior to being added in elasticsearch.yml and the bind being created.

For example, if a user defines elasticsearch:config:path:repo as /nfs/elasticsearch/repos/shortterm/, then as long as /nfs/elasticsearch/repos/shortterm/ exists on the host, it will be bound within the so-elasticsearch container as /nfs/elasticsearch/repos/shortterm/.

Thanks for the feedback.

[abesinger]

Looks good, and thanks for listening.

I've got an idea for a future extension (I described it a ways above in this thread) to allow the user to specify all the repo settings, and then have Salt automatically register the repo on each node that it's supposed to be on. I'll see if I can't work on that at some point and put in a PR for it down the road.