Elasticsearch Snapshot / Backup

[Maji-Shan]

Hello, I am having trouble enabling the snapshot feature of Elastic on our distributed setup (currently on 2.3.260). Going through the documentation, it states 2 relevant ways for this to work for us, one being editing the elasticsearch salt pillar with the path.repo setting, and the other with utilizing the curator.

To me, editing the elasticsearch salt pillar seems to be the most practical, however I have no idea what or where this file is. The official elastic documentation states to edit the 'elasticsearch.yml' file and add the repo path setting. When I edit the file located in /opt/so/conf/elasticsearch/elasticsearch.yml, it resets back to how it was prior to the edit after a so-elasticsearch-restart. Other places reference editing the global.sls, but that also doesn't work.

I also tried looking into the curator way of doing it, and I also hit a hitch where when I do so-curator-restart after editing it states the curator daemon isn't started. Does this mean I cannot use the curator for doing the snapshot after editing the init.sls?

I am so genuinely lost right now and would appreciate help. I am not a expert at this at all, and am trying my best so I am sorry if some of this sounds strange or confusing. I'm trying to set up backups of Elastic data so that at some point if something occurs, it can be off-loaded to a separate storage site for recovery later. I have looked at similar discussions prior and they reference that there's some easy way to edit this 'path.repo' setting after an update for SO but I still cannot find that. Thank you.

[Maji-Shan]

Bump to see if anyone has an answer.

[rjdub2]

Not sure if you are still working on this or found an answer, but we've been trying to set up snapshots and ran into the same confusion you did. Documentation says to look at the Elastic documentation, but what we have found is specific to SO. I believe there are two ways to do this.

1. Add a symlink in /nsm/elasticsearch/repo to the local or network location you want to store snapshots in. I believe this needs to be set on each of your search nodes and you want to set it outside of any docker container. This should create the necessary path.repo in the Elasticsearch config.

2. On your manager node you will see a .sls config file for each search node in /opt/so/saltstack/local/pillar/minions. They'll look like nodename_searchnode.sls. You want to edit the file for each search node and in this file you'll want to add the following under elasticsearch:

config:
    path:
        repo:
            - (location of your repo)

[InfosecGoon]

Some notes from the last time I set this up -- I was only doing a one-time snapshot to transfer the case data to a new install.

Create a directory for the backup/snapshot.

sudo mkdir /backups

Change ownership of backup directory to Elasticsearch.

sudo chown -R elasticsearch:elasticsearch /backups

Elevate to root (sudo -i).

Add path.repo to the minion pillar file in /opt/so/saltstack/local/pillar/minions/

elasticsearch:

es_port: 9200

esclustername: '{{ grains.host }}'

esheap: 5422m

log_size_limit: 164

mainint: eth0

mainip: 192.168.10.10

node_route_type: hot

node_type: ''

config:

path:

  repo: ["/backups"]

Restart Elasticsearch.

so-elasticsearch-restart

Add the repository to Elasticsearch (should return "Acknowledged true").

so-elasticsearch-query _snapshot/backups -d '{ "type": "fs", "settings": { "location": "/backups" }}'

Take a snapshot in Kibana dev tools:

PUT /_snapshot/backups/casebackup?pretty

{

"indices": "so-case*",

"ignore_unavailable": true,

"include_global_state": false

}