Implement basic TLS/SSL features (#179)

* Configured Elasticsearch to work with SSL

* Disable Xpack on Kibana and Ingestor nodes

* Implement SSL OPS file

* Unlink elasticsearch_config job from remote ES cluster and run it against colocated one

* Unbound upload-kibana-objects from ES remote cluster

* Fix scale-to-one-az ops file

* Unbound curator from remote ES cluster and make it use colocated one

* Move ls-router to separate OPS file

* Disable post-start across all instances

* Change dn

* Disable post-start on Kibana also

* Put admin cert to data node

* Re-organize post-start

* Add README

* Split ssl/tls

* Upload blobs

* Fixup upon review

Author: axelaris

README.md

@@ -1,32 +1,35 @@
 # Logsearch
 
-A scalable stack of [Elasticsearch](http://www.elasticsearch.org/overview/elasticsearch/),
-[Logstash](http://www.elasticsearch.org/overview/logstash/), and
-[Kibana](http://www.elasticsearch.org/overview/kibana/) for your
-own [BOSH](http://docs.cloudfoundry.org/bosh/)-managed infrastructure.
+A scalable stack of [Elasticsearch](https://www.elastic.co/elasticsearch), [Logstash](https://www.elastic.co/logstash), and [Kibana](https://www.elastic.co/kibana) for your own [BOSH](https://bosh.io/docs)-managed infrastructure.
+
+![logsearch-scheme](docs/img/logsearch.png)
 
 ## BREAKING CHANGES
 
-Logsearch < v23.0.0 was based on Elasticsearch 1.x and Kibana 3.
+### Logsearch v211 is based on Elastic stack version 7
+In v211.1.0 basic cluster security features were implemented using [Securiry](https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/) plugin from OpenDistro Elasticsearch implementation. For better handling of these features, a following changes was made:
 
-Logsearch > v200 is based on Elasticsearch 2.x and Kibana 4.
+- Additional Elasticsearch job has been colocated on **Maintenance** instance. This allows secure communication over localhost for all singletons also colocated there (all singletons have been unlinked from any remote Elasticsearch cluster, and bound to local one).
+- Since using of Ls-rounter instance is not mandatory - it was moved to separate [ops-file](deployment/operations/enable-router.yml).
+- Secure Elasticsearch node-to-node communication has been implemented using [enable-tls](deployment/operations/enable-tls.yml) ops-file.
+- Secure ingesting logs is implemented using [enable-ssl](deployment/operations/enable-ssl.yml) ops-file.
 
-- There is NO upgrade path from Elasticsearch 1.x to 2.x.  Sorry :(
+### Logsearch v210 is based on Elastic stack version 6
 
-Logsearch > v204.0.0 is based on Elastic stack version 5.
+- Elasticsearch 6.x can use indices created in Elasticsearch 5.x, but not those created in Elasticsearch 2.x or before.
+- **Important**: After upgrading running 5.x cluster to 6.x all existing indicies will be available for reading data. However, writing to these indicies is not possible. In order to write data immediatelly after upgrade you have to [change index naming convention](https://github.com/cloudfoundry-community/logsearch-boshrelease/commit/2f83b41ee14dbe3141e21cc0c40df340d50e0169). As long as index names are usually based on current date, this change can be safely reverted in a day or so.
 
+### Logsearch v204 is based on Elastic stack version 5.
 - For upgrade procedure from Elasticsearch 2.x please refer to [v205.0.0 release notes](https://github.com/cloudfoundry-community/logsearch-boshrelease/releases/tag/v205.0.0#component-updates).
 
-Logsearch > v210.0.0 is based on Elastic stack version 6.
-
-- Elasticsearch 6.x can use indices created in Elasticsearch 5.x, but not those created in Elasticsearch 2.x or before.
-- **Important**: After upgrading running 5.x cluster to 6.x all existing indicies will be available for reading data. However, writing to these indicies is not possible. In order to write data immediatelly after upgrade you have to [change index naming convention](https://github.com/cloudfoundry-community/logsearch-boshrelease/commit/2f83b41ee14dbe3141e21cc0c40df340d50e0169). As long as index names are usually based on current date, this change can be safely reverted in a day or so.
+### Logsearch v200 is based on Elasticsearch 2.x and Kibana 4.
+- There is NO upgrade path from Elasticsearch 1.x to 2.x.  Sorry :(
 
+### Logsearch < v23 was based on Elasticsearch 1.x and Kibana 3.
 
 ## Getting Started
 
-This repo contains Logsearch Core; which deploys an ELK cluster that can receive and parse logs via syslog
-that contain JSON.
+This repo contains Logsearch Core; which deploys an ELK cluster that can receive and parse logs via syslog that contain JSON.
 
 Most users will want to combine Logsearch Core with a Logsearch Addon to customise their cluster for a
 particular type of logs.  Its likely you want to be following an Addon installation guides - see below
@@ -36,7 +39,7 @@ for a list of the common Addons:
 
 
 ## Installing Logsearch Core
-   
+
 Before starting deployment, make sure your BOSH environment is ready, and all `BOSH_` evironment variables are set. We suggest you to use [BBL](https://github.com/cloudfoundry/bosh-bootloader) tool to spin up the BOSH environment.
 
 ```
@@ -45,7 +48,7 @@ $ bosh -d logsearch deploy logsearch-deployment.yml
 ```
 ## Common customisations:
 
-0. Adding new parsing rules:
+Adding new parsing rules:
 
         logstash_parser:
           filters: |
@@ -63,21 +66,6 @@ $ bosh -d logsearch deploy logsearch-deployment.yml
 
 ## Known issues
 
-#### VMs lose connectivity to each other after VM recreation (eg. instance type upgrade)
-
-While this issue is not specific to this boshrelease, it is worth noting.
-
-On certain IAAS'es, (AWS confirmed), the bosh-agent fails to flush the ARP cache of the VMs in the deployment which, in rare cases, results in VMs not being able to communicate with each other after some of them has been recreated. The symptoms of when this happens are varied depending on the affected VMs. It could be anything from HAproxy reporting it couldn't find any backends (eg. Kibana) or the parsers failing to connect to the queue.
-
-To prevent stale ARP entries, set the `director.flush_arp` property of your BOSH deployment to `true`.
-
-The issue, if occurs, should fix itself as the kernel updates incomplete ARP entries, which **should** happen within minutes
-
-This can also be done manually if an immediate manual fix is preferred. This should be done on the VMs that are trying to talk to the VM that has been recreated.
-
-```
-arp -d $recreated_vm_ip
-```
 
 ## License
 

config/blobs.yml

@@ -56,6 +56,7 @@ curator/vendor/voluptuous-0.11.5-py2.py3-none-any.whl:
   sha: b5c82285df0610464a9332c567cf8bd06b8786f6
 elasticsearch/elasticsearch-7.6.1-linux-x86_64.tar.gz:
   size: 296454172
+  object_id: a7447f2a-1772-4892-508e-43e5029c54ba
   sha: sha256:25583ddd44a99437958f7f9410cd9746c8230b367d570cdf69e96824a583748a
 haproxy/haproxy-1.7.5.tar.gz:
   size: 1743979
@@ -67,9 +68,11 @@ haproxy/pcre-8.40.tar.gz:
   sha: 10384eb3d411794cc15f55b9d837d3f69e35391e
 kibana/kibana-7.6.1-linux-x86_64.tar.gz:
   size: 249498863
+  object_id: df54821e-9deb-4c08-4f02-6f160dec5913
   sha: sha256:da636529511e707bbbc621dc131ff2ed18f50fe0df30821c375d16c5ba4248f6
 logstash/logstash-7.6.1.tar.gz:
   size: 172679481
+  object_id: 8f5d650d-58a2-4774-566a-519d8e085ae5
   sha: sha256:6b16f3158829ad820463c7f3ca4cfec433d12d0eafa25be203c92d12ca91da10
 logstash/logstash-filter-alter-3.0.2.zip:
   size: 7425

deployment/logsearch-deployment.yml

@@ -35,7 +35,7 @@ instance_groups:
       elasticsearch:
         node:
           allow_master: true
-        config_options:
+        config_options: &xpack-options
           xpack.monitoring.enabled: false
           xpack.graph.enabled: false
           xpack.ml.enabled: false
@@ -47,6 +47,7 @@ instance_groups:
   - z1
   instances: 1
   vm_type: medium
+  persistent_disk_type: 5GB
   stemcell: default
   update:
     serial: true
@@ -55,10 +56,15 @@ instance_groups:
   jobs:
   - name: bpm
     release: bpm
-  - name: elasticsearch_config
+  - name: elasticsearch
     release: logsearch
     consumes:
       elasticsearch: {from: elasticsearch_master}
+    properties:
+      elasticsearch:
+        config_options: *xpack-options
+  - name: elasticsearch_config
+    release: logsearch
     properties:
       elasticsearch_config:
         index_prefix: logs-
@@ -68,8 +74,6 @@ instance_groups:
           - index-mappings: /var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
   - name: curator
     release: logsearch
-    consumes:
-      elasticsearch: {from: elasticsearch_master}
   - name: smoke_tests
     release: logsearch
     consumes:
@@ -101,12 +105,7 @@ instance_groups:
       elasticsearch:
         node:
           allow_data: true
-        config_options:
-          xpack.monitoring.enabled: false
-          xpack.graph.enabled: false
-          xpack.ml.enabled: false
-          xpack.security.enabled: false
-          xpack.watcher.enabled: false
+        config_options: *xpack-options
 
 - name: kibana
   azs:
@@ -124,6 +123,9 @@ instance_groups:
     release: logsearch
     consumes:
       elasticsearch: {from: elasticsearch_master}
+    properties:
+      elasticsearch:
+        config_options: *xpack-options
   - name: kibana
     release: logsearch
     provides:
@@ -136,12 +138,7 @@ instance_groups:
           timeout: 500
         env:
           - NODE_ENV: production
-        config_options:
-          xpack.monitoring.enabled: false
-          xpack.graph.enabled: false
-          xpack.ml.enabled: false
-          xpack.security.enabled: false
-          xpack.watcher.enabled: false
+        config_options: *xpack-options
 
 - name: ingestor
   azs:
@@ -160,6 +157,9 @@ instance_groups:
     release: logsearch
     consumes:
       elasticsearch: {from: elasticsearch_master}
+    properties:
+      elasticsearch:
+        config_options: *xpack-options
   - name: ingestor_syslog
     release: logsearch
     consumes:
@@ -175,25 +175,6 @@ instance_groups:
         deployment_dictionary:
           - /var/vcap/packages/logsearch-config/deployment_lookup.yml
 
-- name: ls-router
-  azs:
-  - z1
-  instances: 1
-  vm_type: medium
-  stemcell: default
-  networks:
-  - name: default
-  jobs:
-  - name: bpm
-    release: bpm
-  - name: haproxy
-    release: logsearch
-    consumes:
-      elasticsearch: {from: elasticsearch_master}
-      ingestor: {from: ingestor_link}
-      kibana: {from: kibana_link}
-      syslog_forwarder: nil
-
 releases:
 - name: logsearch
   url: https://s3.amazonaws.com/logsearch/logsearch-211.0.3.tgz

deployment/operations/cloudfoundry.yml

@@ -20,7 +20,6 @@
     release: logsearch-for-cloudfoundry
     consumes:
       cloud_controller: {from: cloud_controller, deployment: cf}
-      elasticsearch: {from: elasticsearch_master}
     properties:
       cloudfoundry:
         user: admin
@@ -105,7 +104,7 @@
           skip_ssl_validation: true
 
 - type: replace
-  path: /instance_groups/name=ls-router/jobs/-
+  path: /instance_groups/name=kibana/jobs/-
   value:
     name: route_registrar
     release: routing
@@ -117,7 +116,7 @@
       route_registrar:
         routes:
         - name: kibana
-          port: 80
+          port: 5601
           registration_interval: 60s
           uris:
           - "logs.((system_domain))"

deployment/operations/disable-post-start.yml

@@ -8,3 +8,23 @@
   path: /instance_groups/name=elasticsearch_data/jobs/name=elasticsearch/properties/elasticsearch/health?
   value:
     disable_post_start: true
+
+- type: replace
+  path: /instance_groups/name=maintenance/jobs/name=elasticsearch/properties/elasticsearch/health?
+  value:
+    disable_post_start: true
+
+- type: replace
+  path: /instance_groups/name=ingestor/jobs/name=elasticsearch/properties/elasticsearch/health?
+  value:
+    disable_post_start: true
+
+- type: replace
+  path: /instance_groups/name=kibana/jobs/name=elasticsearch/properties/elasticsearch/health?
+  value:
+    disable_post_start: true
+
+- type: replace
+  path: /instance_groups/name=kibana/jobs/name=kibana/properties/kibana/health?
+  value:
+    disable_post_start: true

deployment/operations/enable-router.yml

@@ -0,0 +1,46 @@
+---
+- type: replace
+  path: /instance_groups/name=ls-router?
+  value:
+    name: ls-router
+    azs:
+    - z1
+    instances: 1
+    vm_type: medium
+    stemcell: default
+    networks:
+    - name: default
+    jobs:
+    - name: bpm
+      release: bpm
+    - name: haproxy
+      release: logsearch
+      consumes:
+        elasticsearch: {from: elasticsearch_master}
+        ingestor: {from: ingestor_link}
+        kibana: {from: kibana_link}
+        syslog_forwarder: nil
+
+# Uncomment following to relocate route-registrar to ls-router
+# Make sure then this OPS file comes after cloudfoundry.yml 
+
+# - type: remove
+#   path: /instance_groups/name=kibana/jobs/name=route_registrar
+
+# - type: replace
+#   path: /instance_groups/name=ls-router/jobs/-
+#   value:
+#     name: route_registrar
+#     release: routing
+#     consumes:
+#       nats:
+#         from: nats
+#         deployment: cf
+#     properties:
+#       route_registrar:
+#         routes:
+#         - name: kibana
+#           port: 80
+#           registration_interval: 60s
+#           uris:
+#           - "logs.((system_domain))"

deployment/operations/enable-ssl.yml

@@ -0,0 +1,9 @@
+---
+# ingestor
+- type: replace
+  path: /instance_groups/name=ingestor/jobs/name=ingestor_syslog/properties?/logstash_ingestor?/syslog_tls?
+  value:
+    port: 6514
+    ssl_cert: ((haproxy-ssl.certificate))
+    ssl_key: ((haproxy-ssl.private_key))
+    skip_ssl_validation: false