Skip to content

Commit 778b582

Browse files
geerlingguygeerlingguy
committed
Issue #196: Move ELK example into this repository. Preliminary tests.
1 parent 68815ac commit 778b582

19 files changed

+391
-0
lines changed

‎.travis.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,9 @@ env:
2929
# - playbook: dynamic-inventory.yml
3030
# distro: ubuntu1604
3131

32+
- playbook: elk.yml
33+
distro: ubuntu1804
34+
3235
- playbook: https-self-signed.yml
3336
distro: ubuntu1604
3437

‎elk/README.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Ansible Vagrant profile for ELK (Elasticsearch, Logstash, Kibana)
2+
3+
## Background
4+
5+
Vagrant and VirtualBox (or some other VM provider) can be used to quickly build or rebuild virtual servers.
6+
7+
This Vagrant profile configures two servers:
8+
9+
1. A server with the ELK stack ([Elasticsearch](http://www.elasticsearch.org/), [Logstash](http://logstash.net/), and [Kibana](http://www.elasticsearch.org/overview/kibana/)) using the [Ansible](http://www.ansible.com/) provisioner.
10+
2. A server with the Nginx and [Filebeat](https://www.elastic.co/products/beats/filebeat) using the [Ansible](http://www.ansible.com/) provisioner, which routes Nginx access logs to the first server.
11+
12+
## Getting Started
13+
14+
This README file is inside a folder that contains a `Vagrantfile` (hereafter this folder shall be called the [vagrant_root]), which tells Vagrant how to set up your virtual machine in VirtualBox.
15+
16+
To use the vagrant file, you will need to have done the following:
17+
18+
1. Download and Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads)
19+
2. Download and Install [Vagrant](https://www.vagrantup.com/downloads.html)
20+
3. Install [Ansible](https://www.ansible.com/) ([guide for installing Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html))
21+
4. Open a shell prompt (Terminal app on a Mac) and cd into the folder containing the `Vagrantfile`
22+
5. Run the following command to install the necessary Ansible roles for this profile: `$ ansible-galaxy install -r requirements.yml`
23+
24+
Once all of that is done, you can simply type in `vagrant up`, and Vagrant will create both new VMs and configure them.
25+
26+
Once the VMs are up and running (after `vagrant up` is complete and you're back at the command prompt), you can log into either one via SSH if you'd like by typing in `vagrant ssh [name]` (either `logs` for the ELK server, or `webs` for the Nginx server). Otherwise, the next steps are below.
27+
28+
### Setting up your hosts file
29+
30+
You need to modify your host machine's hosts file (Mac/Linux: `/etc/hosts`; Windows: `%systemroot%\system32\drivers\etc\hosts`), adding the lines below:
31+
32+
192.168.9.90 logs.test
33+
192.168.9.91 webs.test
34+
35+
(Where `logs.test`/`webs.test` is the hostname you have configured in the `Vagrantfile`).
36+
37+
After that is configured, you could visit http://logs.test/ in a browser, and you'll see the Kibana dashboard, and you can visit http://webs.test/, and you'll see Nginx's default index page.
38+
39+
If you'd like additional assistance editing your hosts file, please read [How do I modify my hosts file?](http://www.rackspace.com/knowledge_center/article/how-do-i-modify-my-hosts-file) from Rackspace.
40+
41+
## Author Information
42+
43+
Created in 2014 by [Jeff Geerling](https://www.jeffgeerling.com/), author of [Ansible for DevOps](https://www.ansiblefordevops.com/).

‎elk/Vagrantfile

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# -*- mode: ruby -*-
2+
# vi: set ft=ruby :
3+
4+
VAGRANTFILE_API_VERSION = "2"
5+
6+
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
7+
config.vm.box = "geerlingguy/ubuntu1804"
8+
config.ssh.insert_key = false
9+
10+
config.vm.provider :virtualbox do |v|
11+
v.memory = 2048
12+
v.cpus = 2
13+
v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
14+
v.customize ["modifyvm", :id, "--ioapic", "on"]
15+
end
16+
17+
# ELK server.
18+
config.vm.define "logs" do |logs|
19+
logs.vm.hostname = "logs.test"
20+
logs.vm.network :private_network, ip: "192.168.9.90"
21+
22+
logs.vm.provision :ansible do |ansible|
23+
ansible.playbook = "provisioning/elk/playbook.yml"
24+
ansible.inventory_path = "provisioning/elk/inventory"
25+
ansible.become = true
26+
end
27+
end
28+
29+
# Web server.
30+
config.vm.define "webs" do |webs|
31+
webs.vm.hostname = "webs.test"
32+
webs.vm.network :private_network, ip: "192.168.9.91"
33+
34+
webs.vm.provision :ansible do |ansible|
35+
ansible.compatibility_mode = "2.0"
36+
ansible.playbook = "provisioning/web/playbook.yml"
37+
ansible.inventory_path = "provisioning/web/inventory"
38+
ansible.become = true
39+
end
40+
end
41+
42+
end

‎elk/ansible.cfg

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[defaults]
2+
roles_path = ./roles
3+
nocows = 1
4+
5+
[ssh_connection]
6+
pipelining = True
7+
control_path = /tmp/ansible-ssh-%%h-%%p-%%r

‎elk/provisioning/elk/elk-vagrant-example.crt

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
-----BEGIN CERTIFICATE-----
2+
MIICpDCCAYwCCQCLxH8FCzXqnjANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAls
3+
b2dzLnRlc3QwHhcNMjAwMTEzMTg1MDU0WhcNMzAwMTEwMTg1MDU0WjAUMRIwEAYD
4+
VQQDDAlsb2dzLnRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCm
5+
/AWIzC+jjImCWpJJf8JftlqC8sjkB6SPBwrKpRqkIfuwIYcOvivcd9/jpUe4MMvw
6+
swy/DQO9gGnopotwaMcGRfYLQDunxjLijgMIdyKi1EQyf6FqHN/zQF5S/hXKJUTf
7+
ULvuWD/zAJPhPqxRAwPVqVnfZgk4qDr8ctnnchJ6fTAUfaHVKAk7IaPjG90lXdS4
8+
wYcdz3umIsXZ+GaYrJfrPQ+hJaHgRvlSlLZBKra7GwvSJBfxYObrsW9cBcOxZ+X+
9+
qWg4pyOew1IIIM00b0rl2KB05xAvmzKfPYOZxxm02W+YVvpxKa+F8ktjLlRWZCBl
10+
mDwLlL/EutIzVo8Hl4UNAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAArsXVWy3BFR
11+
uF5B3wbBaubfHWQ61/+JBbrUzWfaoNvgjtMOyfz3ijJPtFcb+4IgNIo7WMw1JJeV
12+
4PR0CEXnNmDl6qiA0dBqrI4PP2h3clG2U/8A4rFZ3wtAFhV2L1kMIsJ5IKd1Z+Jd
13+
xUY3u4v/MAW6EnH2OZM6VKCDpFPeZA0vEFseLAkxOxW4jj7+eqZjnYVzLxmnxsF5
14+
nQziI1mN0trtofhcLPi8I9gC0uOyRzgBXbQiqBPIDX+16uhmzS2n3tjBOxdKNqxW
15+
b9U+xafYa4k693IdnIhtJPDq9G8fbOhKV0fZ548bmkYMPan/OJ3yJx88o6AH8edh
16+
z48RJYo1BiY=
17+
-----END CERTIFICATE-----

‎elk/provisioning/elk/elk-vagrant-example.key

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
-----BEGIN PRIVATE KEY-----
2+
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCm/AWIzC+jjImC
3+
WpJJf8JftlqC8sjkB6SPBwrKpRqkIfuwIYcOvivcd9/jpUe4MMvwswy/DQO9gGno
4+
potwaMcGRfYLQDunxjLijgMIdyKi1EQyf6FqHN/zQF5S/hXKJUTfULvuWD/zAJPh
5+
PqxRAwPVqVnfZgk4qDr8ctnnchJ6fTAUfaHVKAk7IaPjG90lXdS4wYcdz3umIsXZ
6+
+GaYrJfrPQ+hJaHgRvlSlLZBKra7GwvSJBfxYObrsW9cBcOxZ+X+qWg4pyOew1II
7+
IM00b0rl2KB05xAvmzKfPYOZxxm02W+YVvpxKa+F8ktjLlRWZCBlmDwLlL/EutIz
8+
Vo8Hl4UNAgMBAAECggEAMQj+4w36y08grjvEi3rN7GdTKvL6lSO9ahbKT3muWhZ0
9+
n2MpEuY7YRF+mI3XgropmTQrAQtkVsiX0S7e8ZKDJ09yEhm5qunj1W2YbGr3q9Mj
10+
d6TNFKkIBpmXWU2tvl+1WtKNbU9S8BvyUQmjdmKpRJvnfVesS4vzTmC3pmR0Eq2o
11+
yTZHgOsawPq2n2poOgboJN1XyNi8dFY28Z50tJHjlXNqGi8K3J/EohkIif7wYFna
12+
a2hwpkh1qdcdZWtjCe/wNbfELOL5WzmYqiKT5NxsmU0Ah0tq9QCAirVZ5q61R/lP
13+
5M1qfKyWgk7D284f3ezgLvCXozpBxM3hleB2Fmqc4QKBgQDa4doF5TTQVi4acX+Y
14+
jQu8OIJ0FlaaX8puKUSKR23VTekcXCj/lNZ0s/dhBYpwn8ggqwtGSeQU3qk2WsGz
15+
0MgjGdVWcnGAfPCgVCS/ULNi9yW6zsPcuX84unzfblei+b9E1eRlpcux/lGaitrh
16+
csOqiFhr8cUmY3tjYhkivXgJNQKBgQDDTS0QaIoSs+Zpt2gMZqKmuwx6Q8ORTR7L
17+
LokhOOGJYligEeDHVRNA0f3wItnhazoWJE4sU09hjPLXzIn8QKBDy9xODQqgzTVp
18+
MlNDR1Kxb+hwxbRIx3WLrrGnqu2wPYWfpOLVkx/oIMWTUehsixq4FbhL0DK9eypA
19+
K36T0z7feQKBgQCoMm/woc5w3hi1d0w+tPw0mhQdEuFf5YZQGuQxgmwzQpiEk4lp
20+
xTz7FyRc6P4WR6JpACc8zyE2rFfJjfpVrHVvC7X1pSH1Q41BFwfaADCpNxRGNgcq
21+
gVzzAdNdepRbh6FkSx0kw0ABOlYI9PT309HqJLYV1QbT1hSGLWb1XDV2lQKBgEju
22+
1zdbcsSNPiyYe1i2M2OEUqKOD8iBPlCuLjOj5EXD3kjYpFKMKMZ6JRS6TxdpTaXb
23+
JP5Ulj5dvopsLdNv5umqhbj4EdH/EEUasBFd3zm4CtuiWTMcmX7yXBG7OUwpjW8S
24+
ykrryGyNYHsliWCPigjYw5w8QdaU2jBgjcJyjvYxAoGBALS3NVYsbCtG6nE/Elx6
25+
FoYtxJbA7Hy0AsrgvRlZ0nlIzGUhmFNmnlneUy/22KYMR+vFT7qLxPKg5gtSxysC
26+
EuDv3ODSY0aCT/7m93Nzlp4w/BLjDP0jhSiwkXHkj4vCj37mYrXsAbmzO55M9Cme
27+
kQHqVKo9bXlTWBg5ONYP7cIb
28+
-----END PRIVATE KEY-----

‎elk/provisioning/elk/elk-vagrant-example.p8

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
-----BEGIN PRIVATE KEY-----
2+
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCm/AWIzC+jjImC
3+
WpJJf8JftlqC8sjkB6SPBwrKpRqkIfuwIYcOvivcd9/jpUe4MMvwswy/DQO9gGno
4+
potwaMcGRfYLQDunxjLijgMIdyKi1EQyf6FqHN/zQF5S/hXKJUTfULvuWD/zAJPh
5+
PqxRAwPVqVnfZgk4qDr8ctnnchJ6fTAUfaHVKAk7IaPjG90lXdS4wYcdz3umIsXZ
6+
+GaYrJfrPQ+hJaHgRvlSlLZBKra7GwvSJBfxYObrsW9cBcOxZ+X+qWg4pyOew1II
7+
IM00b0rl2KB05xAvmzKfPYOZxxm02W+YVvpxKa+F8ktjLlRWZCBlmDwLlL/EutIz
8+
Vo8Hl4UNAgMBAAECggEAMQj+4w36y08grjvEi3rN7GdTKvL6lSO9ahbKT3muWhZ0
9+
n2MpEuY7YRF+mI3XgropmTQrAQtkVsiX0S7e8ZKDJ09yEhm5qunj1W2YbGr3q9Mj
10+
d6TNFKkIBpmXWU2tvl+1WtKNbU9S8BvyUQmjdmKpRJvnfVesS4vzTmC3pmR0Eq2o
11+
yTZHgOsawPq2n2poOgboJN1XyNi8dFY28Z50tJHjlXNqGi8K3J/EohkIif7wYFna
12+
a2hwpkh1qdcdZWtjCe/wNbfELOL5WzmYqiKT5NxsmU0Ah0tq9QCAirVZ5q61R/lP
13+
5M1qfKyWgk7D284f3ezgLvCXozpBxM3hleB2Fmqc4QKBgQDa4doF5TTQVi4acX+Y
14+
jQu8OIJ0FlaaX8puKUSKR23VTekcXCj/lNZ0s/dhBYpwn8ggqwtGSeQU3qk2WsGz
15+
0MgjGdVWcnGAfPCgVCS/ULNi9yW6zsPcuX84unzfblei+b9E1eRlpcux/lGaitrh
16+
csOqiFhr8cUmY3tjYhkivXgJNQKBgQDDTS0QaIoSs+Zpt2gMZqKmuwx6Q8ORTR7L
17+
LokhOOGJYligEeDHVRNA0f3wItnhazoWJE4sU09hjPLXzIn8QKBDy9xODQqgzTVp
18+
MlNDR1Kxb+hwxbRIx3WLrrGnqu2wPYWfpOLVkx/oIMWTUehsixq4FbhL0DK9eypA
19+
K36T0z7feQKBgQCoMm/woc5w3hi1d0w+tPw0mhQdEuFf5YZQGuQxgmwzQpiEk4lp
20+
xTz7FyRc6P4WR6JpACc8zyE2rFfJjfpVrHVvC7X1pSH1Q41BFwfaADCpNxRGNgcq
21+
gVzzAdNdepRbh6FkSx0kw0ABOlYI9PT309HqJLYV1QbT1hSGLWb1XDV2lQKBgEju
22+
1zdbcsSNPiyYe1i2M2OEUqKOD8iBPlCuLjOj5EXD3kjYpFKMKMZ6JRS6TxdpTaXb
23+
JP5Ulj5dvopsLdNv5umqhbj4EdH/EEUasBFd3zm4CtuiWTMcmX7yXBG7OUwpjW8S
24+
ykrryGyNYHsliWCPigjYw5w8QdaU2jBgjcJyjvYxAoGBALS3NVYsbCtG6nE/Elx6
25+
FoYtxJbA7Hy0AsrgvRlZ0nlIzGUhmFNmnlneUy/22KYMR+vFT7qLxPKg5gtSxysC
26+
EuDv3ODSY0aCT/7m93Nzlp4w/BLjDP0jhSiwkXHkj4vCj37mYrXsAbmzO55M9Cme
27+
kQHqVKo9bXlTWBg5ONYP7cIb
28+
-----END PRIVATE KEY-----

‎elk/provisioning/elk/inventory

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[logs]
2+
logs.test ansible_ssh_host=192.168.9.90 ansible_ssh_port=22

‎elk/provisioning/elk/playbook.yml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
# Ansible playbook for an ELK (Elasticsearch, Logstash, Kibana) logging server.
3+
#
4+
# @author Jeff Geerling (2014).
5+
6+
- hosts: logs
7+
gather_facts: yes
8+
9+
vars_files:
10+
- vars/main.yml
11+
12+
pre_tasks:
13+
- name: Update apt cache if needed.
14+
apt: update_cache=yes cache_valid_time=86400
15+
16+
roles:
17+
- geerlingguy.java
18+
- geerlingguy.nginx
19+
- geerlingguy.elasticsearch
20+
- geerlingguy.elasticsearch-curator
21+
- geerlingguy.kibana
22+
- geerlingguy.logstash
23+
- geerlingguy.filebeat

‎elk/provisioning/elk/vars/main.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
java_packages:
3+
- openjdk-8-jdk
4+
5+
kibana_server_name: logs.test
6+
kibana_username: kibana
7+
kibana_password: password
8+
9+
nginx_user: www-data
10+
nginx_worker_connections: 1024
11+
nginx_remove_default_vhost: true
12+
nginx_vhosts:
13+
# Kibana proxy.
14+
- listen: "80 default_server"
15+
filename: kibana.conf
16+
server_name: logs.test
17+
extra_parameters: |
18+
location / {
19+
include /etc/nginx/proxy_params;
20+
proxy_pass http://localhost:5601;
21+
proxy_set_header Authorization "";
22+
proxy_read_timeout 90s;
23+
}
24+
25+
logstash_ssl_key_file: elk-vagrant-example.p8
26+
logstash_ssl_certificate_file: elk-vagrant-example.crt
27+
28+
filebeat_output_logstash_enabled: true
29+
filebeat_output_logstash_hosts:
30+
- "logs.test:5044"
31+
32+
filebeat_ssl_key_file: elk-vagrant-example.p8
33+
filebeat_ssl_certificate_file: elk-vagrant-example.crt
34+
filebeat_ssl_insecure: "true"
35+
36+
filebeat_inputs:
37+
- type: log
38+
paths:
39+
- /var/log/auth.log

0 commit comments

Comments
 (0)