Skip to content

Commit b831f38

Browse files
mabrarovmabrarov
committed
Configuration for testing fluent/fluent-bit#1560.
0 parents  commit b831f38

File tree

14 files changed

+471
-0
lines changed

14 files changed

+471
-0
lines changed

‎.gitattributes‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
* text=auto
2+
*.sh text eol=lf
3+
*.conf text eol=lf
4+
*.bat text eol=crlf
5+
*.cmd text eol=crlf
6+
*.ps1 text eol=crlf

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/.idea/
2+
/build/

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Configurations for Elastic stack and related technologies

‎elastic-cluster/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Docker Compose project for Elastic cluster
2+
3+
Docker Compose project for Elastic cluster consisting of:
4+
5+
1. 3 master / data / ingesting Elasticsearch nodes.
6+
1. 1 coordinating Elasticsearch node.
7+
1. 1 Kibana node

‎elastic-cluster/docker-compose.yaml‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
version: '2.1'
2+
3+
x-logging: &default-logging
4+
driver: 'json-file'
5+
options:
6+
max-size: '12m'
7+
max-file: '5'
8+
9+
volumes:
10+
elasticsearch1-data:
11+
elasticsearch2-data:
12+
elasticsearch3-data:
13+
elasticsearch4-data:
14+
kibana-data:
15+
16+
services:
17+
elasticsearch1:
18+
image: 'elasticsearch:7.17.9'
19+
mem_limit: '1024m'
20+
mem_reservation: '512m'
21+
ulimits:
22+
memlock:
23+
soft: -1
24+
hard: -1
25+
environment:
26+
node.name: 'elasticsearch1'
27+
cluster.name: 'elasticsearch'
28+
discovery.seed_hosts: 'elasticsearch1,elasticsearch2,elasticsearch3'
29+
cluster.initial_master_nodes: 'elasticsearch1,elasticsearch2,elasticsearch3'
30+
node.master: 'true'
31+
node.data: 'true'
32+
node.attr.data: 'hot'
33+
node.ingest: 'true'
34+
node.ml: 'false'
35+
node.transform: 'false'
36+
node.remote_cluster_client: 'false'
37+
xpack.ml.enabled: 'false'
38+
xpack.transform.enabled: 'false'
39+
xpack.security.enabled: 'true'
40+
ELASTIC_PASSWORD: 'elastic'
41+
ports:
42+
- '9201:9200'
43+
volumes:
44+
- 'elasticsearch1-data:/usr/share/elasticsearch/data'
45+
logging: *default-logging
46+
47+
elasticsearch2:
48+
image: 'elasticsearch:7.17.9'
49+
mem_limit: '1024m'
50+
mem_reservation: '512m'
51+
ulimits:
52+
memlock:
53+
soft: -1
54+
hard: -1
55+
environment:
56+
node.name: 'elasticsearch2'
57+
cluster.name: 'elasticsearch'
58+
discovery.seed_hosts: 'elasticsearch1,elasticsearch2,elasticsearch3'
59+
cluster.initial_master_nodes: 'elasticsearch1,elasticsearch2,elasticsearch3'
60+
bootstrap.memory_lock: 'true'
61+
node.master: 'true'
62+
node.data: 'true'
63+
node.attr.data: 'hot'
64+
node.ingest: 'true'
65+
node.ml: 'false'
66+
node.transform: 'false'
67+
node.remote_cluster_client: 'false'
68+
xpack.ml.enabled: 'false'
69+
xpack.transform.enabled: 'false'
70+
xpack.security.enabled: 'true'
71+
ELASTIC_PASSWORD: 'elastic'
72+
ports:
73+
- '9202:9200'
74+
volumes:
75+
- 'elasticsearch2-data:/usr/share/elasticsearch/data'
76+
logging: *default-logging
77+
78+
elasticsearch3:
79+
image: 'elasticsearch:7.17.9'
80+
mem_limit: '1024m'
81+
mem_reservation: '512m'
82+
ulimits:
83+
memlock:
84+
soft: -1
85+
hard: -1
86+
environment:
87+
node.name: 'elasticsearch3'
88+
cluster.name: 'elasticsearch'
89+
discovery.seed_hosts: 'elasticsearch1,elasticsearch2,elasticsearch3'
90+
cluster.initial_master_nodes: 'elasticsearch1,elasticsearch2,elasticsearch3'
91+
bootstrap.memory_lock: 'true'
92+
node.master: 'true'
93+
node.data: 'true'
94+
node.attr.data: 'hot'
95+
node.ingest: 'true'
96+
node.ml: 'false'
97+
node.transform: 'false'
98+
node.remote_cluster_client: 'false'
99+
xpack.ml.enabled: 'false'
100+
xpack.transform.enabled: 'false'
101+
xpack.security.enabled: 'true'
102+
ELASTIC_PASSWORD: 'elastic'
103+
ports:
104+
- '9203:9200'
105+
volumes:
106+
- 'elasticsearch3-data:/usr/share/elasticsearch/data'
107+
logging: *default-logging
108+
109+
elasticsearch4:
110+
image: 'elasticsearch:7.17.9'
111+
mem_limit: '1024m'
112+
mem_reservation: '512m'
113+
ulimits:
114+
memlock:
115+
soft: -1
116+
hard: -1
117+
environment:
118+
node.name: 'elasticsearch4'
119+
cluster.name: 'elasticsearch'
120+
discovery.seed_hosts: 'elasticsearch1,elasticsearch2,elasticsearch3'
121+
cluster.initial_master_nodes: 'elasticsearch1,elasticsearch2,elasticsearch3'
122+
bootstrap.memory_lock: 'true'
123+
node.master: 'false'
124+
node.voting_only: 'false'
125+
node.data: 'false'
126+
node.ingest: 'false'
127+
node.ml: 'false'
128+
node.transform: 'false'
129+
node.remote_cluster_client: 'false'
130+
xpack.ml.enabled: 'false'
131+
xpack.transform.enabled: 'false'
132+
xpack.security.enabled: 'true'
133+
ELASTIC_PASSWORD: 'elastic'
134+
volumes:
135+
- 'elasticsearch4-data:/usr/share/elasticsearch/data'
136+
logging: *default-logging
137+
138+
kibana:
139+
image: 'kibana:7.17.9'
140+
mem_limit: '1024m'
141+
mem_reservation: '512m'
142+
ulimits:
143+
memlock:
144+
soft: -1
145+
hard: -1
146+
environment:
147+
SERVER_PUBLICBASEURL: 'http://localhost:5601'
148+
ELASTICSEARCH_HOSTS: 'http://elasticsearch4:9200'
149+
xpack.security.enabled: 'true'
150+
ELASTICSEARCH_USERNAME: 'elastic'
151+
ELASTICSEARCH_PASSWORD: 'elastic'
152+
TELEMETRY_ENABLED: 'false'
153+
NEWSFEED_ENABLED: 'false'
154+
ports:
155+
- '5601:5601'
156+
volumes:
157+
- 'kibana-data:/usr/share/kibana/data'
158+
logging: *default-logging

‎elastic-cluster/init.sh‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
DOCKER_HOST='localhost'
6+
7+
elasticsearch_api_user='elastic'
8+
elasticsearch_api_password='elastic'
9+
elasticsearch_api_url_base="http://${DOCKER_HOST}:9201"
10+
kibana_api_url_base="http://${DOCKER_HOST}:5601"
11+
elasticsearch_index_alias="fluent-bit"
12+
elasticsearch_index_name="${elasticsearch_index_alias}-000001"
13+
kibana_index_pattern_id="${elasticsearch_index_alias}"
14+
kibana_index_pattern_title="${elasticsearch_index_alias}-*"
15+
16+
response_status_code="$(curl -ks -X POST \
17+
-o /dev/null -w '%{http_code}\n' \
18+
--user "${elasticsearch_api_user}:${elasticsearch_api_password}" \
19+
"${elasticsearch_api_url_base}/_license/start_trial?acknowledge=true")"
20+
if [[ "${response_status_code}" -ne 200 ]]; then
21+
echo "Failed to set Elasticsearch trial license" >&2
22+
exit 1
23+
fi
24+
25+
response_status_code="$(curl -s -X PUT \
26+
-o /dev/null -w '%{http_code}\n' \
27+
--user "${elasticsearch_api_user}:${elasticsearch_api_password}" \
28+
-H 'Content-Type: application/json' \
29+
-d \
30+
"{
31+
\"aliases\": {
32+
\"${elasticsearch_index_alias}\": {}
33+
},
34+
\"mappings\": {
35+
\"properties\": {
36+
\"@timestamp\": {
37+
\"type\": \"date\"
38+
},
39+
\"message\": {
40+
\"type\": \"text\",
41+
\"fields\": {
42+
\"keyword\": {
43+
\"type\": \"text\"
44+
}
45+
}
46+
}
47+
}
48+
},
49+
\"settings\": {
50+
\"index\": {
51+
\"number_of_shards\": 2,
52+
\"number_of_replicas\": 1,
53+
\"refresh_interval\": \"30s\"
54+
}
55+
}
56+
}
57+
" \
58+
"${elasticsearch_api_url_base}/${elasticsearch_index_name}")"
59+
if [[ "${response_status_code}" -ne 200 ]]; then
60+
echo "Failed to create ${elasticsearch_index_name} Elasticsearch index" >&2
61+
exit 1
62+
else
63+
echo "Successfully created ${elasticsearch_index_name} Elasticsearch index with ${elasticsearch_index_alias} alias"
64+
fi
65+
66+
response_status_code="$(curl -s -X POST \
67+
-o /dev/null -w '%{http_code}\n' \
68+
--user "${elasticsearch_api_user}:${elasticsearch_api_password}" \
69+
-H 'kbn-xsrf: required_header' \
70+
-H 'Content-Type: application/json' \
71+
-d \
72+
"{
73+
\"attributes\": {
74+
\"title\": \"${kibana_index_pattern_title}\",
75+
\"timeFieldName\": \"@timestamp\",
76+
\"fields\": \"[ { \\\"name\\\": \\\"@timestamp\\\", \\\"type\\\": \\\"date\\\", \\\"esTypes\\\": [ \\\"date\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": true, \\\"readFromDocValues\\\": true, \\\"metadata_field\\\": false }, { \\\"name\\\": \\\"_id\\\", \\\"type\\\": \\\"string\\\", \\\"esTypes\\\": [ \\\"_id\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": true, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": true }, { \\\"name\\\": \\\"_index\\\", \\\"type\\\": \\\"string\\\", \\\"esTypes\\\": [ \\\"_index\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": true, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": true }, { \\\"name\\\": \\\"_score\\\", \\\"type\\\": \\\"number\\\", \\\"searchable\\\": false, \\\"aggregatable\\\": false, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": true }, { \\\"name\\\": \\\"_source\\\", \\\"type\\\": \\\"_source\\\", \\\"esTypes\\\": [ \\\"_source\\\" ], \\\"searchable\\\": false, \\\"aggregatable\\\": false, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": true }, { \\\"name\\\": \\\"_type\\\", \\\"type\\\": \\\"string\\\", \\\"esTypes\\\": [ \\\"_type\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": true, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": true }, { \\\"name\\\": \\\"message\\\", \\\"type\\\": \\\"string\\\", \\\"esTypes\\\": [ \\\"text\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": false, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": false }, { \\\"name\\\": \\\"message.keyword\\\", \\\"type\\\": \\\"string\\\", \\\"esTypes\\\": [ \\\"text\\\" ], \\\"searchable\\\": true, \\\"aggregatable\\\": false, \\\"readFromDocValues\\\": false, \\\"metadata_field\\\": false, \\\"subType\\\": { \\\"multi\\\": { \\\"parent\\\": \\\"message\\\" } } } ]\"
77+
}
78+
}
79+
" \
80+
"${kibana_api_url_base}/api/saved_objects/index-pattern/${kibana_index_pattern_id}")"
81+
if [[ "${response_status_code}" -ne 200 ]]; then
82+
echo "Failed to create ${kibana_index_pattern_id} Kibana index pattern" >&2
83+
exit 1
84+
else
85+
echo "Successfully created ${kibana_index_pattern_id} Kibana index pattern with ${kibana_index_pattern_title} title"
86+
fi
87+
88+
echo "Try login with ${elasticsearch_api_user}/${elasticsearch_api_password} credentials to Kibana located at ${kibana_api_url_base}"

‎fluent-bit-es-cluster/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Fluent Bit with Elasticsearch output and Upstream Servers
2+
3+
Refer to:
4+
5+
1. Fluent Bit documentation for [Upstream Servers](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers).
6+
1. [https://github.com/fluent/fluent-bit/pull/1560](https://github.com/fluent/fluent-bit/pull/1560).
7+
1. [https://github.com/mabrarov/fluent-bit/compare/master...mabrarov:fluent-bit:feature/out_es_upstream_support](https://github.com/mabrarov/fluent-bit/compare/master...mabrarov:fluent-bit:feature/out_es_upstream_support).

‎fluent-bit-es-cluster/fluent-bit.conf‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[SERVICE]
2+
Flush 10
3+
4+
[INPUT]
5+
Name dummy
6+
Dummy { "message" : "this is dummy data" }
7+
8+
[OUTPUT]
9+
Workers 4
10+
Name es
11+
Upstream ./out_es_upstream.conf
12+
TLS Off
13+
Retry_Limit 6
14+
Replace_Dots On
15+
HTTP_User elastic
16+
HTTP_Passwd elastic
17+
Index fluent-bit

‎fluent-bit-es-cluster/upstream.conf‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[UPSTREAM]
2+
name elasticsearch-balancing
3+
4+
[NODE]
5+
name elasticsearch1
6+
host localhost
7+
port 9201
8+
9+
[NODE]
10+
name elasticsearch2
11+
host localhost
12+
port 9202
13+
14+
[NODE]
15+
name elasticsearch3
16+
host localhost
17+
port 9203

‎fluent-bit-es/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Fluent Bit with Elasticsearch output

0 commit comments

Comments
 (0)