Skip to content

Commit 9714e00

Browse files
author
marcin
authored
Ability to spin ES cluster outside of VPC (#56)
* Ability to spin ES cluster outside of VPC * Updated readme * FMT * Introduce vpc_enabled variable to mitigate problems with ES spinning in VPC * Introduce vpc_enabled variable to mitigate problems with ES spinning in VPC * Update modules used in test to prevent deprecations and fix vpc_options block * Prevent creating SG for ingress_cidr_blocks if no ingress_cidr_blocks are provided * Do not create a SG without allowed_cidr_blocks definied * Update build-harness, readme and default ES version to latest 7.4 * Additional readme changes * Additional readme changes
1 parent 6c8ee94 commit 9714e00

File tree

8 files changed

+219
-140
lines changed

8 files changed

+219
-140
lines changed

‎README.md

Lines changed: 83 additions & 64 deletions
Large diffs are not rendered by default.

‎README.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,5 @@ contributors:
117117
github: "goruha"
118118
- name: "Sarkis Varozian"
119119
github: "sarkis"
120+
- name: "Marcin Brański"
121+
github: "3h4x"

‎docs/terraform.md

Lines changed: 79 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,88 @@
1+
## Requirements
2+
3+
| Name | Version |
4+
|------|---------|
5+
| terraform | ~> 0.12.0 |
6+
| aws | ~> 2.0 |
7+
| null | ~> 2.0 |
8+
| template | ~> 2.0 |
9+
10+
## Providers
11+
12+
| Name | Version |
13+
|------|---------|
14+
| aws | ~> 2.0 |
15+
| null | ~> 2.0 |
16+
117
## Inputs
218

319
| Name | Description | Type | Default | Required |
4-
|------|-------------|:----:|:-----:|:-----:|
5-
| advanced_options | Key-value string pairs to specify advanced configuration options | map(string) | `<map>` | no |
6-
| allowed_cidr_blocks | List of CIDR blocks to be allowed to connect to the cluster | list(string) | `<list>` | no |
7-
| attributes | Additional attributes (e.g. `1`) | list(string) | `<list>` | no |
8-
| automated_snapshot_start_hour | Hour at which automated snapshots are taken, in UTC | number | `0` | no |
9-
| availability_zone_count | Number of Availability Zones for the domain to use. | number | `2` | no |
10-
| cognito_authentication_enabled | Whether to enable Amazon Cognito authentication with Kibana | bool | `false` | no |
11-
| cognito_iam_role_arn | ARN of the IAM role that has the AmazonESCognitoAccess policy attached | string | `` | no |
12-
| cognito_identity_pool_id | The ID of the Cognito Identity Pool to use | string | `` | no |
13-
| cognito_user_pool_id | The ID of the Cognito User Pool to use | string | `` | no |
14-
| create_iam_service_linked_role | Whether to create `AWSServiceRoleForAmazonElasticsearchService` service-linked role. Set it to `false` if you already have an ElasticSearch cluster created in the AWS account and AWSServiceRoleForAmazonElasticsearchService already exists. See https://github.com/terraform-providers/terraform-provider-aws/issues/5218 for more info | bool | `true` | no |
15-
| dedicated_master_count | Number of dedicated master nodes in the cluster | number | `0` | no |
16-
| dedicated_master_enabled | Indicates whether dedicated master nodes are enabled for the cluster | bool | `false` | no |
17-
| dedicated_master_type | Instance type of the dedicated master nodes in the cluster | string | `t2.small.elasticsearch` | no |
18-
| delimiter | Delimiter to be used between `namespace`, `environment`, `stage`, `name` and `attributes` | string | `-` | no |
19-
| dns_zone_id | Route53 DNS Zone ID to add hostname records for Elasticsearch domain and Kibana | string | `` | no |
20-
| domain_endpoint_options_enforce_https | Whether or not to require HTTPS | bool | `false` | no |
21-
| domain_endpoint_options_tls_security_policy | The name of the TLS security policy that needs to be applied to the HTTPS endpoint | string | `Policy-Min-TLS-1-0-2019-07` | no |
22-
| ebs_iops | The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Applicable only for the Provisioned IOPS EBS volume type | number | `0` | no |
23-
| ebs_volume_size | EBS volumes for data storage in GB | number | `0` | no |
24-
| ebs_volume_type | Storage type of EBS volumes | string | `gp2` | no |
25-
| elasticsearch_subdomain_name | The name of the subdomain for Elasticsearch in the DNS zone (_e.g._ `elasticsearch`, `ui`, `ui-es`, `search-ui`) | string | `` | no |
26-
| elasticsearch_version | Version of Elasticsearch to deploy (_e.g._ `7.1`, `6.8`, `6.7`, `6.5`, `6.4`, `6.3`, `6.2`, `6.0`, `5.6`, `5.5`, `5.3`, `5.1`, `2.3`, `1.5` | string | `6.8` | no |
27-
| enabled | Set to false to prevent the module from creating any resources | bool | `true` | no |
28-
| encrypt_at_rest_enabled | Whether to enable encryption at rest | bool | `true` | no |
29-
| encrypt_at_rest_kms_key_id | The KMS key ID to encrypt the Elasticsearch domain with. If not specified, then it defaults to using the AWS/Elasticsearch service KMS key | string | `` | no |
30-
| environment | Environment, e.g. 'prod', 'staging', 'dev', 'pre-prod', 'UAT' | string | `` | no |
31-
| iam_actions | List of actions to allow for the IAM roles, _e.g._ `es:ESHttpGet`, `es:ESHttpPut`, `es:ESHttpPost` | list(string) | `<list>` | no |
32-
| iam_authorizing_role_arns | List of IAM role ARNs to permit to assume the Elasticsearch user role | list(string) | `<list>` | no |
33-
| iam_role_arns | List of IAM role ARNs to permit access to the Elasticsearch domain | list(string) | `<list>` | no |
34-
| iam_role_max_session_duration | The maximum session duration (in seconds) for the user role. Can have a value from 1 hour to 12 hours | number | `3600` | no |
35-
| ingress_port_range_end | End number for allowed port range. (e.g. `443`) | number | `65535` | no |
36-
| ingress_port_range_start | Start number for allowed port range. (e.g. `443`) | number | `0` | no |
37-
| instance_count | Number of data nodes in the cluster | number | `4` | no |
38-
| instance_type | Elasticsearch instance type for data nodes in the cluster | string | `t2.small.elasticsearch` | no |
39-
| kibana_subdomain_name | The name of the subdomain for Kibana in the DNS zone (_e.g._ `kibana`, `ui`, `ui-es`, `search-ui`, `kibana.elasticsearch`) | string | `kibana` | no |
40-
| label_order | The naming order of the id output and Name tag | list(string) | `<list>` | no |
41-
| log_publishing_application_cloudwatch_log_group_arn | ARN of the CloudWatch log group to which log for ES_APPLICATION_LOGS needs to be published | string | `` | no |
42-
| log_publishing_application_enabled | Specifies whether log publishing option for ES_APPLICATION_LOGS is enabled or not | bool | `false` | no |
43-
| log_publishing_index_cloudwatch_log_group_arn | ARN of the CloudWatch log group to which log for INDEX_SLOW_LOGS needs to be published | string | `` | no |
44-
| log_publishing_index_enabled | Specifies whether log publishing option for INDEX_SLOW_LOGS is enabled or not | bool | `false` | no |
45-
| log_publishing_search_cloudwatch_log_group_arn | ARN of the CloudWatch log group to which log for SEARCH_SLOW_LOGS needs to be published | string | `` | no |
46-
| log_publishing_search_enabled | Specifies whether log publishing option for SEARCH_SLOW_LOGS is enabled or not | bool | `false` | no |
47-
| name | Solution name, e.g. 'app' or 'jenkins' | string | `` | no |
48-
| namespace | Namespace, which could be your organization name or abbreviation, e.g. 'eg' or 'cp' | string | `` | no |
49-
| node_to_node_encryption_enabled | Whether to enable node-to-node encryption | bool | `false` | no |
50-
| security_groups | List of security group IDs to be allowed to connect to the cluster | list(string) | `<list>` | no |
51-
| stage | Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release' | string | `` | no |
52-
| subnet_ids | Subnet IDs | list(string) | - | yes |
53-
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | map(string) | `<map>` | no |
54-
| vpc_id | VPC ID | string | - | yes |
55-
| zone_awareness_enabled | Enable zone awareness for Elasticsearch cluster | bool | `true` | no |
20+
|------|-------------|------|---------|:--------:|
21+
| advanced\_options | Key-value string pairs to specify advanced configuration options | `map(string)` | `{}` | no |
22+
| allowed\_cidr\_blocks | List of CIDR blocks to be allowed to connect to the cluster | `list(string)` | `[]` | no |
23+
| attributes | Additional attributes (e.g. `1`) | `list(string)` | `[]` | no |
24+
| automated\_snapshot\_start\_hour | Hour at which automated snapshots are taken, in UTC | `number` | `0` | no |
25+
| availability\_zone\_count | Number of Availability Zones for the domain to use. | `number` | `2` | no |
26+
| cognito\_authentication\_enabled | Whether to enable Amazon Cognito authentication with Kibana | `bool` | `false` | no |
27+
| cognito\_iam\_role\_arn | ARN of the IAM role that has the AmazonESCognitoAccess policy attached | `string` | `""` | no |
28+
| cognito\_identity\_pool\_id | The ID of the Cognito Identity Pool to use | `string` | `""` | no |
29+
| cognito\_user\_pool\_id | The ID of the Cognito User Pool to use | `string` | `""` | no |
30+
| create\_iam\_service\_linked\_role | Whether to create `AWSServiceRoleForAmazonElasticsearchService` service-linked role. Set it to `false` if you already have an ElasticSearch cluster created in the AWS account and AWSServiceRoleForAmazonElasticsearchService already exists. See https://github.com/terraform-providers/terraform-provider-aws/issues/5218 for more info | `bool` | `true` | no |
31+
| dedicated\_master\_count | Number of dedicated master nodes in the cluster | `number` | `0` | no |
32+
| dedicated\_master\_enabled | Indicates whether dedicated master nodes are enabled for the cluster | `bool` | `false` | no |
33+
| dedicated\_master\_type | Instance type of the dedicated master nodes in the cluster | `string` | `"t2.small.elasticsearch"` | no |
34+
| delimiter | Delimiter to be used between `namespace`, `environment`, `stage`, `name` and `attributes` | `string` | `"-"` | no |
35+
| dns\_zone\_id | Route53 DNS Zone ID to add hostname records for Elasticsearch domain and Kibana | `string` | `""` | no |
36+
| domain\_endpoint\_options\_enforce\_https | Whether or not to require HTTPS | `bool` | `false` | no |
37+
| domain\_endpoint\_options\_tls\_security\_policy | The name of the TLS security policy that needs to be applied to the HTTPS endpoint | `string` | `"Policy-Min-TLS-1-0-2019-07"` | no |
38+
| ebs\_iops | The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Applicable only for the Provisioned IOPS EBS volume type | `number` | `0` | no |
39+
| ebs\_volume\_size | EBS volumes for data storage in GB | `number` | `0` | no |
40+
| ebs\_volume\_type | Storage type of EBS volumes | `string` | `"gp2"` | no |
41+
| elasticsearch\_subdomain\_name | The name of the subdomain for Elasticsearch in the DNS zone (\_e.g.\_ `elasticsearch`, `ui`, `ui-es`, `search-ui`) | `string` | `""` | no |
42+
| elasticsearch\_version | Version of Elasticsearch to deploy (\_e.g.\_ `7.4`, `7.1`, `6.8`, `6.7`, `6.5`, `6.4`, `6.3`, `6.2`, `6.0`, `5.6`, `5.5`, `5.3`, `5.1`, `2.3`, `1.5` | `string` | `"7.4"` | no |
43+
| enabled | Set to false to prevent the module from creating any resources | `bool` | `true` | no |
44+
| encrypt\_at\_rest\_enabled | Whether to enable encryption at rest | `bool` | `true` | no |
45+
| encrypt\_at\_rest\_kms\_key\_id | The KMS key ID to encrypt the Elasticsearch domain with. If not specified, then it defaults to using the AWS/Elasticsearch service KMS key | `string` | `""` | no |
46+
| environment | Environment, e.g. 'prod', 'staging', 'dev', 'pre-prod', 'UAT' | `string` | `""` | no |
47+
| iam\_actions | List of actions to allow for the IAM roles, _e.g._ `es:ESHttpGet`, `es:ESHttpPut`, `es:ESHttpPost` | `list(string)` | `[]` | no |
48+
| iam\_authorizing\_role\_arns | List of IAM role ARNs to permit to assume the Elasticsearch user role | `list(string)` | `[]` | no |
49+
| iam\_role\_arns | List of IAM role ARNs to permit access to the Elasticsearch domain | `list(string)` | `[]` | no |
50+
| iam\_role\_max\_session\_duration | The maximum session duration (in seconds) for the user role. Can have a value from 1 hour to 12 hours | `number` | `3600` | no |
51+
| ingress\_port\_range\_end | End number for allowed port range. (e.g. `443`) | `number` | `65535` | no |
52+
| ingress\_port\_range\_start | Start number for allowed port range. (e.g. `443`) | `number` | `0` | no |
53+
| instance\_count | Number of data nodes in the cluster | `number` | `4` | no |
54+
| instance\_type | Elasticsearch instance type for data nodes in the cluster | `string` | `"t2.small.elasticsearch"` | no |
55+
| kibana\_subdomain\_name | The name of the subdomain for Kibana in the DNS zone (\_e.g.\_ `kibana`, `ui`, `ui-es`, `search-ui`, `kibana.elasticsearch`) | `string` | `"kibana"` | no |
56+
| label\_order | The naming order of the id output and Name tag | `list(string)` | `[]` | no |
57+
| log\_publishing\_application\_cloudwatch\_log\_group\_arn | ARN of the CloudWatch log group to which log for ES\_APPLICATION\_LOGS needs to be published | `string` | `""` | no |
58+
| log\_publishing\_application\_enabled | Specifies whether log publishing option for ES\_APPLICATION\_LOGS is enabled or not | `bool` | `false` | no |
59+
| log\_publishing\_index\_cloudwatch\_log\_group\_arn | ARN of the CloudWatch log group to which log for INDEX\_SLOW\_LOGS needs to be published | `string` | `""` | no |
60+
| log\_publishing\_index\_enabled | Specifies whether log publishing option for INDEX\_SLOW\_LOGS is enabled or not | `bool` | `false` | no |
61+
| log\_publishing\_search\_cloudwatch\_log\_group\_arn | ARN of the CloudWatch log group to which log for SEARCH\_SLOW\_LOGS needs to be published | `string` | `""` | no |
62+
| log\_publishing\_search\_enabled | Specifies whether log publishing option for SEARCH\_SLOW\_LOGS is enabled or not | `bool` | `false` | no |
63+
| name | Solution name, e.g. 'app' or 'jenkins' | `string` | `""` | no |
64+
| namespace | Namespace, which could be your organization name or abbreviation, e.g. 'eg' or 'cp' | `string` | `""` | no |
65+
| node\_to\_node\_encryption\_enabled | Whether to enable node-to-node encryption | `bool` | `false` | no |
66+
| security\_groups | List of security group IDs to be allowed to connect to the cluster | `list(string)` | `[]` | no |
67+
| stage | Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release' | `string` | `""` | no |
68+
| subnet\_ids | VPC Subnet IDs | `list(string)` | `[]` | no |
69+
| tags | Additional tags (e.g. `map('BusinessUnit','XYZ')` | `map(string)` | `{}` | no |
70+
| vpc\_enabled | Set to false if ES should be deployed outside of VPC. | `bool` | `true` | no |
71+
| vpc\_id | VPC ID | `string` | `null` | no |
72+
| zone\_awareness\_enabled | Enable zone awareness for Elasticsearch cluster | `bool` | `true` | no |
5673

5774
## Outputs
5875

5976
| Name | Description |
6077
|------|-------------|
61-
| domain_arn | ARN of the Elasticsearch domain |
62-
| domain_endpoint | Domain-specific endpoint used to submit index, search, and data upload requests |
63-
| domain_hostname | Elasticsearch domain hostname to submit index, search, and data upload requests |
64-
| domain_id | Unique identifier for the Elasticsearch domain |
65-
| domain_name | Name of the Elasticsearch domain |
66-
| elasticsearch_user_iam_role_arn | The ARN of the IAM role to allow access to Elasticsearch cluster |
67-
| elasticsearch_user_iam_role_name | The name of the IAM role to allow access to Elasticsearch cluster |
68-
| kibana_endpoint | Domain-specific endpoint for Kibana without https scheme |
69-
| kibana_hostname | Kibana hostname |
70-
| security_group_id | Security Group ID to control access to the Elasticsearch domain |
78+
| domain\_arn | ARN of the Elasticsearch domain |
79+
| domain\_endpoint | Domain-specific endpoint used to submit index, search, and data upload requests |
80+
| domain\_hostname | Elasticsearch domain hostname to submit index, search, and data upload requests |
81+
| domain\_id | Unique identifier for the Elasticsearch domain |
82+
| domain\_name | Name of the Elasticsearch domain |
83+
| elasticsearch\_user\_iam\_role\_arn | The ARN of the IAM role to allow access to Elasticsearch cluster |
84+
| elasticsearch\_user\_iam\_role\_name | The name of the IAM role to allow access to Elasticsearch cluster |
85+
| kibana\_endpoint | Domain-specific endpoint for Kibana without https scheme |
86+
| kibana\_hostname | Kibana hostname |
87+
| security\_group\_id | Security Group ID to control access to the Elasticsearch domain |
7188

‎examples/basic/main.tf

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
provider "aws" {
2+
region = "us-east-2"
3+
}
4+
15
module "elasticsearch" {
26
source = "../../"
37
namespace = "eg"
@@ -17,6 +21,6 @@ module "elasticsearch" {
1721
kibana_subdomain_name = "kibana-es"
1822

1923
advanced_options = {
20-
rest.action.multi.allow_explicit_index = "true"
24+
"rest.action.multi.allow_explicit_index" = "true"
2125
}
2226
}

‎examples/complete/main.tf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@ provider "aws" {
33
}
44

55
module "vpc" {
6-
source = "git::https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.7.0"
6+
source = "git::https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.10.0"
77
namespace = var.namespace
88
stage = var.stage
99
name = var.name
1010
cidr_block = "172.16.0.0/16"
1111
}
1212

1313
module "subnets" {
14-
source = "git::https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.16.0"
14+
source = "git::https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.19.0"
1515
availability_zones = var.availability_zones
1616
namespace = var.namespace
1717
stage = var.stage

‎examples/non_vpc/main.tf

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
provider "aws" {
2+
region = "us-east-2"
3+
}
4+
5+
module "elasticsearch" {
6+
source = "../../"
7+
namespace = "eg"
8+
stage = "dev"
9+
name = "es"
10+
dns_zone_id = "Z14EN2YD427LRQ"
11+
security_groups = ["sg-XXXXXXXXX", "sg-YYYYYYYY"]
12+
vpc_enabled = false
13+
zone_awareness_enabled = "true"
14+
elasticsearch_version = "6.5"
15+
instance_type = "t2.small.elasticsearch"
16+
instance_count = 4
17+
iam_role_arns = ["arn:aws:iam::XXXXXXXXX:role/ops", "arn:aws:iam::XXXXXXXXX:role/dev"]
18+
iam_actions = ["es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost"]
19+
encrypt_at_rest_enabled = "true"
20+
kibana_subdomain_name = "kibana-es"
21+
22+
advanced_options = {
23+
"rest.action.multi.allow_explicit_index" = "true"
24+
}
25+
}

‎main.tf

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,15 +25,15 @@ module "user_label" {
2525
}
2626

2727
resource "aws_security_group" "default" {
28-
count = var.enabled ? 1 : 0
28+
count = var.enabled && var.vpc_enabled ? 1 : 0
2929
vpc_id = var.vpc_id
3030
name = module.label.id
3131
description = "Allow inbound traffic from Security Groups and CIDRs. Allow all outbound traffic"
3232
tags = module.label.tags
3333
}
3434

3535
resource "aws_security_group_rule" "ingress_security_groups" {
36-
count = var.enabled ? length(var.security_groups) : 0
36+
count = var.enabled && var.vpc_enabled ? length(var.security_groups) : 0
3737
description = "Allow inbound traffic from Security Groups"
3838
type = "ingress"
3939
from_port = var.ingress_port_range_start
@@ -44,7 +44,7 @@ resource "aws_security_group_rule" "ingress_security_groups" {
4444
}
4545

4646
resource "aws_security_group_rule" "ingress_cidr_blocks" {
47-
count = var.enabled && length(var.allowed_cidr_blocks) > 0 ? 1 : 0
47+
count = var.enabled && var.vpc_enabled && length(var.allowed_cidr_blocks) > 0 ? 1 : 0
4848
description = "Allow inbound traffic from CIDR blocks"
4949
type = "ingress"
5050
from_port = var.ingress_port_range_start
@@ -55,7 +55,7 @@ resource "aws_security_group_rule" "ingress_cidr_blocks" {
5555
}
5656

5757
resource "aws_security_group_rule" "egress" {
58-
count = var.enabled ? 1 : 0
58+
count = var.enabled && var.vpc_enabled ? 1 : 0
5959
description = "Allow all egress traffic"
6060
type = "egress"
6161
from_port = 0
@@ -163,9 +163,13 @@ resource "aws_elasticsearch_domain" "default" {
163163
enabled = var.node_to_node_encryption_enabled
164164
}
165165

166-
vpc_options {
167-
security_group_ids = [join("", aws_security_group.default.*.id)]
168-
subnet_ids = var.subnet_ids
166+
dynamic "vpc_options" {
167+
for_each = var.vpc_enabled ? [true] : []
168+
169+
content {
170+
security_group_ids = [join("", aws_security_group.default.*.id)]
171+
subnet_ids = var.subnet_ids
172+
}
169173
}
170174

171175
snapshot_options {
@@ -229,7 +233,7 @@ resource "aws_elasticsearch_domain_policy" "default" {
229233
module "domain_hostname" {
230234
source = "git::https://github.com/cloudposse/terraform-aws-route53-cluster-hostname.git?ref=tags/0.3.0"
231235
enabled = var.enabled && var.dns_zone_id != "" ? true : false
232-
name = "${var.elasticsearch_subdomain_name == "" ? var.name : var.elasticsearch_subdomain_name}"
236+
name = var.elasticsearch_subdomain_name == "" ? var.name : var.elasticsearch_subdomain_name
233237
ttl = 60
234238
zone_id = var.dns_zone_id
235239
records = [join("", aws_elasticsearch_domain.default.*.endpoint)]

0 commit comments

Comments
 (0)