Skip to content

Commit c8193c7

Browse files
bcarranzabcarranza
authored
Msk features (#451)
1 parent df1abfc commit c8193c7

File tree

23 files changed

+2244
-0
lines changed

23 files changed

+2244
-0
lines changed

‎terraform-modules/aws/msk_1.0.9/README.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
## Requirements
2+
3+
No requirements.
4+
5+
## Providers
6+
7+
| Name | Version |
8+
|------|---------|
9+
| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |
10+
11+
## Modules
12+
13+
| Name | Source | Version |
14+
|------|--------|---------|
15+
| <a name="module_msk"></a> [msk](#module\_msk) | ./module | n/a |
16+
17+
## Resources
18+
19+
| Name | Type |
20+
|------|------|
21+
| [aws_acmpca_certificate.cert](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate) | resource |
22+
| [aws_acmpca_certificate_authority.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate_authority) | resource |
23+
| [aws_acmpca_certificate_authority_certificate.cacert](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate_authority_certificate) | resource |
24+
| [aws_cloudwatch_log_group.msk_cloudwatch_log_group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
25+
| [aws_kms_key.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |
26+
| [aws_s3_bucket.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
27+
| [aws_s3_bucket_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource |
28+
| [aws_iam_policy_document.acmpca_bucket_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
29+
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
30+
31+
## Inputs
32+
33+
| Name | Description | Type | Default | Required |
34+
|------|-------------|------|---------|:--------:|
35+
| <a name="input_allowed_security_group_ids"></a> [allowed\_security\_group\_ids](#input\_allowed\_security\_group\_ids) | The security\_group\_id\_list output from the security\_groups module | `list(string)` | n/a | yes |
36+
| <a name="input_autoscaling_enabled"></a> [autoscaling\_enabled](#input\_autoscaling\_enabled) | To automatically expand your cluster's storage in response to increased usage, you can enable this. [More info](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand.html) | `bool` | `true` | no |
37+
| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | The AWS region you want to deploy to | `string` | n/a | yes |
38+
| <a name="input_broker_instance_type"></a> [broker\_instance\_type](#input\_broker\_instance\_type) | The instance type to use for the Kafka brokers | `string` | `"kafka.t3.small"` | no |
39+
| <a name="input_broker_per_zone"></a> [broker\_per\_zone](#input\_broker\_per\_zone) | Number of Kafka brokers per zone. | `number` | `1` | no |
40+
| <a name="input_broker_volume_size"></a> [broker\_volume\_size](#input\_broker\_volume\_size) | The size in GiB of the EBS volume for the data drive on each broker node | `number` | n/a | yes |
41+
| <a name="input_client_broker"></a> [client\_broker](#input\_client\_broker) | Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS\_PLAINTEXT, and PLAINTEXT | `string` | n/a | yes |
42+
| <a name="input_client_sasl_iam_enabled"></a> [client\_sasl\_iam\_enabled](#input\_client\_sasl\_iam\_enabled) | Enables client authentication via IAM policies (cannot be set to true at the same time as client\_sasl\_*\_enabled). | `bool` | `false` | no |
43+
| <a name="input_client_sasl_scram_enabled"></a> [client\_sasl\_scram\_enabled](#input\_client\_sasl\_scram\_enabled) | Enables SCRAM client authentication via AWS Secrets Manager (cannot be set to `true` at the same time as `client_tls_auth_enabled`). | `bool` | `false` | no |
44+
| <a name="input_client_sasl_scram_secret_association_arns"></a> [client\_sasl\_scram\_secret\_association\_arns](#input\_client\_sasl\_scram\_secret\_association\_arns) | List of AWS Secrets Manager secret ARNs for scram authentication (cannot be set to `true` at the same time as `client_tls_auth_enabled`). | `list(string)` | `[]` | no |
45+
| <a name="input_client_tls_auth_enabled"></a> [client\_tls\_auth\_enabled](#input\_client\_tls\_auth\_enabled) | Set true to enable the Client TLS Authentication | `bool` | n/a | yes |
46+
| <a name="input_cloudwatch_logs_enabled"></a> [cloudwatch\_logs\_enabled](#input\_cloudwatch\_logs\_enabled) | Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs | `bool` | n/a | yes |
47+
| <a name="input_cloudwatch_logs_log_group"></a> [cloudwatch\_logs\_log\_group](#input\_cloudwatch\_logs\_log\_group) | Name of the Cloudwatch Log Group to deliver logs to | `string` | n/a | yes |
48+
| <a name="input_common_name"></a> [common\_name](#input\_common\_name) | The common name for the CA | `string` | `"example.com"` | no |
49+
| <a name="input_create_private_ca"></a> [create\_private\_ca](#input\_create\_private\_ca) | Boolean flag to control if the resources about CA Private should be created | `bool` | `false` | no |
50+
| <a name="input_encryption_at_rest_kms_key_arn"></a> [encryption\_at\_rest\_kms\_key\_arn](#input\_encryption\_at\_rest\_kms\_key\_arn) | You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If null the key created in this module will be used. | `string` | `null` | no |
51+
| <a name="input_encryption_in_cluster"></a> [encryption\_in\_cluster](#input\_encryption\_in\_cluster) | Whether data communication among broker nodes is encrypted | `bool` | n/a | yes |
52+
| <a name="input_enhanced_monitoring"></a> [enhanced\_monitoring](#input\_enhanced\_monitoring) | Specify the desired enhanced MSK CloudWatch monitoring level. Valid values: DEFAULT, PER\_BROKER, and PER\_TOPIC\_PER\_BROKER | `string` | n/a | yes |
53+
| <a name="input_expiration_in_days"></a> [expiration\_in\_days](#input\_expiration\_in\_days) | The CA expiration in days | `number` | `7` | no |
54+
| <a name="input_jmx_exporter_enabled"></a> [jmx\_exporter\_enabled](#input\_jmx\_exporter\_enabled) | Set true to enable the Prometheus JMX Exporter | `bool` | `false` | no |
55+
| <a name="input_kafka_version"></a> [kafka\_version](#input\_kafka\_version) | The desired Kafka software version | `string` | `"2.8.1"` | no |
56+
| <a name="input_key_algorithm"></a> [key\_algorithm](#input\_key\_algorithm) | The CA key algorithm | `string` | `"RSA_4096"` | no |
57+
| <a name="input_multi_vpc_connectivity_enabled"></a> [multi\_vpc\_connectivity\_enabled](#input\_multi\_vpc\_connectivity\_enabled) | Turn on the vpc connectivitiy | `bool` | `false` | no |
58+
| <a name="input_multi_vpc_connectivity_iam_enabled"></a> [multi\_vpc\_connectivity\_iam\_enabled](#input\_multi\_vpc\_connectivity\_iam\_enabled) | Turn on the vpc connectivitiy in IAM mode | `bool` | `false` | no |
59+
| <a name="input_name"></a> [name](#input\_name) | Solution name | `string` | n/a | yes |
60+
| <a name="input_namespace"></a> [namespace](#input\_namespace) | Namespace, which could be your organization name or abbreviation, | `string` | n/a | yes |
61+
| <a name="input_node_exporter_enabled"></a> [node\_exporter\_enabled](#input\_node\_exporter\_enabled) | Set true to enable the Prometheus Node Exporter | `bool` | `false` | no |
62+
| <a name="input_properties"></a> [properties](#input\_properties) | Contents of the server.properties file. Supported properties are documented in the [MSK Developer Guide](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html) | `map(string)` | `{}` | no |
63+
| <a name="input_s3_bucket_create"></a> [s3\_bucket\_create](#input\_s3\_bucket\_create) | Set it to true if you want the process to create the S3 bucket for you, and false if you already have one. | `bool` | `false` | no |
64+
| <a name="input_s3_logs_bucket"></a> [s3\_logs\_bucket](#input\_s3\_logs\_bucket) | Name of the S3 bucket to deliver logs to (only able if (s3\_bucket\_create = true), It is going to create a new resource by you | `string` | n/a | yes |
65+
| <a name="input_s3_logs_enabled"></a> [s3\_logs\_enabled](#input\_s3\_logs\_enabled) | Indicates whether you want to enable or disable streaming broker logs to S3 | `bool` | n/a | yes |
66+
| <a name="input_s3_logs_prefix"></a> [s3\_logs\_prefix](#input\_s3\_logs\_prefix) | Prefix to append to the S3 folder name logs are delivered to | `string` | `""` | no |
67+
| <a name="input_security_groups"></a> [security\_groups](#input\_security\_groups) | DEPRECATED: Use `allowed_security_group_ids` instead.<br>List of security group IDs to be allowed to connect to the cluster | `list(string)` | `[]` | no |
68+
| <a name="input_signing_algorithm"></a> [signing\_algorithm](#input\_signing\_algorithm) | The CA signing algorithm | `string` | `"SHA512WITHRSA"` | no |
69+
| <a name="input_subnet_ids"></a> [subnet\_ids](#input\_subnet\_ids) | Subnet IDs for Client Broker | `list(string)` | n/a | yes |
70+
| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags | `map(any)` | n/a | yes |
71+
| <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | The VPC id of where you want to provision MSK | `string` | n/a | yes |
72+
| <a name="input_years_valid"></a> [years\_valid](#input\_years\_valid) | The Certificate will be valid for a period of x years from the time it is issued. | `number` | `10` | no |
73+
| <a name="input_zone_id"></a> [zone\_id](#input\_zone\_id) | Route53 DNS Zone ID for MSK broker hostnames | `string` | n/a | yes |
74+
75+
## Outputs
76+
77+
| Name | Description |
78+
|------|-------------|
79+
| <a name="output_bootstrap_brokers"></a> [bootstrap\_brokers](#output\_bootstrap\_brokers) | A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster |
80+
| <a name="output_bootstrap_brokers_tls"></a> [bootstrap\_brokers\_tls](#output\_bootstrap\_brokers\_tls) | A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster |
81+
| <a name="output_cluster_arn"></a> [cluster\_arn](#output\_cluster\_arn) | Amazon Resource Name (ARN) of the MSK cluster |
82+
| <a name="output_cluster_name"></a> [cluster\_name](#output\_cluster\_name) | MSK Cluster name |
83+
| <a name="output_cluster_uuid"></a> [cluster\_uuid](#output\_cluster\_uuid) | UUID of the MSK cluster, for use in IAM policies |
84+
| <a name="output_config_arn"></a> [config\_arn](#output\_config\_arn) | Amazon Resource Name (ARN) of the configuration |
85+
| <a name="output_current_version"></a> [current\_version](#output\_current\_version) | Current version of the MSK Cluster used for updates |
86+
| <a name="output_latest_revision"></a> [latest\_revision](#output\_latest\_revision) | Latest revision of the configuration |
87+
| <a name="output_security_group_id"></a> [security\_group\_id](#output\_security\_group\_id) | The ID of the security group rule |
88+
| <a name="output_security_group_name"></a> [security\_group\_name](#output\_security\_group\_name) | The name of the security group rule |
89+
| <a name="output_zookeeper_connect_string"></a> [zookeeper\_connect\_string](#output\_zookeeper\_connect\_string) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster |

‎terraform-modules/aws/msk_1.0.9/main.tf‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
locals{
2+
years_valid = var.years_valid
3+
}
4+
5+
resource "aws_cloudwatch_log_group" "msk_cloudwatch_log_group" {
6+
name = var.cloudwatch_logs_log_group
7+
tags = var.tags
8+
}
9+
10+
#######################################
11+
# S3 bucket
12+
#######################################
13+
resource "aws_kms_key" "this" {
14+
description = "This key is used to encrypt bucket objects"
15+
deletion_window_in_days = 10
16+
}
17+
18+
resource "aws_s3_bucket" "this" {
19+
count = var.s3_bucket_create ? 1 : 0
20+
bucket = var.s3_logs_bucket
21+
tags = var.tags
22+
}
23+
24+
data "aws_iam_policy_document" "acmpca_bucket_access" {
25+
count = var.s3_bucket_create ? 1 : 0
26+
statement {
27+
actions = [
28+
"s3:GetBucketAcl",
29+
"s3:GetBucketLocation",
30+
"s3:PutObject",
31+
"s3:PutObjectAcl",
32+
]
33+
34+
resources = [
35+
aws_s3_bucket.this[0].arn,
36+
"${aws_s3_bucket.this[0].arn}/*",
37+
]
38+
39+
principals {
40+
identifiers = ["acm-pca.amazonaws.com"]
41+
type = "Service"
42+
}
43+
}
44+
}
45+
46+
resource "aws_s3_bucket_policy" "this" {
47+
count = var.s3_bucket_create ? 1 : 0
48+
bucket = aws_s3_bucket.this[0].id
49+
policy = data.aws_iam_policy_document.acmpca_bucket_access[0].json
50+
}
51+
52+
#######################################
53+
# Private CA
54+
#######################################
55+
data "aws_partition" "current" {
56+
}
57+
58+
resource "aws_acmpca_certificate_authority_certificate" "cacert" {
59+
count = var.create_private_ca ? 1 : 0
60+
certificate_authority_arn = aws_acmpca_certificate_authority.this[0].arn
61+
62+
certificate = aws_acmpca_certificate.cert[0].certificate
63+
certificate_chain = aws_acmpca_certificate.cert[0].certificate_chain
64+
}
65+
66+
resource "aws_acmpca_certificate" "cert" {
67+
count = var.create_private_ca ? 1 : 0
68+
certificate_authority_arn = aws_acmpca_certificate_authority.this[0].arn
69+
certificate_signing_request = aws_acmpca_certificate_authority.this[0].certificate_signing_request
70+
signing_algorithm = "SHA512WITHRSA"
71+
72+
template_arn = "arn:${data.aws_partition.current.partition}:acm-pca:::template/RootCACertificate/V1"
73+
74+
validity {
75+
type = "YEARS"
76+
value = local.years_valid
77+
}
78+
}
79+
80+
resource "aws_acmpca_certificate_authority" "this" {
81+
count = var.create_private_ca ? 1 : 0
82+
certificate_authority_configuration {
83+
key_algorithm = var.key_algorithm
84+
signing_algorithm = var.signing_algorithm
85+
86+
subject {
87+
common_name = var.common_name
88+
}
89+
}
90+
91+
type = "ROOT"
92+
93+
revocation_configuration {
94+
crl_configuration {
95+
custom_cname = "crl.${var.common_name}"
96+
enabled = true
97+
expiration_in_days = var.expiration_in_days
98+
s3_bucket_name = var.s3_logs_bucket
99+
}
100+
}
101+
102+
tags = var.tags
103+
104+
depends_on = [aws_s3_bucket_policy.this]
105+
}
106+
107+
#######################################
108+
# MSK Cluster
109+
#######################################
110+
module "msk" {
111+
source = "./module"
112+
#version = "v2.0.0"
113+
namespace = var.namespace
114+
name = var.name
115+
vpc_id = var.vpc_id
116+
client_broker = var.client_broker
117+
zone_id = var.zone_id
118+
security_groups = var.security_groups
119+
subnet_ids = var.subnet_ids
120+
kafka_version = var.kafka_version
121+
#it was deprecated , now the param is number_of_broker_nodes
122+
# number_of_broker_nodes = var.number_of_broker_nodes
123+
broker_per_zone = var.broker_per_zone
124+
broker_instance_type = var.broker_instance_type
125+
broker_volume_size = var.broker_volume_size
126+
tags = var.tags
127+
certificate_authority_arns = var.create_private_ca ? [aws_acmpca_certificate_authority.this[0].arn] : []
128+
client_tls_auth_enabled = var.client_tls_auth_enabled
129+
client_sasl_iam_enabled = var.client_sasl_iam_enabled
130+
client_sasl_scram_enabled = var.client_sasl_scram_enabled
131+
client_sasl_scram_secret_association_arns = var.client_sasl_scram_secret_association_arns
132+
encryption_in_cluster = var.encryption_in_cluster
133+
encryption_at_rest_kms_key_arn = var.encryption_at_rest_kms_key_arn != null ? var.encryption_at_rest_kms_key_arn : aws_kms_key.this.arn
134+
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
135+
cloudwatch_logs_log_group = var.cloudwatch_logs_enabled == true ? var.cloudwatch_logs_log_group : ""
136+
enhanced_monitoring = var.enhanced_monitoring
137+
node_exporter_enabled = var.node_exporter_enabled
138+
jmx_exporter_enabled = var.jmx_exporter_enabled
139+
s3_logs_bucket = var.s3_logs_enabled == true ? var.s3_logs_bucket : ""
140+
s3_logs_enabled = var.s3_logs_enabled
141+
s3_logs_prefix = var.s3_logs_enabled == true ? var.s3_logs_prefix : ""
142+
autoscaling_enabled = var.autoscaling_enabled
143+
multi_vpc_connectivity_enabled = var.multi_vpc_connectivity_enabled
144+
multi_vpc_connectivity_iam_enabled = var.multi_vpc_connectivity_iam_enabled
145+
properties = var.properties
146+
allowed_security_group_ids = var.allowed_security_group_ids
147+
depends_on = [
148+
aws_cloudwatch_log_group.msk_cloudwatch_log_group,
149+
aws_s3_bucket.this,
150+
aws_acmpca_certificate.cert[0]
151+
]
152+
}

0 commit comments

Comments
 (0)