Route53 Record + vpc endpoint extensive by bcarranza · Pull Request #411 · ManagedKube/kubernetes-ops

bcarranza · 2023-03-01T22:48:55Z

Create a new route53 record to handle a generic record in route 53
Create a new record with sftp transfer server

Where I am using this branch

https://github.com/exact-payments/gruntwork-infrastructure-live/pull/1523/files#diff-e4e4cb018f9616b9193a6577d685fa8da8acf6abfa2037d213c0ed1b6177b767R12

Test cases

Test cases about indivual module: record-vpc-enpoint-sftp-transfer
Sftp connection test

root@debian-pod-ssh:~# sftp bcarranza@sftp-dataplatform.epops.cc
Connected to sftp-dataplatform.epops.cc.
sftp> exit
root@debian-pod-ssh:~# sftp bcarranza@sftp-dataplatform.epops.cc
Connected to sftp-dataplatform.epops.cc.
sftp> ls
bcarranza_files  example.txt      example1.txt

Input


inputs = {
  route53_zone_id = dependency.route53.outputs.public_hosted_zone_map[local.domain_name]
  record_name = "sftp-dataplatform.${local.domain_name}"
  vpc_id = dependency.vpc.outputs.vpc_id
  transfer_server_id = element(split(".", dependency.sftp.outputs.transfer_endpoint), 0)
}

How looks like record route 53
Appy

root@cc6aa29f5343:/workspaces/gruntwork-infrastructure-live/ops/us-west-2/ops/p2a/3000-data-platform/3025-sftp-route53# terragrunt apply
WARN[0002] No double-slash (//) found in source URL /cloudposse/terraform-aws-transfer-sftp.git. Relative paths in downloaded Terraform code may not work.  prefix=[/workspaces/gruntwork-infrastructure-live/ops/us-west-2/ops/p2a/3000-data-platform/3020-sftp] 
Initializing modules...
- record in ../record

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/local v2.3.0...
- Installed hashicorp/local v2.3.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.56.0...
- Installed hashicorp/aws v4.56.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Acquiring state lock. This may take a few moments...
data.template_file.log_name: Reading...
data.template_file.log_name: Read complete after 0s [id=6145fa111cd834d1c11d8ed1f63de159901010232de8c4dcfba72373f28bdef4]
data.aws_region.current: Reading...
data.aws_caller_identity.current: Reading...
data.aws_region.current: Read complete after 0s [id=us-west-2]
data.aws_caller_identity.current: Read complete after 1s [id=476264532441]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_vpc_endpoint.this will be read during apply
  # (config refers to values not yet known)
 <= data "aws_vpc_endpoint" "this" {
      + arn                   = (known after apply)
      + cidr_blocks           = (known after apply)
      + dns_entry             = (known after apply)
      + dns_options           = (known after apply)
      + id                    = (known after apply)
      + ip_address_type       = (known after apply)
      + network_interface_ids = (known after apply)
      + owner_id              = (known after apply)
      + policy                = (known after apply)
      + prefix_list_id        = (known after apply)
      + private_dns_enabled   = (known after apply)
      + requester_managed     = (known after apply)
      + route_table_ids       = (known after apply)
      + security_group_ids    = (known after apply)
      + service_name          = (known after apply)
      + state                 = (known after apply)
      + subnet_ids            = (known after apply)
      + tags                  = (known after apply)
      + vpc_endpoint_type     = (known after apply)
      + vpc_id                = "vpc-0254921a46bca1455"

      + timeouts {
          + read = (known after apply)
        }
    }

  # data.local_file.get-vpc-endpoint-id-value will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "local_file" "get-vpc-endpoint-id-value" {
      + content        = (known after apply)
      + content_base64 = (known after apply)
      + filename       = "./vpc-endpoint-id.txt"
      + id             = (known after apply)
    }

  # null_resource.output-vpc-endpoint-id will be created
  + resource "null_resource" "output-vpc-endpoint-id" {
      + id = (known after apply)
    }

  # module.record.aws_route53_record.this will be created
  + resource "aws_route53_record" "this" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "sftp-dataplatform.epops.cc"
      + type            = "A"
      + zone_id         = "Z10308181L9DK26UL0JK3"

      + alias {
          + evaluate_target_health = false
          + name                   = (known after apply)
          + zone_id                = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + fqdn                        = (known after apply)
  + name                        = "sftp-dataplatform.epops.cc"
  + vpc_endpoint_dns_name       = (known after apply)
  + vpc_endpoint_hosted_zone_id = (known after apply)
  + vpc_endpoint_id             = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.output-vpc-endpoint-id: Creating...
null_resource.output-vpc-endpoint-id: Provisioning with 'local-exec'...
null_resource.output-vpc-endpoint-id (local-exec): Executing: ["/bin/sh" "-c" "aws transfer describe-server --server-id s-2c18dc3b67c748018 --query 'Server.EndpointDetails.VpcEndpointId' > ./vpc-endpoint-id.txt"]
null_resource.output-vpc-endpoint-id: Creation complete after 2s [id=4522301945537840821]
data.local_file.get-vpc-endpoint-id-value: Reading...
data.local_file.get-vpc-endpoint-id-value: Read complete after 0s [id=ceb1bf6b4b041b945df013ce44a40dce56c39970]
data.aws_vpc_endpoint.this: Reading...
data.aws_vpc_endpoint.this: Read complete after 1s [id=vpce-055ff2bc56c264089]
module.record.aws_route53_record.this: Creating...
module.record.aws_route53_record.this: Still creating... [10s elapsed]
module.record.aws_route53_record.this: Still creating... [20s elapsed]
module.record.aws_route53_record.this: Still creating... [30s elapsed]
module.record.aws_route53_record.this: Still creating... [40s elapsed]
module.record.aws_route53_record.this: Still creating... [50s elapsed]
module.record.aws_route53_record.this: Creation complete after 53s [id=Z10308181L9DK26UL0JK3_sftp-dataplatform.epops.cc_A]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

fqdn = "sftp-dataplatform.epops.cc"
name = "sftp-dataplatform.epops.cc"
vpc_endpoint_dns_name = "vpce-055ff2bc56c264089-8drouhom.vpce-svc-046efd4167c4038e3.us-west-2.vpce.amazonaws.com"
vpc_endpoint_hosted_zone_id = "Z1YSA3EXCYUU9Z"
vpc_endpoint_id = "vpce-055ff2bc56c264089"

Test cases about indivual module: record

inputs = {
  
  route53_zone_id = dependency.route53.outputs.public_hosted_zone_map[local.domain_name]
  record_name = "sftp-dataplatform.${local.domain_name}"
  vpc_endpoint_dns_name = "vpce-0c5dcb513666d031b-1m3hyfue.vpce-svc-0369525cec7e36a11.us-west-2.vpce.amazonaws.com"
  vpc_endpoint_zone_id = "Z1YSA3EXCYUU9Z"
}

Apply

Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route53_record.this will be created
  + resource "aws_route53_record" "this" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "sftp-dataplatform.epops.cc"
      + type            = "A"
      + zone_id         = "Z10308181L9DK26UL0JK3"

      + alias {
          + evaluate_target_health = false
          + name                   = "vpce-0c5dcb513666d031b-1m3hyfue.vpce-svc-0369525cec7e36a11.us-west-2.vpce.amazonaws.com"
          + zone_id                = "Z1YSA3EXCYUU9Z"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + fqdn = (known after apply)
  + name = "sftp-dataplatform.epops.cc"

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_route53_record.this: Creating...
aws_route53_record.this: Still creating... [10s elapsed]
aws_route53_record.this: Still creating... [19s elapsed]
aws_route53_record.this: Still creating... [29s elapsed]
aws_route53_record.this: Creation complete after 33s [id=Z10308181L9DK26UL0JK3_sftp-dataplatform.epops.cc_A]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

fqdn = "sftp-dataplatform.epops.cc"
name = "sftp-dataplatform.epops.cc"

how looks like the new route 53 record

bcarranza · 2023-03-03T19:13:37Z

terraform-modules/aws/route53/record-vpc-endpoint-sftp-transfer/main.tf

+
+resource "null_resource" "output-vpc-endpoint-id" {
+  provisioner "local-exec" {
+    command = "aws transfer describe-server --server-id ${var.transfer_server_id} --query 'Server.EndpointDetails.VpcEndpointId' > ${data.template_file.log_name.rendered}"


It seems extremely to use a provisioner to get vpc id enpdoint, but it is working well.
aws transfer module in aws and cloud posse doesn't have an output to vpc endpoint id.
Even terraform state has the value, but is a primitive string, so its forbidden to get.

cloud posse: https://github.com/cloudposse/terraform-aws-transfer-sftp
aws terraform provider: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/transfer_server

bcarranza · 2023-03-03T19:19:27Z

terraform-modules/aws/route53/record-vpc-endpoint-sftp-transfer/main.tf

+  depends_on = [
+    data.aws_vpc_endpoint.this
+  ]
+}


I am reusing the first terraform called only record !

bcarranza · 2023-03-03T20:11:06Z

thread: https://3pillarglobal.slack.com/archives/C02N9MVF0/p1677871126411499
The use of local-exec requires running in an environment with AWS CLI and depends on a text file where the data to extract is saved, so it's not very user-friendly. However, it was a desperate measure due to the lack of the VPC endpoint ID in AWS Transfer. We'll document this and discuss it next week with Garland Kan. @sekka1

"It's also good to highlight the positive aspect of this. The PR includes a generic Route53 record module that adds to the functionality of managed Kube. The only issue is with the record-sftp-transfer-server-vpc-endpoint

bcarranza added 30 commits March 1, 2023 16:47

record route53

a15db2d

fix output vars

85dd220

record route 53 aws transfer.

cfe494e

fix aws transfer attributes

1912d6b

record name

1a6d036

fix module

35d8425

region

d48229f

vpc id

2dac7e7

dns_entry

c70252c

output fixes

515604d

arrays

1f83d9f

fix arrays

133ab9f

vpc endpoint id

2432d24

vpc endpoint id

b24b4c6

endpoint details

4674c8e

endpoint_details

fb736e5

other way to get vpc endpoint it.

a5915e7

fix

6cfb974

test harcoding

72addb6

testing local-exec

65107a8

depends on

6357618

typo data

53cb426

null_resource.cluster.triggers.output

47d0664

local exec provisioner

0da8c72

typos in output

9bfdabd

add content

0c9a927

remove quotes

c6b491f

trim function

fd0c374

vpc id

1578e98

change name of txt

07442a1

bcarranza added 4 commits March 3, 2023 11:52

replace trim

e0c1e74

typo trim

d54211e

typo replace

602ea17

remove new lines

d301f73

bcarranza commented Mar 3, 2023

View reviewed changes

bcarranza changed the title ~~record route53~~ Route53 Record + vpc endpoint extensive Mar 3, 2023

bcarranza commented Mar 3, 2023

View reviewed changes

bcarranza added 2 commits March 6, 2023 11:24

route 53 v2

1bf5434

remove second test to vpc endpoint by state

a7e7c7e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route53 Record + vpc endpoint extensive #411

Route53 Record + vpc endpoint extensive #411
bcarranza wants to merge 36 commits intomainfrom
route53-record

bcarranza commented Mar 1, 2023 •

edited

Loading

Uh oh!

bcarranza Mar 3, 2023

Uh oh!

bcarranza Mar 3, 2023

Uh oh!

bcarranza Mar 3, 2023

Uh oh!

bcarranza commented Mar 3, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bcarranza commented Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Where I am using this branch

Test cases

Uh oh!

bcarranza Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

bcarranza Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

bcarranza Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

bcarranza commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bcarranza commented Mar 1, 2023 •

edited

Loading

bcarranza commented Mar 3, 2023 •

edited

Loading