`#cloud-init` should be the very first line in a multipart configuration

[hahuang65]

Terraform CLI and Provider Versions

Terraform v1.4.4
on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v5.26.0
• provider registry.terraform.io/hashicorp/cloudinit v2.3.4

Use Cases or Problem Statement

When using a multipart cloudinit_config such as

data "cloudinit_config" "rails" {
  gzip          = true
  base64_encode = true

  part {
    content    = file("files/user_data/parts/base.yaml")
    merge_type = "list(append)+dict(recurse_array)+str()"
  }

  part {
    content    = file("files/user_data/parts/rails.yaml")
    merge_type = "list(append)+dict(recurse_array)+str()"
  }

  part {
    content    = file("files/user_data/parts/users.yaml")
    merge_type = "list(append)+dict(recurse_array)+str()"
  }
}

it ends up generating a /var/lib/cloud/instance/user-data.txt file that starts with

--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
Mime-Version: 1.0
X-Merge-Type: list(append)+dict(recurse_array)+str()

#cloud-config

This causes cloud-init schema --system --annotate to complain:

# E1: File None needs to begin with "#cloud-config"

The other issue is that merge_type = "list(append)+dict(recurse_array)+str()" doesn't seem to do anything for the parts besides the first.

I had to stick merge_type = "list(append)+dict(recurse_array)+str()" in my part YAML files for cloudinit to not overwrite previous parts.

Proposal

Would it be possible to move the #cloud-init above the --MIMEBOUNDARY, or add an additional one there?

I'm not sure what the feasible solutions might be.

How much impact is this issue causing?

Medium

Additional Information

This was discovered (both problems of the #cloud-init comment, and the merge_type = "list(append)+dict(recurse_array)+str()" not being inserted into each part) when troubleshooting upgrading from Amazon Linux 2 to Amazon Linux 2023 (cloud-init v19 to v22) here

canonical/cloud-init#5533

You'll see that the resolution for that issue was that I needed to manually put merge_type = "list(append)+dict(recurse_array)+str()" into each of my user_data/parts/<part>.yaml, even though I had it in my Terraform block parts:

  part {
    content    = file("files/user_data/parts/base.yaml")
    merge_type = "list(append)+dict(recurse_array)+str()"
  }

Code of Conduct

• I agree to follow this project's Code of Conduct

[bbasata]

Would it be possible to move the #cloud-init above the --MIMEBOUNDARY, or add an additional one there?

Hello, and thank you for your patience with the delay in reviewing this issue.

Based on the cloud-init documentation for user data formats, the current behavior is correct -- the header for MIME multi-part is Content-Type: multipart/mixed and not #cloud-config.

I wonder if the error from cloud-init schema --system --annotate was fixed in cloud-init 22.3 (comment).

I needed to manually put merge_type = "list(append)+dict(recurse_array)+str()" into each of my user_data/parts/<part>.yaml, even though I had it in my Terraform block parts

I will try to duplicate this result.