Skip to content

README.md

Latest commit

 

History

History
174 lines (134 loc) · 16.1 KB

README.md

File metadata and controls

174 lines (134 loc) · 16.1 KB

Description

This module creates a Google Cloud Storage (GCS) bucket.

For more information on this and other network storage options in the Cluster Toolkit, see the extended Network Storage documentation.

Example

The following example will create a bucket named simulation-results-xxxxxxxx, where xxxxxxxx is a randomly generated id.

  - id: bucket
    source: modules/file-system/cloud-storage-bucket
    settings:
      name_prefix: simulation-results
      random_suffix: true

NOTE: Use of random_suffix may cause the following error when used with other modules: value depends on resource attributes that cannot be determined until apply. To resolve this set random_suffix to false (default).

NOTE: Bucket namespace is shared by all users of Google Cloud so it is possible to have a bucket name clash with an existing bucket that is not in your project. To resolve this try to use a more unique name, or set the random_suffix variable to true.

Naming of Bucket

There are potentially three parts to the bucket name. Each of these parts are configurable in the blueprint.

  1. A custom prefix, provided by the user in the blueprint
    Provide the custom prefix using the name_prefix setting.

  2. The deployment name, included by default
    The deployment name can be excluded by setting use_deployment_name_in_bucket_name: false.

  3. A random id suffix, excluded by default
    The random id can be included by setting random_suffix: true.

If none of these are provided (no name_prefix, use_deployment_name_in_bucket_name: false, & random_suffix: false), then the bucket name will default to no-bucket-name-provided.

Since bucket namespace is shared by all users of Google Cloud, it is more likely to experience naming clashes than with other resources. In many cases, adding the random_suffix will resolve the naming clash issue.

Warning: If a bucket is created with a random_suffix and then used as the bucket for a startup script in the same deployment group this will cause a not known at apply time error in terraform. The solution is to either create the bucket in a separate deployment group or to remove the random suffix.

Mounting

To mount the Cloud Storage bucket you must first ensure that the GCS Fuse client has been installed and then call the proper mount command.

Both of these steps are automatically handled with the use of the use command in a selection of Cluster Toolkit modules. See the compatibility matrix in the network storage doc for a complete list of supported modules.

If mounting is not automatically handled as described above, the cloud-storage-bucket module outputs runners that can be used with the startup-script module to install the client and mount the file system. See the following example:

  - id: bucket
    source: modules/file-system/cloud-storage-bucket
    settings: {local_mount: /data}

  - id: mount-at-startup
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(bucket.client_install_runner)
      - $(bucket.mount_runner)

License

Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name Version
terraform = 1.12.2
google >= 3.83
google-beta >= 6.9.0
random ~> 3.0

Providers

Name Version
google >= 3.83
google-beta >= 6.9.0
random ~> 3.0

Modules

No modules.

Resources

Name Type
google-beta_google_storage_anywhere_cache.cache_instances resource
google-beta_google_storage_bucket.bucket resource
google_storage_bucket_iam_binding.viewers resource
random_id.resource_name_suffix resource

Inputs

Name Description Type Default Required
anywhere_cache Anywhere Cache configurations.
When you create a cache for a bucket, the cache must be created in a zone within the location of your bucket.
For example, if your bucket is located in the us-east1 region, you can create a cache in us-east1-b but not us-central1-c.
If your bucket is located in the ASIA dual-region, you can create a cache in any zones that make up the asia-east1 and asia-southeast1 regions.
This validation only works for single regions.		 
object({
    zones            = list(string)
    ttl              = optional(string, "86400s")
    admission_policy = optional(string, "admit-on-first-miss")
  })
 null no
anywhere_cache_create_timeout Timeout for Anywhere Cache creation operations. Can be set to a duration like '1h' or '30m'.
The maximum documented creation time is 48 hours. Please refer to the official documentation for more details on timeouts:
https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_anywhere_cache#timeouts		 string "240m" no
autoclass Configure bucket autoclass setup

The autoclass config supports automatic transitions of objects in the bucket to appropriate storage classes based on each object's access pattern.

The terminal storage class defines that objects in the bucket eventually transition to if they are not read for a certain length of time.
Supported values include: 'NEARLINE', 'ARCHIVE' (Default 'NEARLINE')

See Cloud documentation for more details:

https://cloud.google.com/storage/docs/autoclass		 
object({
    enabled                = optional(bool, false)
    terminal_storage_class = optional(string, null)
  })
 
{
  "enabled": false
}
 no
deployment_name Name of the HPC deployment; used as part of name of the GCS bucket. string n/a yes
enable_hierarchical_namespace If true, enables hierarchical namespace for the bucket. This option must be configured during the initial creation of the bucket. bool false no
enable_object_retention If true, enables retention policy at per object level for the bucket.

See Cloud documentation for more details:

https://cloud.google.com/storage/docs/object-lock		 bool false no
enable_versioning If true, enables versioning for the bucket. bool false no
force_destroy If true will destroy bucket with all objects stored within. bool false no
labels Labels to add to the GCS bucket. Key-value pairs. map(string) n/a yes
lifecycle_rules List of config to manage data lifecycle rules for the bucket. For more details: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket.html#nested_lifecycle_rule 
list(object({
    # Object with keys:
    # - type - The type of the action of this Lifecycle Rule. Supported values: Delete and SetStorageClass.
    # - storage_class - (Required if action type is SetStorageClass) The target Storage Class of objects affected by this Lifecycle Rule.
    action = object({
      type          = string
      storage_class = optional(string)
    })

    # Object with keys:
    # - age - (Optional) Minimum age of an object in days to satisfy this condition.
    # - send_age_if_zero - (Optional) While set true, num_newer_versions value will be sent in the request even for zero value of the field.
    # - created_before - (Optional) Creation date of an object in RFC 3339 (e.g. 2017-06-13) to satisfy this condition.
    # - with_state - (Optional) Match to live and/or archived objects. Supported values include: "LIVE", "ARCHIVED", "ANY".
    # - matches_storage_class - (Optional) Comma delimited string for storage class of objects to satisfy this condition. Supported values include: MULTI_REGIONAL, REGIONAL, NEARLINE, COLDLINE, ARCHIVE, STANDARD, DURABLE_REDUCED_AVAILABILITY.
    # - matches_prefix - (Optional) One or more matching name prefixes to satisfy this condition.
    # - matches_suffix - (Optional) One or more matching name suffixes to satisfy this condition.
    # - num_newer_versions - (Optional) Relevant only for versioned objects. The number of newer versions of an object to satisfy this condition.
    # - custom_time_before - (Optional) A date in the RFC 3339 format YYYY-MM-DD. This condition is satisfied when the customTime metadata for the object is set to an earlier date than the date used in this lifecycle condition.
    # - days_since_custom_time - (Optional) The number of days from the Custom-Time metadata attribute after which this condition becomes true.
    # - days_since_noncurrent_time - (Optional) Relevant only for versioned objects. Number of days elapsed since the noncurrent timestamp of an object.
    # - noncurrent_time_before - (Optional) Relevant only for versioned objects. The date in RFC 3339 (e.g. 2017-06-13) when the object became nonconcurrent.
    condition = object({
      age                        = optional(number)
      send_age_if_zero           = optional(bool)
      created_before             = optional(string)
      with_state                 = optional(string)
      matches_storage_class      = optional(string)
      matches_prefix             = optional(string)
      matches_suffix             = optional(string)
      num_newer_versions         = optional(number)
      custom_time_before         = optional(string)
      days_since_custom_time     = optional(number)
      days_since_noncurrent_time = optional(number)
      noncurrent_time_before     = optional(string)
    })
  }))
 [] no
local_mount The mount point where the contents of the device may be accessed after mounting. string "/mnt" no
mount_options Mount options to be put in fstab. Note: implicit_dirs makes it easier to work with objects added by other tools, but there is a performance impact. See: more information string "defaults,_netdev,implicit_dirs" no
name_prefix Name Prefix. string null no
project_id ID of project in which GCS bucket will be created. string n/a yes
public_access_prevention Bucket public access can be controlled by setting a value of either inherited or enforced.
When set to enforced, public access to the bucket is blocked.
If set to inherited, the bucket's public access prevention depends on whether it is subject to the organization policy constraint for public access prevention.

See Cloud documentation for more details:

https://cloud.google.com/storage/docs/public-access-prevention		 string null no
random_suffix If true, a random id will be appended to the suffix of the bucket name. bool false no
region The region to deploy to string n/a yes
retention_policy_period If defined, this will configure retention_policy with retention_period for the bucket, value must be in between 1 and 3155760000(100 years) seconds.

See Cloud documentation for more details:

https://cloud.google.com/storage/docs/bucket-lock		 number null no
soft_delete_retention_duration If defined, this will configure soft_delete_policy with retention_duration_seconds for the bucket, value can be 0 or in between 604800(7 days) and 7776000(90 days).
Setting a 0 duration disables soft delete, meaning any deleted objects will be permanently deleted.

See Cloud documentation for more details:

https://cloud.google.com/storage/docs/soft-delete		 number null no
storage_class The storage class of the GCS bucket. string "REGIONAL" no
uniform_bucket_level_access Allow uniform control access to the bucket. bool true no
use_deployment_name_in_bucket_name If true, the deployment name will be included as part of the bucket name. This helps prevent naming clashes across multiple deployments. bool true no
viewers A list of additional accounts that can read packages from this bucket set(string) [] no

Outputs

Name Description
anywhere_cache_ids The IDs of the created Anywhere Cache instances.
client_install_runner Runner that performs client installation needed to use gcs fuse.
gcs_bucket_name Bucket name.
gcs_bucket_path The gsutil bucket path with format of gs://<bucket-name>.
mount_runner Runner that mounts the cloud storage bucket with gcs fuse.
network_storage Describes a remote network storage to be mounted by fs-tab.