Terraform module to create Google Cloud infrastructure resources required to run DataRobot.
module "datarobot_infra" {
source = "datarobot-oss/dr-infra/google"
name = "datarobot"
google_project_id = "your-google-project-id"
region = "us-west1"
domain_name = "datarobot.example.com"
cert_manager_letsencrypt_email_address = "you@example.com"
tags = {
environment = "dev"
managed-by = "terraform"
}
}
- Complete - Demonstrates all available input variables.
- Public - Minimal configuration for a publicly accessible deployment (internet-facing load balancer, public GKE API endpoint).
- Private - Minimal configuration for a private deployment (internal load balancer, private-only GKE API endpoint, existing VPC).
- Clone the repo
git clone https://github.com/datarobot-oss/terraform-google-dr-infra.git- Change directories into the example that best aligns with your use-case.
cd terraform-google-dr-infra/examples/public- Modify
main.tfto suit your specific use-case. - Run terraform.
terraform init
terraform applycreate_networkto create a new Google VPCexisting_vpc_name,existing_kubernetes_nodes_subnet, andexisting_kubernetes_pods_range_nameto use an existing VPC and subnet
Create a new Google VPC with one subnet using a /20 slice of network_address_space and a NAT gateway attached.
kubernetes_pod_cidr and kubernetes_service_cidr are secondary ranges within the subnet which will be used for the Kubernetes pod and service IPs, respectively.
Only the priamry the kubernetes_pod_cidr IPs are attached to the Cloud NAT gateway.
TBD
create_dns_zoneto create a new Google Cloud DNS managed zoneexisting_dns_zone_nameto use an existing Google Cloud DNS managed zone
Creates a new Cloud DNS managed zone for domain_name. When dns_zone_public is true (the default) a public zone is created; when false a private zone is created for the given VPC.
The Cloud DNS zone is used by external_dns to manage DNS records for the resources created by the DataRobot application. When the zone is public it is also used for DNS validation when using cert_manager and cert_manager_letsencrypt_clusterissuers.
TBD
create_storageto create a new Google Cloud Storage Bucketexisting_gcs_bucket_nameto use an existing Google Cloud Storage Bucket
Create a new GCS Bucket with prefix name and name datarobot.
The DataRobot application will use this storage account for persistent file storage.
TBD
create_container_registryto create a new Google Artifact Registry Repositoryexisting_artifact_registry_repo_idto use an existing Google Artifact Registry Repository
Create a new GAR repository with name name.
The DataRobot application will use this registry to host custom images created by various services.
TBD
create_kubernetes_clusterto create a new Google Kubernetes Engine Clusterexisting_gke_cluster_nameto use an existing GKE cluster
Create a new GKE cluster to host the DataRobot application and any other helm charts installed by this module.
By default, the Kubernetes cluster API endpoint is accessible both via a private endpoint created within the same VPC as well as publicly over the internet. GKE nodes always communicate with the control plane using the private IP address. Public endpoint access can be restricted using the kubernetes_cluster_endpoint_access_list variable or disabled completely by setting kubernetes_cluster_endpoint_public_access to false.
When kubernetes_cluster_endpoint_public_access is false, Kubernetes management operations such as kubectl and helm commands (including the Helm chart installs performed by this Terraform module) must be run from a host which can access the Kubernetes cluster API private endpoint. By default, any host within the GKE nodes subnet has access but this can be extended using the kubernetes_cluster_endpoint_access_list variable. This can be helpful when running this Terraform module from a host that resides within the same VPC as the GKE cluster but in a different subnet than the GKE nodes.
Two node groups are created:
- A
drcpunode group intended to host the majority of the DataRobot pods - A
drgpunode group intended to host GPU workload pods containing the labeldatarobot.com/node-capability: gpuand taintnvidia.com/gpu:NoSchedule
By default, slices of network_address_space will be used for the cluster nodes and control plane private endpoint IPs. It is best to use a separate address space for kubernetes_pod_cidr as it is a secondary (aliased) range.
TBD
create_app_identityto create a new Google Service account to represent the DataRobot application
Create a new GKE Service Account with roles/storage.admin access to the Google Cloud Storage bucket and roles/artifactregistry.writer access to the Google Artifact Registry Repository.
Workload identities are created for each datarobot_service_accounts within the datarobot_namespace and attached to this Service Account. This allows those pods running with those service accounts to access file storage and the artifact registry.
TBD
create_postgresto create a new Cloud SQL for PostgreSQL instance
Create an Cloud SQL for PostgreSQL connected to via service networking.
TBD
create_redisto create a new Memorystore Redis instance
Create a Memorystore Redis instance connected to via service networking.
TBD
create_mongodbto create a new MongoDB Atlas cluster
Create a MongoDB Atlas project and cluster for use by the DataRobot application.
TBD
ingress_nginxto install theingress-nginxhelm chart
Uses the terraform-helm-release module to install the https://kubernetes.github.io/ingress-nginx/ingress-nginx helm chart into the ingress-nginx namespace.
The ingress-nginx helm chart will trigger the deployment of an Google Network Load Balancer directing traffic to the ingress-nginx-controller Kubernetes services.
Values passed to the helm chart can be overridden by passing a custom values file via the ingress_nginx_values variable as demonstrated in the complete example.
Not required
cert_managerto install thecert-managerhelm chart
Uses the terraform-helm-release module to install the https://charts.jetstack.io/cert-manager helm chart into the cert-manager namespace.
A Google Service Account is created for the cert-manager Kubernetes service account running in the cert-manager namespace that allows the creation of DNS resources within the specified DNS zone.
cert-manager can be used by the DataRobot application to create and manage various certificates including the application.
When cert_manager_letsencrypt_clusterissuers is enabled, letsencrypt-staging and letsencrypt-prod ClusterIssuers will be created which can be used by the datarobot-google umbrella chart to issue certificates used by the DataRobot application. The default values in that helm chart (as of version 10.2) have global.ingress.tls.enabled, global.ingress.tls.certmanager, and global.ingress.tls.issuer as letsencrypt-prod which will use the letsencrypt-prod ClusterIssuer to issue a public ACME certificate as the TLS certificate used by the Kubernetes ingress resources.
Values passed to the helm chart can be overridden by passing a custom values file via the cert_manager_values variable as demonstrated in the complete example.
TBD
external_dnsto install theexternal-dnshelm chart
Uses the terraform-helm-release module to install the https://charts.bitnami.com/bitnami/external-dns helm chart into the external-dns namespace.
A Google Service Account is created for the external-dns Kubernetes service account running in the external-dns namespace that allows the creation of DNS resources within the specified DNS zone.
external-dns is used to automatically create DNS records for ingress resources in the Kubernetes cluster. When the DataRobot application is installed and the ingress resources are created, external-dns will automatically create a DNS record pointing at the ingress resource.
Values passed to the helm chart can be overridden by passing a custom values file via the external_dns_values variable as demonstrated in the complete example.
TBD
nvidia_device_pluginto install thenvidia-device-pluginhelm chart
Uses the terraform-helm-release module to install the https://nvidia.github.io/k8s-device-plugin/nvidia-device-plugin helm chart into the nvidia-device-plugin namespace.
Values passed to the helm chart can be overridden by passing a custom values file via the nvidia_device_plugin_values variable as demonstrated in the complete example.
Not required
deschedulerto install thedeschedulerhelm chart
Uses the terraform-helm-release module to install the descheduler helm chart from the https://kubernetes-sigs.github.io/descheduler/ helm repo into the kube-system namespace.
This helm chart allows for automatic rescheduling of pods for optimizing resource consumption.
Not required
TBD
Currently the only thing coupling a release of this module to a DataRobot Enterprise Release is the default list of datarobot_service_accounts. Technically, this module can be used with any DataRobot version if the user specifies the correct list of datarobot_service_accounts for that version.
The default installation supports DataRobot versions >= 10.0.
| Name | Version |
|---|---|
| terraform | >= 1.3.5 |
| >= 6.6.0 | |
| helm | >= 3.0.2 |
| kubectl | ~> 1.19 |
| mongodbatlas | ~> 2.0 |
| Name | Version |
|---|---|
| >= 6.6.0 |
| Name | Source | Version |
|---|---|---|
| app_identity | terraform-google-modules/service-accounts/google | ~> 4.0 |
| cert_manager | ./modules/cert-manager | n/a |
| cloud_router | terraform-google-modules/cloud-router/google | ~> 6.1 |
| custom_endpoints | ./modules/custom-private-endpoints | n/a |
| descheduler | ./modules/descheduler | n/a |
| dns | terraform-google-modules/cloud-dns/google | ~> 5.0 |
| external_dns | ./modules/external-dns | n/a |
| ingress_nginx | ./modules/ingress-nginx | n/a |
| kubernetes | terraform-google-modules/kubernetes-engine/google//modules/private-cluster | ~> 39.0 |
| mongodb | ./modules/mongodb | n/a |
| network | terraform-google-modules/network/google | ~> 9.0 |
| nvidia_device_plugin | ./modules/nvidia-device-plugin | n/a |
| observability | ./modules/observability | n/a |
| postgres | terraform-google-modules/sql-db/google//modules/postgresql | ~> 26.0 |
| private_link_service | ./modules/private-link-service | n/a |
| redis | terraform-google-modules/memorystore/google | ~> 15.0 |
| storage | terraform-google-modules/cloud-storage/google | ~> 8.0 |
| Name | Type |
|---|---|
| google_artifact_registry_repository.this | resource |
| google_artifact_registry_repository_iam_member.datarobot | resource |
| google_compute_global_address.postgres | resource |
| google_compute_global_address.redis | resource |
| google_service_account_iam_member.datarobot | resource |
| google_service_networking_connection.this | resource |
| google_storage_bucket_iam_member.datarobot | resource |
| google_client_config.default | data source |
| google_compute_network.existing | data source |
| google_compute_subnetwork.existing_ingress_psc | data source |
| google_compute_subnetwork.existing_kubernetes_nodes | data source |
| google_compute_subnetwork.existing_mongodb | data source |
| google_container_cluster.existing | data source |
| google_dns_managed_zone.existing | data source |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| allow_psc_global_access | Whether to allow global access for Private Service Connect | bool |
false |
no |
| cert_manager | Install the cert-manager helm chart. All other cert_manager variables are ignored if this variable is false. | bool |
true |
no |
| cert_manager_letsencrypt_clusterissuers | Whether to create letsencrypt-prod and letsencrypt-staging ClusterIssuers. This will only work if the DNS zone is public. | bool |
true |
no |
| cert_manager_letsencrypt_email_address | Email address for the certificate owner. Let's Encrypt will use this to contact you about expiring certificates, and issues related to your account. Only required if cert_manager_letsencrypt_clusterissuers is true. | string |
"user@example.com" |
no |
| cert_manager_values_overrides | Values in raw yaml format to pass to helm. | string |
null |
no |
| create_app_identity | Create a new user assigned identity for the DataRobot application | bool |
true |
no |
| create_container_registry | Create a new Google Container Registry. Ignored if an existing existing_artifact_registry_repo_id is specified. | bool |
true |
no |
| create_dns_zone | Create a Cloud DNS managed zone for domain_name. Ignored if existing_dns_zone_name is specified. | bool |
true |
no |
| create_ingress_psc | Expose the internal LB created by the ingress-nginx controller as a Google Private Service Connection. Only applies if internet_facing_ingress_lb is false. | bool |
false |
no |
| create_ingress_psc_namespace | Whether to create a namespace for the ingress private service connect | bool |
false |
no |
| create_kubernetes_cluster | Create a new Google Kubernetes Engine cluster. All kubernetes and helm chart variables are ignored if this variable is false. | bool |
true |
no |
| create_mongodb | Whether to create a MongoDB Atlas instance | bool |
false |
no |
| create_network | Create a new Google VPC. Ignored if an existing existing_vpc_id is specified. | bool |
true |
no |
| create_observability | Whether to provision observability resources | bool |
false |
no |
| create_postgres | Whether to create a CloudSQL for PostgreSQL instance | bool |
false |
no |
| create_redis | Whether to create a Google Memorystore Redis instance | bool |
false |
no |
| create_storage | Create a new Google Storage Bucket to use for DataRobot file storage. Ignored if an existing_gcs_bucket_name is specified. | bool |
true |
no |
| custom_private_endpoints | Configuration for the specific endpoint | list(object({ |
[] |
no |
| datarobot_namespace | Kubernetes namespace in which the DataRobot application will be installed | string |
"dr-app" |
no |
| datarobot_service_accounts | Kubernetes service accounts in the datarobot_namespace to provide with Storage Blob Data Contributor and AcrPush access | set(string) |
[ |
no |
| descheduler | Install the descheduler helm chart to enable rescheduling of pods. All other descheduler variables are ignored if this variable is false | bool |
true |
no |
| descheduler_values_overrides | Values in raw yaml format to pass to helm. | string |
null |
no |
| dns_zone_force_destroy | Force destroy the Cloud DNS managed zone. Ignored if an existing_dns_zone_name is specified or create_dns_zone is false. | bool |
false |
no |
| dns_zone_public | Create a public Cloud DNS managed zone. When false, a private zone will be created for the given VPC. |
bool |
true |
no |
| domain_name | Name of the domain to use for the DataRobot application. If create_dns_zone is true then a zone will be created for this domain. It is also used by the cert-manager helm chart for DNS validation and as a domain filter by the external-dns helm chart. | string |
"" |
no |
| existing_artifact_registry_repo_id | ID of existing artifact registry repository to use | string |
null |
no |
| existing_dns_zone_name | Name of an existing Cloud DNS managed zone to use. When specified, all other DNS variables will be ignored. | string |
null |
no |
| existing_gcs_bucket_name | ID of existing Google Storage Bucket to use for DataRobot file storage. When specified, all other storage variables will be ignored. | string |
null |
no |
| existing_gke_cluster_name | Name of existing GKE cluster to use. When specified, all other kubernetes variables will be ignored. | string |
null |
no |
| existing_ingress_pcs_subnet_name | Name of an existing subnet to use for the Private Service Connection used by the ingress-nginx controller. Required when an existing_vpc_name is specified and create_ingress_psc is true. Ignored if no existing_vpc_name is specified or create_ingress_psc is false. | string |
null |
no |
| existing_kubernetes_nodes_subnet | Name of an existing subnet to use for the GKE node pools and control plane private endpoint. Required when create_kubernetes_cluster is true and an existing_vpc_name is specified. Ignored if no existing_vpc_name is specified. |
string |
null |
no |
| existing_kubernetes_pods_range_name | Name of an secondary IP range within subnet defined by existing_kubernetes_nodes_subnet_name to use for the Kubernetes pods. Required when an existing_vpc_name is specified. Ignored if no existing_vpc_name is specified. | string |
null |
no |
| existing_mongodb_subnet_name | Name of an existing subnet to use for MongoDB Atlas VPC Peering. Required when an existing_vpc_name is specified. Ignored if no existing_vpc_name is specified. | string |
null |
no |
| existing_vpc_name | Name of an existing Google VPC to use. When specified, other network variables are ignored. | string |
null |
no |
| external_dns | Install the external_dns helm chart to manage DNS records for resources created by the application. All other external_dns variables are ignored if this variable is false. | bool |
true |
no |
| external_dns_values_overrides | Values in raw yaml format to pass to helm. | string |
null |
no |
| gcr_registry_name | Name of the Artifact Registry repository. Defaults to the name if not specified. | string |
null |
no |
| google_project_id | The ID of the Google Project where these resources will be created | string |
n/a | yes |
| ingress_nginx | Install the ingress-nginx helm chart to use as the ingress controller for the GKE cluster. All other ingress_nginx variables are ignored if this variable is false. | bool |
true |
no |
| ingress_nginx_values_overrides | Values in raw yaml format to pass to helm. | string |
null |
no |
| ingress_psc_consumer_projects | The list of consumer project IDs that are allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. | list(string) |
[] |
no |
| ingress_psc_subnet_cidr | CIDR range to use for the Private Service Connection used by the ingress-nginx controller. Only used when create_network is true and an existing_vpc_name is not specified. |
string |
null |
no |
| ingress_service_name | The name of the ingress service to attach the private link to. | string |
"ingress-nginx-controller" |
no |
| install_helm_charts | Whether to install helm charts into the target EKS cluster. All other helm chart variables are ignored if this is false. |
bool |
true |
no |
| internet_facing_ingress_lb | Determines the type of Load Balancer created for GKE ingress. If true, an external Load Balancer will be created. If false, an internal Load Balancer will be created. | bool |
true |
no |
| kubernetes_cluster_deletion_protection | Enable deletion protection on the GKE cluster | bool |
true |
no |
| kubernetes_cluster_endpoint_access_list | List of CIDRs allowed to access the Kubernetes cluster API endpoint. When kubernetes_cluster_endpoint_public_access is true, these CIDRs specify which public IP addresses are allowed to access the Kubernetes cluster API external endpoint. When kubernetes_cluster_endpoint_public_access is false, these CIDRs specify which private IP addresses are allowed to access the Kubernetes cluster API internal endpoint. By default, only hosts within the kubernetes nodes subnet are allowed to access the Kubernetes cluster API internal endpoint. | list(string) |
[] |
no |
| kubernetes_cluster_endpoint_public_access | Whether the Kubernetes cluster API endpoint can be accessed via an external IP address | bool |
true |
no |
| kubernetes_cluster_grant_registry_access | Grants created cluster-specific service account storage.objectViewer and artifactregistry.reader roles | bool |
true |
no |
| kubernetes_cluster_version | GKE cluster version | string |
"latest" |
no |
| kubernetes_master_ipv4_cidr | The IP range in CIDR notation to use for the hosted master network including the Kubernetes control plane. If you use this flag, GKE creates a new subnet that uses the values you defined in master-ipv4-cidr and uses the new subnet to provision the internal IP address for the control plane. | string |
null |
no |
| kubernetes_node_pools | Map of GKE node pools | any |
{ |
no |
| kubernetes_nodes_cidr | The CIDR to use for Kubernetes nodes IP addresses. This is used as the primary IP range for the Kubernetes nodes subnet. | string |
null |
no |
| kubernetes_pod_cidr | The CIDR to use for Kubernetes pod IP addresses. This is used as a secondary IP range within the Kubernetes nodes subnet. | string |
"172.16.0.0/15" |
no |
| mongodb_admin_username | MongoDB admin username | string |
"pcs-mongodb" |
no |
| mongodb_atlas_auto_scaling_disk_gb_enabled | Enable Atlas disk size autoscaling | bool |
true |
no |
| mongodb_atlas_disk_size | Starting atlas disk size | string |
"20" |
no |
| mongodb_atlas_instance_type | atlas instance type | string |
"M30" |
no |
| mongodb_atlas_org_id | Atlas organization ID | string |
null |
no |
| mongodb_atlas_private_key | Private API key for Mongo Atlas | string |
"" |
no |
| mongodb_atlas_public_key | Public API key for Mongo Atlas | string |
"" |
no |
| mongodb_audit_enable | Enable database auditing for production instances only(cost incurred 10%) | bool |
false |
no |
| mongodb_backup_schedule | Configuration for the MongoDB Atlas cloud backup schedule policy items and cross-region copy settings | object({ |
{} |
no |
| mongodb_enable_slack_alerts | Enable alert notifications to a Slack channel. When true, slack_api_token and slack_notification_channel must be set. |
string |
false |
no |
| mongodb_network_reservation_ip_offset | Value to offset the network reservation IP | number |
2 |
no |
| mongodb_slack_api_token | Slack API token to use for alert notifications. Required when enable_slack_alerts is true. |
string |
null |
no |
| mongodb_slack_notification_channel | Slack channel to send alert notifications to. Required when enable_slack_alerts is true. |
string |
null |
no |
| mongodb_subnet_cidr | CIDR range to use for MongoDB Atlas VPC Peering. Only used when create_network is true and an existing_vpc_name is not specified. |
string |
null |
no |
| mongodb_termination_protection_enabled | Enable protection to avoid accidental production cluster termination | bool |
false |
no |
| mongodb_version | MongoDB version | string |
"7.0" |
no |
| name | Name to use as a prefix for created resources | string |
n/a | yes |
| network_address_space | The CIDR to use for the VPC and subnets created by this module | string |
"10.0.0.0/20" |
no |
| nvidia_device_plugin | Install the nvidia-device-plugin helm chart to expose node GPU resources to the GKE cluster. All other nvidia_device_plugin variables are ignored if this variable is false. | bool |
true |
no |
| nvidia_device_plugin_values_overrides | Values in raw yaml format to pass to helm. | string |
null |
no |
| postgres_availability_type | The availability type of the Cloud SQL instance, high availability (REGIONAL) or single zone (ZONAL) | string |
"REGIONAL" |
no |
| postgres_cidr | CIDR range to use for PostgreSQL private IP | string |
null |
no |
| postgres_database_flags | The database flags for the Cloud SQL instance. | list(object({ |
[ |
no |
| postgres_database_version | The PostgreSQL version to use | string |
"POSTGRES_13" |
no |
| postgres_deletion_protection | Whether Terraform will be prevented from destroying the instance. When the field is set to true or unset in Terraform state, a terraform apply or terraform destroy that would delete the instance will fail. When the field is set to false, deleting the instance is allowed | bool |
false |
no |
| postgres_disk_autoresize_limit | The maximum size to which storage capacity can be automatically increased. The default value is 0, which specifies that there is no limit. | number |
0 |
no |
| postgres_disk_size | The size of data disk, in GB. Size of a running instance cannot be reduced but can be increased | number |
20 |
no |
| postgres_disk_type | The type of data disk: PD_SSD, PD_HDD, or HYPERDISK_BALANCED | string |
"PD_SSD" |
no |
| postgres_tier | The machine type to use. See tiers for more details and supported versions. Postgres supports only shared-core machine types, and custom machine types such as db-custom-2-13312. | string |
"db-custom-4-16384" |
no |
| redis_cidr | CIDR range to use for Redis private IP | string |
null |
no |
| redis_memory_size_gb | Redis memory size in GiB. Defaulted to 1 GiB | number |
8 |
no |
| redis_tier | The service tier of the instance: BASIC or STANDARD_HA | string |
"STANDARD_HA" |
no |
| redis_transit_encryption_mode | The TLS mode of the Redis instance, If not provided, TLS is enabled for the instance. Possible values are: SERVER_AUTHENTICATION, DISABLED. | string |
"SERVER_AUTHENTICATION" |
no |
| region | Google region to create the resources in | string |
n/a | yes |
| release_channel | The release channel of this cluster. Accepted values are UNSPECIFIED, RAPID, REGULAR and STABLE. Defaults to STABLE. |
string |
"STABLE" |
no |
| storage_force_destroy | Force destroy the Google Storage Bucket when terminating, deleting all objects it contains. Ignored if an existing_gcs_bucket_name is specified or create_storage is false. | bool |
false |
no |
| tags | A map of tags to add to all created resources | map(string) |
{ |
no |
| Name | Description |
|---|---|
| artifact_registry_repo_id | ID of the Artifact Registry repository |
| artifact_registry_repo_path | Path to the Artifact Registry repository |
| datarobot_service_account_email | Email of the DataRobot service account |
| datarobot_service_account_key | DataRobot service account key |
| dns_zone_name | Name of the DNS zone |
| dns_zone_name_servers | Name servers of the DNS zone |
| gke_cluster_name | Name of the GKE cluster |
| mongodb_endpoint | MongoDB endpoint |
| mongodb_password | MongoDB admin password |
| postgres_endpoint | PostgreSQL endpoint |
| postgres_password | PostgreSQL admin password |
| redis_endpoint | Google Memorystore Redis endpoint |
| redis_password | Google Memorystore Redis instance primary access key |
| redis_port | Google Memorystore Redis port |
| storage_bucket_name | Name of the storage bucket |
| vpc_name | Name of the VPC |