diff --git a/doc/user/content/installation/_index.md b/doc/user/content/installation/_index.md index aea86cb8a07cb..6024d29829a9b 100644 --- a/doc/user/content/installation/_index.md +++ b/doc/user/content/installation/_index.md @@ -1,5 +1,5 @@ --- -title: "Install/Upgrade (Self-Managed)" +title: "Self-Managed Deployments" description: "Installation and upgrade guides for Self-Managed Materialize." disable_list: true menu: diff --git a/doc/user/content/installation/appendix-terraforms.md b/doc/user/content/installation/appendix-terraforms.md index cfb6ae78dd999..de7ea46f42952 100644 --- a/doc/user/content/installation/appendix-terraforms.md +++ b/doc/user/content/installation/appendix-terraforms.md @@ -1,6 +1,6 @@ --- -title: "Appendix: Terraforms" +title: "Terraform Modules" description: "List of template Terraform modules that are available as a starting point." menu: @@ -10,7 +10,7 @@ menu: weight: 95 --- -To help you get started, Materialize provides some template Terraforms. +To help you get started, Materialize provides Terraform modules. {{< important >}} These modules are intended for evaluation/demonstration purposes and for serving @@ -25,11 +25,18 @@ your own production deployment, either: {{}} +### **Terraform Modules** + +Materialize provides a [**unified Terraform module**](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main?tab=readme-ov-file#materialize-self-managed-terraform-modules) in order to provide concrete examples and an opinionated model for deploying materialize. +This module supports deployments for AWS + {{< yaml-table data="self_managed/terraform_list" >}} -## Releases +### *Legacy Terraform Modules* + +{{< yaml-table data="self_managed/legacy_terraform_list" >}} -### Materialize on AWS Terraform module +#### Materialize on AWS Terraform module {{< yaml-table data="self_managed/aws_terraform_versions" >}} @@ -40,7 +47,7 @@ https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#u for release-specific upgrade notes. -### Materialize on Azure Terraform module +#### Materialize on Azure Terraform module {{< yaml-table data="self_managed/azure_terraform_versions" >}} diff --git a/doc/user/content/installation/concepts.md b/doc/user/content/installation/concepts.md new file mode 100644 index 0000000000000..aef4c72a182fd --- /dev/null +++ b/doc/user/content/installation/concepts.md @@ -0,0 +1,249 @@ +--- +title: "Self-Managed Deployment Concepts" +description: "Learn about the key components and architecture of self-managed Materialize deployments." +aliases: + - /self-hosted/concepts/ +menu: + main: + parent: 'installation' + weight: 90 +--- + +## Overview + +Self-managed Materialize deployments on Kubernetes consist of several layers of components that work together to provide a fully functional database environment. Understanding these components and how they interact is essential for deploying, managing, and troubleshooting your self-managed installation. + +This page provides an overview of the core architectural components in a self-managed deployment, from the infrastructure level (Helm chart) down to the application level (clusters and replicas). + +## Architecture layers + +A self-managed Materialize deployment is organized into the following layers: + +Layer | Component | Description +------|-----------|------------ +**Infrastructure** | [Helm Chart](#helm-chart) | Package manager component that bootstraps the Kubernetes deployment +**Orchestration** | [Materialize Operator](#materialize-operator) | Kubernetes operator that manages Materialize instances +**Database** | [Materialize Instance](#materialize-instance) | The Materialize database instance itself +**Compute** | [Clusters and Replicas](#clusters-and-replicas) | Isolated compute resources for workloads + +## Helm chart + +The Helm chart is the entry point for deploying Materialize in a self-managed Kubernetes environment. It serves as a package manager component that defines and deploys the Materialize Operator. + +For configuration options for the the help chart and operator, consulte the [Materialize Operator Configuration page](/installation/configuration/). + +### What gets installed + +The Helm chart itself is relatively simple and focused. When you install the Materialize Helm chart, it: + +- Deploys the **Materialize Operator** as a Kubernetes deployment +- Creates necessary cluster-wide resources (CRDs, RBAC roles, service accounts) +- Configures operator settings and permissions + +### Working with the Helm chart + +You interact with the Helm chart through standard Helm commands: + +```bash +# Install the Materialize operator +helm install materialize materialize/materialize-operator + +# Upgrade the operator +helm upgrade materialize materialize/materialize-operator + +# Uninstall the operator +helm uninstall materialize +``` + +Once installed, the operator handles the deployment and management of Materialize instances. + +## Materialize Operator + +The Materialize Operator (implemented as `orchestratord`) is a Kubernetes operator that automates the deployment and lifecycle management of Materialize instances. It implements the [Kubernetes operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to extend Kubernetes with domain-specific knowledge about Materialize. + +### Managed resources + +The operator watches for Materialize custom resources and creates/manages all the Kubernetes resources required to run a Materialize instance, including: + +- **Namespaces**: Isolated Kubernetes namespaces for each instance +- **Services**: Network services for connecting to Materialize +- **Network Policies**: Network isolation and security rules +- **Certificates**: TLS certificates for secure connections +- **ConfigMaps and Secrets**: Configuration and sensitive data +- **Deployments**: These support the `Balancerd` and `Consoled` pod used as the ingress layer for materialize. +- **StatefulSets**: `Environmentd` and `Clusterd` which are the database control plane and compute resources respectively. + +### Working with the operator + +To deploy Materialize instances with the operator you must create Materialize custom resources. + +```yaml +apiVersion: materialize.cloud/v1alpha1 +kind: Materialize +metadata: + name: my-environment +spec: + environmentdImageRef: materialize/environmentd:v26.0.0 + ... +``` + +When you create this resource the operator detects the change and automatically creates or updates all necessary Kubernetes resources. + +Updates to the resource are watched, but only rolled out when a new `requestRollout` UUID is provided. + +For detailed configuration options, see: +- [Operator Configuration](/installation/configuration/) +- [Materialize CRD Field Descriptions](/installation/appendix-materialize-crd-field-descriptions/) +- [Upgrade Overview](/installation/upgrading/) + +## Materialize Instance + +A Materialize instance is the actual database that you connect to and interact with. Each instance is an isolated Materialize deployment with its own data, configuration, and compute resources. + +### Components + +When you create a Materialize instance, the operator deploys three core components as Kubernetes resources: + +- **environmentd**: The main database control plane, deployed as a StatefulSet +- **balancerd**: A pgwire and http proxy used to connect to environmentd, deployed as a Deployment +- **console**: Web-based administration interface, deployed as a Deployment + +### The environmentd component + +The primary component of a Materialize instance is **environmentd**, which runs as a Kubernetes pod. `Environmentd` houses the control plane and contains: + +- **Adapter**: The SQL interface that handles client connections, query parsing, and planning +- **Storage Controller**: Maintains durable metadata for storage +- **Compute Controller**: Orchestrates compute resources and manages system state + +When you connect to Materialize with a SQL client (e.g., `psql`), you're connecting to `environmentd`. + +On startup Environmentd will create several built-in clusters. + +### Instance responsibilities + +A Materialize instance manages: + +- **SQL objects**: Sources, views, materialized views, indexes, sinks +- **Schemas and databases**: Logical organization of objects +- **User connections**: SQL client connections and authentication +- **Catalog metadata**: System information about all objects and configuration +- **Compute orchestration**: Coordination of work across clusters and replicas + +### Connecting to an instance + +You interact with a Materialize instance through standard PostgreSQL-compatible tools and drivers: + +```bash +# Connect with psql +psql "postgres://materialize@:6875/materialize" +``` + +Once connected, you can issue SQL commands to create sources, define views, run queries, and manage the database: + +```sql +-- Create a source +CREATE SOURCE my_source FROM KAFKA ...; + +-- Create a materialized view +CREATE MATERIALIZED VIEW my_view AS + SELECT ... FROM my_source ...; + +-- Query the view +SELECT * FROM my_view; +``` + +## Clusters and Replicas + +Clusters are isolated pools of compute resources that execute workloads in Materialize. They provide resource isolation and fault tolerance for your data processing pipelines. + +For a comprehensive overview of clusters in Materialize, see the [Clusters concept page](/concepts/clusters/). + +### Cluster architecture + +- **Clusters**: Logical groupings of compute resources dedicated to specific workloads (sources, sinks, indexes, materialized views, queries) +- **Replicas**: Physical instantiations of a cluster's compute resources, deployed as Kubernetes StatefulSets + +Each replica contains identical compute resources and processes the same data independently, providing fault tolerance and high availability. + +### Kubernetes resources + +When you create a cluster with one or more replicas in Materialize, the instance coordinates with the operator to create: + +- One or more **StatefulSet** resources (one per replica) +- **Pods** within each StatefulSet that execute the actual compute workload +- **Persistent volumes** (if configured) for scratch disk space + +For example: + +```sql +-- Create a cluster with 2 replicas +CREATE CLUSTER my_cluster SIZE = '100cc', REPLICATION FACTOR = 2; +``` + +This creates two separate StatefulSets in Kubernetes, each running compute processes. + +### Managing clusters + +You interact with clusters primarily through SQL: + +```sql +-- Create a cluster +CREATE CLUSTER ingest_cluster SIZE = '50cc', REPLICATION FACTOR = 1; + +-- Use a cluster for a source +CREATE SOURCE my_source + IN CLUSTER ingest_cluster + FROM KAFKA ...; + +-- Create a cluster for materialized views +CREATE CLUSTER compute_cluster SIZE = '100cc', REPLICATION FACTOR = 2; + +-- Use a cluster for a materialized view +CREATE MATERIALIZED VIEW my_view + IN CLUSTER compute_cluster AS + SELECT ... FROM my_source ...; + +-- Resize a cluster +ALTER CLUSTER compute_cluster SET (SIZE = '200cc'); + +-- Drop a cluster +DROP CLUSTER ingest_cluster; +``` + +Materialize handles the underlying Kubernetes resource creation and management automatically. + +## Deployment workflow + +Understanding how these components work together helps clarify the deployment process: + +1. **Install the Helm chart**: This deploys the Materialize Operator to your Kubernetes cluster. + +2. **Create a Materialize instance**: Apply a Materialize custom resource. The operator detects this and creates all necessary Kubernetes resources, including the `environmentd`, `balancerd`, and `console` pods. + +3. **Connect to the instance**: Use a SQL client to connect to the `environmentd` service endpoint. + +4. **Create clusters**: Issue SQL commands to create clusters. Materialize coordinates with the operator to provision StatefulSets for replicas. + +5. **Deploy workloads**: Create sources, materialized views, indexes, and sinks on your clusters. + +## Relationship to Materialize concepts + +Self-managed deployments implement the same core Materialize concepts as the cloud offering: + +- [**Clusters**](/concepts/clusters/): Identical behavior, but backed by Kubernetes StatefulSets +- [**Sources**](/concepts/sources/): Same functionality for ingesting data +- [**Views**](/concepts/views/): Same query semantics and incremental maintenance +- [**Indexes**](/concepts/indexes/): Same in-memory query acceleration +- [**Sinks**](/concepts/sinks/): Same data egress capabilities + +The self-managed deployment model adds the Kubernetes infrastructure layer (Helm chart and operator) but doesn't change how you interact with Materialize at the SQL level. + +## Related pages + +- [Installation guides](/installation/) +- [Operator Configuration](/installation/configuration/) +- [Materialize CRD Field Descriptions](/installation/appendix-materialize-crd-field-descriptions/) +- [Operational guidelines](/installation/operational-guidelines/) +- [Clusters concept page](/concepts/clusters/) +- [Materialize architecture overview](/concepts/) diff --git a/doc/user/content/installation/install-on-aws/_index.md b/doc/user/content/installation/install-on-aws/_index.md index 4c16e703b7763..4e84093d164d2 100644 --- a/doc/user/content/installation/install-on-aws/_index.md +++ b/doc/user/content/installation/install-on-aws/_index.md @@ -1,6 +1,6 @@ --- -title: "Install on AWS (via Terraform)" -description: "" +title: "Install on AWS" +description: "Install and upgrade Materialize on AWS" aliases: - /self-hosted/install-on-aws/ - /self-managed/v25.1/installation/install-on-aws/ @@ -14,476 +14,14 @@ menu: {{% self-managed/materialize-components-sentence %}} -The tutorial deploys Materialize to AWS Elastic Kubernetes Service (EKS) with a -PostgreSQL RDS database as the metadata database and AWS S3 for blob storage. -The tutorial uses [Materialize on AWS Terraform -module](https://github.com/MaterializeInc/terraform-aws-materialize) to: - -- Set up the AWS Kubernetes environment. -- Call - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) - module to deploy Materialize Operator and Materialize instances to that EKS - cluster. - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< /warning >}} - -{{% self-managed/aws-recommended-instances %}} - -See [Appendix: AWS Deployment -guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) for -more information. - -## Prerequisites - -### Terraform - -If you don't have Terraform installed, [install -Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). - -### AWS CLI - -If you do not have the AWS CLI installed, install. For details, see the [AWS -documentation](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). - -### kubectl - -If you do not have `kubectl`, install. See the [Amazon EKS: install `kubectl` -documentation](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) -for details. - -### Helm 3.2.0+ - -If you do not have Helm 3.2.0+, install. For details, see the [Helm -documentation](https://helm.sh/docs/intro/install/). - -### License key - -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} - -## Set up AWS Kubernetes environment and install Materialize - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< self-managed/tutorial-disclaimer >}} - -{{< /warning >}} - -{{< tabs >}} - -{{< tab "Deployed components" >}} - -[Materialize on AWS Terraform -module](https://github.com/MaterializeInc/terraform-aws-materialize/blob/main/README.md) -deploys a sample infrastructure on AWS (region `us-east-1`) with the following -components: - -{{< yaml-table data="self_managed/aws_terraform_deployed_components" >}} - -{{< tip >}} -{{% self-managed/aws-terraform-configs %}} -{{< /tip >}} - -{{}} -{{< tab "Releases" >}} - -{{< yaml-table data="self_managed/aws_terraform_versions" >}} - -{{}} -{{}} - -1. Open a Terminal window. - -1. Configure AWS CLI with your AWS credentials. For details, see the [AWS - documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). - -{{% self-managed/versions/step-clone-aws-terraform-repo %}} - -1. Go to the `examples/simple` folder in the Materialize Terraform repo - directory. - - ```bash - cd terraform-aws-materialize/examples/simple - ``` - - {{< tip >}} - {{< self-managed/aws-terraform-configs >}} - {{< /tip >}} - -1. Create a `terraform.tfvars` file (you can copy from the - `terraform.tfvars.example` file) and specify the following variables: - - | Variable | Description | - |--------------------|-------------| - | `namespace` | A namespace (e.g., `my-demo`) that will be used to form part of the prefix for your AWS resources.
**Requirements:**
- Maximum of 12 characters
- Must start with a lowercase letter
- Must be lowercase alphanumeric and hyphens only | - | `environment` | An environment name (e.g., `dev`, `test`) that will be used to form part of the prefix for your AWS resources.
**Requirements:**
- Maximum of 8 characters
- Must be lowercase alphanumeric only | - - - ```bash - # The namespace and environment variables are used to construct the names of the resources - # e.g. ${namespace}-${environment}-storage, ${namespace}-${environment}-db etc. - - namespace = "enter-namespace" // maximum 12 characters, start with a letter, contain lowercase alphanumeric and hyphens only (e.g. my-demo) - environment = "enter-environment" // maximum 8 characters, lowercase alphanumeric only (e.g., dev, test) - ``` - - {{< tip >}} - {{< self-managed/aws-terraform-configs >}} - {{< /tip >}} - -1. Initialize the terraform directory. - - ```bash - terraform init - ``` - -1. Use terraform plan to review the changes to be made. - - ```bash - terraform plan - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply - ``` - - To approve the changes and apply, enter `yes`. - - - - Upon successful completion, various fields and their values are output: - - ```none - Apply complete! Resources: 89 added, 0 changed, 0 destroyed. - - Outputs: - - cluster_certificate_authority_data = - database_endpoint = "my-demo-dev-db.abcdefg8dsto.us-east-1.rds.amazonaws.com:5432" - eks_cluster_endpoint = "https://0123456789A00BCD000E11BE12345A01.gr7.us-east-1.eks.amazonaws.com" - eks_cluster_name = "my-demo-dev-eks" - materialize_s3_role_arn = "arn:aws:iam::000111222333:role/my-demo-dev-mz-role" - metadata_backend_url = - nlb_details = [] - oidc_provider_arn = "arn:aws:iam::000111222333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/7D14BCA3A7AA896A836782D96A24F958" - persist_backend_url = "s3://my-demo-dev-storage-f2def2a9/dev:serviceaccount:materialize-environment:12345678-1234-1234-1234-12345678912" - s3_bucket_name = "my-demo-dev-storage-f2def2a9" - vpc_id = "vpc-0abc000bed1d111bd" - ``` - -1. Note your specific values for the following fields: - - - `eks_cluster_name` (Used to configure `kubectl`) - -1. Configure `kubectl` to connect to your EKS cluster, replacing: - - - `` with the name of your EKS cluster. Your cluster - name has the form `{namespace}-{environment}-eks`; e.g., - `my-demo-dev-eks`. - - - `` with the region of your EKS cluster. The - simple example uses `us-east-1`. - - ```bash - aws eks update-kubeconfig --name --region - ``` - - To verify that you have configured correctly, run the following command: - - ```bash - kubectl get nodes - ``` - - For help with `kubectl` commands, see [kubectl Quick - reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). - -1. By default, the example Terraform installs the Materialize Operator and, - starting in v0.4.0, a `cert-manager`. Verify the installation and check the - status: - - {{< tabs >}} - {{< tab "Materialize Operator" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n materialize - ``` - - Wait for the components to be in the `Running` state: - - ```none - NAME READY STATUS RESTARTS AGE - pod/my-demo-dev-materialize-operator-84ff4b4648-brjhl 1/1 Running 0 12s - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/my-demo-dev-materialize-operator 1/1 1 1 12s - - NAME DESIRED CURRENT READY AGE - replicaset.apps/my-demo-dev-materialize-operator-84ff4b4648 1 1 1 12s - ``` - - {{}} - {{< tab "cert-manager (Starting in version 0.4.0)" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n cert-manager - ``` - Wait for the components to be in the `Running` state: - ``` - NAME READY STATUS RESTARTS AGE - pod/cert-manager-cainjector-686546c9f7-v9hwp 1/1 Running 0 4m20s - pod/cert-manager-d6746cf45-cdmb5 1/1 Running 0 4m20s - pod/cert-manager-webhook-5f79cd6f4b-rcjbq 1/1 Running 0 4m20s - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/cert-manager ClusterIP 172.20.2.136 9402/TCP 4m20s - service/cert-manager-cainjector ClusterIP 172.20.154.137 9402/TCP 4m20s - service/cert-manager-webhook ClusterIP 172.20.63.217 443/TCP,9402/TCP 4m20s - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/cert-manager 1/1 1 1 4m20s - deployment.apps/cert-manager-cainjector 1/1 1 1 4m20s - deployment.apps/cert-manager-webhook 1/1 1 1 4m20s - NAME DESIRED CURRENT READY AGE - replicaset.apps/cert-manager-cainjector-686546c9f7 1 1 1 4m20s - replicaset.apps/cert-manager-d6746cf45 1 1 1 4m20s - replicaset.apps/cert-manager-webhook-5f79cd6f4b 1 1 1 - 4m20s - ``` - - {{}} - {{}} - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting) guide. - -1. Once the Materialize operator is deployed and running, you can deploy the - Materialize instances. To deploy Materialize instances, create a - `mz_instances.tfvars` file with the [Materialize instance - configuration](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances). - - For example, the following specifies the configuration for a `demo`: - - ```bash - cat < mz_instances.tfvars - - materialize_instances = [ - { - name = "demo" - namespace = "materialize-environment" - database_name = "demo_db" - cpu_request = "1" - memory_request = "2Gi" - memory_limit = "2Gi" - license_key = "" - } - ] - EOF - ``` - - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - - **Starting in v0.3.0**, the Materialize on AWS Terraform module also - deploys, by default, Network Load Balancers (NLBs) for each Materialize - instance (i.e., the - [`create_nlb`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) - flag defaults to `true`). The NLBs, by default, are configured to be - internal (i.e., the - [`internal_nlb`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) - flag defaults to `true`). See [`materialize_instances`]( - https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) - for the Materialize instance configuration options. - - - **Starting in v0.4.0**, a self-signed `ClusterIssuer` is deployed by - default. The `ClusterIssuer` is deployed on subsequent after the - `cert-manager` is running. - - - **Starting in v0.4.6**, you can specify addition configuration options via - `environmentd_extra_args`. - - {{< tip >}} - {{% self-managed/aws-terraform-upgrade-notes %}} - - See [Materialize on AWS releases](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) for notable changes. - {{}} - -1. Run `terraform plan` with both `.tfvars` files and review the changes to be - made. - - ```bash - terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - The plan should show the changes to be made, with a summary similar to the - following: - - ``` - Plan: 17 to add, 1 to change, 0 to destroy. - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - To approve the changes and apply, enter `yes`. - - Upon successful completion, you should see output with a summary similar to - the following: - - - - ```bash - Apply complete! Resources: 17 added, 1 changed, 0 destroyed. - - Outputs: - - cluster_certificate_authority_data = - database_endpoint = "my-demo-dev-db.abcdefg8dsto.us-east-1.rds.amazonaws.com:5432" - eks_cluster_endpoint = "https://0123456789A00BCD000E11BE12345A01.gr7.us-east-1.eks.amazonaws.com" - eks_cluster_name = "my-demo-dev-eks" - materialize_s3_role_arn = "arn:aws:iam::000111222333:role/my-demo-dev-mz-role" - metadata_backend_url = - nlb_details = [ - "demo" = { - "arn" = "arn:aws:elasticloadbalancing:us-east-1:000111222333:loadbalancer/net/my-demo-dev/aeae3d936afebcfe" - "dns_name" = "my-demo-dev-aeae3d936afebcfe.elb.us-east-1.amazonaws.com" - } - ] - oidc_provider_arn = "arn:aws:iam::000111222333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/7D14BCA3A7AA896A836782D96A24F958" - persist_backend_url = "s3://my-demo-dev-storage-f2def2a9/dev:serviceaccount:materialize-environment:12345678-1234-1234-1234-12345678912" - s3_bucket_name = "my-demo-dev-storage-f2def2a9" - vpc_id = "vpc-0abc000bed1d111bd" - ``` - - The Network Load Balancer (NLB) details `nlb_details` are available when - running the Terraform module v0.3.0+. - -1. Verify the installation and check the status: - - ```bash - kubectl get all -n materialize-environment - ``` - - Wait for the components to be in the `Running` state. - - ```none - NAME READY STATUS RESTARTS AGE - pod/create-db-demo-db-6swk7 0/1 Completed 0 33s - pod/mzutd2fbabf5-balancerd-6c9755c498-28kcw 1/1 Running 0 11s - pod/mzutd2fbabf5-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 11s - pod/mzutd2fbabf5-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 11s - pod/mzutd2fbabf5-console-57f94b4588-6lg2x 1/1 Running 0 4s - pod/mzutd2fbabf5-console-57f94b4588-v65lk 1/1 Running 0 4s - pod/mzutd2fbabf5-environmentd-1-0 1/1 Running 0 16s - - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/mzutd2fbabf5-balancerd ClusterIP None 6876/TCP,6875/TCP 11s - service/mzutd2fbabf5-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 12s - service/mzutd2fbabf5-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 12s - service/mzutd2fbabf5-console ClusterIP None 8080/TCP 4s - service/mzutd2fbabf5-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 11s - service/mzutd2fbabf5-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 16s - service/mzutd2fbabf5-persist-pubsub-1 ClusterIP None 6879/TCP 16s - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/mzutd2fbabf5-balancerd 1/1 1 1 11s - deployment.apps/mzutd2fbabf5-console 2/2 2 2 4s - - NAME DESIRED CURRENT READY AGE - replicaset.apps/mzutd2fbabf5-balancerd-6c9755c498 1 1 1 11s - replicaset.apps/mzutd2fbabf5-console-57f94b4588 2 2 2 4s - - NAME READY AGE - statefulset.apps/mzutd2fbabf5-cluster-s2-replica-s1-gen-1 1/1 12s - statefulset.apps/mzutd2fbabf5-cluster-u1-replica-u1-gen-1 1/1 11s - statefulset.apps/mzutd2fbabf5-environmentd-1 1/1 16s - - NAME STATUS COMPLETIONS DURATION AGE - job.batch/create-db-demo-db Complete 1/1 11s 33s - ``` - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). - -1. Open the Materialize Console in your browser: - - {{< tabs >}} - - {{< tab "Via Network Load Balancer" >}} - - Starting in v0.3.0, for each Materialize instance, Materialize on AWS - Terraform module also deploys AWS Network Load Balancers (by default, - internal) with the following listeners, including a listener on port 8080 for - the Materialize Console: - - | Port | Description | - | ---- | ------------| - | 6875 | For SQL connections to the database | - | 6876 | For HTTP(S) connections to the database | - | **8080** | **For HTTP(S) connections to Materialize Console** | - - The Network Load Balancer (NLB) details are found in the `nlb_details` in - the [Terraform output](#aws-terrafrom-output). - - The example uses a self-signed ClusterIssuer. As such, you may encounter a - warning with regards to the certificate. In production, run with certificates - from an official Certificate Authority (CA) rather than self-signed - certificates. - - {{}} - - {{< tab "Via port forwarding" >}} - - {{% self-managed/port-forwarding-handling %}} - - {{}} - {{}} - - {{< tip >}} - - {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} - - {{< /tip >}} - -## Next steps - -{{% self-managed/next-steps %}} - -## Cleanup - -{{% self-managed/cleanup-cloud %}} - - {{< tip >}} - - - To delete your S3 bucket, you may need to empty the S3 bucket first. If the - `terraform destroy` command is unable to delete the S3 bucket and does not - progress beyond "Still destroying...", empty the S3 bucket first and rerun - the `terraform destroy` command. - - - Upon successful destroy, you may receive some informational messages with - regards to CustomResourceDefinition(CRD). You may safely ignore these - messages as your whole deployment has been destroyed, including the CRDs. - - {{}} +| Guide | Description | +|-------|-------------| +| [Terraform Provider](/installation/install-on-aws/terraform-module/) | Install Materialize on AWS using our new Unified Terraform Provider | +| [Terraform Provider (legacy)](/installation/install-on-aws/legacy-terraform-module/) | Install Materialize on AWS using our Terraform Provider (legacy) | +| [Appendix: AWS deployment guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) | Additional guidelines for AWS deployments | ## See also - [Materialize Operator Configuration](/installation/configuration/) - [Troubleshooting](/installation/troubleshooting/) -- [Appendix: AWS Deployment -guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) - [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md b/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md index f8f8252b35695..755f7bfb926d1 100644 --- a/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md +++ b/doc/user/content/installation/install-on-aws/appendix-deployment-guidelines.md @@ -14,47 +14,39 @@ menu: As a general guideline, we recommend: -- Processor Type: ARM-based CPU - -- Sizing: - - - If spill-to-disk is not enabled: 1:8 ratio of vCPU to GiB memory - - - If spill-to-disk is enabled (*Recommended*): 1:16 ratio of vCPU to GiB local - instance storage +- ARM-based CPU +- A 1:8 ratio of vCPU to GiB memory is recommended. +- When using swap, it is recommended to use a 8:1 ratio of GiB local instance storage to GiB Ram. {{% self-managed/aws-recommended-instances %}} ## Locally-attached NVMe storage -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. +Configuring swap on nodes to using locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. ### Swap support -Starting in v0.6.1 of Materialize on AWS Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: +***New Unified Terraform*** -- Creates a node group for Materialize. -- Configures NVMe instance store volumes as swap using a daemonset. -- Enables swap at the Kubelet. +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws/examples/simple) supports configuring swap out of the box. -For swap support, the following configuration option is available: +***Legacy Terraform*** -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) +The Legacy Terraform provider, adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: + - Creates a node group for Materialize. + - Configures NVMe instance store volumes as swap using a daemonset. + - Enables swap at the Kubelet. See [Upgrade Notes](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061). - -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. +{{< note >}} +If deploying `v25.2` Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26` this will be handled automatically. +{{< /note >}} ## TLS diff --git a/doc/user/content/installation/install-on-aws/legacy-terraform-module/_index.md b/doc/user/content/installation/install-on-aws/legacy-terraform-module/_index.md new file mode 100644 index 0000000000000..8c91d1c7a569c --- /dev/null +++ b/doc/user/content/installation/install-on-aws/legacy-terraform-module/_index.md @@ -0,0 +1,23 @@ +--- +title: "Terraform Module (legacy)" +description: "" +disable_list: true +disable_toc: true +menu: + main: + parent: "install-on-aws" + identifier: "install-on-aws-legacy-terraform-module" + weight: 5 + + +--- + +The tutorials in this section show you how to deploy Materialize using the [Materialize on AWS Legacy Terraform +module](https://github.com/MaterializeInc/terraform-aws-materialize). + + +| Guide | Description | +|-------|-------------| +| [Install](/installation/install-on-aws/legacy-terraform-module/install/) | Install Materialize on AWS | +| [Upgrade](/installation/install-on-aws/legacy-terraform-module/upgrade/) | Upgrade your Materialize deployment on AWS | +| [Appendix: AWS configuration](/installation/install-on-aws/legacy-terraform-module/appendix-configuration/) | Configuration for AWS deployments | diff --git a/doc/user/content/installation/install-on-aws/appendix-aws-configuration.md b/doc/user/content/installation/install-on-aws/legacy-terraform-module/appendix-configuration.md similarity index 73% rename from doc/user/content/installation/install-on-aws/appendix-aws-configuration.md rename to doc/user/content/installation/install-on-aws/legacy-terraform-module/appendix-configuration.md index e9e520e2c9cb8..70db19b5f3048 100644 --- a/doc/user/content/installation/install-on-aws/appendix-aws-configuration.md +++ b/doc/user/content/installation/install-on-aws/legacy-terraform-module/appendix-configuration.md @@ -3,7 +3,7 @@ title: "Appendix: Required configuration" description: "Required configuration for Materialize on AWS Terraform." menu: main: - parent: "install-on-aws" + parent: "install-on-aws-legacy-terraform-module" identifier: "appendix-aws-provider-config" weight: 50 aliases: @@ -60,3 +60,19 @@ provider "helm" { } } ``` + +## Swap support + +Starting in v0.6.1 of Materialize on AWS Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration option is available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061). diff --git a/doc/user/content/installation/install-on-aws/legacy-terraform-module/install.md b/doc/user/content/installation/install-on-aws/legacy-terraform-module/install.md new file mode 100644 index 0000000000000..dc04c39636c1e --- /dev/null +++ b/doc/user/content/installation/install-on-aws/legacy-terraform-module/install.md @@ -0,0 +1,481 @@ +--- +title: "Install" +description: "" +aliases: + - /self-hosted/install-on-aws/ +menu: + main: + parent: "install-on-aws-legacy-terraform-module" + identifier: "install-aws" + weight: 5 +--- + +{{% self-managed/materialize-components-sentence %}} + +The tutorial deploys Materialize to AWS Elastic Kubernetes Service (EKS) with a +PostgreSQL RDS database as the metadata database and AWS S3 for blob storage. +The tutorial uses [Materialize on AWS Terraform +module](https://github.com/MaterializeInc/terraform-aws-materialize) to: + +- Set up the AWS Kubernetes environment. +- Call + [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) + module to deploy Materialize Operator and Materialize instances to that EKS + cluster. + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +{{% self-managed/aws-recommended-instances %}} + +See [Appendix: AWS Deployment +guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) for +more information. + +## Prerequisites + +### Terraform + +If you don't have Terraform installed, [install +Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). + +### AWS CLI + +If you do not have the AWS CLI installed, install. For details, see the [AWS +documentation](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + +### kubectl + +If you do not have `kubectl`, install. See the [Amazon EKS: install `kubectl` +documentation](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) +for details. + +### Helm 3.2.0+ + +If you do not have Helm 3.2.0+, install. For details, see the [Helm +documentation](https://helm.sh/docs/intro/install/). + +### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +## Set up AWS Kubernetes environment and install Materialize + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< self-managed/tutorial-disclaimer >}} + +{{< /warning >}} + +{{< tabs >}} + +{{< tab "Deployed components" >}} + +[Materialize on AWS Terraform +module](https://github.com/MaterializeInc/terraform-aws-materialize/blob/main/README.md) +deploys a sample infrastructure on AWS (region `us-east-1`) with the following +components: + +{{< yaml-table data="self_managed/aws_terraform_deployed_components" >}} + +{{< tip >}} +{{% self-managed/aws-terraform-configs %}} +{{< /tip >}} + +{{}} +{{< tab "Releases" >}} + +{{< yaml-table data="self_managed/aws_terraform_versions" >}} + +{{}} +{{}} + +1. Open a Terminal window. + +1. Configure AWS CLI with your AWS credentials. For details, see the [AWS + documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). + +{{% self-managed/versions/step-clone-aws-terraform-repo %}} + +1. Go to the `examples/simple` folder in the Materialize Terraform repo + directory. + + ```bash + cd terraform-aws-materialize/examples/simple + ``` + + {{< tip >}} + {{< self-managed/aws-terraform-configs >}} + {{< /tip >}} + +1. Create a `terraform.tfvars` file (you can copy from the + `terraform.tfvars.example` file) and specify the following variables: + + | Variable | Description | + |--------------------|-------------| + | `namespace` | A namespace (e.g., `my-demo`) that will be used to form part of the prefix for your AWS resources.
**Requirements:**
- Maximum of 12 characters
- Must start with a lowercase letter
- Must be lowercase alphanumeric and hyphens only | + | `environment` | An environment name (e.g., `dev`, `test`) that will be used to form part of the prefix for your AWS resources.
**Requirements:**
- Maximum of 8 characters
- Must be lowercase alphanumeric only | + + + ```bash + # The namespace and environment variables are used to construct the names of the resources + # e.g. ${namespace}-${environment}-storage, ${namespace}-${environment}-db etc. + + namespace = "enter-namespace" // maximum 12 characters, start with a letter, contain lowercase alphanumeric and hyphens only (e.g. my-demo) + environment = "enter-environment" // maximum 8 characters, lowercase alphanumeric only (e.g., dev, test) + ``` + + {{< tip >}} + {{< self-managed/aws-terraform-configs >}} + {{< /tip >}} + +1. Initialize the terraform directory. + + ```bash + terraform init + ``` + +1. Use terraform plan to review the changes to be made. + + ```bash + terraform plan + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply + ``` + + To approve the changes and apply, enter `yes`. + + + + Upon successful completion, various fields and their values are output: + + ```none + Apply complete! Resources: 89 added, 0 changed, 0 destroyed. + + Outputs: + + cluster_certificate_authority_data = + database_endpoint = "my-demo-dev-db.abcdefg8dsto.us-east-1.rds.amazonaws.com:5432" + eks_cluster_endpoint = "https://0123456789A00BCD000E11BE12345A01.gr7.us-east-1.eks.amazonaws.com" + eks_cluster_name = "my-demo-dev-eks" + materialize_s3_role_arn = "arn:aws:iam::000111222333:role/my-demo-dev-mz-role" + metadata_backend_url = + nlb_details = [] + oidc_provider_arn = "arn:aws:iam::000111222333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/7D14BCA3A7AA896A836782D96A24F958" + persist_backend_url = "s3://my-demo-dev-storage-f2def2a9/dev:serviceaccount:materialize-environment:12345678-1234-1234-1234-12345678912" + s3_bucket_name = "my-demo-dev-storage-f2def2a9" + vpc_id = "vpc-0abc000bed1d111bd" + ``` + +1. Note your specific values for the following fields: + + - `eks_cluster_name` (Used to configure `kubectl`) + +1. Configure `kubectl` to connect to your EKS cluster, replacing: + + - `` with the name of your EKS cluster. Your cluster + name has the form `{namespace}-{environment}-eks`; e.g., + `my-demo-dev-eks`. + + - `` with the region of your EKS cluster. The + simple example uses `us-east-1`. + + ```bash + aws eks update-kubeconfig --name --region + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl get nodes + ``` + + For help with `kubectl` commands, see [kubectl Quick + reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +1. By default, the example Terraform installs the Materialize Operator and, + starting in v0.4.0, a `cert-manager`. Verify the installation and check the + status: + + {{< tabs >}} + {{< tab "Materialize Operator" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n materialize + ``` + + Wait for the components to be in the `Running` state: + + ```none + NAME READY STATUS RESTARTS AGE + pod/my-demo-dev-materialize-operator-84ff4b4648-brjhl 1/1 Running 0 12s + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/my-demo-dev-materialize-operator 1/1 1 1 12s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/my-demo-dev-materialize-operator-84ff4b4648 1 1 1 12s + ``` + + {{}} + {{< tab "cert-manager (Starting in version 0.4.0)" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n cert-manager + ``` + Wait for the components to be in the `Running` state: + ``` + NAME READY STATUS RESTARTS AGE + pod/cert-manager-cainjector-686546c9f7-v9hwp 1/1 Running 0 4m20s + pod/cert-manager-d6746cf45-cdmb5 1/1 Running 0 4m20s + pod/cert-manager-webhook-5f79cd6f4b-rcjbq 1/1 Running 0 4m20s + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/cert-manager ClusterIP 172.20.2.136 9402/TCP 4m20s + service/cert-manager-cainjector ClusterIP 172.20.154.137 9402/TCP 4m20s + service/cert-manager-webhook ClusterIP 172.20.63.217 443/TCP,9402/TCP 4m20s + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/cert-manager 1/1 1 1 4m20s + deployment.apps/cert-manager-cainjector 1/1 1 1 4m20s + deployment.apps/cert-manager-webhook 1/1 1 1 4m20s + NAME DESIRED CURRENT READY AGE + replicaset.apps/cert-manager-cainjector-686546c9f7 1 1 1 4m20s + replicaset.apps/cert-manager-d6746cf45 1 1 1 4m20s + replicaset.apps/cert-manager-webhook-5f79cd6f4b 1 1 1 + 4m20s + ``` + + {{}} + {{}} + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting) guide. + +1. Once the Materialize operator is deployed and running, you can deploy the + Materialize instances. To deploy Materialize instances, create a + `mz_instances.tfvars` file with the [Materialize instance + configuration](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances). + + For example, the following specifies the configuration for a `demo` instance. + + ```bash + cat < mz_instances.tfvars + + materialize_instances = [ + { + name = "demo" + namespace = "materialize-environment" + database_name = "demo_db" + cpu_request = "1" + memory_request = "2Gi" + memory_limit = "2Gi" + license_key = "" + } + ] + EOF + ``` + + - **Starting in v0.3.0**, the Materialize on AWS Terraform module also + deploys, by default, Network Load Balancers (NLBs) for each Materialize + instance (i.e., the + [`create_nlb`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) + flag defaults to `true`). The NLBs, by default, are configured to be + internal (i.e., the + [`internal_nlb`](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) + flag defaults to `true`). See [`materialize_instances`]( + https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#input_materialize_instances) + for the Materialize instance configuration options. + + - **Starting in v0.4.0**, a self-signed `ClusterIssuer` is deployed by + default. The `ClusterIssuer` is deployed on subsequent after the + `cert-manager` is running. + + - **Starting in v0.4.6**, you can specify addition configuration options via + `environmentd_extra_args`. + + {{< tip >}} + {{% self-managed/aws-terraform-upgrade-notes %}} + + See [Materialize on AWS releases](/installation/appendix-terraforms/#materialize-on-aws-terraform-module) for notable changes. + {{}} + +1. Run `terraform plan` with both `.tfvars` files and review the changes to be + made. + + ```bash + terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + The plan should show the changes to be made, with a summary similar to the + following: + + ``` + Plan: 17 to add, 1 to change, 0 to destroy. + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + To approve the changes and apply, enter `yes`. + + Upon successful completion, you should see output with a summary similar to + the following: + + + + ```bash + Apply complete! Resources: 17 added, 1 changed, 0 destroyed. + + Outputs: + + cluster_certificate_authority_data = + database_endpoint = "my-demo-dev-db.abcdefg8dsto.us-east-1.rds.amazonaws.com:5432" + eks_cluster_endpoint = "https://0123456789A00BCD000E11BE12345A01.gr7.us-east-1.eks.amazonaws.com" + eks_cluster_name = "my-demo-dev-eks" + materialize_s3_role_arn = "arn:aws:iam::000111222333:role/my-demo-dev-mz-role" + metadata_backend_url = + nlb_details = [ + "demo" = { + "arn" = "arn:aws:elasticloadbalancing:us-east-1:000111222333:loadbalancer/net/my-demo-dev/aeae3d936afebcfe" + "dns_name" = "my-demo-dev-aeae3d936afebcfe.elb.us-east-1.amazonaws.com" + } + ] + oidc_provider_arn = "arn:aws:iam::000111222333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/7D14BCA3A7AA896A836782D96A24F958" + persist_backend_url = "s3://my-demo-dev-storage-f2def2a9/dev:serviceaccount:materialize-environment:12345678-1234-1234-1234-12345678912" + s3_bucket_name = "my-demo-dev-storage-f2def2a9" + vpc_id = "vpc-0abc000bed1d111bd" + ``` + + The Network Load Balancer (NLB) details `nlb_details` are available when + running the Terraform module v0.3.0+. + +1. Verify the installation and check the status: + + ```bash + kubectl get all -n materialize-environment + ``` + + Wait for the components to be in the `Running` state. + + ```none + NAME READY STATUS RESTARTS AGE + pod/create-db-demo-db-6swk7 0/1 Completed 0 33s + pod/mzutd2fbabf5-balancerd-6c9755c498-28kcw 1/1 Running 0 11s + pod/mzutd2fbabf5-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 11s + pod/mzutd2fbabf5-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 11s + pod/mzutd2fbabf5-console-57f94b4588-6lg2x 1/1 Running 0 4s + pod/mzutd2fbabf5-console-57f94b4588-v65lk 1/1 Running 0 4s + pod/mzutd2fbabf5-environmentd-1-0 1/1 Running 0 16s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/mzutd2fbabf5-balancerd ClusterIP None 6876/TCP,6875/TCP 11s + service/mzutd2fbabf5-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 12s + service/mzutd2fbabf5-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 12s + service/mzutd2fbabf5-console ClusterIP None 8080/TCP 4s + service/mzutd2fbabf5-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 11s + service/mzutd2fbabf5-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 16s + service/mzutd2fbabf5-persist-pubsub-1 ClusterIP None 6879/TCP 16s + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/mzutd2fbabf5-balancerd 1/1 1 1 11s + deployment.apps/mzutd2fbabf5-console 2/2 2 2 4s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/mzutd2fbabf5-balancerd-6c9755c498 1 1 1 11s + replicaset.apps/mzutd2fbabf5-console-57f94b4588 2 2 2 4s + + NAME READY AGE + statefulset.apps/mzutd2fbabf5-cluster-s2-replica-s1-gen-1 1/1 12s + statefulset.apps/mzutd2fbabf5-cluster-u1-replica-u1-gen-1 1/1 11s + statefulset.apps/mzutd2fbabf5-environmentd-1 1/1 16s + + NAME STATUS COMPLETIONS DURATION AGE + job.batch/create-db-demo-db Complete 1/1 11s 33s + ``` + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting/). + +1. Open the Materialize Console in your browser: + + {{< tabs >}} + + {{< tab "Via Network Load Balancer" >}} + + Starting in v0.3.0, for each Materialize instance, Materialize on AWS + Terraform module also deploys AWS Network Load Balancers (by default, + internal) with the following listeners, including a listener on port 8080 for + the Materialize Console: + + | Port | Description | + | ---- | ------------| + | 6875 | For SQL connections to the database | + | 6876 | For HTTP(S) connections to the database | + | **8080** | **For HTTP(S) connections to Materialize Console** | + + The Network Load Balancer (NLB) details are found in the `nlb_details` in + the [Terraform output](#aws-terrafrom-output). + + The example uses a self-signed ClusterIssuer. As such, you may encounter a + warning with regards to the certificate. In production, run with certificates + from an official Certificate Authority (CA) rather than self-signed + certificates. + + {{}} + + {{< tab "Via port forwarding" >}} + + {{% self-managed/port-forwarding-handling %}} + + {{}} + {{}} + + {{< tip >}} + + {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} + + {{< /tip >}} + +## Next steps + +{{% self-managed/next-steps %}} + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + + {{< tip >}} + + - To delete your S3 bucket, you may need to empty the S3 bucket first. If the + `terraform destroy` command is unable to delete the S3 bucket and does not + progress beyond "Still destroying...", empty the S3 bucket first and rerun + the `terraform destroy` command. + + - Upon successful destroy, you may receive some informational messages with + regards to CustomResourceDefinition(CRD). You may safely ignore these + messages as your whole deployment has been destroyed, including the CRDs. + + {{}} + +## See also + +- [Materialize Operator Configuration](/installation/configuration/) +- [Troubleshooting](/installation/troubleshooting/) +- [Appendix: AWS Deployment +guidelines](/installation/install-on-aws/appendix-deployment-guidelines/) +- [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-aws/upgrade-on-aws.md b/doc/user/content/installation/install-on-aws/legacy-terraform-module/upgrade.md similarity index 98% rename from doc/user/content/installation/install-on-aws/upgrade-on-aws.md rename to doc/user/content/installation/install-on-aws/legacy-terraform-module/upgrade.md index 6deb0e7ee509c..de77568fc4085 100644 --- a/doc/user/content/installation/install-on-aws/upgrade-on-aws.md +++ b/doc/user/content/installation/install-on-aws/legacy-terraform-module/upgrade.md @@ -1,9 +1,9 @@ --- -title: "Upgrade on AWS (Terraform)" +title: "Upgrade" description: "Procedure to upgrade your Materialize operator and instances running on AWS" menu: main: - parent: "install-on-aws" + parent: "install-on-aws-legacy-terraform-module" identifier: "upgrade-on-aws" weight: 10 --- diff --git a/doc/user/content/installation/install-on-aws/terraform-module/_index.md b/doc/user/content/installation/install-on-aws/terraform-module/_index.md new file mode 100644 index 0000000000000..475455fa85d85 --- /dev/null +++ b/doc/user/content/installation/install-on-aws/terraform-module/_index.md @@ -0,0 +1,148 @@ +--- +title: "Terraform Module" +description: "" +menu: + main: + parent: "install-on-aws" + identifier: "install-aws-terraform" + weight: 5 +--- + +Materialize provides a set of modular Terraform modules that can be used to +deploy all services required for a production ready Materialize database. +The module is intended to provide a simple set of examples on how to deploy +materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +The repository can be found at: + +***[Materialize Terraform Self-Managed AWS](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws)*** + +Please see the [top level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) and [cloud specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws) documentation for a full understanding +of the module structure and customizations. + +Also check out the [AWS deployment guide](/installation/install-on-aws/appendix-deployment-guidelines/) for details on recommended instance sizing and configuration. + +{{< note >}} +{{% self-managed/materialize-components-sentence %}} +{{< /note >}} + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + + +## Prerequisites + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [AWS Cli ](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- [`kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +#### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +--- + +# Example: Simple Materialize Deployment on AWS + +This example demonstrates how to deploy a complete Materialize environment on AWS using the modular Terraform setup from this repository. + + +## Setup +```shell +git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git +cd materialize-terraform-self-managed/aws/examples/simple +```` + + +## What Gets Created + +This example provisions the following infrastructure: + +### Networking +- **VPC**: 10.0.0.0/16 with DNS hostnames and support enabled +- **Subnets**: 3 private subnets (10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24) and 3 public subnets (10.0.101.0/24, 10.0.102.0/24, 10.0.103.0/24) across availability zones us-east-1a, us-east-1b, us-east-1c +- **NAT Gateway**: Single NAT Gateway for all private subnets +- **Internet Gateway**: For public subnet connectivity + +### Compute +- **EKS Cluster**: Version 1.32 with CloudWatch logging (API, audit) +- **Base Node Group**: 2 nodes (t4g.medium) for Karpenter and CoreDNS +- **Karpenter**: Auto-scaling controller with two node classes: + - Generic nodepool: t4g.xlarge instances for general workloads + - Materialize nodepool: r7gd.2xlarge instances with swap enabled and dedicated taints to run materialize instance workloads. + +### Database +- **RDS PostgreSQL**: Version 15, db.t3.large instance +- **Storage**: 50GB allocated, autoscaling up to 100GB +- **Deployment**: Single-AZ (non-production configuration) +- **Backups**: 7-day retention +- **Security**: Dedicated security group with access from EKS cluster and nodes + +### Storage +- **S3 Bucket**: Dedicated bucket for Materialize persistence +- **Encryption**: Disabled (for testing; enable in production) +- **Versioning**: Disabled (for testing; enable in production) +- **IAM Role**: IRSA role for Kubernetes service account access + +### Kubernetes Add-ons +- **AWS Load Balancer Controller**: For managing Network Load Balancers +- **cert-manager**: Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal +- **Self-signed ClusterIssuer**: Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. + +### Materialize +- **Operator**: Materialize Kubernetes operator +- **Instance**: Single Materialize instance in `materialize-environment` namespace +- **Network Load Balancer**: Dedicated internal NLB for Materialize access (ports 6875, 6876, 8080) + +--- + +## Getting Started + +### Step 1: Set Required Variables + +Before running Terraform, create a `terraform.tfvars` file with the following variables: + +```hcl +name_prefix = "simple-demo" +aws_region = "us-east-1" +aws_profile = "your-aws-profile" +license_key = "your-materialize-license-key" # Get from https://materialize.com/self-managed/ +tags = { + environment = "demo" +} +``` + +**Required Variables:** +- `name_prefix`: Prefix for all resource names +- `aws_region`: AWS region for deployment +- `aws_profile`: AWS CLI profile to use +- `tags`: Map of tags to apply to resources +- `license_key`: Materialize license key + +--- + +### Step 2: Deploy Materialize + +Run the usual Terraform workflow: + +```bash +terraform init +terraform apply +``` + +--- + +## Notes + +* You can customize each module independently. +* To reduce cost in your demo environment, you can tweak subnet CIDRs and instance types in `main.tf`. + +***Don't forget to destroy resources when finished:*** +```bash +terraform destroy +``` diff --git a/doc/user/content/installation/install-on-azure/_index.md b/doc/user/content/installation/install-on-azure/_index.md index d61eba7238635..be733f8a566ac 100644 --- a/doc/user/content/installation/install-on-azure/_index.md +++ b/doc/user/content/installation/install-on-azure/_index.md @@ -1,6 +1,6 @@ --- -title: "Install on Azure (via Terraform)" -description: "Install Materialize on Azure Kubernetes Service (AKS) using Terraform" +title: "Install on Azure" +description: "Install and upgrade Materialize on Azure" disable_list: true menu: main: @@ -13,518 +13,14 @@ aliases: {{% self-managed/materialize-components-sentence blobstorage="blob storage; specifically **block** blob storage on Azure" %}} -The tutorial deploys Materialize to Azure Kubernetes Service (AKS) with a -PostgreSQL database as the metadata database and Azure premium block blob -storage for blob storage. The tutorial uses [Materialize on Azure Terraform -modules](https://github.com/MaterializeInc/terraform-azurerm-materialize) to: - -- Set up the Azure Kubernetes environment -- Call - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) - module to deploy Materialize Operator and Materialize instances to that AKS - cluster - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< self-managed/tutorial-disclaimer >}} - -{{< /warning >}} - -## Prerequisites - -### Azure subscription - -If you do not have an Azure subscription to use for this tutorial, create one. - -### Azure CLI - -If you don't have Azure CLI installed, [install Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). - -### Terraform - -If you don't have Terraform installed, [install Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). - -### kubectl - -If you do not have `kubectl`, install `kubectl`. - -### Python (v3.12+) and pip - -If you don't have Python (v3.12 or greater) installed, install it. See -[Python.org](https://www.python.org/downloads/). If `pip` is not included with -your version of Python, install it. - -### Helm 3.2.0+ - -If you don't have Helm version 3.2.0+ installed, install. For details, see to -the [Helm documentation](https://helm.sh/docs/intro/install/). - -### jq (Optional) - -*Optional*. `jq` is used to parse the AKS cluster name and region from the -Terraform outputs. Alternatively, you can manually specify the name and region. -If you want to use `jq` and do not have `jq` installed, install. - -### License key - -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} - -## A. Authenticate with Azure - -1. Open a Terminal window. - -1. Authenticate with Azure. - - ```bash - az login - ``` - - The command opens a browser window to sign in to Azure. Sign in. - -1. Select the subscription and tenant to use. After you have signed in, back in - the terminal, your tenant and subscription information is displayed. - - ```none - Retrieving tenants and subscriptions for the selection... - - [Tenant and subscription selection] - - No Subscription name Subscription ID Tenant - ----- ------------------- ------------------------------------ ---------------- - [1]* ... ... ... - - The default is marked with an *; the default tenant is '' and - subscription is '' (). - ``` - - Select the subscription and tenant. - -1. Set `ARM_SUBSCRIPTION_ID` to the subscription ID. - - ```bash - export ARM_SUBSCRIPTION_ID= - ``` - -## B. Set up Azure Kubernetes environment and install Materialize - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< /warning >}} - -{{< tabs >}} - -{{< tab "Deployed components" >}} - -[Materialize on Azure Terraform -module](https://github.com/MaterializeInc/terraform-azurerm-materialize) for -deploys a sample infrastructure on Azure with the following components: - -{{< yaml-table data="self_managed/azure_terraform_deployed_components" >}} - -{{< tip >}} - -{{% self-managed/azure-terraform-configs %}} - -{{< /tip >}} - -{{}} -{{< tab "Releases" >}} - -{{< yaml-table data="self_managed/azure_terraform_versions" >}} - -{{}} -{{}} - -1. Open a Terminal window. - -{{% self-managed/versions/step-clone-azure-terraform-repo %}} - -1. Go to the `examples/simple` folder in the Materialize Terraform repo - directory. - - ```bash - cd terraform-azurerm-materialize/examples/simple - ``` - - {{< tip >}} - - {{% self-managed/azure-terraform-configs %}} - - {{< /tip >}} - - -1. Optional. Create a virtual environment, specifying a path for the new virtual - environment: - - ```bash - python3 -m venv - - ``` - - Activate the virtual environment: - ```bash - source /bin/activate - ``` - -1. Install the required packages. - - ```bash - pip install -r requirements.txt - ``` - -1. Create a `terraform.tfvars` file (you can copy from the - `terraform.tfvars.example` file) and specify: - - - The prefix for the resources. Prefix has a maximum of 12 characters and - contains only alphanumeric characters and hyphens; e.g., `mydemo`. - - - The location for the AKS cluster. - - ```bash - prefix="enter-prefix" // maximum 12 characters, containing only alphanumeric characters and hyphens; e.g. mydemo - location="eastus2" - ``` - - {{< tip >}} - - {{% self-managed/azure-terraform-configs %}} - - {{< /tip >}} - -1. Initialize the terraform directory. - - ```bash - terraform init - ``` - -1. Use terraform plan to review the changes to be made. - - ```bash - terraform plan - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply - ``` - - To approve the changes and apply, enter `yes`. - - Upon successful completion, various fields and their values are output: - - ```bash - Apply complete! Resources: 33 added, 0 changed, 0 destroyed. - - Outputs: - - aks_cluster = - connection_strings = - kube_config = - load_balancer_details = {} - resource_group_name = "mydemo-rg" - ``` - -1. Configure `kubectl` to connect to your cluster: - - - ``. Your cluster name has the form `-aks`; e.g., - `mz-simple-aks`. - - - ``, as specified in the output. - - ```bash - az aks get-credentials --resource-group --name - ``` - - Alternatively, you can use the following command to get the cluster name and - resource group name from the Terraform output: - - ```bash - az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -json aks_cluster | jq -r '.name') - ``` - - To verify that you have configured correctly, run the following command: - - ```bash - kubectl cluster-info - ``` - - For help with `kubectl` commands, see [kubectl Quick - reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). - -1. By default, the example Terraform installs the Materialize Operator and, - starting in v0.3.0, a `cert-manager`. Verify the - installation and check the status: - - {{< tabs >}} - {{< tab "Materialize Operator" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n materialize - ``` - - Wait for the components to be in the `Running` state: - - ```none - NAME READY STATUS RESTARTS AGE - pod/materialize-mydemo-materialize-operator-74d8f549d6-lkjjf 1/1 Running 0 36m - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/materialize-mydemo-materialize-operator 1/1 1 1 36m - - NAME DESIRED CURRENT READY AGE - replicaset.apps/materialize-mydemo-materialize-operator-74d8f549d6 1 1 1 36m - ``` - - {{}} - - {{< tab "cert-manager (Starting in version 0.3.0)" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n cert-manager - ``` - Wait for the components to be in the `Running` state: - ``` - NAME READY STATUS RESTARTS AGE - pod/cert-manager-8576d99cc8-xqxbc 1/1 Running 0 4m22s - pod/cert-manager-cainjector-664b5878d6-wc4tz 1/1 Running 0 4m22s - pod/cert-manager-webhook-6ddb7bd6c5-vrm2p 1/1 Running 0 4m22s - - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/cert-manager ClusterIP 10.1.227.230 9402/TCP 4m22s - service/cert-manager-cainjector ClusterIP 10.1.222.156 9402/TCP 4m22s - service/cert-manager-webhook ClusterIP 10.1.84.207 443/TCP,9402/TCP 4m22s - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/cert-manager 1/1 1 1 4m23s - deployment.apps/cert-manager-cainjector 1/1 1 1 4m23s - deployment.apps/cert-manager-webhook 1/1 1 1 4m23s - - NAME DESIRED CURRENT READY AGE - replicaset.apps/cert-manager-8576d99cc8 1 1 1 4m23s - replicaset.apps/cert-manager-cainjector-664b5878d6 1 1 1 4m23s - replicaset.apps/cert-manager-webhook-6ddb7bd6c5 1 1 1 4m23s - ``` - - {{}} - {{}} - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). - -1. Once the Materialize operator is deployed and running, you can deploy the - Materialize instances. To deploy Materialize instances, create a - `mz_instances.tfvars` file with the Materialize instance configuration. - - For example, the following specifies the configuration for a `demo` instance. - - ```bash - cat < mz_instances.tfvars - - materialize_instances = [ - { - name = "demo" - namespace = "materialize-environment" - database_name = "demo_db" - cpu_request = "1" - memory_request = "2Gi" - memory_limit = "2Gi" - license_key = "" - } - ] - EOF - ``` - - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - - **Starting in v0.3.0**, the Materialize on Azure Terraform module also - deploys, by default, a self-signed `ClusterIssuer`. The `ClusterIssuer` is - deployed after the `cert-manager` is deployed and running. - - - **Starting in v0.3.1**, the Materialize on Azure Terraform module also - deploys, by default, [Load - balancers](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) - for Materialize instances (i.e., the - [`create_load_balancer`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) - flag defaults to `true`). The load balancers, by default, are configured to - be internal (i.e., the - [`internal_load_balancer`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) - flag defaults to `true`). - - - **Starting in v0.4.3**, you can specify addition configuration options via - `environmentd_extra_args`. - - {{< tip >}} - {{% self-managed/azure-terraform-upgrade-notes %}} - - See [Materialize on Azure releases](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) for notable changes. - {{}} - -1. Run `terraform plan` with both `.tfvars` files and review the changes to be - made. - - ```bash - terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - The plan should show the changes to be made, with a summary similar to the - following: - - ``` - Plan: 9 to add, 1 to change, 0 to destroy. - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - To approve the changes and apply, enter `yes`. - - - - Upon successful completion, you should see output with a summary similar to the following: - - ```bash - Apply complete! Resources: 9 added, 1 changed, 0 destroyed. - - Outputs: - - aks_cluster = - connection_strings = - kube_config = - load_balancer_details = { - "demo" = { - "balancerd_load_balancer_ip" = "192.0.2.10" - "console_load_balancer_ip" = "192.0.2.254" - } - } - resource_group_name = "mydemo-rg" - ``` - -1. Verify the installation and check the status: - - ```bash - kubectl get all -n materialize-environment - ``` - - Wait for the components to be ready and in the `Running` state. - - ```none - NAME READY STATUS RESTARTS AGE - pod/db-demo-db-l6ss8 0/1 Completed 0 2m21s - pod/mz62lr3yltj8-balancerd-6d5dd6d4cf-r9nf4 1/1 Running 0 111s - pod/mz62lr3yltj8-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 114s - pod/mz62lr3yltj8-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 114s - pod/mz62lr3yltj8-console-bfc797745-6nlwv 1/1 Running 0 96s - pod/mz62lr3yltj8-console-bfc797745-tk9vm 1/1 Running 0 96s - pod/mz62lr3yltj8-environmentd-1-0 1/1 Running 0 2m4s - - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/mz62lr3yltj8-balancerd ClusterIP None 6876/TCP,6875/TCP 111s - service/mz62lr3yltj8-balancerd-lb LoadBalancer 10.1.201.77 192.0.2.10 6875:30890/TCP,6876:31750/TCP 2m4s - service/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s - service/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s - service/mz62lr3yltj8-console ClusterIP None 8080/TCP 96s - service/mz62lr3yltj8-console-lb LoadBalancer 10.1.130.212 192.0.2.254 8080:30379/TCP 2m4s - service/mz62lr3yltj8-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 111s - service/mz62lr3yltj8-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 2m5s - service/mz62lr3yltj8-persist-pubsub-1 ClusterIP None 6879/TCP 2m4s - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/mz62lr3yltj8-balancerd 1/1 1 1 111s - deployment.apps/mz62lr3yltj8-console 2/2 2 2 96s - - NAME DESIRED CURRENT READY AGE - replicaset.apps/mz62lr3yltj8-balancerd-6d5dd6d4cf 1 1 1 111s - replicaset.apps/mz62lr3yltj8-console-bfc797745 2 2 2 96s - - NAME READY AGE - statefulset.apps/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 1/1 114s - statefulset.apps/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 1/1 114s - statefulset.apps/mz62lr3yltj8-environmentd-1 1/1 2m4s - - NAME STATUS COMPLETIONS DURATION AGE - job.batch/db-demo-db Complete 1/1 10s 2m21s - - ``` - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). - -1. Open the Materialize Console in your browser: - - - {{< tabs >}} - - {{< tab "Via Network Load Balancer" >}} - - Starting in v0.3.1, for each Materialize instance, Materialize on Azure - Terraform module also deploys load balancers (by default, internal) with the - following listeners, including a listener on port 8080 for the Materialize - Console: - - | Port | Description | - | ---- | ------------| - | 6875 | For SQL connections to the database | - | 6876 | For HTTP(S) connections to the database | - | **8080** | **For HTTP(S) connections to Materialize Console** | - - The load balancer details are found in the `load_balancer_details` in - the [Terraform output](#azure-terraform-output). - - The example uses a self-signed ClusterIssuer. As such, you may encounter a - warning with regards to the certificate. In production, run with certificates - from an official Certificate Authority (CA) rather than self-signed - certificates. - - {{}} - - {{< tab "Via port forwarding" >}} - - {{% self-managed/port-forwarding-handling %}} - - {{}} - {{}} - - {{< tip >}} - - {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} - - {{< /tip >}} - -## Next steps - -{{% self-managed/next-steps %}} - -## Cleanup - -{{% self-managed/cleanup-cloud %}} - - {{< tip>}} - - If the `terraform destroy` command is unable to delete the subnet because it - is in use, you can rerun the `terraform destroy` command. - - {{}} +| Guide | Description | +|-------|-------------| +| [Terraform Provider](/installation/install-on-azure/terraform-module/) | Install Materialize on Azure using our new Unified Terraform Provider | +| [Terraform Provider (legacy)](/installation/install-on-azure/legacy-terraform-module/) | Install Materialize on Azure using our Terraform Provider (legacy) | +| [Appendix: Azure deployment guidelines](/installation/install-on-azure/appendix-deployment-guidelines/) | Additional guidelines for Azure deployments | ## See also - [Materialize Operator Configuration](/installation/configuration/) - [Troubleshooting](/installation/troubleshooting/) -- [Appendix: Azure deployment guidelines](/installation/install-on-azure/ - appendix-deployment-guidelines) - [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md b/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md index b0be5c7180155..cb3f16f1a01c9 100644 --- a/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md +++ b/doc/user/content/installation/install-on-azure/appendix-deployment-guidelines.md @@ -12,9 +12,9 @@ menu: As a general guideline, we recommend: -- Processor Type: ARM-based CPU - -- Sizing: 2:1 disk-to-RAM ratio with spill-to-disk enabled. +- ARM-based CPU +- A 1:8 ratio of vCPU to GiB memory is recommended. +- When using swap, it is recommended to use a 8:1 ratio of GiB local instance storage to GiB Ram. ### Recommended Azure VM Types with Local NVMe Disks @@ -37,43 +37,39 @@ when the VM is stopped or deleted. {{}} -See also [Locally attached NVMe storage](#locally-attached-nvme-storage). - ## Locally-attached NVMe storage -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. +Configuring swap on nodes to using locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. ### Swap support -Starting in v0.6.1 of Materialize on Azure Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: +***New Unified Terraform*** + +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure/examples/simple) supports configuring swap out of the box. +***Legacy Terraform*** + +The Legacy Terraform provider, adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: - Creates a node group for Materialize. - Configures NVMe instance store volumes as swap using a daemonset. - Enables swap at the Kubelet. -For swap support, the following configuration option is available: - -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) - See [Upgrade Notes](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061). +{{< note >}} +If deploying `v25.2` Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26` this will be handled automatically. +{{< /note >}} + ## Recommended Azure Blob Storage Materialize writes **block** blobs on Azure. As a general guideline, we recommend **Premium block blob** storage accounts. -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## TLS When running with TLS in production, run with certificates from an official @@ -84,11 +80,3 @@ Certificate Authority (CA) rather than self-signed certificates. - [Configuration](/installation/configuration/) - [Installation](/installation/) - [Troubleshooting](/installation/troubleshooting/) - -[`enable_disk_support`]: https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#disk-support-for-materialize-on-azure - -[`disk_support_config`]: - https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_disk_support_config - -[`disk_setup_image`]: - https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_disk_setup_image diff --git a/doc/user/content/installation/install-on-azure/legacy-terraform-module/_index.md b/doc/user/content/installation/install-on-azure/legacy-terraform-module/_index.md new file mode 100644 index 0000000000000..9588822487a23 --- /dev/null +++ b/doc/user/content/installation/install-on-azure/legacy-terraform-module/_index.md @@ -0,0 +1,23 @@ +--- +title: "Terraform Module (legacy)" +description: "" +disable_list: true +disable_toc: true +menu: + main: + parent: "install-on-azure" + identifier: "install-on-azure-legacy-terraform-module" + weight: 5 + + +--- + +The tutorials in this section show you how to deploy Materialize using the [Materialize on Azure Legacy Terraform +modules](https://github.com/MaterializeInc/terraform-azurerm-materialize). + + +| Guide | Description | +|-------|-------------| +| [Install](/installation/install-on-azure/legacy-terraform-module/install/) | Install Materialize on Azure | +| [Upgrade](/installation/install-on-azure/legacy-terraform-module/upgrade/) | Upgrade your Materialize deployment on Azure | +| [Appendix: Azure configuration](/installation/install-on-azure/legacy-terraform-module/appendix-configuration/) | Configuration for Azure deployments | diff --git a/doc/user/content/installation/install-on-azure/appendix-azure-configuration.md b/doc/user/content/installation/install-on-azure/legacy-terraform-module/appendix-configuration.md similarity index 82% rename from doc/user/content/installation/install-on-azure/appendix-azure-configuration.md rename to doc/user/content/installation/install-on-azure/legacy-terraform-module/appendix-configuration.md index c2613724d1f97..2146660c94f74 100644 --- a/doc/user/content/installation/install-on-azure/appendix-azure-configuration.md +++ b/doc/user/content/installation/install-on-azure/legacy-terraform-module/appendix-configuration.md @@ -3,7 +3,7 @@ title: "Appendix: Required configuration" description: "Required configuration for Materialize on Azure Terraform." menu: main: - parent: "install-on-azure" + parent: "install-on-azure-legacy-terraform-module" identifier: "appendix-azure-config" weight: 50 aliases: @@ -83,5 +83,21 @@ provider "helm" { } ``` +## Swap support + +Starting in v0.6.1 of Materialize on Azure Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration option is available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061). + [^1]: If using the `examples/simple/main.tf`, the example configuration handles them for you. diff --git a/doc/user/content/installation/install-on-azure/legacy-terraform-module/install.md b/doc/user/content/installation/install-on-azure/legacy-terraform-module/install.md new file mode 100644 index 0000000000000..4d340e0ee08fc --- /dev/null +++ b/doc/user/content/installation/install-on-azure/legacy-terraform-module/install.md @@ -0,0 +1,522 @@ +--- +title: "Install" +description: "Install Materialize on Azure Kubernetes Service (AKS) using Terraform" +menu: + main: + parent: "install-on-azure-legacy-terraform-module" + identifier: "install-azure" + weight: 5 +--- + + +{{% self-managed/materialize-components-sentence blobstorage="blob storage; specifically **block** blob storage on Azure" %}} + +The tutorial deploys Materialize to Azure Kubernetes Service (AKS) with a +PostgreSQL database as the metadata database and Azure premium block blob +storage for blob storage. The tutorial uses [Materialize on Azure Terraform +modules](https://github.com/MaterializeInc/terraform-azurerm-materialize) to: + +- Set up the Azure Kubernetes environment +- Call + [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) + module to deploy Materialize Operator and Materialize instances to that AKS + cluster + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< self-managed/tutorial-disclaimer >}} + +{{< /warning >}} + +## Prerequisites + +### Azure subscription + +If you do not have an Azure subscription to use for this tutorial, create one. + +### Azure CLI + +If you don't have Azure CLI installed, [install Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). + +### Terraform + +If you don't have Terraform installed, [install Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). + +### kubectl + +If you do not have `kubectl`, install `kubectl`. + +### Python (v3.12+) and pip + +If you don't have Python (v3.12 or greater) installed, install it. See +[Python.org](https://www.python.org/downloads/). If `pip` is not included with +your version of Python, install it. + +### Helm 3.2.0+ + +If you don't have Helm version 3.2.0+ installed, install. For details, see to +the [Helm documentation](https://helm.sh/docs/intro/install/). + +### jq (Optional) + +*Optional*. `jq` is used to parse the AKS cluster name and region from the +Terraform outputs. Alternatively, you can manually specify the name and region. +If you want to use `jq` and do not have `jq` installed, install. + +### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +## A. Authenticate with Azure + +1. Open a Terminal window. + +1. Authenticate with Azure. + + ```bash + az login + ``` + + The command opens a browser window to sign in to Azure. Sign in. + +1. Select the subscription and tenant to use. After you have signed in, back in + the terminal, your tenant and subscription information is displayed. + + ```none + Retrieving tenants and subscriptions for the selection... + + [Tenant and subscription selection] + + No Subscription name Subscription ID Tenant + ----- ------------------- ------------------------------------ ---------------- + [1]* ... ... ... + + The default is marked with an *; the default tenant is '' and + subscription is '' (). + ``` + + Select the subscription and tenant. + +1. Set `ARM_SUBSCRIPTION_ID` to the subscription ID. + + ```bash + export ARM_SUBSCRIPTION_ID= + ``` + +## B. Set up Azure Kubernetes environment and install Materialize + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +{{< tabs >}} + +{{< tab "Deployed components" >}} + +[Materialize on Azure Terraform +module](https://github.com/MaterializeInc/terraform-azurerm-materialize) for +deploys a sample infrastructure on Azure with the following components: + +{{< yaml-table data="self_managed/azure_terraform_deployed_components" >}} + +{{< tip >}} + +{{% self-managed/azure-terraform-configs %}} + +{{< /tip >}} + +{{}} +{{< tab "Releases" >}} + +{{< yaml-table data="self_managed/azure_terraform_versions" >}} + +{{}} +{{}} + +1. Open a Terminal window. + +{{% self-managed/versions/step-clone-azure-terraform-repo %}} + +1. Go to the `examples/simple` folder in the Materialize Terraform repo + directory. + + ```bash + cd terraform-azurerm-materialize/examples/simple + ``` + + {{< tip >}} + + {{% self-managed/azure-terraform-configs %}} + + {{< /tip >}} + + +1. Optional. Create a virtual environment, specifying a path for the new virtual + environment: + + ```bash + python3 -m venv + + ``` + + Activate the virtual environment: + ```bash + source /bin/activate + ``` + +1. Install the required packages. + + ```bash + pip install -r requirements.txt + ``` + +1. Create a `terraform.tfvars` file (you can copy from the + `terraform.tfvars.example` file) and specify: + + - The prefix for the resources. Prefix has a maximum of 12 characters and + contains only alphanumeric characters and hyphens; e.g., `mydemo`. + + - The location for the AKS cluster. + + ```bash + prefix="enter-prefix" // maximum 12 characters, containing only alphanumeric characters and hyphens; e.g. mydemo + location="eastus2" + ``` + + {{< tip >}} + + {{% self-managed/azure-terraform-configs %}} + + {{< /tip >}} + +1. Initialize the terraform directory. + + ```bash + terraform init + ``` + +1. Use terraform plan to review the changes to be made. + + ```bash + terraform plan + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply + ``` + + To approve the changes and apply, enter `yes`. + + Upon successful completion, various fields and their values are output: + + ```bash + Apply complete! Resources: 33 added, 0 changed, 0 destroyed. + + Outputs: + + aks_cluster = + connection_strings = + kube_config = + load_balancer_details = {} + resource_group_name = "mydemo-rg" + ``` + +1. Configure `kubectl` to connect to your cluster: + + - ``. Your cluster name has the form `-aks`; e.g., + `mz-simple-aks`. + + - ``, as specified in the output. + + ```bash + az aks get-credentials --resource-group --name + ``` + + Alternatively, you can use the following command to get the cluster name and + resource group name from the Terraform output: + + ```bash + az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -json aks_cluster | jq -r '.name') + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl cluster-info + ``` + + For help with `kubectl` commands, see [kubectl Quick + reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +1. By default, the example Terraform installs the Materialize Operator and, + starting in v0.3.0, a `cert-manager`. Verify the + installation and check the status: + + {{< tabs >}} + {{< tab "Materialize Operator" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n materialize + ``` + + Wait for the components to be in the `Running` state: + + ```none + NAME READY STATUS RESTARTS AGE + pod/materialize-mydemo-materialize-operator-74d8f549d6-lkjjf 1/1 Running 0 36m + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/materialize-mydemo-materialize-operator 1/1 1 1 36m + + NAME DESIRED CURRENT READY AGE + replicaset.apps/materialize-mydemo-materialize-operator-74d8f549d6 1 1 1 36m + ``` + + {{}} + + {{< tab "cert-manager (Starting in version 0.3.0)" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n cert-manager + ``` + Wait for the components to be in the `Running` state: + ``` + NAME READY STATUS RESTARTS AGE + pod/cert-manager-8576d99cc8-xqxbc 1/1 Running 0 4m22s + pod/cert-manager-cainjector-664b5878d6-wc4tz 1/1 Running 0 4m22s + pod/cert-manager-webhook-6ddb7bd6c5-vrm2p 1/1 Running 0 4m22s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/cert-manager ClusterIP 10.1.227.230 9402/TCP 4m22s + service/cert-manager-cainjector ClusterIP 10.1.222.156 9402/TCP 4m22s + service/cert-manager-webhook ClusterIP 10.1.84.207 443/TCP,9402/TCP 4m22s + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/cert-manager 1/1 1 1 4m23s + deployment.apps/cert-manager-cainjector 1/1 1 1 4m23s + deployment.apps/cert-manager-webhook 1/1 1 1 4m23s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/cert-manager-8576d99cc8 1 1 1 4m23s + replicaset.apps/cert-manager-cainjector-664b5878d6 1 1 1 4m23s + replicaset.apps/cert-manager-webhook-6ddb7bd6c5 1 1 1 4m23s + ``` + + {{}} + {{}} + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting/). + +1. Once the Materialize operator is deployed and running, you can deploy the + Materialize instances. To deploy Materialize instances, create a + `mz_instances.tfvars` file with the Materialize instance configuration. + + For example, the following specifies the configuration for a `demo` instance. + + ```bash + cat < mz_instances.tfvars + + materialize_instances = [ + { + name = "demo" + namespace = "materialize-environment" + database_name = "demo_db" + cpu_request = "1" + memory_request = "2Gi" + memory_limit = "2Gi" + license_key = "" + } + ] + EOF + ``` + + - **Starting in v0.3.0**, the Materialize on Azure Terraform module also + deploys, by default, a self-signed `ClusterIssuer`. The `ClusterIssuer` is + deployed after the `cert-manager` is deployed and running. + + - **Starting in v0.3.1**, the Materialize on Azure Terraform module also + deploys, by default, [Load + balancers](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) + for Materialize instances (i.e., the + [`create_load_balancer`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) + flag defaults to `true`). The load balancers, by default, are configured to + be internal (i.e., the + [`internal_load_balancer`](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#input_materialize_instances) + flag defaults to `true`). + + - **Starting in v0.4.3**, you can specify addition configuration options via + `environmentd_extra_args`. + + {{< tip >}} + {{% self-managed/azure-terraform-upgrade-notes %}} + + See [Materialize on Azure releases](/installation/appendix-terraforms/#materialize-on-azure-terraform-module) for notable changes. + {{}} + +1. Run `terraform plan` with both `.tfvars` files and review the changes to be + made. + + ```bash + terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + The plan should show the changes to be made, with a summary similar to the + following: + + ``` + Plan: 9 to add, 1 to change, 0 to destroy. + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + To approve the changes and apply, enter `yes`. + + + + Upon successful completion, you should see output with a summary similar to the following: + + ```bash + Apply complete! Resources: 9 added, 1 changed, 0 destroyed. + + Outputs: + + aks_cluster = + connection_strings = + kube_config = + load_balancer_details = { + "demo" = { + "balancerd_load_balancer_ip" = "192.0.2.10" + "console_load_balancer_ip" = "192.0.2.254" + } + } + resource_group_name = "mydemo-rg" + ``` + +1. Verify the installation and check the status: + + ```bash + kubectl get all -n materialize-environment + ``` + + Wait for the components to be ready and in the `Running` state. + + ```none + NAME READY STATUS RESTARTS AGE + pod/db-demo-db-l6ss8 0/1 Completed 0 2m21s + pod/mz62lr3yltj8-balancerd-6d5dd6d4cf-r9nf4 1/1 Running 0 111s + pod/mz62lr3yltj8-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 114s + pod/mz62lr3yltj8-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 114s + pod/mz62lr3yltj8-console-bfc797745-6nlwv 1/1 Running 0 96s + pod/mz62lr3yltj8-console-bfc797745-tk9vm 1/1 Running 0 96s + pod/mz62lr3yltj8-environmentd-1-0 1/1 Running 0 2m4s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/mz62lr3yltj8-balancerd ClusterIP None 6876/TCP,6875/TCP 111s + service/mz62lr3yltj8-balancerd-lb LoadBalancer 10.1.201.77 192.0.2.10 6875:30890/TCP,6876:31750/TCP 2m4s + service/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s + service/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 114s + service/mz62lr3yltj8-console ClusterIP None 8080/TCP 96s + service/mz62lr3yltj8-console-lb LoadBalancer 10.1.130.212 192.0.2.254 8080:30379/TCP 2m4s + service/mz62lr3yltj8-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 111s + service/mz62lr3yltj8-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 2m5s + service/mz62lr3yltj8-persist-pubsub-1 ClusterIP None 6879/TCP 2m4s + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/mz62lr3yltj8-balancerd 1/1 1 1 111s + deployment.apps/mz62lr3yltj8-console 2/2 2 2 96s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/mz62lr3yltj8-balancerd-6d5dd6d4cf 1 1 1 111s + replicaset.apps/mz62lr3yltj8-console-bfc797745 2 2 2 96s + + NAME READY AGE + statefulset.apps/mz62lr3yltj8-cluster-s2-replica-s1-gen-1 1/1 114s + statefulset.apps/mz62lr3yltj8-cluster-u1-replica-u1-gen-1 1/1 114s + statefulset.apps/mz62lr3yltj8-environmentd-1 1/1 2m4s + + NAME STATUS COMPLETIONS DURATION AGE + job.batch/db-demo-db Complete 1/1 10s 2m21s + + ``` + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting/). + +1. Open the Materialize Console in your browser: + + + {{< tabs >}} + + {{< tab "Via Network Load Balancer" >}} + + Starting in v0.3.1, for each Materialize instance, Materialize on Azure + Terraform module also deploys load balancers (by default, internal) with the + following listeners, including a listener on port 8080 for the Materialize + Console: + + | Port | Description | + | ---- | ------------| + | 6875 | For SQL connections to the database | + | 6876 | For HTTP(S) connections to the database | + | **8080** | **For HTTP(S) connections to Materialize Console** | + + The load balancer details are found in the `load_balancer_details` in + the [Terraform output](#azure-terraform-output). + + The example uses a self-signed ClusterIssuer. As such, you may encounter a + warning with regards to the certificate. In production, run with certificates + from an official Certificate Authority (CA) rather than self-signed + certificates. + + {{}} + + {{< tab "Via port forwarding" >}} + + {{% self-managed/port-forwarding-handling %}} + + {{}} + {{}} + + {{< tip >}} + + {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} + + {{< /tip >}} + +## Next steps + +{{% self-managed/next-steps %}} + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + + {{< tip>}} + + If the `terraform destroy` command is unable to delete the subnet because it + is in use, you can rerun the `terraform destroy` command. + + {{}} + +## See also + +- [Materialize Operator Configuration](/installation/configuration/) +- [Troubleshooting](/installation/troubleshooting/) +- [Appendix: Azure deployment guidelines](/installation/install-on-azure/ + appendix-deployment-guidelines) +- [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-azure/upgrade-on-azure.md b/doc/user/content/installation/install-on-azure/legacy-terraform-module/upgrade.md similarity index 98% rename from doc/user/content/installation/install-on-azure/upgrade-on-azure.md rename to doc/user/content/installation/install-on-azure/legacy-terraform-module/upgrade.md index 763d5004888cc..c11593bfe4014 100644 --- a/doc/user/content/installation/install-on-azure/upgrade-on-azure.md +++ b/doc/user/content/installation/install-on-azure/legacy-terraform-module/upgrade.md @@ -1,9 +1,9 @@ --- -title: "Upgrade on Azure (Terraform)" +title: "Upgrade" description: "Procedure to upgrade your Materialize operator and instances running on Azure" menu: main: - parent: "install-on-azure" + parent: "install-on-azure-legacy-terraform-module" identifier: "upgrade-on-azure" weight: 10 --- diff --git a/doc/user/content/installation/install-on-azure/terraform-module/_index.md b/doc/user/content/installation/install-on-azure/terraform-module/_index.md new file mode 100644 index 0000000000000..ca72c1f6a607b --- /dev/null +++ b/doc/user/content/installation/install-on-azure/terraform-module/_index.md @@ -0,0 +1,154 @@ +--- +title: "Terraform Module" +description: "" +menu: + main: + parent: "install-on-azure" + identifier: "install-azure-terraform" + weight: 5 +--- + +Materialize provides a set of modular Terraform modules that can be used to +deploy all services required for a production ready Materialize database. +The module is intended to provide a simple set of examples on how to deploy +materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +The repository can be found at: + +***[Materialize Terraform Self-Managed Azure](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure)*** + +Please see the [top level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) and [cloud specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure) documentation for a full understanding +of the module structure and customizations. + +Also check out the [Azure deployment guide](/installation/install-on-azure/appendix-deployment-guidelines/) for details on recommended instance sizing and configuration. + +{{< note >}} +{{% self-managed/materialize-components-sentence %}} +{{< /note >}} + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + + +## Prerequisites + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [Azure Cli ](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) +- [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) + +#### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +--- + +# Example: Simple Materialize Deployment on Azure + +This example demonstrates how to deploy a complete Materialize environment on Azure using the modular Terraform setup from this repository. + + +## Setup +```shell +git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git +cd materialize-terraform-self-managed/azure/examples/simple +```` + +## What Gets Created + +This example provisions the following infrastructure: + +### Resource Group +- **Resource Group**: New resource group to contain all resources + +### Networking +- **Virtual Network**: 20.0.0.0/16 address space +- **AKS Subnet**: 20.0.0.0/20 with NAT Gateway association and service endpoints for Storage and SQL +- **PostgreSQL Subnet**: 20.0.16.0/24 delegated to PostgreSQL Flexible Server +- **NAT Gateway**: Standard SKU with static public IP for outbound connectivity +- **Private DNS Zone**: For PostgreSQL private endpoint resolution with VNet link + +### Compute +- **AKS Cluster**: Version 1.32 with Cilium networking (network plugin: azure, data plane: cilium, policy: cilium) +- **Default Node Pool**: Standard_D4pds_v6 VMs, autoscaling 2-5 nodes, labeled for generic workloads +- **Materialize Node Pool**: Standard_E4pds_v6 VMs with 100GB disk, autoscaling 2-5 nodes, swap enabled, dedicated taints for Materialize workloads +- **Managed Identities**: + - AKS cluster identity: Used by AKS control plane to provision Azure resources (creating load balancers when Materialize LoadBalancer services are created, managing network interfaces) + - Workload identity: Used by Materialize pods for secure, passwordless authentication to Azure Storage (no storage account keys stored in cluster) + +### Database +- **Azure PostgreSQL Flexible Server**: Version 15 +- **SKU**: GP_Standard_D2s_v3 (2 vCores, 4GB memory) +- **Storage**: 32GB with 7-day backup retention +- **Network Access**: Public Network Access is disabled, Private access only (no public endpoint) +- **Database**: `materialize` database pre-created + +### Storage +- **Storage Account**: Premium BlockBlobStorage with LRS replication for Materialize persistence +- **Container**: `materialize` blob container +- **Access Control**: Workload Identity federation for Kubernetes service account (passwordless authentication via OIDC) +- **Network Access**: Currently allows all traffic (production deployments should restrict to AKS subnet only traffic) + +### Kubernetes Add-ons +- **cert-manager**: Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal +- **Self-signed ClusterIssuer**: Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. + +### Materialize +- **Operator**: Materialize Kubernetes operator +- **Instance**: Single Materialize instance in `materialize-environment` namespace +- **Load Balancers**: Internal Azure Load Balancers for Materialize access + +--- + +## Getting Started + +### Step 1: Set Required Variables + +Before running Terraform, create a `terraform.tfvars` file with the following variables: + +```hcl +subscription_id = "12345678-1234-1234-1234-123456789012" +resource_group_name = "materialize-demo-rg" +name_prefix = "simple-demo" +location = "westus2" +license_key = "your-materialize-license-key" # Optional: Get from https://materialize.com/self-managed/ +tags = { + environment = "demo" +} +``` + +**Required Variables:** +- `subscription_id`: Azure subscription ID +- `resource_group_name`: Name for the resource group (will be created) +- `name_prefix`: Prefix for all resource names +- `location`: Azure region for deployment +- `tags`: Map of tags to apply to resources +- `license_key`: Materialize license key + +--- + +### Step 2: Deploy Materialize + +Run the usual Terraform workflow: + +```bash +terraform init +terraform apply +``` + +## Notes + +*Autoscaling: Uses Azure's native cluster autoscaler that integrates directly with Azure Virtual Machine Scale Sets for automated node scaling. In future we are planning to enhance this by making use of karpenter-provider-azure* + +* You can customize each module independently. +* To reduce cost in your demo environment, you can tweak VM sizes and database tiers in `main.tf`. + +***Don't forget to destroy resources when finished:*** + +```bash +terraform destroy +``` diff --git a/doc/user/content/installation/install-on-gcp/_index.md b/doc/user/content/installation/install-on-gcp/_index.md index b6c9117b1f6e8..c62a95c53eec1 100644 --- a/doc/user/content/installation/install-on-gcp/_index.md +++ b/doc/user/content/installation/install-on-gcp/_index.md @@ -1,6 +1,6 @@ --- -title: "Install on GCP (via Terraform)" -description: "" +title: "Install on GCP" +description: "Install and upgrade Materialize on GCP" aliases: - /self-hosted/install-on-gcp/ - /self-managed/v25.1/installation/install-on-gcp/ @@ -14,577 +14,14 @@ menu: {{% self-managed/materialize-components-sentence %}} -This tutorial deploys Materialize to GCP Google Kubernetes Engine (GKE) cluster -with a Cloud SQL PostgreSQL database as the metadata database and Cloud Storage -bucket for blob storage. Specifically, the tutorial uses [Materialize on Google -Cloud Provider Terraform -module](https://github.com/MaterializeInc/terraform-google-materialize) to: - -- Set up the GCP environment. - -- Call - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) - module to deploy Materialize Operator and Materialize instances to the GKE - cluster. - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< self-managed/tutorial-disclaimer >}} - -{{< /warning >}} - -## Prerequisites - -### Google cloud provider project - -You need a GCP project for which you have a role (such as -`roles/resourcemanager.projectIamAdmin` or `roles/owner`) that includes [ -permissions to manage access to the -project](https://cloud.google.com/iam/docs/granting-changing-revoking-access). - -### gcloud CLI - -If you do not have gcloud CLI, install. For details, see the [Install the gcloud -CLI documentation](https://cloud.google.com/sdk/docs/install). - -### Google service account - -The tutorial assumes the use of a service account. If you do not have a service -account to use for this tutorial, create a service account. For details, see -[Create service -accounts](https://cloud.google.com/iam/docs/service-accounts-create#creating). - -### Terraform - -If you do not have Terraform installed, [install -Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). - -### kubectl and plugins - -{{< tip >}} - -Using `gcloud` to install `kubectl` will also install the needed plugins. -Otherwise, you will need to manually install the `gke-gcloud-auth-plugin` for -`kubectl`. - -{{< /tip >}} - -- If you do not have `kubectl`, install `kubectl`. To install, see [Install - kubectl and configure cluster - access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) - for details. You will configure `kubectl` to interact with your GKE cluster - later in the tutorial. - -- If you do not have `gke-gcloud-auth-plugin` for `kubectl`, install the - `gke-gcloud-auth-plugin`. For details, see [Install the - gke-gcloud-auth-plugin](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin). - -### Helm 3.2.0+ - -If you do not have Helm version 3.2.0+ installed, install. For details, see the -[Helm documentation](https://helm.sh/docs/intro/install/). - -### jq (Optional) - -*Optional*. `jq` is used to parse the GKE cluster name and region from the -Terraform outputs. Alternatively, you can manually specify the name and region. -If you want to use `jq` and do not have `jq` installed, install. - -### License key - -Starting in v26.0, Self-Managed Materialize requires a license key. - -{{< yaml-table data="self_managed/license_key" >}} - -## A. Configure GCP project and service account - -1. Open a Terminal window. - -1. Initialize the gcloud CLI (`gcloud init`) to specify the GCP project you want - to use. For details, see the [Initializing the gcloud CLI - documentation](https://cloud.google.com/sdk/docs/initializing#initialize_the). - - {{< tip >}} - You do not need to configure a default Compute Region and Zone as you will - specify the region. - {{}} - -1. Enable the following services for your GCP project, if not already enabled: - - ```bash - gcloud services enable container.googleapis.com # For creating Kubernetes clusters - gcloud services enable sqladmin.googleapis.com # For creating databases - gcloud services enable cloudresourcemanager.googleapis.com # For managing GCP resources - gcloud services enable servicenetworking.googleapis.com # For private network connections - gcloud services enable iamcredentials.googleapis.com # For security and authentication - ``` - -1. To the service account that will run the Terraform script, - grant the following IAM roles: - - - `roles/editor` - - `roles/iam.serviceAccountAdmin` - - `roles/servicenetworking.networksAdmin` - - `roles/storage.admin` - - `roles/container.admin` - - 1. Enter your GCP project ID. - - ```bash - read -s PROJECT_ID - ``` - - 1. Find your service account email for your GCP project - - ```bash - gcloud iam service-accounts list --project $PROJECT_ID - ``` - - 1. Enter your service account email. - - ```bash - read -s SERVICE_ACCOUNT - ``` - - 1. Grant the service account the neccessary IAM roles. - - ```bash - gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SERVICE_ACCOUNT" \ - --role="roles/editor" - - gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SERVICE_ACCOUNT" \ - --role="roles/iam.serviceAccountAdmin" - - gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SERVICE_ACCOUNT" \ - --role="roles/servicenetworking.networksAdmin" - - gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SERVICE_ACCOUNT" \ - --role="roles/storage.admin" - - gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SERVICE_ACCOUNT" \ - --role="roles/container.admin" - ``` - -1. For the service account, authenticate to allow Terraform to - interact with your GCP project. For details, see [Terraform: Google Cloud - Provider Configuration - reference](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication). - - For example, if using [User Application Default - Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default), - you can run the following command: - - ```bash - gcloud auth application-default login - ``` - - {{< tip >}} - If using `GOOGLE_APPLICATION_CREDENTIALS`, use absolute path to your key file. - {{}} - -## B. Set up GCP Kubernetes environment and install Materialize - -{{< warning >}} - -{{< self-managed/terraform-disclaimer >}} - -{{< /warning >}} - -{{< tabs >}} - -{{< tab "Deployed components" >}} -[Materialize on GCP Terraform -module](https://github.com/MaterializeInc/terraform-google-materialize) deploys -a sample infrastructure on GCP (region `us-central1`) with the following -components: - -{{< yaml-table data="self_managed/gcp_terraform_deployed_components" >}} - -{{< tip >}} -{{< self-managed/gcp-terraform-configs >}} -{{< /tip >}} -{{}} -{{< tab "Releases" >}} - -{{< yaml-table data="self_managed/gcp_terraform_versions" >}} - -{{}} -{{}} - -{{% self-managed/versions/step-clone-google-terraform-repo %}} - -1. Go to the `examples/simple` folder in the Materialize Terraform repo - directory. - - ```bash - cd terraform-google-materialize/examples/simple - ``` - - {{< tip >}} - {{< self-managed/gcp-terraform-configs >}} - {{< /tip >}} - -1. Create a `terraform.tfvars` file (you can copy from the - `terraform.tfvars.example` file) and specify the following variables: - - | **Variable** | **Description** | - |--------------|-----------------| - | `project_id` | Your GCP project ID. | - | `prefix` | A prefix (e.g., `mz-simple`) for your resources. Prefix has a maximum of 15 characters and contains only alphanumeric characters and dashes. | - | `region` | The region for the GKE cluster. | - - ```bash - project_id = "enter-your-gcp-project-id" - prefix = "enter-your-prefix" // Maximum of 15 characters, contain lowercase alphanumeric and hyphens only (e.g., mz-simple) - region = "us-central1" - ``` - - {{< tip >}} - - {{< self-managed/gcp-terraform-configs >}} - - {{< /tip >}} - -1. Initialize the terraform directory. - - ```bash - terraform init - ``` - -1. Run terraform plan and review the changes to be made. - - ```bash - terraform plan - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply - ``` - - To approve the changes and apply, enter `yes`. - - Upon successful completion, various fields and their values are output: - - ```bash - Apply complete! Resources: 27 added, 0 changed, 0 destroyed. - - Outputs: - - connection_strings = - gke_cluster = - load_balancer_details = {} - network = { - "network_id" = "projects/my-project/global/networks/mz-simple-network" - "network_name" = "mz-simple-network" - "subnet_name" = "mz-simple-subnet" - } - service_accounts = { - "gke_sa" = "mz-simple-gke-sa@my-project.iam.gserviceaccount.com" - "materialize_sa" = "mz-simple-materialize-sa@my-project.iam.gserviceaccount.com" - } - ``` - -1. Configure `kubectl` to connect to your GKE cluster, specifying: - - - ``. Your cluster name has the form `-gke`; e.g., - `mz-simple-gke`. - - - ``. By default, the example Terraform module uses the `us-central1` - region. - - - ``. Your GCP project ID. - - ```bash - gcloud container clusters get-credentials \ - --region \ - --project - ``` - - Alternatively, you can use the following command to get the cluster name and - region from the Terraform output and the project ID from the environment - variable set earlier. - - ```bash - gcloud container clusters get-credentials $(terraform output -json gke_cluster | jq -r .name) \ - --region $(terraform output -json gke_cluster | jq -r .location) --project $PROJECT_ID - ``` - - To verify that you have configured correctly, run the following command: - - ```bash - kubectl cluster-info - ``` - - For help with `kubectl` commands, see [kubectl Quick - reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). - -1. By default, the example Terraform installs the Materialize Operator and, - starting in v0.3.0, a `cert-manager`. Verify the - installation and check the status: - - {{< tabs >}} - {{< tab "Materialize Operator" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n materialize - ``` - - Wait for the components to be in the `Running` state: - - ```none - NAME READY STATUS RESTARTS AGE - pod/materialize-mz-simple-materialize-operator-74d8f549d6-lkjjf 1/1 Running 0 36m - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/materialize-mz-simple-materialize-operator 1/1 1 1 36m - - NAME DESIRED CURRENT READY AGE - replicaset.apps/materialize-mz-simple-materialize-operator-74d8f549d6 1 1 1 36m - ``` - - {{}} - {{< tab "cert-manager (Starting in version 0.3.0)" >}} - - Verify the installation and check the status: - - ```shell - kubectl get all -n cert-manager - ``` - Wait for the components to be in the `Running` state: - ``` - NAME READY STATUS RESTARTS AGE - pod/cert-manager-6794b8d569-vt264 1/1 Running 0 22m - pod/cert-manager-cainjector-7f69cd69f7-7brqw 1/1 Running 0 22m - pod/cert-manager-webhook-6cc5dccc4b-7tmd4 1/1 Running 0 22m - - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/cert-manager ClusterIP 10.52.3.63 9402/TCP 22m - service/cert-manager-cainjector ClusterIP 10.52.15.171 9402/TCP 22m - service/cert-manager-webhook ClusterIP 10.52.5.148 443/TCP,9402/TCP 22m - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/cert-manager 1/1 1 1 22m - deployment.apps/cert-manager-cainjector 1/1 1 1 22m - deployment.apps/cert-manager-webhook 1/1 1 1 22m - - NAME DESIRED CURRENT READY AGE - replicaset.apps/cert-manager-6794b8d569 1 1 1 22m - replicaset.apps/cert-manager-cainjector-7f69cd69f7 1 1 1 22m - replicaset.apps/cert-manager-webhook-6cc5dccc4b 1 1 1 22m - ``` - - {{}} - {{}} - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). - -1. Once the Materialize operator is deployed and running, you can deploy the - Materialize instances. To deploy Materialize instances, create a - `mz_instances.tfvars` file with the Materialize instance configuration. - - For example, the following specifies the configuration for a `demo` instance. - - ```bash - cat < mz_instances.tfvars - - materialize_instances = [ - { - name = "demo" - namespace = "materialize-environment" - database_name = "demo_db" - cpu_request = "1" - memory_request = "2Gi" - memory_limit = "2Gi" - license_key = "" - } - ] - EOF - ``` - - - **Starting in v26.0**, Self-Managed Materialize requires a license key. To - get your license key: - {{% yaml-table data="self_managed/license_key" %}} - - - **Starting in v0.3.0**, the Materialize on GCP Terraform module also - deploys, by default: - - - [Load balancers](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) for Materialize instances (i.e., the [`create_load_balancer`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) flag defaults to `true`). The load balancers, by default, are configured to be internal (i.e., the [`internal_load_balancer`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) flag defaults to `true`). - - - A self-signed `ClusterIssuer`. The `ClusterIssuer` is deployed after the - `cert-manager` is deployed and running. - - - **Starting in v0.4.3**, you can specify addition configuration options via - `environmentd_extra_args`. - - {{< tip >}} - {{% self-managed/gcp-terraform-upgrade-notes %}} - - See [Materialize on GCP releases](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) for notable changes. - {{}} - -1. Run `terraform plan` with both `.tfvars` files and review the changes to be - made. - - ```bash - terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - The plan should show the changes to be made, with a summary similar to the - following: - - ``` - Plan: 9 to add, 1 to change, 0 to destroy. - ``` - -1. If you are satisfied with the changes, apply. - - ```bash - terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars - ``` - - To approve the changes and apply, enter `yes`. - - - Upon successful completion, you should see output with a summary similar to the following: - - ```bash - Apply complete! Resources: 9 added, 1 changed, 0 destroyed. - - Outputs: - - connection_strings = - gke_cluster = - load_balancer_details = { - "demo" = { - "balancerd_load_balancer_ip" = "192.0.2.10" - "console_load_balancer_ip" = "192.0.2.254" - } - } - network = { - "network_id" = "projects/my-project/global/networks/mz-simple-network" - "network_name" = "mz-simple-network" - "subnet_name" = "mz-simple-subnet" - } - service_accounts = { - "gke_sa" = "mz-simple-gke-sa@my-project.iam.gserviceaccount.com" - "materialize_sa" = "mz-simple-materialize-sa@my-project.iam.gserviceaccount.com" - } - ``` - -1. Verify the installation and check the status: - - ```bash - kubectl get all -n materialize-environment - ``` - - Wait for the components to be in the `Running` state. - - ```none - NAME READY STATUS RESTARTS AGE - pod/db-demo-db-wrvhw 0/1 Completed 0 4m26s - pod/mzdtwvu4qe4q-balancerd-6989df5c75-mpmqx 1/1 Running 0 3m54s - pod/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 3m53s - pod/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 3m52s - pod/mzdtwvu4qe4q-console-7c9bc94bcb-6t7lg 1/1 Running 0 3m41s - pod/mzdtwvu4qe4q-console-7c9bc94bcb-9x5qq 1/1 Running 0 3m41s - pod/mzdtwvu4qe4q-environmentd-1-0 1/1 Running 0 4m9s - - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/mzdtwvu4qe4q-balancerd ClusterIP None 6876/TCP,6875/TCP 3m54s - service/mzdtwvu4qe4q-balancerd-lb LoadBalancer 10.52.5.105 192.0.2.10 6875:30844/TCP,6876:32307/TCP 4m9s - service/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 3m53s - service/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 3m52s - service/mzdtwvu4qe4q-console ClusterIP None 8080/TCP 3m41s - service/mzdtwvu4qe4q-console-lb LoadBalancer 10.52.4.2 192.0.2.254 8080:32193/TCP 4m9s - service/mzdtwvu4qe4q-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 3m54s - service/mzdtwvu4qe4q-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 4m9s - service/mzdtwvu4qe4q-persist-pubsub-1 ClusterIP None 6879/TCP 4m9s - - NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/mzdtwvu4qe4q-balancerd 1/1 1 1 3m54s - deployment.apps/mzdtwvu4qe4q-console 2/2 2 2 3m41s - - NAME DESIRED CURRENT READY AGE - replicaset.apps/mzdtwvu4qe4q-balancerd-6989df5c75 1 1 1 3m54s - replicaset.apps/mzdtwvu4qe4q-console-7c9bc94bcb 2 2 2 3m41s - - NAME READY AGE - statefulset.apps/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1 1/1 3m53s - statefulset.apps/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1 1/1 3m52s - statefulset.apps/mzdtwvu4qe4q-environmentd-1 1/1 4m9s - - NAME STATUS COMPLETIONS DURATION AGE - job.batch/db-demo-db Complete 1/1 12s 4m27s - - ``` - - If you run into an error during deployment, refer to the - [Troubleshooting](/installation/troubleshooting/). - -1. Open the Materialize Console in your browser: - - {{< tabs >}} - - {{< tab "Via Network Load Balancer" >}} - - Starting in v0.3.0, for each Materialize instance, Materialize on GCP - Terraform module also deploys load balancers (by default, internal) with the - following listeners, including a listener on port 8080 for the Materialize - Console: - - | Port | Description | - | ---- | ------------| - | 6875 | For SQL connections to the database | - | 6876 | For HTTP(S) connections to the database | - | **8080** | **For HTTP(S) connections to Materialize Console** | - - The load balancer details are found in the `load_balancer_details` in - the [Terraform output](#gcp-terraform-output). - - The example uses a self-signed ClusterIssuer. As such, you may encounter a - warning with regards to the certificate. In production, run with certificates - from an official Certificate Authority (CA) rather than self-signed - certificates. - - {{}} - - {{< tab "Via port forwarding" >}} - - {{% self-managed/port-forwarding-handling console="console-lb"%}} - - {{}} - {{}} - - - {{< tip >}} - - {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} - - {{< /tip >}} - -## Next steps - -{{% self-managed/next-steps %}} - -## Cleanup - -{{% self-managed/cleanup-cloud %}} +| Guide | Description | +|-------|-------------| +| [Terraform Provider](/installation/install-on-gcp/terraform-module/) | Install Materialize on GCP using our new Unified Terraform Provider | +| [Terraform Provider (legacy)](/installation/install-on-gcp/legacy-terraform-module/) | Install Materialize on GCP using our Terraform Provider (legacy) | +| [Appendix: Google deployment guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/) | Additional guidelines for Google Cloud deployments | ## See also -- [Troubleshooting](/installation/troubleshooting/) - [Materialize Operator Configuration](/installation/configuration/) -- [Appendix: Google deployment guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/) +- [Troubleshooting](/installation/troubleshooting/) - [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md b/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md index e2a051a92772c..69eae78bce5cf 100644 --- a/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md +++ b/doc/user/content/installation/install-on-gcp/appendix-deployment-guidelines.md @@ -12,9 +12,9 @@ menu: As a general guideline, we recommend: -- Processor Type: ARM-based CPU - -- Sizing: 2:1 disk-to-RAM ratio with spill-to-disk enabled. +- ARM-based CPU +- A 1:8 ratio of vCPU to GiB memory is recommended. +- When using swap, it is recommended to use a 8:1 ratio of GiB local instance storage to GiB Ram. When operating on GCP in production, we recommend the following machine types that support local SSD attachment: @@ -24,20 +24,16 @@ that support local SSD attachment: | [N2 high-memory series] | `n2-highmem-16` or `n2-highmem-32` with local NVMe SSDs | | [N2D high-memory series] | `n2d-highmem-16` or `n2d-highmem-32` with local NVMe SSDs | -To maintain the recommended 2:1 disk-to-RAM ratio for your machine type, see +To maintain the recommended 8:1 disk-to-RAM ratio for your machine type, see [Number of local SSDs](#number-of-local-ssds) to determine the number of local -SSDs -([`disk_support_config.local_ssd_count`](https://github.com/MaterializeInc/terraform-google-materialize/blob/main/README.md#input_disk_support_config)) -to use. +SSDs to use. See also [Locally attached NVMe storage](#locally-attached-nvme-storage). ## Number of local SSDs Each local NVMe SSD in GCP provides 375GB of storage. Use the appropriate number -of local SSDs -([`disk_support_config.local_ssd_count`](https://github.com/MaterializeInc/terraform-google-materialize/blob/main/README.md#input_disk_support_config)) -to ensure your total disk space is at least twice the amount of RAM in your +of local SSDs to ensure your total disk space is at least twice the amount of RAM in your machine type for optimal Materialize performance. {{< note >}} @@ -65,32 +61,36 @@ type](https://cloud.google.com/compute/docs/disks/local-ssd#lssd_disk_options). [N2D high-memory series]: https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machine_types -[enables spill-to-disk]: https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#disk-support-for-materialize-on-gcp ## Locally-attached NVMe storage -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. +Configuring swap on nodes to using locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. ### Swap support -Starting in v0.6.1 of Materialize on Google Cloud PRovider (GCP) Terraform, -disk support (using swap on NVMe instance storage) may be enabled for -Materialize. With this change, the Terraform: +***New Unified Terraform*** -- Creates a node group for Materialize. -- Configures NVMe instance store volumes as swap using a daemonset. -- Enables swap at the Kubelet. +The unified Materialize [Terraform module](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp/examples/simple) supports configuring swap out of the box. -For swap support, the following configuration options are available: +***Legacy Terraform*** -- [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) +The Legacy Terraform provider, adds preliminary swap support in v0.6.1, via the [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) variable. +With this change, the Terraform: + - Creates a node group for Materialize. + - Configures NVMe instance store volumes as swap using a daemonset. + - Enables swap at the Kubelet. See [Upgrade Notes](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#v061). +{{< note >}} +If deploying `v25.2` Materialize clusters will not automatically use swap unless they are configured with a `memory_request` less than their `memory_limit`. In `v26` this will be handled automatically. +{{< /note >}} + + ## CPU affinity It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). @@ -102,15 +102,6 @@ to substantially improve the performance of compute-bound workloads. When running with TLS in production, run with certificates from an official Certificate Authority (CA) rather than self-signed certificates. -## Storage bucket versioning - -Starting in v0.3.1 of Materialize on GCP Terraform, storage bucket versioning is -disabled (i.e., -[`storage_bucket_versioning`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_versioning) -is set to `false` by default) to facilitate cleanup of resources during testing. -When running in production, versioning should be turned on with a sufficient TTL -([`storage_bucket_version_ttl`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_version_ttl)) -to meet any data-recovery requirements. ## See also diff --git a/doc/user/content/installation/install-on-gcp/legacy-terraform-module/_index.md b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/_index.md new file mode 100644 index 0000000000000..04a23c89bb532 --- /dev/null +++ b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/_index.md @@ -0,0 +1,24 @@ +--- +title: "Terraform Module (legacy)" +description: "" +disable_list: true +disable_toc: true +menu: + main: + parent: "install-on-gcp" + identifier: "install-on-gcp-legacy-terraform-module" + weight: 5 + + +--- + +The tutorials in this section show you how to deploy Materialize using the [Materialize on Google +Cloud Module Legacy Terraform +module](https://github.com/MaterializeInc/terraform-google-materialize). + + +| Guide | Description | +|-------|-------------| +| [Install](/installation/install-on-gcp/legacy-terraform-module/install/) | Install Materialize on GCP | +| [Upgrade](/installation/install-on-gcp/legacy-terraform-module/upgrade/) | Upgrade your Materialize deployment on GCP | +| [Appendix: GCP configuration](/installation/install-on-gcp/legacy-terraform-module/appendix-configuration/) | Configuration for Google Cloud deployments | diff --git a/doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/appendix-configuration.md similarity index 52% rename from doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md rename to doc/user/content/installation/install-on-gcp/legacy-terraform-module/appendix-configuration.md index 1522237789e44..288ac69839a61 100644 --- a/doc/user/content/installation/install-on-gcp/appendix-gcp-configuration.md +++ b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/appendix-configuration.md @@ -1,13 +1,13 @@ --- -title: "Appendix: Required configuration" -description: "Required configuration for Materialize on GCP Terraform." +title: "Appendix: Configuration" +description: "Required configuration for Materialize on GCP Terraform (legacy)." menu: main: - parent: "install-on-gcp" - identifier: "appendix-gcp-config" + parent: "install-on-gcp-legacy-terraform-module" + identifier: "legacy-terraform-module-appendix-configuration" weight: 50 aliases: - - /installation/install-on-gcp/appendix-gcp-provider-configuration + - /installation/install-on-gcp/appendix-gcp-configuration/ --- ## Required variables @@ -58,3 +58,29 @@ you need to declare: ```hcl data "google_client_config" "current" {} ``` + +## Swap support + +Starting in v0.6.1 of Materialize on Google Cloud Provider (GCP) Terraform, +disk support (using swap on NVMe instance storage) may be enabled for +Materialize. With this change, the Terraform: + +- Creates a node group for Materialize. +- Configures NVMe instance store volumes as swap using a daemonset. +- Enables swap at the Kubelet. + +For swap support, the following configuration options are available: + +- [`swap_enabled`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_swap_enabled) + +See [Upgrade Notes](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#v061). + +## Storage bucket versioning + +Starting in v0.3.1 of Materialize on GCP Terraform, storage bucket versioning is +disabled (i.e., +[`storage_bucket_versioning`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_versioning) +is set to `false` by default) to facilitate cleanup of resources during testing. +When running in production, versioning should be turned on with a sufficient TTL +([`storage_bucket_version_ttl`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_storage_bucket_version_ttl)) +to meet any data-recovery requirements. diff --git a/doc/user/content/installation/install-on-gcp/legacy-terraform-module/install.md b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/install.md new file mode 100644 index 0000000000000..0eec79daba6ab --- /dev/null +++ b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/install.md @@ -0,0 +1,583 @@ +--- +title: "Install" +description: "" +aliases: + - /self-hosted/install-on-gcp/ + - /installation/install-on-gcp/ +menu: + main: + parent: "install-on-gcp-legacy-terraform-module" + identifier: "legacy-terraform-module-install" + weight: 5 +--- + +{{% self-managed/materialize-components-sentence %}} + +This tutorial deploys Materialize to GCP Google Kubernetes Engine (GKE) cluster +with a Cloud SQL PostgreSQL database as the metadata database and Cloud Storage +bucket for blob storage. Specifically, the tutorial uses [Materialize on Google +Cloud Provider Terraform +module](https://github.com/MaterializeInc/terraform-google-materialize) to: + +- Set up the GCP environment. + +- Call + [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) + module to deploy Materialize Operator and Materialize instances to the GKE + cluster. + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< self-managed/tutorial-disclaimer >}} + +{{< /warning >}} + +## Prerequisites + +### Google cloud provider project + +You need a GCP project for which you have a role (such as +`roles/resourcemanager.projectIamAdmin` or `roles/owner`) that includes [ +permissions to manage access to the +project](https://cloud.google.com/iam/docs/granting-changing-revoking-access). + +### gcloud CLI + +If you do not have gcloud CLI, install. For details, see the [Install the gcloud +CLI documentation](https://cloud.google.com/sdk/docs/install). + +### Google service account + +The tutorial assumes the use of a service account. If you do not have a service +account to use for this tutorial, create a service account. For details, see +[Create service +accounts](https://cloud.google.com/iam/docs/service-accounts-create#creating). + +### Terraform + +If you do not have Terraform installed, [install +Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform). + +### kubectl and plugins + +{{< tip >}} + +Using `gcloud` to install `kubectl` will also install the needed plugins. +Otherwise, you will need to manually install the `gke-gcloud-auth-plugin` for +`kubectl`. + +{{< /tip >}} + +- If you do not have `kubectl`, install `kubectl`. To install, see [Install + kubectl and configure cluster + access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) + for details. You will configure `kubectl` to interact with your GKE cluster + later in the tutorial. + +- If you do not have `gke-gcloud-auth-plugin` for `kubectl`, install the + `gke-gcloud-auth-plugin`. For details, see [Install the + gke-gcloud-auth-plugin](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin). + +### Helm 3.2.0+ + +If you do not have Helm version 3.2.0+ installed, install. For details, see the +[Helm documentation](https://helm.sh/docs/intro/install/). + +### jq (Optional) + +*Optional*. `jq` is used to parse the GKE cluster name and region from the +Terraform outputs. Alternatively, you can manually specify the name and region. +If you want to use `jq` and do not have `jq` installed, install. + +### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +## A. Configure GCP project and service account + +1. Open a Terminal window. + +1. Initialize the gcloud CLI (`gcloud init`) to specify the GCP project you want + to use. For details, see the [Initializing the gcloud CLI + documentation](https://cloud.google.com/sdk/docs/initializing#initialize_the). + + {{< tip >}} + You do not need to configure a default Compute Region and Zone as you will + specify the region. + {{}} + +1. Enable the following services for your GCP project, if not already enabled: + + ```bash + gcloud services enable container.googleapis.com # For creating Kubernetes clusters + gcloud services enable sqladmin.googleapis.com # For creating databases + gcloud services enable cloudresourcemanager.googleapis.com # For managing GCP resources + gcloud services enable servicenetworking.googleapis.com # For private network connections + gcloud services enable iamcredentials.googleapis.com # For security and authentication + ``` + +1. To the service account that will run the Terraform script, + grant the following IAM roles: + + - `roles/editor` + - `roles/iam.serviceAccountAdmin` + - `roles/servicenetworking.networksAdmin` + - `roles/storage.admin` + - `roles/container.admin` + + 1. Enter your GCP project ID. + + ```bash + read -s PROJECT_ID + ``` + + 1. Find your service account email for your GCP project + + ```bash + gcloud iam service-accounts list --project $PROJECT_ID + ``` + + 1. Enter your service account email. + + ```bash + read -s SERVICE_ACCOUNT + ``` + + 1. Grant the service account the neccessary IAM roles. + + ```bash + gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/editor" + + gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/iam.serviceAccountAdmin" + + gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/servicenetworking.networksAdmin" + + gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/storage.admin" + + gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/container.admin" + ``` + +1. For the service account, authenticate to allow Terraform to + interact with your GCP project. For details, see [Terraform: Google Cloud + Provider Configuration + reference](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication). + + For example, if using [User Application Default + Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default), + you can run the following command: + + ```bash + gcloud auth application-default login + ``` + + {{< tip >}} + If using `GOOGLE_APPLICATION_CREDENTIALS`, use absolute path to your key file. + {{}} + +## B. Set up GCP Kubernetes environment and install Materialize + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + +{{< tabs >}} + +{{< tab "Deployed components" >}} +[Materialize on GCP Terraform +module](https://github.com/MaterializeInc/terraform-google-materialize) deploys +a sample infrastructure on GCP (region `us-central1`) with the following +components: + +{{< yaml-table data="self_managed/gcp_terraform_deployed_components" >}} + +{{< tip >}} +{{< self-managed/gcp-terraform-configs >}} +{{< /tip >}} +{{}} +{{< tab "Releases" >}} + +{{< yaml-table data="self_managed/gcp_terraform_versions" >}} + +{{}} +{{}} + +{{% self-managed/versions/step-clone-google-terraform-repo %}} + +1. Go to the `examples/simple` folder in the Materialize Terraform repo + directory. + + ```bash + cd terraform-google-materialize/examples/simple + ``` + + {{< tip >}} + {{< self-managed/gcp-terraform-configs >}} + {{< /tip >}} + +1. Create a `terraform.tfvars` file (you can copy from the + `terraform.tfvars.example` file) and specify the following variables: + + | **Variable** | **Description** | + |--------------|-----------------| + | `project_id` | Your GCP project ID. | + | `prefix` | A prefix (e.g., `mz-simple`) for your resources. Prefix has a maximum of 15 characters and contains only alphanumeric characters and dashes. | + | `region` | The region for the GKE cluster. | + + ```bash + project_id = "enter-your-gcp-project-id" + prefix = "enter-your-prefix" // Maximum of 15 characters, contain lowercase alphanumeric and hyphens only (e.g., mz-simple) + region = "us-central1" + ``` + + {{< tip >}} + + {{< self-managed/gcp-terraform-configs >}} + + {{< /tip >}} + +1. Initialize the terraform directory. + + ```bash + terraform init + ``` + +1. Run terraform plan and review the changes to be made. + + ```bash + terraform plan + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply + ``` + + To approve the changes and apply, enter `yes`. + + Upon successful completion, various fields and their values are output: + + ```bash + Apply complete! Resources: 27 added, 0 changed, 0 destroyed. + + Outputs: + + connection_strings = + gke_cluster = + load_balancer_details = {} + network = { + "network_id" = "projects/my-project/global/networks/mz-simple-network" + "network_name" = "mz-simple-network" + "subnet_name" = "mz-simple-subnet" + } + service_accounts = { + "gke_sa" = "mz-simple-gke-sa@my-project.iam.gserviceaccount.com" + "materialize_sa" = "mz-simple-materialize-sa@my-project.iam.gserviceaccount.com" + } + ``` + +1. Configure `kubectl` to connect to your GKE cluster, specifying: + + - ``. Your cluster name has the form `-gke`; e.g., + `mz-simple-gke`. + + - ``. By default, the example Terraform module uses the `us-central1` + region. + + - ``. Your GCP project ID. + + ```bash + gcloud container clusters get-credentials \ + --region \ + --project + ``` + + Alternatively, you can use the following command to get the cluster name and + region from the Terraform output and the project ID from the environment + variable set earlier. + + ```bash + gcloud container clusters get-credentials $(terraform output -json gke_cluster | jq -r .name) \ + --region $(terraform output -json gke_cluster | jq -r .location) --project $PROJECT_ID + ``` + + To verify that you have configured correctly, run the following command: + + ```bash + kubectl cluster-info + ``` + + For help with `kubectl` commands, see [kubectl Quick + reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/). + +1. By default, the example Terraform installs the Materialize Operator and, + starting in v0.3.0, a `cert-manager`. Verify the + installation and check the status: + + {{< tabs >}} + {{< tab "Materialize Operator" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n materialize + ``` + + Wait for the components to be in the `Running` state: + + ```none + NAME READY STATUS RESTARTS AGE + pod/materialize-mz-simple-materialize-operator-74d8f549d6-lkjjf 1/1 Running 0 36m + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/materialize-mz-simple-materialize-operator 1/1 1 1 36m + + NAME DESIRED CURRENT READY AGE + replicaset.apps/materialize-mz-simple-materialize-operator-74d8f549d6 1 1 1 36m + ``` + + {{}} + {{< tab "cert-manager (Starting in version 0.3.0)" >}} + + Verify the installation and check the status: + + ```shell + kubectl get all -n cert-manager + ``` + Wait for the components to be in the `Running` state: + ``` + NAME READY STATUS RESTARTS AGE + pod/cert-manager-6794b8d569-vt264 1/1 Running 0 22m + pod/cert-manager-cainjector-7f69cd69f7-7brqw 1/1 Running 0 22m + pod/cert-manager-webhook-6cc5dccc4b-7tmd4 1/1 Running 0 22m + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/cert-manager ClusterIP 10.52.3.63 9402/TCP 22m + service/cert-manager-cainjector ClusterIP 10.52.15.171 9402/TCP 22m + service/cert-manager-webhook ClusterIP 10.52.5.148 443/TCP,9402/TCP 22m + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/cert-manager 1/1 1 1 22m + deployment.apps/cert-manager-cainjector 1/1 1 1 22m + deployment.apps/cert-manager-webhook 1/1 1 1 22m + + NAME DESIRED CURRENT READY AGE + replicaset.apps/cert-manager-6794b8d569 1 1 1 22m + replicaset.apps/cert-manager-cainjector-7f69cd69f7 1 1 1 22m + replicaset.apps/cert-manager-webhook-6cc5dccc4b 1 1 1 22m + ``` + + {{}} + {{}} + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting/). + +1. Once the Materialize operator is deployed and running, you can deploy the + Materialize instances. To deploy Materialize instances, create a + `mz_instances.tfvars` file with the Materialize instance configuration. + + For example, the following specifies the configuration for a `demo` instance. + + ```bash + cat < mz_instances.tfvars + + materialize_instances = [ + { + name = "demo" + namespace = "materialize-environment" + database_name = "demo_db" + cpu_request = "1" + memory_request = "2Gi" + memory_limit = "2Gi" + license_key = "" + } + ] + EOF + ``` + + - **Starting in v0.3.0**, the Materialize on GCP Terraform module also + deploys, by default: + + - [Load balancers](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) for Materialize instances (i.e., the [`create_load_balancer`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) flag defaults to `true`). The load balancers, by default, are configured to be internal (i.e., the [`internal_load_balancer`](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#input_materialize_instances) flag defaults to `true`). + + - A self-signed `ClusterIssuer`. The `ClusterIssuer` is deployed after the + `cert-manager` is deployed and running. + + - **Starting in v0.4.3**, you can specify addition configuration options via + `environmentd_extra_args`. + + {{< tip >}} + {{% self-managed/gcp-terraform-upgrade-notes %}} + + See [Materialize on GCP releases](/installation/appendix-terraforms/#materialize-on-gcp-terraform-module) for notable changes. + {{}} + +1. Run `terraform plan` with both `.tfvars` files and review the changes to be + made. + + ```bash + terraform plan -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + The plan should show the changes to be made, with a summary similar to the + following: + + ``` + Plan: 9 to add, 1 to change, 0 to destroy. + ``` + +1. If you are satisfied with the changes, apply. + + ```bash + terraform apply -var-file=terraform.tfvars -var-file=mz_instances.tfvars + ``` + + To approve the changes and apply, enter `yes`. + + + Upon successful completion, you should see output with a summary similar to the following: + + ```bash + Apply complete! Resources: 9 added, 1 changed, 0 destroyed. + + Outputs: + + connection_strings = + gke_cluster = + load_balancer_details = { + "demo" = { + "balancerd_load_balancer_ip" = "192.0.2.10" + "console_load_balancer_ip" = "192.0.2.254" + } + } + network = { + "network_id" = "projects/my-project/global/networks/mz-simple-network" + "network_name" = "mz-simple-network" + "subnet_name" = "mz-simple-subnet" + } + service_accounts = { + "gke_sa" = "mz-simple-gke-sa@my-project.iam.gserviceaccount.com" + "materialize_sa" = "mz-simple-materialize-sa@my-project.iam.gserviceaccount.com" + } + ``` + +1. Verify the installation and check the status: + + ```bash + kubectl get all -n materialize-environment + ``` + + Wait for the components to be in the `Running` state. + + ```none + NAME READY STATUS RESTARTS AGE + pod/db-demo-db-wrvhw 0/1 Completed 0 4m26s + pod/mzdtwvu4qe4q-balancerd-6989df5c75-mpmqx 1/1 Running 0 3m54s + pod/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1-0 1/1 Running 0 3m53s + pod/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1-0 1/1 Running 0 3m52s + pod/mzdtwvu4qe4q-console-7c9bc94bcb-6t7lg 1/1 Running 0 3m41s + pod/mzdtwvu4qe4q-console-7c9bc94bcb-9x5qq 1/1 Running 0 3m41s + pod/mzdtwvu4qe4q-environmentd-1-0 1/1 Running 0 4m9s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/mzdtwvu4qe4q-balancerd ClusterIP None 6876/TCP,6875/TCP 3m54s + service/mzdtwvu4qe4q-balancerd-lb LoadBalancer 10.52.5.105 192.0.2.10 6875:30844/TCP,6876:32307/TCP 4m9s + service/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 3m53s + service/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1 ClusterIP None 2100/TCP,2103/TCP,2101/TCP,2102/TCP,6878/TCP 3m52s + service/mzdtwvu4qe4q-console ClusterIP None 8080/TCP 3m41s + service/mzdtwvu4qe4q-console-lb LoadBalancer 10.52.4.2 192.0.2.254 8080:32193/TCP 4m9s + service/mzdtwvu4qe4q-environmentd ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 3m54s + service/mzdtwvu4qe4q-environmentd-1 ClusterIP None 6875/TCP,6876/TCP,6877/TCP,6878/TCP 4m9s + service/mzdtwvu4qe4q-persist-pubsub-1 ClusterIP None 6879/TCP 4m9s + + NAME READY UP-TO-DATE AVAILABLE AGE + deployment.apps/mzdtwvu4qe4q-balancerd 1/1 1 1 3m54s + deployment.apps/mzdtwvu4qe4q-console 2/2 2 2 3m41s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/mzdtwvu4qe4q-balancerd-6989df5c75 1 1 1 3m54s + replicaset.apps/mzdtwvu4qe4q-console-7c9bc94bcb 2 2 2 3m41s + + NAME READY AGE + statefulset.apps/mzdtwvu4qe4q-cluster-s2-replica-s1-gen-1 1/1 3m53s + statefulset.apps/mzdtwvu4qe4q-cluster-u1-replica-u1-gen-1 1/1 3m52s + statefulset.apps/mzdtwvu4qe4q-environmentd-1 1/1 4m9s + + NAME STATUS COMPLETIONS DURATION AGE + job.batch/db-demo-db Complete 1/1 12s 4m27s + + ``` + + If you run into an error during deployment, refer to the + [Troubleshooting](/installation/troubleshooting/). + +1. Open the Materialize Console in your browser: + + {{< tabs >}} + + {{< tab "Via Network Load Balancer" >}} + + Starting in v0.3.0, for each Materialize instance, Materialize on GCP + Terraform module also deploys load balancers (by default, internal) with the + following listeners, including a listener on port 8080 for the Materialize + Console: + + | Port | Description | + | ---- | ------------| + | 6875 | For SQL connections to the database | + | 6876 | For HTTP(S) connections to the database | + | **8080** | **For HTTP(S) connections to Materialize Console** | + + The load balancer details are found in the `load_balancer_details` in + the [Terraform output](#gcp-terraform-output). + + The example uses a self-signed ClusterIssuer. As such, you may encounter a + warning with regards to the certificate. In production, run with certificates + from an official Certificate Authority (CA) rather than self-signed + certificates. + + {{}} + + {{< tab "Via port forwarding" >}} + + {{% self-managed/port-forwarding-handling console="console-lb"%}} + + {{}} + {{}} + + + {{< tip >}} + + {{% self-managed/troubleshoot-console-mz_catalog_server_blurb %}} + + {{< /tip >}} + +## Next steps + +{{% self-managed/next-steps %}} + +## Cleanup + +{{% self-managed/cleanup-cloud %}} + +## See also + +- [Troubleshooting](/installation/troubleshooting/) +- [Materialize Operator Configuration](/installation/configuration/) +- [Appendix: Google deployment guidelines](/installation/install-on-gcp/appendix-deployment-guidelines/) +- [Installation](/installation/) diff --git a/doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/upgrade.md similarity index 97% rename from doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md rename to doc/user/content/installation/install-on-gcp/legacy-terraform-module/upgrade.md index eab4ab0ebe91b..92f9f2d0b93d2 100644 --- a/doc/user/content/installation/install-on-gcp/upgrade-on-gcp.md +++ b/doc/user/content/installation/install-on-gcp/legacy-terraform-module/upgrade.md @@ -1,11 +1,13 @@ --- -title: "Upgrade on GCP (Terraform)" +title: "Upgrade" description: "Procedure to upgrade your Materialize operator and instances running on GCP" menu: main: - parent: "install-on-gcp" - identifier: "upgrade-on-gcp" + parent: "install-on-gcp-legacy-terraform-module" + identifier: "legacy-terraform-module-upgrade" weight: 10 +aliases: + - /installation/install-on-gcp/upgrade-on-gcp --- {{< annotation type="Disambiguation" >}} diff --git a/doc/user/content/installation/install-on-gcp/terraform-module/_index.md b/doc/user/content/installation/install-on-gcp/terraform-module/_index.md new file mode 100644 index 0000000000000..72acc7d697199 --- /dev/null +++ b/doc/user/content/installation/install-on-gcp/terraform-module/_index.md @@ -0,0 +1,164 @@ +--- +title: "Terraform Module" +description: "" +menu: + main: + parent: "install-on-gcp" + identifier: "install-gcp-terraform" + weight: 5 +--- + +Materialize provides a set of modular Terraform modules that can be used to +deploy all services required for a production ready Materialize database. +The module is intended to provide a simple set of examples on how to deploy +materialize. It can be used as is or modules can be taken from the example and +integrated with existing DevOps tooling. + +The repository can be found at: + +***[Materialize Terraform Self-Managed GCP](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp)*** + +Please see the [top level](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main) and [cloud specific](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp) documentation for a full understanding +of the module structure and customizations. + +Also check out the [GCP deployment guide](/installation/install-on-gcp/appendix-deployment-guidelines/) for details on recommended instance sizing and configuration. + +{{< note >}} +{{% self-managed/materialize-components-sentence %}} +{{< /note >}} + +{{< warning >}} + +{{< self-managed/terraform-disclaimer >}} + +{{< /warning >}} + + +## Prerequisites + +- [Terraform](https://developer.hashicorp.com/terraform/install?product_intent=terraform) +- [GCloud Cli](https://cloud.google.com/sdk/docs/install) +- [`kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) +- [Helm 3.2.0+](https://helm.sh/docs/intro/install/) +- [kubectl gke plugin](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin) +- [a Google Cloud service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) + +#### License key + +{{< include-md file="shared-content/license-key-required.md" >}} + +--- + +# Example: Simple Materialize Deployment on GCP + +This example demonstrates how to deploy a complete Materialize environment on GCP using the modular Terraform setup from this repository. + + +## Setup +```shell +git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git +cd materialize-terraform-self-managed/gcp/examples/simple +```` + +## What Gets Created + +This example provisions the following infrastructure: + +### Networking +- **VPC Network**: Custom VPC with auto-create subnets disabled +- **Subnet**: 192.168.0.0/20 primary range with private Google access enabled +- **Secondary Ranges**: + - Pods: 192.168.64.0/18 + - Services: 192.168.128.0/20 +- **Cloud Router**: For NAT and routing configuration +- **Cloud NAT**: For outbound internet access from private nodes +- **VPC Peering**: Service networking connection for Cloud SQL private access + +### Compute +- **GKE Cluster**: Regional cluster with Workload Identity enabled +- **Generic Node Pool**: e2-standard-8 machines, autoscaling 2-5 nodes, 50GB disk, for general workloads +- **Materialize Node Pool**: n2-highmem-8 machines, autoscaling 2-5 nodes, 100GB disk, 1 local SSD, swap enabled, dedicated taints for Materialize workloads +- **Service Account**: GKE service account with workload identity binding + +### Database +- **Cloud SQL PostgreSQL**: Private IP only (no public IP) +- **Tier**: db-custom-2-4096 (2 vCPUs, 4GB memory) +- **Database**: `materialize` database with UTF8 charset +- **User**: `materialize` user with auto-generated password +- **Network**: Connected via VPC peering for private access + +### Storage +- **Cloud Storage Bucket**: Regional bucket for Materialize persistence +- **Access**: HMAC keys for S3-compatible access (Workload Identity service account with storage permissions is configured but not currently used by Materialize for GCS access, in future we will remove HMAC keys and support access to GCS either via Workload Identity Federation or via Kubernetes ServiceAccounts that impersonate IAM service accounts) +- **Versioning**: Disabled (for testing; enable in production) + +### Kubernetes Add-ons +- **cert-manager**: Certificate management controller for Kubernetes that automates TLS certificate provisioning and renewal +- **Self-signed ClusterIssuer**: Provides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication. + +### Materialize +- **Operator**: Materialize Kubernetes operator in `materialize` namespace +- **Instance**: Single Materialize instance in `materialize-environment` namespace +- **Load Balancers**: GCP Load Balancers for Materialize access + +--- + +## Required APIs +Your GCP project needs several APIs enabled. Here's what each API does in simple terms: + +```bash +# Enable these APIs in your project +gcloud services enable container.googleapis.com # For creating Kubernetes clusters +gcloud services enable sqladmin.googleapis.com # For creating databases +gcloud services enable cloudresourcemanager.googleapis.com # For managing GCP resources +gcloud services enable servicenetworking.googleapis.com # For private network connections +gcloud services enable iamcredentials.googleapis.com # For security and authentication +``` + +## Getting Started + +### Step 1: Set Required Variables + +Before running Terraform, create a `terraform.tfvars` file with the following variables: + +```hcl +project_id = "my-gcp-project" +name_prefix = "simple-demo" +region = "us-central1" +license_key = "your-materialize-license-key" # Optional: Get from https://materialize.com/self-managed/ +labels = { + environment = "demo" + created_by = "terraform" +} +``` + +**Required Variables:** +- `project_id`: GCP project ID +- `name_prefix`: Prefix for all resource names +- `region`: GCP region for deployment +- `labels`: Map of labels to apply to resources +- `license_key`: Materialize license key (required for production use) + +--- + +### Step 2: Deploy Materialize + +Run the usual Terraform workflow: + +```bash +terraform init +terraform apply +``` + +## Notes + +* ***GCP Storage Authentication Limitation:*** Materialize currently only supports HMAC key authentication for GCS access (S3-compatible API). + Current State: The modules configure both HMAC keys and Workload Identity, but Materialize uses HMAC keys for actual storage access. + Future: Native GCS access via Workload Identity Federation or Kubernetes service account impersonation will be supported in a future release, eliminating the need for static credentials. +* You can customize each module independently. +* To reduce cost in your demo environment, you can tweak machine types and database tiers in `main.tf`. + +***Don't forget to destroy resources when finished:*** +```bash +terraform destroy +``` diff --git a/doc/user/content/installation/operational-guidelines.md b/doc/user/content/installation/operational-guidelines.md index de246686733c0..0ad4a3baf4ae2 100644 --- a/doc/user/content/installation/operational-guidelines.md +++ b/doc/user/content/installation/operational-guidelines.md @@ -12,9 +12,11 @@ menu: ## Recommended instance types +As a general guideline, we recommend: + - ARM-based CPU -- 1:8 ratio of vCPU to GiB memory (if spill-to-disk is not enabled) -- 1:16 ratio of vCPU to GiB local instance storage (if spill-to-disk is enabled) +- A 1:8 ratio of vCPU to GiB memory is recommend. +- When using swap, it is recommend to use a 1:16 ratio of vCPU to GiB local instance storage See also the specific cloud provider guidance: @@ -27,12 +29,6 @@ See also the specific cloud provider guidance: - [Azure Deployment guidelines](/installation/install-on-azure/appendix-deployment-guidelines/#recommended-instance-types) -## CPU affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## TLS When running with TLS in production, run with certificates from an official @@ -40,11 +36,11 @@ Certificate Authority (CA) rather than self-signed certificates. ## Locally-attached NVMe storage -For optimal performance, Materialize requires fast, locally-attached NVMe -storage. Having a locally-attached storage allows Materialize to spill to disk -when operating on datasets larger than main memory as well as allows for a more -graceful degradation rather than OOMing. Network-attached storage (like EBS -volumes) can significantly degrade performance and is not supported. +Configuring swap on nodes to using locally-attached NVMe storage allows +Materialize to spill to disk when operating on datasets larger than main memory. +This setup can provide significant cost savings and provides a more graceful +degradation rather than OOMing. Network-attached storage (like EBS volumes) can +significantly degrade performance and is not supported. Refer to the specific cloud provider guidelines: diff --git a/doc/user/content/installation/upgrading.md b/doc/user/content/installation/upgrading.md new file mode 100644 index 0000000000000..424ac916f7d69 --- /dev/null +++ b/doc/user/content/installation/upgrading.md @@ -0,0 +1,187 @@ +--- +title: "Upgrade Overview" +description: "Upgrading Self-Managed Materialize." +menu: + main: + parent: "installation" +--- + +The following provides a general outline and examples for upgrading Materialize. + +For a more specific set of steps, please consult the deployment-specific upgrade +documentation: + - [Minikube](/installation/install-on-local-minikube/upgrade-on-local-minikube/) + - [Kind](/installation/install-on-local-kind/upgrade-on-local-kind/) + - [AWS](/installation/install-on-aws/legacy-terraform-module/upgrade/) + - [GCP](/installation/install-on-gcp/legacy-terraform-module/upgrade/) + - [Azure](/installation/install-on-azure/legacy-terraform-module/upgrade/) + +***When upgrading always***: +- Upgrade the operator first and ensure version compatibility between the operator and the Materialize instance you are upgrading to. +- Upgrade your Materialize instances after upgrading the operator to ensure compatibility. +- Check the [version specific upgrade notes](#version-specific-upgrade-notes). + +### Upgrading the Helm Chart and Kubernetes Operator + +{{< important >}} + +When upgrading Materialize, always upgrade the operator first. + +{{}} + +The Materialize Kubernetes operator is deployed via Helm and can be updated through standard Helm upgrade commands. + +```shell +helm upgrade my-materialize-operator materialize/misc/helm-charts/operator +``` + +If you have custom values, make sure to include your values file: + +```shell +helm upgrade my-materialize-operator materialize/misc/helm-charts/operator -f my-values.yaml +``` + +### Upgrading Materialize Instances + +In order to minimize unexpected downtime and avoid connection drops at critical +periods for your application, changes are not immediately and automatically +rolled out by the Operator. Instead, the upgrade process involves two steps: +- First, staging spec changes to the Materialize custom resource. +- Second, applying the changes via a `rolloutRequest`. + +When upgrading your Materialize instances, you'll first want to update the `environmentdImageRef` field in the Materialize custom resource spec. + +#### Updating the `environmentdImageRef` +To find a compatible version with your currently deployed Materialize operator, check the `appVersion` in the Helm repository. + +```shell +helm list -n materialize +``` + +Using the returned version, we can construct an image ref. +We always recommend using the official Materialize image repository +`docker.io/materialize/environmentd`. + +``` +environmentdImageRef: docker.io/materialize/environmentd:v26.0.0 +``` + +The following is an example of how to patch the version. +```shell +# For version updates, first update the image reference +kubectl patch materialize \ + -n \ + --type='merge' \ + -p "{\"spec\": {\"environmentdImageRef\": \"materialize/environmentd:v26.0.0\"}}" +``` + +#### Applying the changes via `rolloutRequest` + +To apply changes and kick off the Materialize instance upgrade, you must update the `requestRollout` field in the Materialize custom resource spec to a new UUID. +Be sure to consult the [Rollout Configurations](#rollout-configuration) to ensure you've selected the correct rollout behavior. +```shell +# Then trigger the rollout with a new UUID +kubectl patch materialize \ + -n \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\"}}" +``` + + +It is possible to combine both operations in a single command if preferred: + +```shell +kubectl patch materialize 12345678-1234-1234-1234-123456789012 \ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"environmentdImageRef\": \"materialize/environmentd:v26.0.0\", \"requestRollout\": \"$(uuidgen)\"}}" +``` + +#### Using YAML Definition + +Alternatively, you can update your Materialize custom resource definition directly: + +```yaml +apiVersion: materialize.cloud/v1alpha1 +kind: Materialize +metadata: + name: 12345678-1234-1234-1234-123456789012 + namespace: materialize-environment +spec: + environmentdImageRef: materialize/environmentd:v26.0.0 # Update version as needed + requestRollout: 22222222-2222-2222-2222-222222222222 # Generate new UUID + forceRollout: 33333333-3333-3333-3333-333333333333 # Optional: for forced rollouts + inPlaceRollout: false # In Place rollout is deprecated and ignored. Please use rolloutStrategy + rolloutStrategy: WaitUntilReady # The mechanism to use when rolling out the new version. Can be WaitUntilReady or ImmediatelyPromoteCausingDowntime + backendSecretName: materialize-backend +``` + +Apply the updated definition: + +```shell +kubectl apply -f materialize.yaml +``` + +### Rollout Configuration + +#### Forced Rollouts + +If you need to force a rollout even when there are no changes to the instance: + +```shell +kubectl patch materialize \ + -n materialize-environment \ + --type='merge' \ + -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}" +``` + +#### Rollout Strategies +The behavior of the new version rollout follows your `rolloutStrategy` setting: + +`WaitUntilReady` (default): + +New instances are created and all dataflows are determined to be ready before cutover and terminating the old version, temporarily requiring twice the resources during the transition. + +`ImmediatelyPromoteCausingDowntime`: + +Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. + +#### In Place Rollout + +`inPlaceRollout` has been deprecated and will be ignored. + + +### Verifying the Upgrade + +After initiating the rollout, you can monitor the status field of the Materialize custom resource to check on the upgrade. + +```shell +# Watch the status of your Materialize environment +kubectl get materialize -n materialize-environment -w + +# Check the logs of the operator +kubectl logs -l app.kubernetes.io/name=materialize-operator -n materialize +``` +### Version Specific Upgrade Notes + +#### Upgrading to `v26.0` +- This is a major version upgrade. In order to upgrade to `v26.0`, you must first upgrade to `v25.2.15`, then upgrade to `v26.0.0`. +- New requirements were introduced for license keys. In order to upgrade, you will + first need to add a license key to the `backendSecret` used in the spec for your + Materialize resource. Please refer to our [instructions on how to get and install a license keys](/installation/faq#how-do-i-get-a-license-key). +- Swap is now enabled by default. Swap reduces the memory required to + operate Materialize and improves cost efficiency. Upgrading to `v26.0` + requires some preparation to ensure kubernetes nodes are labeled + and configured correctly. Please refer to our guides: + + {{< yaml-table data="self_managed/enable_swap_upgrade_guides" >}} + + +#### Upgrading between minor versions less than `v26` + - Prior to `v26`, you must upgrade at most one minor version at a time. For example, upgrading from `v25.1.5` to `v25.2.15` is permitted. + +## See also + +- [Configuration](/installation/configuration/) +- [Installation](/installation/) +- [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/data/self_managed/legacy_terraform_list.yml b/doc/user/data/self_managed/legacy_terraform_list.yml new file mode 100644 index 0000000000000..8170eb218d06f --- /dev/null +++ b/doc/user/data/self_managed/legacy_terraform_list.yml @@ -0,0 +1,29 @@ +columns: + - column: "Sample Module" + - column: "Description" + +rows: +- "Sample Module": | + [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) + "Description": | + A sample Terraform module for installing the Materialize Helm chart into a Kubernetes cluster. + +- "Sample Module": | + [Materialize on AWS](https://github.com/MaterializeInc/terraform-aws-materialize) + "Description": | + A sample Terraform module for deploying Materialize on AWS Cloud Platform + with all required infrastructure components. + See [Install on AWS](/installation/install-on-aws/) for an example usage. + +- "Sample Module": | + [Materialize on Azure](https://github.com/MaterializeInc/terraform-azurerm-materialize) + "Description": | + A sample Terraform module for deploying Materialize on Azure with all + required infrastructure components. See [Install on + Azure](/installation/install-on-azure/) for an example usage. + +- "Sample Module": | + [Materialize on Google Cloud Platform (GCP)](https://github.com/MaterializeInc/terraform-google-materialize) + "Description": | + A sample Terraform module for deploying Materialize on Google Cloud Platform + (GCP) with all required infrastructure components. See [Install on GCP](/installation/install-on-gcp/) for an example usage. diff --git a/doc/user/data/self_managed/terraform_list.yml b/doc/user/data/self_managed/terraform_list.yml index 8170eb218d06f..94c2481bafd32 100644 --- a/doc/user/data/self_managed/terraform_list.yml +++ b/doc/user/data/self_managed/terraform_list.yml @@ -1,29 +1,22 @@ columns: - - column: "Sample Module" + - column: "Module" - column: "Description" rows: -- "Sample Module": | - [terraform-helm-materialize](https://github.com/MaterializeInc/terraform-helm-materialize) +- "Module": | + [Amazon Web Services (AWS)](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/aws) "Description": | - A sample Terraform module for installing the Materialize Helm chart into a Kubernetes cluster. + An example Terraform module for deploying Materialize on AWS + See [Install on Azure](/installation/install-on-gcp/) for detailed instructions usage. -- "Sample Module": | - [Materialize on AWS](https://github.com/MaterializeInc/terraform-aws-materialize) +- "Module": | + [Azure](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/azure) "Description": | - A sample Terraform module for deploying Materialize on AWS Cloud Platform - with all required infrastructure components. - See [Install on AWS](/installation/install-on-aws/) for an example usage. + An example Terraform module for deploying Materialize on Azure + See [Install on Azure](/installation/install-on-azure/) for detailed instructions usage. -- "Sample Module": | - [Materialize on Azure](https://github.com/MaterializeInc/terraform-azurerm-materialize) +- "Module": | + [Google Cloud Platform (GCP)](https://github.com/MaterializeInc/materialize-terraform-self-managed/tree/main/gcp) "Description": | - A sample Terraform module for deploying Materialize on Azure with all - required infrastructure components. See [Install on - Azure](/installation/install-on-azure/) for an example usage. - -- "Sample Module": | - [Materialize on Google Cloud Platform (GCP)](https://github.com/MaterializeInc/terraform-google-materialize) - "Description": | - A sample Terraform module for deploying Materialize on Google Cloud Platform - (GCP) with all required infrastructure components. See [Install on GCP](/installation/install-on-gcp/) for an example usage. + An example Terraform module for deploying Materialize on GCP + See [Install on Azure](/installation/install-on-gcp/) for detailed instructions usage. diff --git a/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html b/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html index 82e7dc968bf68..83c84ec02eca3 100644 --- a/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html +++ b/doc/user/layouts/shortcodes/self-managed/aws-terraform-configs.html @@ -1,9 +1,9 @@ The tutorial uses the `main.tf` found in the `examples/simple/` directory, which -requires minimal user input. For details on the `examples/simple/` +requires minimal user input. For details on the `examples/simple/` infrastructure configuration (such as the node instance type, etc.), see the [examples/simple/main.tf](https://github.com/MaterializeInc/terraform-aws-materialize/blob/main/examples/simple/main.tf). For more configuration options, you can use the `main.tf` file at the [root of the repository](https://github.com/MaterializeInc/terraform-aws-materialize/) instead. When running with the root `main.tf`, see [AWS required -configuration](/installation/install-on-aws/appendix-aws-configuration/). +configuration](/installation/install-on-aws/legacy-terraform-module/appendix-configuration/). \ No newline at end of file diff --git a/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html b/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html index f87f789efa914..85e18a616dae6 100644 --- a/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html +++ b/doc/user/layouts/shortcodes/self-managed/azure-terraform-configs.html @@ -7,4 +7,4 @@ the repository](https://github.com/MaterializeInc/terraform-azurerm-materialize/) instead. When running with the root `main.tf`, see [Azure required -configuration](/installation/install-on-azure/appendix-azure-configuration/). +configuration](/installation/install-on-azure/legacy-terraform-module/appendix-configuration/). \ No newline at end of file diff --git a/misc/helm-charts/operator/README.md b/misc/helm-charts/operator/README.md index 19ec22df1f5d2..5237620f0a100 100644 --- a/misc/helm-charts/operator/README.md +++ b/misc/helm-charts/operator/README.md @@ -406,12 +406,6 @@ Materialize has been vetted to work on instances with the following properties: When operating in AWS, we recommend using the `r7gd` and `r6gd` families of instances (and `r8gd` once available) when running with local disk, and the `r8g`, `r7g`, and `r6g` families when running without local disk. -## CPU Affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## Learn More - [Materialize Documentation](https://materialize.com/docs) diff --git a/misc/helm-charts/operator/README.md.gotmpl b/misc/helm-charts/operator/README.md.gotmpl index a69d25b3ab480..7b5af4ad70f89 100644 --- a/misc/helm-charts/operator/README.md.gotmpl +++ b/misc/helm-charts/operator/README.md.gotmpl @@ -347,12 +347,6 @@ Materialize has been vetted to work on instances with the following properties: When operating in AWS, we recommend using the `r7gd` and `r6gd` families of instances (and `r8gd` once available) when running with local disk, and the `r8g`, `r7g`, and `r6g` families when running without local disk. -## CPU Affinity - -It is strongly recommended to enable the Kubernetes `static` [CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy). -This ensures that each worker thread of Materialize is given exclusively access to a vCPU. Our benchmarks have shown this -to substantially improve the performance of compute-bound workloads. - ## Learn More - [Materialize Documentation](https://materialize.com/docs)