Merge pull request #237370 from MicrosoftDocs/release-arc-data

Arc data - May 2023 release

Author: huypub

articles/azure-arc/data/adding-exporters-and-pipelines.md

@@ -71,6 +71,8 @@ The following properties are currently configurable during the Public Preview:
 
 The Telemetry Router supports logs and metrics pipelines. These pipelines are exposed in the custom resource specification of the Arc telemetry router and available for modification.
 
+You can't remove the last pipeline from the telemetry router. If you apply a yaml file that removes the last pipeline, the service rejects the update.
+
 #### Pipeline Settings
 
 |  Setting     | Description |
@@ -100,19 +102,19 @@ metadata:
 spec:
   credentials:
     certificates:
-    - certificateName: arcdata-msft-elasticsearch-exporter-internal
+    - certificateName: arcdata-elasticsearch-exporter
     - certificateName: cluster-ca-certificate
   exporters:
     elasticsearch:
     - caCertificateName: cluster-ca-certificate
-      certificateName: arcdata-msft-elasticsearch-exporter-internal
+      certificateName: arcdata-elasticsearch-exporter
       endpoint: https://logsdb-svc:9200
       index: logstash-otel
-      name: arcdata/msft/internal
+      name: arcdata
   pipelines:
     logs:
       exporters:
-      - elasticsearch/arcdata/msft/internal
+      - elasticsearch/arcdata
 ```
 
 
@@ -204,7 +206,7 @@ spec:
       secretName: <name_of_secret>
       secretNamespace: <namespace_with_secret>
   exporters:
-    elasticsearch:
+    Elasticsearch:
     # Step 2. Declare your Elasticsearch exporter with the needed settings 
     # (certificates, endpoint, and index to export to)
     - name: myLogs

articles/azure-arc/data/configure-managed-instance.md

@@ -7,15 +7,14 @@ ms.subservice: azure-arc-data-sqlmi
 author: dnethi
 ms.author: dinethi
 ms.reviewer: mikeray
-ms.date: 05/27/2022
+ms.date: 05/05/2023
 ms.topic: how-to
 ---
 
 # Configure Azure Arc-enabled SQL managed instance
 
 This article explains how to configure Azure Arc-enabled SQL managed instance.
 
-
 ## Configure resources such as cores, memory
 
 
@@ -74,30 +73,130 @@ The following example will scale down the number of replicas from 3 to 2.
 az sql mi-arc update --name sqlmi1 --replicas 2 --k8s-namespace mynamespace --use-k8s
 ```
 
-> [Note]
+> [!Note]
 > If you scale down from 2 replicas to 1 replica, you may run into a conflict with the pre-configured `--readable--secondaries` setting. You can first edit the `--readable--secondaries` before scaling down the replicas. 
 
 
 ## Configure Server options
 
-You can configure server configuration settings for Azure Arc-enabled SQL managed instance after creation time. This article describes how to configure settings like enabling or disabling mssql Agent, enable specific trace flags for troubleshooting scenarios.
+You can configure certain server configuration settings for Azure Arc-enabled SQL managed instance either during or after creation time. This article describes how to configure settings like enabling "Ad Hoc Distributed Queries" or "backup compression default" etc. 
+
+Currently the following server options can be configured:
+- Ad Hoc Distributed Queries
+- Default Trace Enabled
+- Database Mail XPs
+- Backup compression default
+- Cost threshold for parallelism
+- Optimize for ad hoc workloads
+
+> [!Note]
+> - Currently these options can only be specified via YAML file, either during Arc SQL MI creation or post deployment.
+> - The Arc SQL MI image tag has to be at least version v1.19.x or above
+
+Add the following to your YAML file during deployment to configure any of these options.
+
+```yml
+spec:
+  serverConfigurations:
+  - name: "Ad Hoc Distributed Queries"
+    value: 1
+  - name: "Default Trace Enabled"
+    value: 0
+  - name: "Database Mail XPs"
+    value: 1
+  - name: "backup compression default"
+    value: 1
+  - name: "cost threshold for parallelism"
+    value: 50
+  - name: "optimize for ad hoc workloads"
+    value: 1
+```
+
+If you already have an existing Arc SQL MI, you can run `kubectl edit sqlmi <sqlminame> -n <namespace>` and add the above options into the spec.
+
+
+Sample Arc SQL MI YAML file:
+
+```yml
+apiVersion: sql.arcdata.microsoft.com/v13
+kind: SqlManagedInstance
+metadata:
+  name: sql1
+  annotations:
+    exampleannotation1: exampleannotationvalue1
+    exampleannotation2: exampleannotationvalue2
+  labels:
+    examplelabel1: examplelabelvalue1
+    examplelabel2: examplelabelvalue2
+spec:
+  dev: true #options: [true, false]
+  licenseType: LicenseIncluded #options: [LicenseIncluded, BasePrice].  BasePrice is used for Azure Hybrid Benefits.
+  tier: GeneralPurpose #options: [GeneralPurpose, BusinessCritical]
+  serverConfigurations:
+  - name: "Ad Hoc Distributed Queries"
+    value: 1
+  - name: "Default Trace Enabled"
+    value: 0
+  - name: "Database Mail XPs"
+    value: 1
+  - name: "backup compression default"
+    value: 1
+  - name: "cost threshold for parallelism"
+    value: 50
+  - name: "optimize for ad hoc workloads"
+    value: 1
+  security:
+    adminLoginSecret: sql1-login-secret
+  scheduling:
+    default:
+      resources:
+        limits:
+          cpu: "2"
+          memory: 4Gi
+        requests:
+          cpu: "1"
+          memory: 2Gi
+  services:
+    primary:
+      type: LoadBalancer
+  storage:
+    backups:
+      volumes:
+      - className: azurefile # Backup volumes require a ReadWriteMany (RWX) capable storage class
+        size: 5Gi
+    data:
+      volumes:
+      - className: default # Use default configured storage class or modify storage class based on your Kubernetes environment
+        size: 5Gi
+    datalogs:
+      volumes:
+      - className: default # Use default configured storage class or modify storage class based on your Kubernetes environment
+        size: 5Gi
+    logs:
+      volumes:
+      - className: default # Use default configured storage class or modify storage class based on your Kubernetes environment
+        size: 5Gi
+```
 
 
-### Enable SQL Server agent
+## Enable SQL Server agent
 
-SQL Server agent is disabled by default. It can be enabled by running the following command:
+SQL Server agent is disabled during a default deployment of Arc SQL MI. It can be enabled by running the following command:
 
 ```azurecli
 az sql mi-arc update -n <NAME_OF_SQL_MI> --k8s-namespace <namespace> --use-k8s --agent-enabled true
 ```
+
 As an example:
+
 ```azurecli
 az sql mi-arc update -n sqlinstance1 --k8s-namespace arc --use-k8s --agent-enabled true
 ```
 
-### Enable Trace flags
+## Enable trace flags
 
 Trace flags can be enabled as follows:
+
 ```azurecli
 az sql mi-arc update -n <NAME_OF_SQL_MI> --k8s-namespace <namespace> --use-k8s --trace-flags "3614,1234" 
 ```

articles/azure-arc/data/configure-transparent-data-encryption-sql-managed-instance.md

@@ -16,8 +16,7 @@ ms.custom: template-how-to, event-tier1-build-2022
 
 This article describes how to enable and disable transparent data encryption (TDE) at-rest on an Azure Arc-enabled SQL Managed Instance. In this article, the term *managed instance* refers to a deployment of Azure Arc-enabled SQL Managed Instance and enabling/disabling TDE will apply to all databases running on a managed instance.
 
-Enabling service-managed transparent data encryption will require the managed instance to use a service-managed database master key as well as the service-managed server certificate. These credentials will be automatically created when service-managed transparent data encryption is enabled. For more info on TDE, please refer to [Transparent data encryption](/sql/relational-databases/security/encryption/transparent-data-encryption).
-
+For more info on TDE, please refer to [Transparent data encryption](/sql/relational-databases/security/encryption/transparent-data-encryption).
 
 Turning on the TDE feature does the following:
 
@@ -35,24 +34,48 @@ Before you proceed with this article, you must have an Azure Arc-enabled SQL Man
 
 ## Limitations
 
-The following limitations must be considered when deploying Service-Managed TDE:
+The following limitations apply when you enable automatic TDE:
 
 - Only General Purpose Tier is supported.
-- Failover Groups are not supported.
+- Failover groups aren't supported.
 
 ## Turn on transparent data encryption on the managed instance
-### Prerequisites
 
-Turning on TDE on the managed instance will result in the following operations taking place:
+When TDE is enabled on Arc-enabled SQL Managed Instance, the data service automatically does the following tasks:
+
+1. Adds the service-managed database master key in the `master` database.
+2. Adds the service-managed certificate protector.
+3. Adds the associated Database Encryption Keys (DEK) on all databases on the managed instance.
+4. Enables encryption on all databases on the managed instance.
+
+You can set Azure Arc-enabled SQL Managed Instance TDE in one of two modes:
+
+- Service-managed
+- Customer-managed
+
+In service-managed mode, transparent data encryption requires the managed instance to use a service-managed database master key as well as the service-managed server certificate. These credentials are automatically created when service-managed transparent data encryption is enabled. 
+
+In customer-managed mode, transparent data encryption uses a service-managed database master key and uses keys you provide for the server certificate. To configure customer-managed mode:
 
-1. Adding the service-managed database master key in the `master` database.
-2. Adding the service-managed certificate protector.
-3. Adding the associated Database Encryption Keys (DEK) on all databases on the managed instance.
-4. Enabling encryption on all databases on the managed instance.
+1. Create a certificate.
+1. Store the certificate as a secret in the same Kubernetes namespace as the instance.
+
+> [!NOTE]
+> If you need to change from one mode to the other, you must disable TDE from the current mode before you apply the new mode. For details, see [Turn off transparent data encryption on the managed instance](#turn-off-transparent-data-encryption-on-the-managed-instance).
+>
+> For example, if the service is encrypted using service-managed mode, go to `Disabled` mode before you enable customer-managed mode. 
+>
+>  ```console
+>  kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "Disabled" } } } }'
+>  ```
+
+
+To proceed, select the mode you want to use.
 
 ### [Service-managed mode](#tab/service-managed-mode)
 
-Run kubectl patch to enable service-managed TDE
+
+To enable TDE in service-managed mode, run kubectl patch to enable service-managed TDE
 
 ```console
 kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "ServiceManaged" } } } }'
@@ -61,31 +84,81 @@ kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{
 Example:
 
 ```console
-kubectl patch sqlmi contososqlmi --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "ServiceManaged" } } } }'
+kubectl patch sqlmi sqlmi-tde --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "ServiceManaged" } } } }'
 ```
+
+### [Customer-managed mode](#tab/customer-managed-mode)
+
+To enable TDE in customer managed mode:
+
+1. Create a certificate. 
+
+   ```console
+   openssl req -x509 -newkey rsa:2048 -nodes -keyout <key-file> -days 365 -out <cert-file>
+   ```
+
+1. Create a secret for the certificate.
+
+   > [!IMPORTANT]
+   > Store the secret in the same namespace as the managed instance
+
+   ```console
+   kubectl create secret generic <tde-secret-name> --from-literal=privatekey.pem="$(cat <key-file>)" --from-literal=certificate.pem="$(cat <cert-file>) --namespace <namespace>"
+   ```
+
+1. Run `kubectl patch ...` to enable customer-managed TDE
+
+   ```console
+   kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "CustomerManaged", "protectorSecret": "<tde-secret-name>" } } } }'
+   ```
+
+   Example:
+
+   ```console
+   kubectl patch sqlmi sqlmi-tde --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "CustomerManaged", "protectorSecret": "sqlmi-tde-protector-cert-secret" } } } }'
+   ```
+
 ---
 
 ## Turn off transparent data encryption on the managed instance
 
-Turning off TDE on the managed instance will result in the following operations taking place:
+When TDE is disabled on Arc-enabled SQL Managed Instance, the data service automatically does the following tasks:
 
-1. Disabling encryption on all databases on the managed instance.
-2. Dropping the associated DEKs on all databases on the managed instance.
-3. Dropping the service-managed certificate protector.
-4. Dropping the service-managed database master key in the `master` database.
+1. Disables encryption on all databases on the managed instance.
+2. Drops the associated DEKs on all databases on the managed instance.
+3. Drops the service-managed certificate protector.
+4. Drops the service-managed database master key in the `master` database.
 
 ### [Service-managed mode](#tab/service-managed-mode)
 
-Run kubectl patch to disable service-managed TDE
+Run kubectl patch to disable service-managed TDE.
+
+```console
+kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "Disabled" } } } }'
+```
+
+Example:
+```console
+kubectl patch sqlmi sqlmi-tde --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "Disabled" } } } }'
+```
+
+### [Customer-managed mode](#tab/customer-managed-mode)
+
+Run kubectl patch to disable customer-managed TDE.
+
+When you disable TDE in customer-managed mode, you need to set `"protectorSecret" : null`.  
 
 ```console
-kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": null } } } }'
+kubectl patch sqlmi <sqlmi-name> --namespace <namespace> --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "Disabled" , "protectorSecret": null } } } }'
 ```
 
+
 Example:
+
 ```console
-kubectl patch sqlmi contososqlmi --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": null } } } }'
+kubectl patch sqlmi sqlmi-tde --namespace arc --type merge --patch '{ "spec": { "security": { "transparentDataEncryption": { "mode": "Disabled" , "protectorSecret": null } } } }'
 ```
+
 ---
 
 ## Back up a transparent data encryption credential