docs: mTLS documentation with example sample

Signed-off-by: [email protected] <[email protected]>

Author: vishwa-uber

docs/03-concepts/14-mutual-tls.md

@@ -75,14 +75,131 @@ The mTLS handshake process involves the following steps:
 
 ---
 
+## Implementing mTLS in Cadence
+
+### Server Configuration
+
+To enable mTLS in Cadence server, you need to configure TLS settings and start the server with the appropriate environment configuration.
+
+#### Starting the Server with TLS
+
+Use the `--zone` flag to specify the TLS configuration when starting the Cadence server:
+
+```bash
+./cadence-server --env development --zone tls start
+```
+
+**Command breakdown:**
+- `--env development`: Specifies the environment configuration to use (corresponds to `config/development.yaml`)
+- `--zone tls`: Specifies the zone configuration to use (corresponds to the `tls` zone in `development_tls.yaml`)
+- `start`: Starts all Cadence services
+
+The `--zone tls` flag tells the server to load additional configuration from the zone-specific file. In this case, it will look for `config/development_tls.yaml` which contains the TLS-specific settings.
+
+#### TLS Configuration File
+
+The server uses a YAML configuration file to define TLS settings. Here's an example from [`development_tls.yaml`](https://github.com/cadence-workflow/cadence/blob/master/config/development_tls.yaml):
+
+```bash
+services:
+  frontend:
+    rpc:
+      tls:
+        enabled: true
+        certFile: config/credentials/keytest.crt
+        keyFile: config/credentials/keytest
+        caFiles:
+          - config/credentials/client.crt
+        requireClientAuth: true
+
+  matching:
+    rpc:
+      tls:
+        enabled: true
+        certFile: config/credentials/keytest.crt
+        keyFile: config/credentials/keytest
+
+  history:
+    rpc:
+      tls:
+        enabled: true
+        certFile: config/credentials/keytest.crt
+        keyFile: config/credentials/keytest
+
+clusterGroupMetadata:
+  clusterGroup:
+    cluster0:
+      tls:
+        enabled: true
+            
+```
+---
+
+### Client Implementation
+
+To connect a Cadence client with mTLS, you need to configure TLS credentials and pass them to the Cadence client. Here's the essential code from the [helloworld_tls sample](https://github.com/cadence-workflow/cadence-samples/blob/master/new_samples/client_samples/helloworld_tls/hello_world_tls.go):
+
+#### Setting up TLS Client Connection
+
+```go
+import (
+    "crypto/tls"
+    "crypto/x509"
+    "os"
+    
+    "go.uber.org/cadence/client"
+    "google.golang.org/grpc"
+    "google.golang.org/grpc/credentials"
+)
+
+func createCadenceClient() (client.Client, error) {
+    // Load client certificate and key
+    clientCert, err := tls.LoadX509KeyPair("credentials/client.crt", "credentials/client.key")
+    if err != nil {
+        return nil, err
+    }
+
+    // Load CA certificate to verify server
+    caCert, err := os.ReadFile("credentials/keytest.crt")
+    if err != nil {
+        return nil, err
+    }
+    
+    caCertPool := x509.NewCertPool()
+    caCertPool.AppendCertsFromPEM(caCert)
+
+    // Configure TLS
+    tlsConfig := &tls.Config{
+        Certificates: []tls.Certificate{clientCert},
+        RootCAs:      caCertPool,
+        ServerName:   "cadence-frontend", // Must match server certificate CN
+    }
+
+    // Create Cadence client with TLS
+    return client.NewClient(client.Options{
+        HostPort: "127.0.0.1:7833", // gRPC port
+        Domain:   "samples-domain",
+        Transport: &client.TransportConfiguration{
+            GRPCDialOptions: []grpc.DialOption{
+                grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
+            },
+        },
+    })
+}
+```
+
+---
+
 ## Complete Working Example
 
-For a complete working example with detailed code and configuration, refer to the [helloworld_tls sample](https://github.com/cadence-workflow/cadence-samples/tree/master/new_samples/client_samples/helloworld_tls) in the Cadence samples repository. This sample demonstrates how to:
+The [helloworld_tls sample](https://github.com/cadence-workflow/cadence-samples/tree/master/new_samples/client_samples/helloworld_tls) provides a complete, tested implementation of mTLS with Cadence, including:
+
+- Certificate generation scripts
+- Complete client implementation with mTLS
+- Instructions for running with a TLS-enabled server
+- Step-by-step setup guide
 
-- Generate test certificates using OpenSSL
-- Configure both server and client for mTLS
-- Implement a simple workflow with mTLS authentication
-- Test the mTLS connection
+For additional server configuration examples, refer to the [Cadence server repository](https://github.com/cadence-workflow/cadence)
 
 ---