Add HashiCorp Vault integration for multi-cloud secret management

[Copilot]

Switches secret management from Azure Key Vault to HashiCorp Vault to enable deployment across Azure, AWS, GCP, and on-premises. Maintains Azure Key Vault as fallback for backward compatibility.

Infrastructure

Kubernetes deployment (infra/vault/values.yaml)

• HA configuration: 3-node Raft cluster
• TLS-enabled with persistent audit logs
• Integrated metrics and pod disruption budgets

Azure Container Instances (infra/modules/vault-aci.bicep)

• Alternative for Azure-only deployments
• File-based storage with Azure Files integration

Bicep orchestration (infra/main.bicep)

• New secretProvider parameter: azurekeyvault | hashicorpvault | none
• Conditional resource deployment based on provider selection

Application Integration

Shared configuration provider (src/shared/Configuration/VaultConfigurationExtensions.cs)

// Program.cs - single line addition
builder.Configuration.AddVaultConfiguration(builder.Configuration);

Features:

• Kubernetes auth (service account token) and AppRole auth (CI/CD)
• Automatic secret reload with configurable interval
• Graceful degradation if Vault unavailable
• Hierarchical secret path traversal with flattening

CI/CD

GitHub Actions workflow (deploy.yml)

• Intelligent fallback chain: HashiCorp Vault → Azure Key Vault → GitHub Secrets → defaults
• AppRole authentication with Vault CLI installation
• Proper secret masking and environment variable handling

Automation

scripts/setup-vault.sh

• One-command Vault initialization (5-key Shamir, 3-threshold)
• Kubernetes and AppRole auth configuration
• Policy creation and secret population

scripts/add-vault-packages.sh

• Bulk VaultSharp package injection to all 17 microservices

Kubernetes

ServiceAccounts (infra/k8s/vault-serviceaccount.yaml)

• Per-service accounts for least-privilege access
• Token review ClusterRoleBinding for Kubernetes auth
• ConfigMap with Vault endpoint configuration

Documentation

• Complete deployment guide: docs/security/HASHICORP-VAULT-INTEGRATION.md
• Application integration: src/shared/Configuration/README.md
• Updated security policy: SECURITY.md

Migration Path

Existing Azure Key Vault deployments continue working. To adopt HashiCorp Vault:

1. Deploy Vault: helm install vault hashicorp/vault --values infra/vault/values.yaml
2. Initialize: ./scripts/setup-vault.sh
3. Add packages: ./scripts/add-vault-packages.sh
4. Update Program.cs in each service with configuration line above
5. Set GitHub variables: VAULT_ADDR, VAULT_ROLE_ID, VAULT_SECRET_ID

No forced migration - deployments choose via secretProvider parameter.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[v4.0] HashiCorp Key Vault Integration - Production Security Hardening</issue_title>
<issue_description>---
name: 'Azure Key Vault Integration'
about: Migrate all secrets to Azure Key Vault for production-grade security
title: '[v4.0] Azure Key Vault Integration - Production Security Hardening'
labels: 'security, infrastructure, priority:critical'
assignees: ''

🎯 Objective

Migrate all application secrets from environment variables and configuration files to Azure Key Vault with managed identity authentication. This is a BLOCKER for Beta launch and clearinghouse integration.

Priority: 🔴 CRITICAL
Effort: 1-2 weeks (1 developer)
Depends On: Azure subscription with Key Vault quota
Blocks: Clearinghouse integration, customer onboarding

---

📋 Success Criteria

• All production secrets stored in Azure Key Vault (Premium SKU with HSM)
• All 17 microservices read secrets from Key Vault via managed identity
• Portal reads Stripe API keys from Key Vault
• SFTP credentials migrated from local config to Key Vault
• Cosmos DB connection strings use Key Vault references
• Zero secrets in code, config files, or environment variables
• Secret rotation automated (90-day policy)
• Audit logging enabled for all secret access
• Smoke tests pass in staging environment

---

🔧 Implementation Steps

Phase 1: Azure Key Vault Setup (Day 1)

1.1 Create Key Vault Resource

# Create Key Vault (Premium SKU for HSM-backed keys)
az keyvault create \
  --name "kv-cho-prod-${LOCATION}" \
  --resource-group "rg-cloudhealthoffice-prod" \
  --location "eastus" \
  --sku "Premium" \
  --enable-rbac-authorization true \
  --enable-purge-protection true \
  --retention-days 90

# Configure network access (allow Azure services + deny public)
az keyvault network-rule add \
  --name "kv-cho-prod-eastus" \
  --resource-group "rg-cloudhealthoffice-prod" \
  --vnet-name "vnet-cho-prod" \
  --subnet "snet-aks"

az keyvault update \
  --name "kv-cho-prod-eastus" \
  --resource-group "rg-cloudhealthoffice-prod" \
  --default-action Deny

1.2 Enable Audit Logging

# Send Key Vault logs to Log Analytics
az monitor diagnostic-settings create \
  --name "KeyVaultAudit" \
  --resource $(az keyvault show --name "kv-cho-prod-eastus" -g "rg-cloudhealthoffice-prod" --query id -o tsv) \
  --workspace $(az monitor log-analytics workspace show --name "law-cho-prod" -g "rg-cloudhealthoffice-prod" --query id -o tsv) \
  --logs '[{"category": "AuditEvent", "enabled": true, "retentionPolicy": {"enabled": true, "days": 365}}]' \
  --metrics '[{"category": "AllMetrics", "enabled": true}]'

1.3 Configure Managed Identity for AKS

# Enable managed identity on AKS cluster
az aks update \
  --resource-group "rg-cloudhealthoffice-prod" \
  --name "cho-aks-prod" \
  --enable-managed-identity

# Get AKS managed identity
AKS_IDENTITY=$(az aks show -g "rg-cloudhealthoffice-prod" -n "cho-aks-prod" --query identityProfile.kubeletidentity.clientId -o tsv)

# Grant Key Vault Secrets User role
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee $AKS_IDENTITY \
  --scope $(az keyvault show --name "kv-cho-prod-eastus" -g "rg-cloudhealthoffice-prod" --query id -o tsv)

---

Phase 2: Migrate Secrets (Days 2-3)

2.1 Audit Current Secrets
Create inventory of all secrets currently in:

• GitHub Secrets (production)
• appsettings.json files
• Kubernetes ConfigMaps/Secrets
• Environment variables

2.2 Upload Secrets to Key Vault

# Cosmos DB secrets
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "CosmosDb--ConnectionString" --value "${COSMOS_CONNECTION_STRING}"

# Stripe secrets
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "Stripe--SecretKey" --value "${STRIPE_SECRET_KEY}"
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "Stripe--PublishableKey" --value "${STRIPE_PUBLISHABLE_KEY}"
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "Stripe--WebhookSecret" --value "${STRIPE_WEBHOOK_SECRET}"

# Azure AD secrets (portal)
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "AzureAd--ClientSecret" --value "${AZURE_AD_CLIENT_SECRET}"

# SFTP credentials (per tenant)
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "SFTP--clouddentaloffice--Password" --value "${SFTP_PASSWORD}"

# Application Insights
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "ApplicationInsights--ConnectionString" --value "${APPINSIGHTS_CONNECTION_STRING}"

# Service Bus (if still using)
az keyvault secret set --vault-name "kv-cho-prod-eastus" --name "ServiceBus--ConnectionString" --value "${SERVICEBUS_CONNECTION_STRING}"

2.3 Clearinghouse Credentials (for future integration)

# Availity (sandbox then production)
az keyvault secret set --vault-name "k...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes aurelianware/cloudhealthoffice#373

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

[aurelianware]

@claude can you fix the checks and deployment

[Copilot]

Pull request overview

This PR introduces HashiCorp Vault as a multi-cloud secrets backend (with Azure Key Vault as a fallback), adding a shared .NET configuration provider plus supporting infrastructure (Helm/Bicep), automation scripts, workflow updates, and documentation.

Changes:

• Add a shared IConfigurationProvider implementation to load/refresh secrets from Vault via Kubernetes or AppRole auth.
• Add Vault deployment artifacts: Helm values for Kubernetes HA Raft and an Azure Container Instances (ACI) option via Bicep, wired through a new secretProvider parameter.
• Add automation (Vault setup + package injection) and expand docs/security guidance for Vault usage.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
src/shared/Configuration/VaultConfigurationExtensions.cs	Adds Vault-backed configuration source/provider with optional periodic reload.
src/shared/Configuration/README.md	Documents app integration steps and expected secret structure.
scripts/setup-vault.sh	Automates Vault init/unseal/auth/policy setup and initial secret seeding.
scripts/add-vault-packages.sh	Adds VaultSharp packages to all microservice .csproj files.
infra/vault/values.yaml	Helm values for Vault HA Raft + TLS + injector + PDB/affinity.
infra/modules/vault-aci.bicep	Bicep module to run Vault in Azure Container Instances with Azure Files persistence.
infra/main.bicep	Adds secretProvider selector and conditional Vault/Key Vault deployment.
infra/k8s/vault-serviceaccount.yaml	ServiceAccounts + token review binding + Vault config ConfigMap for Kubernetes auth.
docs/security/HASHICORP-VAULT-ISSUE-SUMMARY.md	High-level implementation summary and operational checklist.
docs/security/HASHICORP-VAULT-INTEGRATION.md	Detailed Vault deployment/integration guide.
SECURITY.md	Updates secrets management section to include Vault guidance.
.github/workflows/deploy.yml	Adds Vault→Key Vault→GitHub Secrets retrieval flow for deployment secrets.

[Copilot]

Vault KV v2 paths are being passed with a "secret/..." prefix while also hard-coding mountPoint: "secret". With VaultSharp, the V2 path is typically relative to the mount (e.g., "cloudhealthoffice"), so this will likely call "secret/metadata/secret/..." and fail to list/read secrets. Consider splitting mount point from the configured base path (default mountPoint="secret"), and normalize BasePath to be mount-relative (strip leading "secret/" and "secret/data/").

[Copilot]

LoadAsync always assigns Data = data even when the Vault load fails, which clears previously loaded secrets on transient outages. This contradicts the “continues with existing secrets” behavior described in docs. Preserve the previous Data on failure (only overwrite Data when a full load succeeds), or merge new values into existing Data.

[Copilot]

This writes auth/kubernetes/config using kubernetes_ca_cert=@/var/run/secrets/... and token_reviewer_jwt=@/var/run/secrets/..., but the command is executed on the operator machine (via port-forward), where those file paths typically don’t exist. Run this vault write from داخل a Vault pod (kubectl exec) or fetch the SA CA/token via kubectl and pass the contents explicitly; otherwise Kubernetes auth setup will fail.

Suggested change

	# Configure Kubernetes auth
	vault write auth/kubernetes/config \
	kubernetes_host="$KUBERNETES_HOST" \
	kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
	token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
	# Determine Vault namespace and service account used for Kubernetes auth
	VAULT_NAMESPACE="${VAULT_NAMESPACE:-vault}"
	VAULT_SA_NAME="${VAULT_SA_NAME:-vault}"
	# Fetch service account secret containing token and CA certificate
	SA_SECRET_NAME=$(kubectl get sa "${VAULT_SA_NAME}" -n "${VAULT_NAMESPACE}" -o jsonpath='{.secrets[0].name}')
	SA_CA_CRT=$(kubectl get secret "${SA_SECRET_NAME}" -n "${VAULT_NAMESPACE}" -o jsonpath='{.data.ca\.crt}' | base64 --decode)
	SA_TOKEN=$(kubectl get secret "${SA_SECRET_NAME}" -n "${VAULT_NAMESPACE}" -o jsonpath='{.data.token}' | base64 --decode)
	# Configure Kubernetes auth using explicit CA cert and token contents
	vault write auth/kubernetes/config \
	kubernetes_host="$KUBERNETES_HOST" \
	kubernetes_ca_cert="$SA_CA_CRT" \
	token_reviewer_jwt="$SA_TOKEN" \

[Copilot]

Port-forward is started in the background and only killed on the success path. If any command fails after starting it (set -e), the background process can be left running. Add a trap to always kill $PORT_FORWARD_PID on EXIT/ERR once it’s set.

[Copilot]

This script uses sed -i without a backup suffix, which works on GNU sed but fails on macOS/BSD sed. Other repo scripts use sed -i.bak and remove the backup afterwards; consider matching that pattern here for portability.

Suggested change

	sed -i '/<\/ItemGroup>/i \ <PackageReference Include="VaultSharp" Version="'"$VAULTSHARP_VERSION"'" />\n <PackageReference Include="VaultSharp.Extensions.Configuration" Version="'"$VAULTSHARP_VERSION"'" />' "$CSPROJ_FILE"
	sed -i.bak '/<\/ItemGroup>/i \ <PackageReference Include="VaultSharp" Version="'"$VAULTSHARP_VERSION"'" />\n <PackageReference Include="VaultSharp.Extensions.Configuration" Version="'"$VAULTSHARP_VERSION"'" />' "$CSPROJ_FILE"
	rm "$CSPROJ_FILE.bak"

[Copilot]

Vault UI is configured as a public LoadBalancer (azure-load-balancer-internal: "false"). Exposing Vault’s UI/API publicly by default is a high-risk posture for a secrets system. Prefer ClusterIP with an ingress restricted by IP allowlists/auth, or at minimum default to an internal LB and document how to intentionally expose it.

Suggested change

	serviceType: LoadBalancer # Change to ClusterIP for ingress
	serviceNodePort: null
	externalPort: 8200
	# Annotations for Load Balancer
	annotations:
	service.beta.kubernetes.io/azure-load-balancer-internal: "false" # Public LB for access
	# For AWS: service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
	# For GCP: cloud.google.com/load-balancer-type: "External"
	serviceType: ClusterIP # Secure default: internal access; use ingress or override to expose externally
	serviceNodePort: null
	externalPort: 8200
	# Annotations for Service / Load Balancer
	# Default: no public load balancer exposure. Use ingress with IP allowlists/auth for external access.
	annotations: {}
	# To use an internal Azure Load Balancer instead of ClusterIP, override these values:
	# ui:
	# serviceType: LoadBalancer
	# annotations:
	# service.beta.kubernetes.io/azure-load-balancer-internal: "true" # Internal LB
	# For AWS Network Load Balancer (example only, not enabled by default):
	# ui:
	# serviceType: LoadBalancer
	# annotations:
	# service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
	# For GCP external load balancer (example only, not enabled by default):
	# ui:
	# serviceType: LoadBalancer
	# annotations:
	# cloud.google.com/load-balancer-type: "External"

[Copilot]

enableDeploymentKeyVault and enableHashiCorpVault are still declared as overridable parameters even though secretProvider is intended to be the single selector. This allows inconsistent combinations (e.g., secretProvider=hashicorpvault but enableDeploymentKeyVault=true) and makes deployments harder to reason about. Consider making these vars derived from secretProvider (or explicitly document/validate precedence).

Suggested change

	param enableDeploymentKeyVault bool = (secretProvider == 'azurekeyvault')
	param deploymentKeyVaultName string = '${baseName}-deploy-kv'
	// HashiCorp Vault parameters (when secretProvider = hashicorpvault)
	param enableHashiCorpVault bool = (secretProvider == 'hashicorpvault')
	param vaultContainerName string = 'vault-${baseName}'
	param vaultDnsLabel string = 'cho-vault-${uniqueString(resourceGroup().id)}'
	param vaultVersion string = '1.15.4'
	// =========================
	// Variables
	param deploymentKeyVaultName string = '${baseName}-deploy-kv'
	// HashiCorp Vault parameters (when secretProvider = hashicorpvault)
	param vaultContainerName string = 'vault-${baseName}'
	param vaultDnsLabel string = 'cho-vault-${uniqueString(resourceGroup().id)}'
	param vaultVersion string = '1.15.4'
	// =========================
	// Variables
	var enableDeploymentKeyVault = (secretProvider == 'azurekeyvault')
	var enableHashiCorpVault = (secretProvider == 'hashicorpvault')

[Copilot]

The Vault ACI configuration disables TLS for the Vault listener (tls_disable = 1) and binds it to 0.0.0.0:8200, which means all Vault traffic (including tokens and secrets) is exposed in cleartext over HTTP. An attacker on the network or the public internet (given the public IP on this container group) can eavesdrop on or tamper with Vault API calls and potentially steal credentials or data. Configure Vault to require TLS (tls_disable = 0), use HTTPS addresses for api_addr/cluster_addr, and restrict exposure to a private network or internal endpoint instead of a public HTTP listener.

[Copilot]

The setup script prints the Vault root_token directly to standard output, which can end up in shell history or centralized logs and exposes the highest‑privilege credential for the Vault cluster. Anyone with access to those logs could fully compromise Vault, read or modify all secrets, and change policies. Avoid ever logging the root token; instead, instruct operators to retrieve it manually from a secure file or password manager and ensure it is only stored in dedicated secure secret storage.

Suggested change

	echo " Token: $(jq -r '.root_token' vault-keys.json 2>/dev/null || echo 'See vault-keys.json')"
	echo " Token: Retrieve the initial root token securely from vault-keys.json or your secret store (never expose it in logs or shell history)"

[Copilot]

The cho-microservices Vault policy and Kubernetes role allow any bound service account to read and list all secrets under secret/data/cloudhealthoffice/* and use the PHI transit key, so compromise of a single microservice pod yields access to every application secret. This breaks least‑privilege isolation between services and significantly amplifies the impact of any RCE or token theft. Define per‑service policies and roles that restrict each service account to only the specific secret paths and transit operations it requires instead of a shared, wide‑open policy.