Add PostgreSQL WAL listener component

AGENTS.md

@@ -32,7 +32,7 @@ lfx-v1-sync-helper/
 │   ├── meltano.yml            # Main Meltano configuration
 │   └── load/target-nats-kv/   # Custom NATS KV target plugin
 ├── cmd/lfx-v1-sync-helper/    # Go microservice source
-├── charts/lfx-v1-sync-helper/ # Helm deployment charts
+├── charts/lfx-v1-sync-helper/ # Helm deployment charts (Chart.yaml version is dynamic on release)
 ├── docker/                    # Docker build configurations
 │   ├── Dockerfile.v1-sync-helper  # Go service container
 │   └── Dockerfile.meltano         # Python ETL container
@@ -142,7 +142,7 @@ lfx-v1-sync-helper/
 
 #### Data Format Support
 - **JSON** (default): Standard JSON encoding for record storage
-- **MessagePack**: Compact binary serialization with `msgpack: true` configuration
+- **MessagePack**: Compact binary serialization with `msgpack: true` configuration (Meltano) or `USE_MSGPACK=true` (WAL handler)
 - **Automatic Detection**: Both Go service and Python plugin automatically detect format when reading existing data
 
 ## CI/CD Integration
@@ -189,8 +189,11 @@ lfx-v1-sync-helper/
 ### Data Serialization
 - **target-nats-kv** supports both JSON and MessagePack encoding
 - Set `msgpack: true` in Meltano configuration to enable MessagePack
+- Set `USE_MSGPACK=true` environment variable for WAL handler to use MessagePack
+- Boolean environment variables accept truthy values: "true", "yes", "t", "y", "1" (case-insensitive)
 - Automatic format detection when reading existing data for compatibility
 - Go service handles both formats transparently
+- WAL handler respects the same encoding configuration as Meltano for consistency
 
 ### Container Standards
 - Multi-stage builds for size optimization

README.md

@@ -1,6 +1,6 @@
 # Data sync components for LFX One
 
-This repository contains tools and services for synchronizing data between LFX v1 and LFX One (v2) platforms. This solution uses Meltano for data replication into the v2 ecosystem, after which a sync helper service handles data mapping and ingestion.
+This repository contains tools and services for synchronizing data between LFX v1 and LFX One (v2) platforms. This solution uses Meltano for data extraction and loading, a WAL listener for real-time PostgreSQL change streaming, and a sync helper service that handles data mapping and ingestion into the v2 ecosystem.
 
 ## Overview
 
@@ -69,12 +69,12 @@ Data extraction and loading pipeline that extracts data from LFX v1 sources (Dyn
 Go service that monitors NATS KV stores for replicated v1 data and synchronizes it with the LFX v2 platform APIs, handling data transformation and conflict resolution.
 
 ### [Helm charts](./charts/lfx-v1-sync-helper/README.md)
-Kubernetes deployment manifests for the v1-sync-helper service, providing scalable deployment options for production environments.
+Kubernetes deployment manifests for the custom app service and WAL listener component, providing scalable deployment options for production environments.
 
 ## Architecture Diagrams
 
 Regarding the following diagrams:
-- The planned realtime sync for PostgreSQL is included in the diagrams.
+
 - The DynamoDB source (incremental or realtime) is not currently included in the diagrams.
 - The planned bidirectional sync (LFX One changes back to v1) is included in the diagrams.
 - "Projects API" is representative of most data entities. However, v1 Meetings push straight to OpenSearch and OpenFGA (via platform services)—this is not shown.
@@ -134,7 +134,7 @@ sequenceDiagram
     participant opensearch as OpenSearch
 
     v1_kv-)+v1-sync-helper: notification on KV bucket subject
-    v1-sync-helper->>v1-sync-helper: check if delete or upsert
+    v1-sync-helper->>v1-sync-helper: check if delete (hard or soft) or upsert
     v1-sync-helper->>v1-sync-helper: check if upsert was by v1-sync-helper's M2M client ID
     v1-sync-helper->>+mapping-db: check for v1->v2 ID mapping
     mapping-db--)-v1-sync-helper: v2 ID, deletion tombstone, or empty
@@ -155,8 +155,8 @@ sequenceDiagram
     projects-api -) opensearch: index deletion transection (via indexer)
     Note right of v1-sync-helper: if the DELETE fails, notify team and abort
     projects-api --)- v1-sync-helper: 204 (no body)
-    v1-sync-helper -) mapping-db: delete v1->v2 mapping
-    v1-sync-helper -) mapping-db: delete v2->v1 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
     else item upsert & NOT last-modified-by v1-sync-helper & mapping empty
     Note right of v1-sync-helper: This is a "create" from v1
     v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
@@ -291,8 +291,8 @@ sequenceDiagram
     projects-api -) opensearch: index deletion transection (via indexer)
     Note right of v1-sync-helper: if the DELETE fails, notify team and abort
     projects-api --)- v1-sync-helper: 204 (no body)
-    v1-sync-helper -) mapping-db: delete v1->v2 mapping
-    v1-sync-helper -) mapping-db: delete v2->v1 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
     else item upsert & NOT last-modified-by v1-sync-helper & mapping empty
     Note right of v1-sync-helper: This is a "create" from v1
     v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
@@ -339,8 +339,8 @@ sequenceDiagram
     mapping-db--)-v1-sync-helper: v1 ID
     v1-sync-helper->>+lfx_v1: delete in v1
     lfx_v1->>-v1-sync-helper: 204 (no content)
-    v1-sync-helper -) mapping-db: delete v1->v2 mapping
-    v1-sync-helper -) mapping-db: delete v2->v1 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
+    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
     end
     deactivate v1-sync-helper
 ```

charts/lfx-v1-sync-helper/README.md

@@ -1,6 +1,6 @@
 # LFX v1 Sync Helper Helm Chart
 
-This Helm chart deploys the LFX v1 Sync Helper service, which monitors NATS KV stores for v1 data and synchronizes it with the LFX v2 platform APIs, handling data transformation and conflict resolution.
+This Helm chart deploys the LFX v1 Sync Helper service, which monitors NATS KV stores for v1 data and synchronizes it with the LFX v2 platform APIs, handling data transformation and conflict resolution. The chart also includes an optional PostgreSQL WAL listener component for real-time database change streaming.
 
 ## Prerequisites
 
@@ -32,11 +32,11 @@ kubectl create secret generic v1-sync-helper-auth0-credentials \
 # Install the chart with required image tag and AUTH0_TENANT
 helm install -n lfx lfx-v1-sync-helper \
     ./charts/lfx-v1-sync-helper \
-    --set image.tag=latest \
+    --set app.image.tag=latest \
     --set app.environment.AUTH0_TENANT.value=my_tenant
 ```
 
-**Note**: When using the local chart, you must specify `--set image.tag=latest` because the committed chart does not have an appVersion, so a version must always be specified when not using the published chart. The AUTH0_TENANT environment variable and Auth0 secret are also required.
+**Note**: When using the local chart, you must specify `--set app.image.tag=latest` because the committed chart does not have an appVersion, so a version must always be specified when not using the published chart. The AUTH0_TENANT environment variable and Auth0 secret are also required.
 
 ### Installing from OCI registry
 
@@ -52,6 +52,13 @@ kubectl create secret generic v1-sync-helper-auth0-credentials \
     --from-literal=client_private_key="$(cat auth0-private-key.pem)" \
     -n lfx
 
+# Create PostgreSQL credentials secret (for wal-listener component)
+kubectl create secret generic v1-platform-db-credentials \
+    --from-literal=host=your-postgres-host \
+    --from-literal=username=your-postgres-user \
+    --from-literal=password=your-postgres-password \
+    -n lfx
+
 # Create values.yaml with required AUTH0_TENANT
 cat > values.yaml << EOF
 app:
@@ -96,9 +103,19 @@ The chart requires the following secrets to be created before installation (if t
        -n lfx
    ```
 
-### Environment Variables
+3. **PostgreSQL credentials** (default name: `v1-platform-db-credentials`):
+   Required for the WAL listener component to connect to the PostgreSQL database.
+   ```bash
+   kubectl create secret generic v1-platform-db-credentials \
+       --from-literal=host=your-postgres-host \
+       --from-literal=username=your-postgres-user \
+       --from-literal=password=your-postgres-password \
+       -n lfx
+   ```
+
+### App Component
 
-The following environment variables have defaults configured in the chart's `app.environment` section:
+The following environment variables for the custom app component have defaults configured in the chart's `app.environment` section:
 
 | Variable                | Default                                                                    | Description               |
 |-------------------------|----------------------------------------------------------------------------|---------------------------|
@@ -113,7 +130,49 @@ The following environment variables have defaults configured in the chart's `app
 
 For a complete list of all supported environment variables, including required ones like `AUTH0_TENANT`, see the [v1-sync-helper README](../../cmd/lfx-v1-sync-helper/README.md#environment-variables).
 
+### WAL Listener Component
+
+The chart includes an optional PostgreSQL WAL (Write-Ahead Log) listener component that provides real-time streaming of database changes to NATS. This component is enabled by default and can be configured or disabled as needed.
+
+#### WAL Listener Configuration
+
+| Parameter                                 | Default                                        | Description                            |
+|-------------------------------------------|------------------------------------------------|----------------------------------------|
+| `walListener.enabled`                     | `true`                                         | Enable/disable WAL listener deployment |
+| `walListener.replicas`                    | `1`                                            | Number of WAL listener replicas        |
+| `walListener.image.repository`            | `ihippik/wal-listener`                         | WAL listener container image           |
+| `walListener.image.tag`                   | `latest`                                       | WAL listener image tag                 |
+| `walListener.config.listener.slotName`    | `lfx_v2`                                       | PostgreSQL replication slot name       |
+| `walListener.config.database.secret.name` | `v1-platform-db-credentials`                   | Secret containing database credentials |
+| `walListener.config.publisher.address`    | `lfx-platform-nats.lfx.svc.cluster.local:4222` | NATS server address                    |
+| `walListener.config.publisher.topic`      | `wal_listener`                                 | NATS topic for publishing changes      |
+
+The WAL listener monitors the following PostgreSQL tables by default (matching the meltano.yml tap-postgres configuration):
+- `collaboration__c` (platform schema)
+- `community__c` (platform schema)
+- `project__c` (salesforce schema)
+- `alternate_email__c` (salesforce schema)
+- `merged_user` (salesforce schema)
+
+To disable the WAL listener:
+```yaml
+walListener:
+  enabled: false
+```
+
+To customize monitored tables:
+```yaml
+walListener:
+  config:
+    listener:
+      filter:
+        tables:
+          your_table:
+            - insert
+            - update
+            - delete
+```
+
 ### Additional Configuration
 
 For all available configuration options and their default values, please see the [values.yaml](values.yaml) file in this chart directory. You can override these values in your own `values.yaml` file or by using the `--set` flag when installing the chart.
-

charts/lfx-v1-sync-helper/templates/deployment.yaml → charts/lfx-v1-sync-helper/templates/app-deployment.yaml

@@ -4,29 +4,31 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: {{ .Chart.Name }}
+  name: {{ .Chart.Name }}-app
   namespace: {{ .Release.Namespace }}
   {{- with .Values.annotations }}
   annotations:
     {{- toYaml . | nindent 4 }}
   {{- end }}
 spec:
-  replicas: {{ .Values.replicas | default 1 }}
+  replicas: {{ .Values.app.replicas | default 1 }}
   selector:
     matchLabels:
       app: {{ .Chart.Name }}
+      component: app
   template:
     metadata:
       labels:
         app: {{ .Chart.Name }}
+        component: app
     spec:
       serviceAccountName: {{ .Values.serviceAccount.name | default .Chart.Name }}
       containers:
         - name: app
-          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+          image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag | default .Chart.AppVersion }}"
           securityContext:
             allowPrivilegeEscalation: false
-          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          imagePullPolicy: {{ .Values.app.image.pullPolicy }}
           env:
           {{- range $name, $config := .Values.app.environment }}
           - name: {{ $name }}
@@ -39,23 +41,23 @@ spec:
           {{- end }}
           # JWT configuration from Heimdall
           - name: HEIMDALL_CLIENT_ID
-            value: {{ .Values.heimdall.clientId | quote }}
+            value: {{ .Values.app.heimdall.clientId | quote }}
           - name: HEIMDALL_PRIVATE_KEY
             valueFrom:
               secretKeyRef:
-                name: {{ .Values.heimdall.secret.name }}
-                key: {{ .Values.heimdall.secret.privateKeyKey }}
+                name: {{ .Values.app.heimdall.secret.name }}
+                key: {{ .Values.app.heimdall.secret.privateKeyKey }}
           # Auth0 configuration for v1 API authentication
           - name: AUTH0_CLIENT_ID
             valueFrom:
               secretKeyRef:
-                name: {{ .Values.auth0.secret.name }}
-                key: {{ .Values.auth0.secret.clientIdKey }}
+                name: {{ .Values.app.auth0.secret.name }}
+                key: {{ .Values.app.auth0.secret.clientIdKey }}
           - name: AUTH0_PRIVATE_KEY
             valueFrom:
               secretKeyRef:
-                name: {{ .Values.auth0.secret.name }}
-                key: {{ .Values.auth0.secret.privateKeyKey }}
+                name: {{ .Values.app.auth0.secret.name }}
+                key: {{ .Values.app.auth0.secret.privateKeyKey }}
           ports:
             - containerPort: 8080
               name: web