v1.0.0 - Production! Acoustic batched processing, metrics

This release updates how the Acoustic synchronization job processes large backlogs, and adds metrics.

Tag v0.8.3 has been running in production without Salesforce for a few weeks, so we're bumping the version number to 1.0.0. Scripts used during the final import have been updated in this release.

Acoustic Synchronization Job

• Pending updates are now processed in batches, rather than all pending updates. This avoids long processing runtime without feedback. The default is 20 updates per batch.
• Prometheus metrics are pushed to the pushgateway, if configured.

Deployments

• Two new environment variables to tune the Acoustic Synchronization Job:
  • ACOUSTIC_BATCH_LIMIT - set the number of updates per batch
  • PROMETHEUS_PUSHGATEWAY_URL - set the URL of the Prometheus push gateway
• New metrics are available, if configured:
  • ctms_background_acoustic_request_total - Total count of acoustic requests by method and status
  • ctms_background_acoustic_requests_duration - Histogram of requests processing time by method (in seconds)
  • ctms_background_acoustic_sync_total - Total count of contacts synced to acoustic
  • ctms_background_acoustic_sync_retries - Gauge of pending records with >0 retries to acoustic
  • ctms_background_acoustic_sync_backlog - Gauge of the number of contacts in the sync backlog. Not counting over-retried records.

Other Changes

• The import script scripts/importers/setup.sql and scripts/importers/finish.sql includes updates for the final import, such as index dropping and creation, case-insensitive duplicate email dropping, and newsletter source column cleanup.