v1.3.0: DB Model Migration

What's Changed

• Version Bump to new minor version by @bsieber-mozilla in #364
• Add leplatrem to Reviewer List by @bsieber-mozilla in #365
• Adding db migration by @bsieber-mozilla in #358
• Version v1.3.0: db migration by @bsieber-mozilla in #367

Full Changelog: v1.2.0...v1.3.0

v1.2.0

What's Changed

• (v1.1.4) Resolve when receiving None value: in #355
• (v1.1.4) Version bump: 1.1.4: #357
• Adding new newsletter: #361
• Newsletter mapping update: #363
• Version Bump to new minor version: #364

Full Changelog: v1.1.4...v1.2.0

v1.1.4

V1.1.4 Release

Enables graceful acknowledgement of webhook payloads that have null values.

What's Changed

• Resolve when receiving None value in #355
• Version bump: 1.1.4 in #357

Full Changelog: v1.1.3...v1.1.4

v1.1.3 - Bugfix for ingesting new deleted Stripe customer

This release fixes a bug when PubSub sends a new-to-CTMS Stripe customer as deleted.

API

• When a Stripe customer object is a deleted customer, and CTMS does not have an existing record for that customer, CTMS returns a 200 OK response and does not add the customer to the database. Previously, this was a 500 Server Error due to CTMS trying to examine the database object, which was None. Fixes #351.

Other

• Update docs/diagrams/ctms_general_overview.svg to make it clear that Basket does not interact directly with Cinchy in the context of the CTMS architecture.

v1.1.2 - Improvements to Stripe object ingestion

Improvements to POST /stripe_from_pubsub, for better behavior with production-level traffic.

API

• POST /stripe_from_pubsub and POST /stripe return a 409 Conflict if the changes fail due to a database error, such as an IntegrityError due to duplicate IDs, or a deadlock. Previously, these returned 500 Server Error.
• If a Stripe customer is submitted that has the same Firefox Account ID (FxA ID) as an existing Stripe customer, the existing Stripe customer is deleted. This was seen on stage, but not production, and may be due to a bug or direct interaction with Stripe. The deletion most closely matches what we believe happens in the FxA Stripe cache, which is indexed by FxA ID.
• Stripe ingest now correctly updates invoice.default_source_id and invoice_line_item.stripe_subscription_item_id.

Deployments

• Database changes that may impact request timing and success:
  • Stripe ingest endpoints now use SELECT ... FOR UPDATE. This will hopefully reduce write conflicts, but may lead to increased deadlocks.
  • Duplicate FxA ID detection adds an extra database SELECT to customer creation and some customer updates (when the FxA ID changes), potentially slowing requests.
  • Stripe ingest updates now correctly parse Stripe timestamps as UTC timestamps rather than timezone-naive timestamps, which avoids database writes that do not change the data.
• Log changes:
  • Structured request logs for /stripe_from_pubsub and /stripe have context changes:
    • Added fxa_id_conflict, listing the FxA ID that was found on a new and existing Stripe customer.
    • Added ingest_actions to detail the contents of the Stripe payload. The keys are the action taken: created, updated, no_change, deleted, and skipped), and the values are a list of objects, represented as object_type:object_id, such as ["subscription:sub_abc123"].
    • Removed stripe_unknown. Unknown objects are now found in ingest_actions["skipped"].
  • An error-level log ("Severity": 3) is emitted with message "IntegrityError converted to 409", or "OperationalError converted to 409", for database exceptions handled by the Stripe endpoints. The log contains the context, and they are no longer sent to Sentry. This is followed by the standard request log for the 409 Conflict returned to the caller.
  • In the Acoustic sync service log message "sync_service cycle complete", the context retry_backlog now has the correct value. Previously, it was a duplicate of the sync_backlog value. The associated metric gauge ctms_background_acoustic_sync_retries had the correct value.

Other

• Updated from Python 3.9.7 to 3.9.9.
• Updated fastapi from 0.65.3 to 0.70.0, starlette from 0.14.2 to 0.16.0, and lxml from 4.6.4 to 4.7.1.
• Updated several documents:
  • Updated overview architecture diagram for Cinchy interaction and Stripe ingestion from FxA.
  • Added a Stripe entity diagram.
  • Synced docs/configuration.md with the current environment configuration, including adding the Acoustic Sync configuration variables.
  • Updated docs/deployment_guide.md, syncing with the current deployment methods, updating the logging section, and adding overview, metrics, and dashboards sections.
  • Updated docs/developer_setup.md with an "Updating Dependencies" section.

v1.1.1 - Bugfix for product segments

This release fixes a bug when generating the product segment for some users.

API Changes

• When a Stripe customer has multiple subscriptions to a product, and the latest status was a failure such as incomplete_expired, the code attempted to set the product segment to re-other, which was an invalid value. In these cases, the segment is now other, the same as when they have a single subscription with a status other than active or canceled.

v1.1.0 - Stripe objects and product subscriptions

This release adds the ability to ingest Stripe objects from the Firefox Accounts (FxA) Firestore cache via a PubSub queue. These are processed to determine the product subscriptions for a contact, and these are synced to a relational table in Acoustic. The product subscriptions are not exposed on the contact in the API.

API

• A new endpoint, POST /stripe, takes Stripe objects and adds them to the CTMS database. The supported objects are customer, subscription, and invoice. This endpoint takes CTMS OAuth2 credentials.
• A new endpoint, POST /stripe_from_pubsub, takes PubSub push requests with a Stripe object, or dictionary of keys to Stripe objects, as payload. This endpoint checks the Javascript Web Token (JWT) authentication header, and verifies the claimed audience and email. The endpoint also takes a client "secret" as a URL parameter. This endpoint returns 202 for content issues to prevent PubSub from submitting again.
• Loading contacts now loads the related Stripe data, and converts them to products. This will increase the number of database requests to read or update a contact.

Acoustic Sync Service

• A contact's product subscriptions are synced to a new Acoustic relational table. This includes placeholder columns for future subscription data.
• The Acoustic sync service does not sleep if it processed a full batch of contacts, to speed up processing a backlog of contacts.
• Added a timeout to Acoustic requests, with a default of 5.0 seconds. If the timeout is reached, syncing fails for that contact and it is retried later.

Deployments

• The database includes new tables for Stripe data, added by migrations: stripe_customer, stripe_price, stripe_invoice, stripe_invoice_line_item, stripe_subscription, and stripe_subscription_item. The primary key is the stripe_id column. The tables refer to each other - stripe_subscription.stripe_customer_id refers to a stripe_customer.stripe_id - but foreign keys are not used because the data may come in an unexpected order from FxA.
• The API __heartbeat__ endpoint now includes details of the Acoustic sync backlog. Optional settings sets maximum levels for the backlog and the retry backlog, to make the heartbeat fail. The default is no maximum.
• The API process now reads the background process settings from environment variables as well. Some are reported in the __heartbeat__ endpoint.
• The background sync service can optionally write the current time to a file, at startup and once per loop. This can be checked by a new process ctms/bin/healthcheck_sync.py as Kubernetes startup and liveness check.
• Environment Variables:
  • Added CTMS_PUBSUB_AUDIENCE and CTMS_PUBSUB_EMAIL, to validate the JWT claim for POST /stripe_from_pubsub.
  • Added CTMS_PUBSUB_CLIENT, checked against the query string parameter in POST /stripe_from_pubsub?pubsub_client=<client_id>.
  • Added CTMS_ACOUSTIC_PRODUCT_SUBSCRIPTIONS_ID, required in the background process, for the product relational table ID.
  • Added optional CTMS_ACOUSTIC_MAX_BACKLOG and CTMS_ACOUSTIC_MAX_RETRY_BACKLOG. If set, __healthcheck__ will fail if the backlog or the retry backlog exceeds these limits.
  • Added optional CTMS_BACKGROUND_HEALTHCHECK_PATH and CTMS_BACKGROUND_HEALTHCHECK_AGE_S. If the path is set, the background process will write the current timestamp. If both are set, ctms/bin/healthcheck_sync.py will read the timestamp file and exit with a failing code if it is older than the age in seconds.
  • Added optional CTMS_ACOUSTIC_TIMEOUT_S, to set the timeout for requests to Acoustic. The default is 5.0 seconds.
• Metrics updates:
  • The new counter ctms_pending_acoustic_sync_total is incremented when an Acoustic sync is scheduled, from an existing endpoint like POST /ctms or PATCH /ctms/<email_id>, as well as the new Stripe ingest endpoints.
  • The ctms_background_acoustic_requests_duration and ctms_background_acoustic_sync_loopsmetrics now include tag table, to identify the table synced (main for the main contact table, newsletter and product for the relational tables).
  • The new counter ctms_background_acoustic_sync_loops increments when a sync loop completes processing a batch of contacts and before sleeping (if requested). This can be used to detect if the sync process is stuck.
  • The new gauge ctms_background_acoustic_sync_age_s gives the age of the sync request for the last synced item that was not re-queued for retrying. This can be used to determine the impact of Acoustic API slowdowns or large backlogs.
• Log updates:
  • The background process now emits structured logs, and the log lines have been reduced.
  • The background process emits on INFO message at startup, "Setting up sync_service.", with the sync_feature_flag in context.
  • The background process emits one INFO message per loop, "sync_service cycle complete". The log context includes:
    • How many contacts were synced, and and the count by sync status.
    • "trivial": true if no contact were synced.
    • The duration of the loop, and the planned sleep duration.
  • The background process emits one DEBUG message per contact ("Successfully sync'd contact to acoustic..." or "Failure for contact in sync to acoustic..."). The log context includes:
    • The email_id.
    • The email address, if a contact's email matches the +trace_me_mozilla_ pattern.
    • The names of skipped columns, except for known columns, such as update_timestamp, which are silently skipped.
    • If the fxa_created date was successfully parsed into a datetime, or what went wrong.
    • The slugs of any skipped newsletters.
    • The status and duration of Acoustic sync requests.
    • The count of rows for the newsletter and product relational tables.
  • The new Stripe endpoints log the payload if the Stripe object has an email that matches the +trace_me_mozilla_ pattern

Other

• Added adminer to the development database as postgres-admin, to allow viewing the database.
• Added new script ctms/bin/ingest_stripe_data.py that can import one or more Stripe objects from a JSON file.
• Updated to Python 3.9.7. The accepted range is 3.7.x to 3.10.x (raised from 3.9.x).
• The PostgreSQL client psycopg2 is now built from source rather than installed as a wheel, meaning that libpq5 is shipped in the deployment object, and development libraries are needed when building on a local developer's machine. This allows an arm64 build for Apple Silicon.
• Updated several dependencies, such as fastapi 0.65.3, alembic 1.7.5, google-cloud-core 2.2.1, psycopg2 2.9.2, and uvicorn 0.15.0.
• Updated several development tools, such as black 21.10b0, bandit 1.7.1, mypy 0.910, pylint 2.12.1, and black 21.11.b1.
• Switched pre-commit to the Poetry environment, to avoid out-of-date dependencies.
• Moved documentation from guides/ to docs/, and refreshed and reworded documentation.
• Added docs/adrs for Architectural Decision Records, with ADR for Stripe syncing.
• Moved scripts/lint.sh to docker/lint.sh and scripts/test.sh to docker/test.sh. Removed some unused scripts.
• Removed auto-documentation stubs and documentation deploy to Github pages.
• Set CODEOWNERS from a team to the current development staff .

v1.0.2 - Adding a newsletter!

This really shouldn't require a release!

Fixing of Date-Formatting in CTMS to Acoustic Sync

This version includes changes desired by Marketing to enable time-based queries for VPN-based offers.

The data in Acoustic previously was not queryable as it was in string-timestamps that Acoustic did not understand.

v1.0.0 - Production! Acoustic batched processing, metrics

This release updates how the Acoustic synchronization job processes large backlogs, and adds metrics.

Tag v0.8.3 has been running in production without Salesforce for a few weeks, so we're bumping the version number to 1.0.0. Scripts used during the final import have been updated in this release.

Acoustic Synchronization Job

• Pending updates are now processed in batches, rather than all pending updates. This avoids long processing runtime without feedback. The default is 20 updates per batch.
• Prometheus metrics are pushed to the pushgateway, if configured.

Deployments

• Two new environment variables to tune the Acoustic Synchronization Job:
  • ACOUSTIC_BATCH_LIMIT - set the number of updates per batch
  • PROMETHEUS_PUSHGATEWAY_URL - set the URL of the Prometheus push gateway
• New metrics are available, if configured:
  • ctms_background_acoustic_request_total - Total count of acoustic requests by method and status
  • ctms_background_acoustic_requests_duration - Histogram of requests processing time by method (in seconds)
  • ctms_background_acoustic_sync_total - Total count of contacts synced to acoustic
  • ctms_background_acoustic_sync_retries - Gauge of pending records with >0 retries to acoustic
  • ctms_background_acoustic_sync_backlog - Gauge of the number of contacts in the sync backlog. Not counting over-retried records.

Other Changes

• The import script scripts/importers/setup.sql and scripts/importers/finish.sql includes updates for the final import, such as index dropping and creation, case-insensitive duplicate email dropping, and newsletter source column cleanup.