EFIAL-458-config/dependencies and EFIAL-459-partner-sync by rtavernaea · Pull Request #76 · edanalytics/runway

rtavernaea · 2026-06-08T15:42:02Z

This PR addresses EDFIAL-458 and EDFIAL-459.

EDFIAL-458:

Add config for UM & Tx sync
- Texas specific secret & TX_SYNC_CRON
  - I think both syncs could potentially end up using the same SYNC_CRON value but kept them separate for now
PartnerSyncCoordinator handles scheduling and kicking off each sync based on what config values are available.
- Right now it goes through all of the possible handlers each time the app starts up. Alternatively it could check the db first for only sync handlers that are currently implemented. This might be more efficient but would mean we have to add an initial partner to register the sync method.

EDFIAL-459:

Both syncs share the same pg-boss connection

user_management_sync
- Mirrors exsiting connection and transactions from Podium sync
- No children/metatenancy for now
- Partners not sync-managed (i.e., sync_managed: null) are left alone by the sync
tx_sync
- Empty for now. Need to land on the best approach for this between options laid out in EDFIAL-460

snyk-io-us · 2026-06-26T13:14:01Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

edandylytics

A couple more comments.

edandylytics · 2026-06-25T21:27:29Z

+    const url = this.get('UM_URL');
+    const auth0Domain = this.get('UM_AUTH0_DOMAIN');
+    const clientId = this.get('UM_CLIENT_ID');
+    const clientSecret = this.get('UM_CLIENT_SECRET');


Deployed environments will need to look up credentials in an AWS secret. You can check out other methods on this service for how to use a secret and fall back to env vars if a secret doesn't exist. postgresPoolConfig is a good example.

edandylytics · 2026-06-26T14:54:01Z

+        }
+
+        for (const partner of existingPartners) {
+          if (!!partner.managedBy && !partner.deletedOn && !apiPartnerCodes.has(partner.id)) {


Yep -- TX being in the UM results is a clearer case than the one I mentioned. Here's how I think about it: It's Runway's job to determine which partners each sync manages. Runway shouldn't trust external systems to say which partners it can update. That means we need to ensure that it's Runway's internal state (partner.magagedBy) that scopes the sync, not what the external systems report.

edandylytics · 2026-06-26T14:59:35Z

I think we should probably refactor away from this coordinator class. It looks like it does two things:

Initialize PgBoss and manage its lifecycle

Start up the sync

For PgBoss, I expect we'll be using that in other places in the app soon and it'd be helpful to have that as it's own injectable component that can be shared.

Once you remove the PgBoss initialization, do we need a coordinator? It seems each sync service could start itself in its own OnModuleInit. Then we don't need the getSyncConfig wrapper either.

Let me know if I'm missing something there, though.

Yeah I think that makes sense, I added the pg-boss service in its own folder because I'm not sure where else it will be used

edandylytics

Broad strokes look good. Still spending time with it, but a few things for you to check out

edandylytics · 2026-06-30T21:38:37Z

+      host ?? 'localhost'
+    }:${port ?? 5432}/${database}?sslmode=${sslMode}`;
+
+    this.boss = new PgBoss(connStr);


Do we need to construct a connection string? Looks like the constructor accepts an options object that's the same shape as the pool config.

edandylytics · 2026-06-30T21:52:48Z

+
+@Injectable()
+export class PgBossService implements OnModuleInit, OnModuleDestroy {
+  boss!: PgBoss;


The assertion that boss is populated could cause issues down the road since it hides the fact that boss is not really available until after onModuleInit runs. Based on this typing, it looks like you should be able to reference boss in a constructor. Or you might think that you can reference boss in another provider's onModuleInit and not have to figure out whether your provider's init runs before or after PgBossService's.

edandylytics · 2026-06-30T21:55:50Z

+
+  TX_SYNC_CRON?: string;
+  TX_CLIENT_SECRET?: string;
+  TX_CONFIG_SECRET?: string; // deployed envs: name of AWS secret containing TX credentials


Let's keep TX stuff out of this PR

edandylytics · 2026-06-30T21:56:42Z

+  audience: string;
+};
+
+export type TxConfig = {


We can remove this TX stuff too

edandylytics · 2026-06-30T21:58:39Z

+      if (typeof secret !== 'object') {
+        throw new Error(`Value for AWS secret ${configSecret} must be an object`);
+      }
+      if (!syncCron) return null;


Why not allow the caller to default a cron schedule? I expect we'll only really need an env var for local testing and if we happen to get the default wrong for some reason

Yeah makes sense. What makes sense as a default? Every 24 hours?

That sounds good to me. Sometime in the early morning or a bit after UM syncs tenants from Heimdall

SC prod syncs at 10:45 pm, went with 2 here

edandylytics · 2026-06-30T22:03:31Z

+    const syncCron = this.get('UM_SYNC_CRON');
+    const configSecret = this.get('UM_CONFIG_SECRET');
+    if (configSecret) {
+      const secret = await this.getAWSSecret(configSecret);


Let's use fetchAWSSecret instead. getAWSSecret does some caching that I now think is unhelpful since it requires an app restart to pick up new values from the secret. I'd like to move away from getAWSSecret but haven't removed it fully.

edandylytics · 2026-06-30T22:04:38Z

+    if (configSecret) {
+      const secret = await this.getAWSSecret(configSecret);
+      if (typeof secret !== 'object') {
+        throw new Error(`Value for AWS secret ${configSecret} must be an object`);


What would happen if this were to throw?

I added a try catch here to log that the sync didn't get scheduled so the app won't fail to start up if this throws. Not sure if it's necessary to add in the sync() method too

edandylytics

I still need to spend some time with the tests and running locally. Functionally, this looks like it should work well. The Boss module updates are clean and I like how simple it is now.

The one correctness issue we need to update is checking managedBy for undeletes. I'd also like to add a managedBy filter to the DB queries as a backstop against bugs.

The remaining comments are mostly aimed at simplification: reducing the number of intermediate data structures, aligning the form of data structures with how they'll be used, removing layers of indirection. None of these are correctness issues, just ways to simplify and hopefully make the sync easier to update down the road. (And I realize much of this is inherited from Podium! Still, good to tighten up if we can)

edandylytics · 2026-07-01T19:19:21Z

+          tenantsByPartner.set(tenant.partnerId, new Map());
+        }
+        tenantsByPartner.get(tenant.partnerId)!.set(tenant.code, tenant);
+      }


I'm wondering if there's a way to simplify how we construct tenantsByPartner. It feels like a lot of data structure manipulation that all looks correct, but if we can be more concise I'd find it easier to to reason about.

What if

We had one query on partner that does an include on tenants?

Then we just do something like this to create tenantsByPartner

const tenantsByPartner = new Map( existingPartners.map(p => [p.id, new Map(p.tenant.map(t => [t.code, t]))]) )

Now that I've read the full sync, I'm not sure we need tenantsByPartner as it's constructed here. We use it twice:

on line 224 we convert the tenant map back into an array of tenants (so might as well use partner.tenants from an include)

on line 245 we do want a map of tenants, but that's easy enough to just create down there where we use it and skip the partner layer (e.g. const tenantMap = new Map(p.tenant.map(t => [t.code, t]))

edandylytics · 2026-07-02T15:22:36Z

+      this.logger.warn('UM sync config not set — skipping sync');
+      return;
+    }
+    await this.runSync(config);


Why split sync and runSync?

I see that tests call runSync but they need to cast the service as any to invoke the private method, which is a bit of a smell. In general, I'd like tests to focus on testing the public interface rather than private internals as that allows you to use existing tests while refactoring. That consideration doesn't really apply here, but if we're testing the private method, then it's worth questioning whether it really need to be private. (Note that we can mock umConfig to pass mock config to the sync.)

rtavernaea added 22 commits June 5, 2026 08:40

migration

c47ff3a

tests

ab75b20

add fields to created-modified

e148d6d

sync module

37aa864

envs

321debe

alconfig

03ee108

remove erroring line

93bbedc

test file

84e1ef1

spec

a253024

add top level error boundary

c520d39

revisit

faafabf

wip

b165636

remove children

8695346

tests

5b78956

add enum

e701747

merge

a0ea1d8

global

31a1cef

merge

c37b904

fix test

d4d0919

config

405fac6

tx

fd076f4

fix test

a8e3b8d

rtavernaea changed the title ~~Edfial 458~~ EFIAL-458-config/dependencies and EFIAL-459-partner-sync Jun 18, 2026

rtavernaea added 7 commits June 25, 2026 11:06

merge

1ad5dff

rename manager

ec20826

remove duplicate const

323dc4b

remove managedby from tenant

2cd998a

remove managedby

b8ddc88

dont delete non sync managed partners absent from UM

16c6398

iterate over handlers

c4972f2

rtavernaea added 2 commits June 26, 2026 08:15

remove newline

bb8b7fb

test

5a04bd8

edandylytics reviewed Jun 26, 2026

View reviewed changes

rtavernaea added 3 commits June 30, 2026 15:31

use aws secret

cabe3e7

remove coordinator

5478232

move pg boss setvice

6c1cb82

rtavernaea requested a review from edandylytics June 30, 2026 21:00

edandylytics reviewed Jun 30, 2026

View reviewed changes

rtavernaea added 5 commits July 1, 2026 06:11

fetchaws

1f9ec5a

default cron

49f70b8

add tyr catch

0b8f568

remove cron from umconfig

dee569e

default cron

71db26b

edandylytics reviewed Jul 2, 2026

View reviewed changes

rtavernaea added 14 commits July 6, 2026 08:53

missed al->um

54570f0

flip condition for readability

ab87784

make boss private

3a16825

remove tenantsbypartner

d9d2e43

missed rename in test

a30f431

filter and check managedby

70ea096

remove getpartners and gettenants

d399a94

partner array

c46aaea

move apipartnercodes up

2b4e282

promise all for fetch only

2c85bd4

add managedby to where clause

a3c380a

remove bypartner map

5d05e97

remove runsync

5f809f0

refactor tenantsToDelete for readability

b17793b

rtavernaea requested a review from edandylytics July 7, 2026 12:20

Uh oh!

Conversation

rtavernaea commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

EDFIAL-458:

EDFIAL-459:

Uh oh!

snyk-io-us Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

edandylytics left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edandylytics left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edandylytics left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rtavernaea commented Jun 8, 2026 •

edited

Loading

snyk-io-us Bot commented Jun 26, 2026 •

edited

Loading