Skip to content

Commit 6136836

Browse files
dimikotdimikot
committed
v2.21.1: remove accidentally forgotten debug crutches; rename retry-related Cluster options; improve tests
1 parent 3ca471a commit 6136836

18 files changed

+97
-111
lines changed

‎docs/classes/Cluster.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
# Class: Cluster\<TClient, TNode\>
88

9-
Defined in: [src/abstract/Cluster.ts:124](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L124)
9+
Defined in: [src/abstract/Cluster.ts:125](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L125)
1010

1111
Cluster is a collection of Islands and an orchestration of shardNo -> Island
1212
resolution.
@@ -29,7 +29,7 @@ Shard 0 is a special "global" Shard.
2929

3030
> **new Cluster**\<`TClient`, `TNode`\>(`options`): [`Cluster`](Cluster.md)\<`TClient`, `TNode`\>
3131
32-
Defined in: [src/abstract/Cluster.ts:166](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L166)
32+
Defined in: [src/abstract/Cluster.ts:167](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L167)
3333

3434
Initializes the Cluster, but doesn't send any queries yet, even discovery
3535
queries (also, no implicit prewarming).
@@ -57,7 +57,7 @@ queries (also, no implicit prewarming).
5757

5858
> **prewarm**(`randomizedDelayMs`, `onInitialPrewarm`?): `void`
5959
60-
Defined in: [src/abstract/Cluster.ts:291](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L291)
60+
Defined in: [src/abstract/Cluster.ts:292](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L292)
6161

6262
Signals the Cluster to keep the Clients pre-warmed, e.g. open. (It's up to
6363
the particular Client's implementation, what does a "pre-warmed Client"
@@ -87,7 +87,7 @@ pgbouncer or when DB is accessed over SSL).
8787

8888
> **globalShard**(): [`Shard`](Shard.md)\<`TClient`\>
8989
90-
Defined in: [src/abstract/Cluster.ts:322](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L322)
90+
Defined in: [src/abstract/Cluster.ts:323](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L323)
9191

9292
Returns a global Shard of the Cluster. This method is made synchronous
9393
intentionally, to defer the I/O and possible errors to the moment of the
@@ -103,7 +103,7 @@ actual query.
103103

104104
> **nonGlobalShards**(): `Promise`\<readonly [`Shard`](Shard.md)\<`TClient`\>[]\>
105105
106-
Defined in: [src/abstract/Cluster.ts:329](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L329)
106+
Defined in: [src/abstract/Cluster.ts:330](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L330)
107107

108108
Returns all currently known (discovered) non-global Shards in the Cluster.
109109

@@ -117,7 +117,7 @@ Returns all currently known (discovered) non-global Shards in the Cluster.
117117

118118
> **shard**(`id`): [`Shard`](Shard.md)\<`TClient`\>
119119
120-
Defined in: [src/abstract/Cluster.ts:349](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L349)
120+
Defined in: [src/abstract/Cluster.ts:350](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L350)
121121

122122
Returns Shard of a particular id. This method is made synchronous
123123
intentionally, to defer the I/O and possible errors to the moment of the
@@ -149,7 +149,7 @@ the query), no matter whether it was an immediate call or a deferred one.
149149

150150
> **shardByNo**(`shardNo`): [`Shard`](Shard.md)\<`TClient`\>
151151
152-
Defined in: [src/abstract/Cluster.ts:364](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L364)
152+
Defined in: [src/abstract/Cluster.ts:365](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L365)
153153

154154
Returns a Shard if we know its number. The idea: for each Shard number
155155
(even for non-discovered yet Shards), we keep the corresponding Shard
@@ -174,7 +174,7 @@ Shard hasn't been discovered actually).
174174

175175
> **randomShard**(`seed`?): `Promise`\<[`Shard`](Shard.md)\<`TClient`\>\>
176176
177-
Defined in: [src/abstract/Cluster.ts:372](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L372)
177+
Defined in: [src/abstract/Cluster.ts:373](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L373)
178178

179179
Returns a random Shard among the ones which are currently known
180180
(discovered) in the Cluster.
@@ -195,7 +195,7 @@ Returns a random Shard among the ones which are currently known
195195

196196
> **island**(`islandNo`): `Promise`\<[`Island`](Island.md)\<`TClient`\>\>
197197
198-
Defined in: [src/abstract/Cluster.ts:396](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L396)
198+
Defined in: [src/abstract/Cluster.ts:397](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L397)
199199

200200
Returns an Island by its number.
201201

@@ -215,7 +215,7 @@ Returns an Island by its number.
215215

216216
> **islands**(): `Promise`\<[`Island`](Island.md)\<`TClient`\>[]\>
217217
218-
Defined in: [src/abstract/Cluster.ts:407](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L407)
218+
Defined in: [src/abstract/Cluster.ts:408](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L408)
219219

220220
Returns all Islands in the Cluster.
221221

@@ -229,7 +229,7 @@ Returns all Islands in the Cluster.
229229

230230
> **rediscover**(`what`?): `Promise`\<`void`\>
231231
232-
Defined in: [src/abstract/Cluster.ts:417](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L417)
232+
Defined in: [src/abstract/Cluster.ts:418](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L418)
233233

234234
Triggers shards rediscovery and finishes as soon as it's done. To be used
235235
in unit tests mostly, because in real life, it's enough to just modify the

‎docs/classes/PgShardNamer.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
# Class: PgShardNamer
88

9-
Defined in: src/pg/PgShardNamer.ts:9
9+
Defined in: [src/pg/PgShardNamer.ts:9](https://github.com/clickup/ent-framework/blob/master/src/pg/PgShardNamer.ts#L9)
1010

1111
ShardNamer implementation for PG.
1212

@@ -20,7 +20,7 @@ ShardNamer implementation for PG.
2020

2121
> **new PgShardNamer**(`options`): [`PgShardNamer`](PgShardNamer.md)
2222
23-
Defined in: src/abstract/ShardNamer.ts:34
23+
Defined in: [src/abstract/ShardNamer.ts:34](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L34)
2424

2525
Initializes an instance of ShardNamer.
2626

@@ -51,7 +51,7 @@ Initializes an instance of ShardNamer.
5151

5252
> **shardNoByName**(`name`): `null` \| `number`
5353
54-
Defined in: src/abstract/ShardNamer.ts:46
54+
Defined in: [src/abstract/ShardNamer.ts:46](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L46)
5555

5656
Converts a Shard name to Shard number. Returns null if it's not a correct
5757
Shard name.
@@ -76,7 +76,7 @@ Shard name.
7676

7777
> **shardNameByNo**(`no`): `string`
7878
79-
Defined in: src/abstract/ShardNamer.ts:58
79+
Defined in: [src/abstract/ShardNamer.ts:58](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L58)
8080

8181
Builds the Shard name (e.g. for PG, "schema name") by Shard number using
8282
`ShardNamerOptions#nameFormat`.
@@ -103,7 +103,7 @@ E.g. nameFormat="sh%04d" generates names like "sh0042".
103103

104104
> **shardNoByID**(`id`): `number`
105105
106-
Defined in: src/pg/PgShardNamer.ts:14
106+
Defined in: [src/pg/PgShardNamer.ts:14](https://github.com/clickup/ent-framework/blob/master/src/pg/PgShardNamer.ts#L14)
107107

108108
Synchronously extracts Shard number from an ID. Can also extract from PG
109109
composite rows (to support composite IDs).

‎docs/classes/ShardNamer.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
# Class: `abstract` ShardNamer
88

9-
Defined in: src/abstract/ShardNamer.ts:19
9+
Defined in: [src/abstract/ShardNamer.ts:19](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L19)
1010

1111
Client-specific logic on how to synchronously convert an ID into Shard number
1212
(only for the use cases when ID is prefixed with a Shard number), how to
@@ -22,7 +22,7 @@ build Shard names, and how to extract Shard number from a Shard name.
2222

2323
> **new ShardNamer**(`options`): [`ShardNamer`](ShardNamer.md)
2424
25-
Defined in: src/abstract/ShardNamer.ts:34
25+
Defined in: [src/abstract/ShardNamer.ts:34](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L34)
2626

2727
Initializes an instance of ShardNamer.
2828

@@ -49,7 +49,7 @@ Initializes an instance of ShardNamer.
4949

5050
> `abstract` **shardNoByID**(`id`): `number`
5151
52-
Defined in: src/abstract/ShardNamer.ts:24
52+
Defined in: [src/abstract/ShardNamer.ts:24](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L24)
5353

5454
Synchronously extracts Shard number from an ID prefix, for the use cases
5555
where IDs have this information.
@@ -70,7 +70,7 @@ where IDs have this information.
7070

7171
> **shardNoByName**(`name`): `null` \| `number`
7272
73-
Defined in: src/abstract/ShardNamer.ts:46
73+
Defined in: [src/abstract/ShardNamer.ts:46](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L46)
7474

7575
Converts a Shard name to Shard number. Returns null if it's not a correct
7676
Shard name.
@@ -91,7 +91,7 @@ Shard name.
9191

9292
> **shardNameByNo**(`no`): `string`
9393
94-
Defined in: src/abstract/ShardNamer.ts:58
94+
Defined in: [src/abstract/ShardNamer.ts:58](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L58)
9595

9696
Builds the Shard name (e.g. for PG, "schema name") by Shard number using
9797
`ShardNamerOptions#nameFormat`.

‎docs/interfaces/ClusterOptions.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,6 @@ Options for Cluster constructor.
2929
| <a id="shardnamer"></a> `shardNamer?` | `null` \| [`ShardNamer`](../classes/ShardNamer.md) | Info on how to build/parse Shard names. |
3030
| <a id="shardsdiscoverintervalms"></a> `shardsDiscoverIntervalMs?` | `MaybeCallable`\<`number`\> | How often to run Shards rediscovery in normal circumstances. |
3131
| <a id="shardsdiscoverintervaljitter"></a> `shardsDiscoverIntervalJitter?` | `MaybeCallable`\<`number`\> | Jitter for shardsDiscoverIntervalMs and reloadIslandsIntervalMs. |
32-
| <a id="locateislanderrorretrycount"></a> `locateIslandErrorRetryCount?` | `MaybeCallable`\<`number`\> | Used in the following situations: 1. If we think that we know Island of a particular Shard, but an attempt to access it fails, this means that maybe the Shard is migrating to another Island. In this case, we wait a bit and retry that many times. We should not do it too many times though, because all DB requests will be blocked waiting for the resolution. 2. If we sent a write request to a Client, but it appeared that this Client is a replica, and the master moved to some other Client. In this case, we wait a bit and ping all Clients of the Island to refresh, who is master and who is replica. |
33-
| <a id="locateislanderrorrediscoverclusterdelayms"></a> `locateIslandErrorRediscoverClusterDelayMs?` | `MaybeCallable`\<`number`\> | How much time to wait before we retry rediscovering the entire Cluster. The time here should be just enough to wait for switching the Shard from one Island to another (typically quick). |
34-
| <a id="locateislanderrorrediscoverislanddelayms"></a> `locateIslandErrorRediscoverIslandDelayMs?` | `MaybeCallable`\<`number`\> | How much time to wait before sending discover requests to all Clients of the Island trying to find the new master. The time here may reach several seconds, since some DBs shut down the old master and promote some replica to it not simultaneously. |
32+
| <a id="runonsharderrorretrycount"></a> `runOnShardErrorRetryCount?` | `MaybeCallable`\<`number`\> | Used in the following situations: 1. If we think that we know the Island of a particular Shard, but an attempt to access it fails. This means that maybe the Shard is migrating to another Island. So, we wait a bit and retry that many times. We should not do it too many times though, because all DB requests will be blocked waiting for the resolution. 2. If we sent a WRITE request to a Client, but it appeared that this Client is a replica, and the master moved to some other Client. In this case, we wait a bit and ping all Clients of the Island to refresh, who is master and who is replica. |
33+
| <a id="runonsharderrorrediscoverclusterdelayms"></a> `runOnShardErrorRediscoverClusterDelayMs?` | `MaybeCallable`\<`number`\> | How much time to wait before we retry rediscovering the entire Cluster after a Shard-to-Island resolution error. The time here should be just enough to wait for switching the Shard from one Island to another (typically quick). |
34+
| <a id="runonsharderrorrediscoverislanddelayms"></a> `runOnShardErrorRediscoverIslandDelayMs?` | `MaybeCallable`\<`number`\> | How much time to wait before sending discover requests to all Clients of the Island trying to find the new master (or to reconnect). The time here may reach several seconds, since some DBs shut down the old master and promote some replica to it not simultaneously. |

‎docs/interfaces/ShardNamerOptions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
# Interface: ShardNamerOptions
88

9-
Defined in: src/abstract/ShardNamer.ts:6
9+
Defined in: [src/abstract/ShardNamer.ts:6](https://github.com/clickup/ent-framework/blob/master/src/abstract/ShardNamer.ts#L6)
1010

1111
Options for ShardNamer constructor.
1212

‎docs/type-aliases/ClusterIslands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
> **ClusterIslands**\<`TNode`\>: `ReadonlyArray`\<\{ `no`: `number`; `nodes`: readonly `TNode`[]; \}\>
1010
11-
Defined in: [src/abstract/Cluster.ts:96](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L96)
11+
Defined in: [src/abstract/Cluster.ts:97](https://github.com/clickup/ent-framework/blob/master/src/abstract/Cluster.ts#L97)
1212

1313
A type of `ClusterOptions#islands` property. Represents the full list of
1414
Islands and their corresponding Nodes (masters and replicas).

‎internal/build.sh‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
11
#!/bin/bash
22
set -e
33

4-
tsc --build
4+
if [[ -d ../../packages ]]; then
5+
tsc --build
6+
else
7+
cat tsconfig.json | sed 's/"references"/"references-off"/'> tsconfig.json.tmp
8+
trap 'rm -f tsconfig.json.tmp' EXIT
9+
tsc -p tsconfig.json.tmp
10+
fi

‎package-lock.json‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "@clickup/ent-framework",
33
"description": "A PostgreSQL graph-database-alike library with microsharding and row-level security",
4-
"version": "2.20.1",
4+
"version": "2.21.1",
55
"license": "MIT",
66
"keywords": [
77
"postgresql",

‎src/abstract/Cluster.ts‎

Lines changed: 24 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { inspect, types } from "util";
1+
import { types } from "util";
22
import delayMod from "delay";
33
import { Memoize } from "fast-typescript-memoize";
44
import defaults from "lodash/defaults";
@@ -68,25 +68,26 @@ export interface ClusterOptions<TClient extends Client, TNode> {
6868
/** Jitter for shardsDiscoverIntervalMs and reloadIslandsIntervalMs. */
6969
shardsDiscoverIntervalJitter?: MaybeCallable<number>;
7070
/** Used in the following situations:
71-
* 1. If we think that we know Island of a particular Shard, but an attempt to
72-
* access it fails, this means that maybe the Shard is migrating to another
73-
* Island. In this case, we wait a bit and retry that many times. We should
74-
* not do it too many times though, because all DB requests will be blocked
75-
* waiting for the resolution.
76-
* 2. If we sent a write request to a Client, but it appeared that this Client
71+
* 1. If we think that we know the Island of a particular Shard, but an
72+
* attempt to access it fails. This means that maybe the Shard is migrating
73+
* to another Island. So, we wait a bit and retry that many times. We
74+
* should not do it too many times though, because all DB requests will be
75+
* blocked waiting for the resolution.
76+
* 2. If we sent a WRITE request to a Client, but it appeared that this Client
7777
* is a replica, and the master moved to some other Client. In this case,
7878
* we wait a bit and ping all Clients of the Island to refresh, who is
7979
* master and who is replica. */
80-
locateIslandErrorRetryCount?: MaybeCallable<number>;
81-
/** How much time to wait before we retry rediscovering the entire Cluster.
82-
* The time here should be just enough to wait for switching the Shard from
83-
* one Island to another (typically quick). */
84-
locateIslandErrorRediscoverClusterDelayMs?: MaybeCallable<number>;
80+
runOnShardErrorRetryCount?: MaybeCallable<number>;
81+
/** How much time to wait before we retry rediscovering the entire Cluster
82+
* after a Shard-to-Island resolution error. The time here should be just
83+
* enough to wait for switching the Shard from one Island to another
84+
* (typically quick). */
85+
runOnShardErrorRediscoverClusterDelayMs?: MaybeCallable<number>;
8586
/** How much time to wait before sending discover requests to all Clients of
86-
* the Island trying to find the new master. The time here may reach several
87-
* seconds, since some DBs shut down the old master and promote some replica
88-
* to it not simultaneously. */
89-
locateIslandErrorRediscoverIslandDelayMs?: MaybeCallable<number>;
87+
* the Island trying to find the new master (or to reconnect). The time here
88+
* may reach several seconds, since some DBs shut down the old master and
89+
* promote some replica to it not simultaneously. */
90+
runOnShardErrorRediscoverIslandDelayMs?: MaybeCallable<number>;
9091
}
9192

9293
/**
@@ -131,9 +132,9 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
131132
shardsDiscoverIntervalMs: 10000,
132133
shardsDiscoverIntervalJitter: 0.2,
133134
reloadIslandsIntervalMs: NaN,
134-
locateIslandErrorRetryCount: 2,
135-
locateIslandErrorRediscoverClusterDelayMs: 1000,
136-
locateIslandErrorRediscoverIslandDelayMs: 5000,
135+
runOnShardErrorRetryCount: 2,
136+
runOnShardErrorRediscoverClusterDelayMs: 1000,
137+
runOnShardErrorRediscoverIslandDelayMs: 5000,
137138
};
138139

139140
/** The complete registry of all initialized Clients. Cluster nodes may change
@@ -434,8 +435,8 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
434435
body: (island: Island<TClient>, attempt: number) => Promise<TRes>,
435436
onAttemptError?: (error: unknown, attempt: number) => void,
436437
): Promise<TRes> {
437-
let island: Island<TClient>;
438438
for (let attempt = 0; ; attempt++) {
439+
let island: Island<TClient>;
439440
try {
440441
// Re-read Islands map on every retry, because it might change.
441442
const startTime = performance.now();
@@ -463,18 +464,6 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
463464

464465
if (typeof error?.stack === "string") {
465466
const suffix = `\n after ${attempt + 1} attempt${attempt > 0 ? "s" : ""}`;
466-
process.stdout.write(
467-
`DEBUG: ${inspect(
468-
{
469-
error:
470-
error instanceof ClientError
471-
? error
472-
: error?.constructor?.name,
473-
island: island!?.options.clients.map((c) => c.options),
474-
},
475-
{ depth: null, compact: true, breakLength: 100000000 },
476-
)}\n`,
477-
);
478467
if (!error.stack.endsWith(suffix)) {
479468
error.stack = error.stack.trimEnd() + suffix;
480469
}
@@ -485,7 +474,7 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
485474

486475
if (
487476
!(error instanceof ClientError) ||
488-
attempt >= maybeCall(this.options.locateIslandErrorRetryCount)
477+
attempt >= maybeCall(this.options.runOnShardErrorRetryCount)
489478
) {
490479
throw error;
491480
}
@@ -520,7 +509,7 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
520509
@Memoize({ clearOnResolve: true })
521510
private async rediscoverCluster(): Promise<void> {
522511
await delay(
523-
maybeCall(this.options.locateIslandErrorRediscoverClusterDelayMs),
512+
maybeCall(this.options.runOnShardErrorRediscoverClusterDelayMs),
524513
);
525514
// We don't want to wait forever if some Island is completely down.
526515
const startTime = performance.now();
@@ -556,9 +545,7 @@ export class Cluster<TClient extends Client, TNode = DesperateAny> {
556545
*/
557546
@Memoize((island) => island.no, { clearOnResolve: true })
558547
private async rediscoverIsland(island: Island<TClient>): Promise<void> {
559-
await delay(
560-
maybeCall(this.options.locateIslandErrorRediscoverIslandDelayMs),
561-
);
548+
await delay(maybeCall(this.options.runOnShardErrorRediscoverIslandDelayMs));
562549
// We don't want to wait forever if the Island is completely down.
563550
const startTime = performance.now();
564551
await pTimeout(

0 commit comments

Comments
 (0)