feat: make track deletes work across execution (#5517)

The track deletes features currently assume the entire dataset is being
fetched/saved within a single execution.
Which means it isn't compatible with checkpoints and fetching/saving
across multiple executions.
To solve this problem, this PR is introducing a way to explicitly define
the start and the end of the track deletes interval, storing the
starting point in a special checkpoint that survive across executions,
instead of assuming the start is always the beginning of the current
execution

cc @bastienbeurier to agree on the naming
`trackDeletesStart/trackDeletesEnd`

ex:
```
exec: async (nango) => {
        await nango.trackDeletesStart('MyModel');

        ...
        await nango.batchSave([...], 'MyModel');
        ...

        const deleted = await nango.trackDeletesEnd('MyModel');
}
```

<!-- Summary by @propel-code-bot -->

---

The runner stores a per-model delete-window checkpoint keyed off the
sync checkpoint and, when the window closes, uses it to invoke deletion
of outdated records before clearing the checkpoint.

<details>
<summary><strong>Key Changes</strong></summary>

• Added `trackDeletesStart`/`trackDeletesEnd` to `NangoSyncBase` and
`NangoSyncRunner`, with per-model checkpoint keys and
`deleteOutdatedRecords` using stored `syncJobId`
• Refactored `Checkpointing` in
`packages/runner/lib/sdk/checkpointing.ts` to manage per-key state and
accept `key` arguments for checkpoint operations
• Expanded CLI parser/compiler validations to track `trackDeletes` calls
per model and enforce ordering around `batchSave`
• Updated docs and examples to use `trackDeletesStart`/`trackDeletesEnd`
and mark `deleteRecordsFromPreviousExecutions` as deprecated
• Reset behavior in `packages/shared/lib/clients/orchestrator.ts` now
hard-deletes all checkpoints with a key prefix

</details>

<details>
<summary><strong>Possible Issues</strong></summary>

• A run that calls `trackDeletesStart` but exits before
`trackDeletesEnd` will leave the delete-window checkpoint in place for
future runs
• The per-key `stateByKey` map in `Checkpointing` is not pruned, which
could grow in long-lived processes with many dynamic keys
• Full reset now returns an error if `hardDeleteCheckpoints` fails,
which could block user-initiated resets

</details>

---
*This summary was automatically generated by @propel-code-bot*

Author: TBonnin

docs/implementation-guides/use-cases/syncs/checkpoints.mdx

@@ -152,7 +152,7 @@ After the initial run completes, subsequent runs are fast and incremental — on
 Not every sync can be incremental. You need to fetch the full dataset when:
 
 - **The external API doesn't support filtering by modification date** — Some APIs have no way to ask "give me everything that changed since X." In this case, you must fetch all records on every run.
-- **You need automated delete detection** — To detect deleted records automatically using [`deleteRecordsFromPreviousExecutions()`](/reference/functions#detect-deletions-automatically), Nango must compare the full current dataset against the previous one. This requires a full sync. (If the external API exposes deleted records, you can use [`batchDelete()`](/reference/functions#delete-records) with checkpoints instead — see [deletion detection guide](/implementation-guides/use-cases/syncs/deletion-detection).)
+- **You need automated delete detection** — To detect deleted records automatically using [`trackDeletesStart`/`trackDeletesEnd`](/reference/functions#detect-deletions-automatically), Nango compares the records that existed before `trackDeletesStart` against what was saved between `trackDeletesStart` and `trackDeletesEnd`. This requires fetching the complete dataset between the two calls. (If the external API exposes deleted records, you can use [`batchDelete()`](/reference/functions#delete-records) instead — see [deletion detection guide](/implementation-guides/use-cases/syncs/deletion-detection).)
 
 For small datasets (e.g., a list of Slack users for an organization with fewer than 100 employees), a full sync on every run is perfectly fine:
 
@@ -173,7 +173,7 @@ export default createSync({
 As datasets grow, full syncs become unscalable — taking longer to run, triggering rate limits, and consuming more compute and memory. If your dataset has more than a few thousand records and the API supports filtering by date or cursor, use checkpoints.
 
 <Warning>
-Checkpoints are currently incompatible with `deleteRecordsFromPreviousExecutions()` because that function requires comparing the full dataset between consecutive runs. Support for checkpoints with full syncs is coming soon.
+`deleteRecordsFromPreviousExecutions()` is incompatible with checkpoints because it requires comparing the full dataset between consecutive runs. Use `trackDeletesStart`/`trackDeletesEnd` instead, which explicitly bounds the deletion detection window and supports checkpointed syncs.
 </Warning>
 
 # Avoiding memory overuse

docs/implementation-guides/use-cases/syncs/deletion-detection.mdx

@@ -84,17 +84,13 @@ export default createSync({
 
 Syncs that fetch all records on every run can automatically detect deletions.
 
-Nango can therefore detect removals by computing the diff between two consecutive result sets. Enable this behaviour by calling the `deleteRecordsFromPreviousExecutions` function. ([full reference](/reference/functions#detect-deletions-automatically)).
-
-<Note>
-  `deleteRecordsFromPreviousExecutions` does not work with checkpoint-based syncs because fetching the data incrementally prevents performing a diff and automatically detecting deletions.
-</Note>
+Nango detects removals by computing the diff between what existed before `trackDeletesStart` and what was saved between `trackDeletesStart` and `trackDeletesEnd`. ([full reference](/reference/functions#detect-deletions-automatically)).
 
 ### Example sync with deletion detection
 
 ```ts
-import { createSync } from 'nango';
-import * as z from 'zod';
+import { createSync } from ‘nango’;
+import * as z from ‘zod’;
 
 const TicketSchema = z.object({
   id: z.string(),
@@ -103,55 +99,57 @@ const TicketSchema = z.object({
 });
 
 export default createSync({
-  description: 'Fetch all help-desk tickets',
-  frequency: 'every day',
-  endpoints: [{ method: 'GET', path: '/tickets', group: 'Tickets' }],
+  description: ‘Fetch all help-desk tickets’,
+  frequency: ‘every day’,
+  endpoints: [{ method: ‘GET’, path: ‘/tickets’, group: ‘Tickets’ }],
   models: { Ticket: TicketSchema },
 
   exec: async (nango) => {
+    // Mark the start of deletion tracking
+    await nango.trackDeletesStart(‘Ticket’);
+
     const tickets = await nango.paginate<{ id: string; subject: string; status: string }>({
-      endpoint: '/tickets',
-      paginate: { type: 'cursor', cursorPathInResponse: 'next', cursorNameInRequest: 'cursor', responsePath: 'tickets' }
+      endpoint: ‘/tickets’,
+      paginate: { type: ‘cursor’, cursorPathInResponse: ‘next’, cursorNameInRequest: ‘cursor’, responsePath: ‘tickets’ }
     });
 
     for await (const page of tickets) {
-      await nango.batchSave(page, 'Ticket');
+      await nango.batchSave(page, ‘Ticket’);
     }
 
-    // Delete records that don't exist anymore
-    await nango.deleteRecordsFromPreviousExecutions('Ticket');
+    // Detect and mark deleted records
+    await nango.trackDeletesEnd(‘Ticket’);
   }
 });
 ```
 
 ### How the algorithm works
 
-1. During the execution, Nango stores the list of record IDs.
-2. When calling `deleteRecordsFromPreviousExecutions`, Nango compares the new list with the old one.
-3. Any records missing in the new list are marked as deleted (soft delete). They remain accessible from the Nango cache, but with `record._metadata.deleted === true`.
+1. When `trackDeletesStart` is called, Nango marks the beginning of the deletion tracking window for the model.
+2. Records saved with `batchSave` between `trackDeletesStart` and `trackDeletesEnd` are tracked.
+3. When `trackDeletesEnd` is called, Nango compares what existed before `trackDeletesStart` with what was saved in the window.
+4. Any records missing from the new dataset are marked as deleted (soft delete). They remain accessible from the Nango cache, but with `record._metadata.deleted === true`.
 
 <Warning>
-**Be careful with exception handling when using** `deleteRecordsFromPreviousExecutions`
+**Be careful with exception handling when using** `trackDeletesStart`/`trackDeletesEnd`
 
 Nango only performs deletion detection (the “diff”) if a sync run completes successfully without any uncaught exceptions.
 
-If you’re using `deleteRecordsFromPreviousExecutions`, exception handling becomes critical:
-- If your sync doesn’t fetch the full dataset, but still call `deleteRecordsFromPreviousExecutions` (e.g. you catch and swallow an exception), Nango will attempt the diff on an incomplete dataset.
+Exception handling is critical:
+- If your sync doesn’t fetch the full dataset between the two calls (e.g. you catch and swallow an exception), Nango will attempt the diff on an incomplete dataset.
 - This leads to false positives, where valid records are mistakenly considered deleted.
 
 **What You Should Do**
 
-If a failure prevents full data retrieval, make sure the sync run fails and `deleteRecordsFromPreviousExecutions` is not being called:
+If a failure prevents full data retrieval, make sure the sync run fails and `trackDeletesEnd` is not being called:
 - Let exceptions bubble up and interrupt the run.
 - If you’re using `try/catch`, re-throw exceptions that indicate incomplete data.
 </Warning>
 
 <Tip>
-**How to use** `deleteRecordsFromPreviousExecutions` **safely**
-
-If some records are incorrectly marked as deleted due to calling `deleteRecordsFromPreviousExecutions` improperly, you can trigger a full resync (via the UI or API) to restore the correct data state.
+**How to use** `trackDeletesStart`/`trackDeletesEnd` **safely**
 
-However, because `deleteRecordsFromPreviousExecutions` relies on logic in your sync functions, mistakes there can lead to false deletions.
+If some records are incorrectly marked as deleted, you can trigger a full resync (via the UI or API) to restore the correct data state.
 
 We strongly recommend not performing irreversible destructive actions (like hard-deleting records in your system) based solely on deletions reported by Nango. A full resync should always be able to recover from issues.
 </Tip>
@@ -160,7 +158,7 @@ We strongly recommend not performing irreversible destructive actions (like hard
 
 | Symptom                                                                    | Likely cause                                                                             |
 | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
-| Records that still exist in the source API are shown as `deleted` in Nango | sync didn't save all records (silent failures) before calling `deleteRecordsFromPreviousExecutions` |
+| Records that still exist in the source API are shown as `deleted` in Nango | sync didn’t save all records (silent failures) between `trackDeletesStart` and `trackDeletesEnd` |
 | You never see deleted records                                              | Check if deletion detection is implemented for the sync.                                 |
 
 <Tip>

docs/reference/api/sync/prune-records.mdx

@@ -15,7 +15,7 @@ This is useful for compliance requirements, ensuring Nango doesn't hold onto you
 Pruning is not the same as marking a record as deleted from the external API.
 This endpoint prunes data from Nango’s cache only. It does not delete anything on the external API, and it is not the same as [detecting a deletion from the source](/implementation-guides/use-cases/syncs/implement-a-sync#deletion-detection).
 
-If you need to tell your customers that a record was deleted on the external API while keeping its last-known payload in cache, use `batchDelete` or `deleteRecordsFromPreviousExecutions` in your sync functions instead.
+If you need to tell your customers that a record was deleted on the external API while keeping its last-known payload in cache, use `batchDelete` or `trackDeletesStart`/`trackDeletesEnd` in your sync functions instead.
 </Tip>
 
 ## Cursor behavior

docs/reference/functions.mdx

@@ -27,6 +27,9 @@ description: "Full reference of the SDK available in Nango Functions."
       },
 
       exec: async (nango) => {
+        // Mark the start of deletion tracking
+        await nango.trackDeletesStart('GithubIssueDemo');
+
         // Fetch issues from GitHub.
         const res = await nango.get({
             endpoint: '/repos/NangoHQ/interactive-demo/issues?labels=demo&sort=created&direction=asc'
@@ -40,8 +43,8 @@ description: "Full reference of the SDK available in Nango Functions."
         // Persist issues to the Nango cache.
         await nango.batchSave(issues, 'GithubIssueDemo');
 
-        // Delete records that don't exist anymore
-        await nango.deleteRecordsFromPreviousExecutions('GithubIssueDemo')
+        // Detect and mark deleted records
+        await nango.trackDeletesEnd('GithubIssueDemo');
       },
     });
     ```
@@ -172,7 +175,7 @@ Read more about [integration functions](/guides/primitives/functions) to underst
 </ResponseField>
 
 <ResponseField name="trackDeletes" type="boolean" deprecated>
-  [DEPRECATED] Instead use `await nango.deleteRecordsFromPreviousExecutions('modelName')` inside the sync `exec` function.
+  [DEPRECATED] Instead use `await nango.trackDeletesStart('modelName')` and `await nango.trackDeletesEnd('modelName')` inside the sync `exec` function.
 
   When `trackDeletes` is set to `true`, Nango automatically detects deleted records **during full syncs only** and marks them as deleted in each record’s metadata (soft delete). These records remain stored in the cache.
 
@@ -883,23 +886,27 @@ await nango.batchDelete(githubIssuesToDelete, 'GitHubIssue');
 
 ### Detect deletions automatically
 
-Automatically detects and marks records as deleted by comparing the current sync execution with the previous one.
+Automatically detects and marks records as deleted by comparing what existed before `trackDeletesStart` with what was saved between `trackDeletesStart` and `trackDeletesEnd`.
 
 <Tip>
 This does not remove cached payloads.
 
-`nango.deleteRecordsFromPreviousExecutions()` is used to mark records as deleted in Nango because they were not returned by the external API. Nango may still keep the last-known payload so your customer can react to the deletion event.
+`nango.trackDeletesStart()`/`nango.trackDeletesEnd()` are used to mark records as deleted in Nango because they were not returned by the external API. Nango may still keep the last-known payload so your customer can react to the deletion event.
 
 If you want to permanently remove data from Nango storage for cost or compliance reasons, use [record pruning instead](/reference/api/sync/prune-records).
 </Tip>
 
 ```js
-await nango.deleteRecordsFromPreviousExecutions('ModelName');
+await nango.trackDeletesStart('ModelName');
+
+// ... fetch and save all records ...
+
+await nango.trackDeletesEnd('ModelName');
 ```
 
-This function should be called at the end of your sync execution, after all records have been saved with `batchSave`. Nango will compare the records saved in the current execution with those from the previous execution and mark any missing records as deleted.
+Call `trackDeletesStart` at the beginning of your sync execution, before fetching any data. Call `trackDeletesEnd` after all records have been saved with `batchSave`. Nango will compare the records that existed before `trackDeletesStart` with those saved in the window and mark any missing records as deleted.
 
-**Parameters**
+**Parameters** (both functions)
 
 <Expandable>
   <ResponseField name="modelType" type="string" required>
@@ -909,11 +916,14 @@ This function should be called at the end of your sync execution, after all reco
 
 **Important considerations:**
 
-- Only use within syncs that fetch the complete dataset
-- Call this function only after successfully saving all records
-- If your sync fails or doesn't fetch the complete dataset, avoid calling this function as it may cause false deletions
+- Only use within syncs that fetch the complete dataset between `trackDeletesStart` and `trackDeletesEnd`
+- If your sync fails or doesn't fetch the complete dataset, avoid calling `trackDeletesEnd` as it may cause false deletions
 - Records are soft deleted (marked with `_metadata.deleted = true`) and remain in the cache
 
+<Note>
+`deleteRecordsFromPreviousExecutions` is deprecated. Use `trackDeletesStart`/`trackDeletesEnd` instead.
+</Note>
+
 For more details on deletion detection strategies, see the [detecting deletes guide](/implementation-guides/use-cases/syncs/deletion-detection.mdx).
 
 ### Update records

packages/cli/example/github/syncs/fetchIssues.ts

@@ -32,6 +32,8 @@ const sync = createSync({
 
     // Sync execution
     exec: async (nango) => {
+        await nango.trackDeletesStart('GithubIssue');
+
         const repos = await getAllRepositories(nango);
 
         for (const repo of repos) {
@@ -65,7 +67,7 @@ const sync = createSync({
             }
         }
 
-        await nango.deleteRecordsFromPreviousExecutions('GithubIssue');
+        await nango.trackDeletesEnd('GithubIssue');
     },
 
     // Webhook handler

packages/cli/lib/services/__snapshots__/model.service.unit.test.ts.snap

@@ -474,7 +474,12 @@ export declare class NangoSync<TCheckpoint = Checkpoint> extends NangoAction {
     batchUpdate<T extends object>(results: T[], model: string): Promise<boolean | null>;
     getMetadata<T = Metadata>(): Promise<T>;
     setMergingStrategy(merging: { strategy: 'ignore_if_modified_after' | 'override' }, model: string): Promise<void>;
+    /**
+     * @deprecated please use trackDeletesStart and trackDeletesEnd
+     */
     deleteRecordsFromPreviousExecutions(model: string): Promise<{ deletedKeys: string[] }>;
+    trackDeletesStart(model: string): Promise<void>;
+    trackDeletesEnd(model: string): Promise<{ deletedKeys: string[] }>;
     getRecordsByIds<K = string | number, T = any>(ids: K[], model: string): Promise<Map<K, T>>;
     getCheckpoint(): Promise<TCheckpoint | null>;
     saveCheckpoint(checkpoint: TCheckpoint): Promise<void>;