Skip to content

feat(sync): add --select flag to sync specific schemas/tables without deleting the rest - #924

Merged
Bl3f merged 4 commits into
mainfrom
claude/nifty-sutherland-774837
Jun 30, 2026
Merged

feat(sync): add --select flag to sync specific schemas/tables without deleting the rest#924
Bl3f merged 4 commits into
mainfrom
claude/nifty-sutherland-774837

Conversation

@ClaireGz

@ClaireGz ClaireGz commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Problem

There is no way to refresh a single table or schema with nao sync without affecting everything else:

  • If you only put one table/schema in the include config, nao sync deletes all other previously-synced tables — the post-sync cleanup (cleanup_stale_paths) removes any schema/table directory that wasn't part of the current run.
  • So the only safe option is to sync everything, which can be slow on large warehouses.

Closes #923

Solution

Add a --select / -s flag to nao sync that restricts the sync to a selection of schemas/tables and skips stale-path cleanup, so nothing outside the selection is removed.

# Sync a single table
nao sync --select analytics.orders

# Sync a whole schema
nao sync --select analytics

# Multiple selections + glob patterns
nao sync -s analytics.orders -s staging.dim_*

# Combine with a specific connection
nao sync -p databases:my-warehouse -s analytics.orders

Behaviour:

  • A pattern with no dot (analytics) selects every table in that schema (analytics.*).
  • A pattern with a dot (analytics.orders) selects a specific table; glob wildcards are supported (staging.dim_*).
  • Selection is applied on top of the existing include/exclude config — it narrows further, never widens.
  • When --select is passed, cleanup_stale_paths is skipped, preserving previously-synced tables/schemas outside the selection.
  • Tables and Snowflake semantic views both respect the selection.

Implementation

  • cli/nao_core/commands/sync/__init__.py — new --select/-s parameter, passed through to the provider.
  • cli/.../providers/databases/provider.py_matches_selection() helper; sync_database() filters tables/semantic views by the selection; provider skips cleanup and prints a Select: … (stale cleanup skipped) note when active.
  • cli/.../providers/base.py, notion, repositories — accept the select kwarg for a consistent interface (only databases honors it).

Tests

  • TestMatchesSelection — schema-only, schema.table, and glob pattern matching.
  • TestSelectSkipsCleanup — cleanup runs without --select, is skipped with it, and select is threaded into sync_database.

make lint passes; the sync test suite is green (218 passed; the integration/ dir is excluded due to a local unixodbc/pyodbc issue unrelated to this change).

🤖 Generated with Claude Code


Note

Low Risk
Scoped CLI and database sync filtering with explicit cleanup skip; other providers unchanged in parent pre_sync database cleanup still runs.

Overview
Adds nao sync -s / --select so you can refresh only chosen database schemas or tables (e.g. analytics, analytics.orders, staging.dim_*) without a full warehouse sync.

Selection is narrowed on top of existing include/exclude config via _matches_selection (schema-only patterns map to schema.*; globs use fnmatch). The databases provider applies it to tables and Snowflake semantic views. When --select is active, cleanup_stale_paths is skipped so previously synced objects outside the selection stay on disk; the CLI prints a note that stale cleanup was skipped.

The select argument is threaded through the SyncProvider.sync interface (repos/Notion accept it but ignore it). Unit tests cover pattern matching and cleanup-on vs cleanup-off behavior.

Reviewed by Cursor Bugbot for commit 9dbd501. Bugbot is set up for automated code reviews on this repo. Configure here.

Review in cubic

… deleting the rest

`nao sync` previously deleted any previously-synced table/schema that wasn't
part of the current run (cleanup_stale_paths). The only way to refresh a
single table was to either narrow `include` (which wipes everything else) or
sync the whole warehouse.

Add a `--select`/`-s` flag that restricts the sync to given schemas/tables and
skips stale-path cleanup so nothing outside the selection is removed:

  nao sync --select analytics.orders
  nao sync --select analytics
  nao sync -s analytics.orders -s staging.dim_*

A pattern without a dot selects a whole schema; glob wildcards are supported.
Selection is applied on top of the existing include/exclude config. Tables and
Snowflake semantic views both respect it.

Closes #923

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧹 Preview Removed

The preview deployment for this PR has been cleaned up.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9dbd501. Configure here.

Comment thread cli/nao_core/commands/sync/providers/databases/provider.py
Comment thread cli/nao_core/commands/sync/providers/databases/provider.py

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 6 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread cli/nao_core/commands/sync/providers/databases/provider.py Outdated
Comment thread cli/nao_core/commands/sync/providers/databases/provider.py Outdated
ClaireGz and others added 2 commits June 25, 2026 14:30
…tern

Surface that `--select` composes with `--provider databases:<conn>` so users
discover they can scope a refresh to one connection and avoid connecting to
their other databases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback on the --select filter:

- Case-sensitivity (P1): fnmatch is case-sensitive on Linux/macOS, so
  `--select analytics.orders` failed to match Snowflake's uppercased
  `ANALYTICS.ORDERS`. Lowercase both sides before matching.
- Catalog-qualified schemas (P2): the previous "dot means schema.table"
  heuristic broke when a schema itself contains a dot (e.g. StarRocks
  `catalog.schema`). Match the pattern against the full name directly and
  treat a pattern as a whole-schema selector via `<pattern>.*`, so both
  `catalog.analytics` and `catalog.analytics.orders` work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@socallmebertille socallmebertille left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny changes can be made and it will look awesome

Comment thread cli/nao_core/commands/sync/providers/notion/provider.py Outdated
Comment thread cli/nao_core/commands/sync/providers/repositories/provider.py Outdated
Comment thread cli/nao_core/commands/sync/providers/base.py Outdated
Comment thread cli/nao_core/commands/sync/providers/base.py Outdated
Comment thread cli/nao_core/commands/sync/__init__.py Outdated
Comment thread cli/nao_core/commands/sync/__init__.py
Address review feedback: --select is database-only, so it no longer pollutes
the abstract SyncProvider.sync() nor the notion/repositories implementations.

- Remove the `select` parameter from the base abstract `sync` and from the
  Notion and Repository providers.
- The CLI passes `select` only to DatabaseSyncProvider (via isinstance) and
  warns when `--select` is given but no databases provider is active.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@socallmebertille socallmebertille left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@Bl3f
Bl3f merged commit 6ea2530 into main Jun 30, 2026
7 checks passed
@Bl3f
Bl3f deleted the claude/nifty-sutherland-774837 branch June 30, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nao sync: add a --select flag to sync only specific schemas/tables without deleting the rest

3 participants