feat(social-listening): stable mentions paging across rebuilds (GH-1918)

[audigregorie]

Summary

The social-listening mentions feed paged by numeric offset over a Snowflake table that is rebuilt by an hourly full refresh — absolute row positions shift mid-session, so deep Load More paging could skip or repeat rows, and a hard cap stopped browsing past ~100,000 rows. This PR switches the feed to keyset (cursor) pagination: each page returns an opaque page_token derived from the last shown row's content, and the next page continues strictly after that row, so paging stays stable across rebuilds and the cursor chain serves the feed to its end.

Resolves #1918

Behavior changes

Before	After
As users clicked Load More, the feed requested rows by position ("rows 200–300"). Because the mention list is rebuilt hourly, positions could shift while someone scrolled — rows could repeat or vanish — and browsing stopped entirely after ~100,000 rows with the total shown as capped.	The feed bookmarks the last row shown and continues from that exact row, so pages stay consistent even while the list refreshes. The feed itself is no longer depth-capped — the cursor chain serves it to its true end; past 500 rendered rows the list keeps the page responsive by suggesting narrower filters instead of rendering further — and the "Showing X of Y" total is no longer capped.

Technical changes

packages/shared/src/interfaces/social-listening.interface.ts, packages/shared/src/constants/social-listening.constants.ts — Shared contracts

• Added SocialListeningFeedCursor — the (MENTION_TS, _KEY) compound sort key of the last shown row, content-relative so it survives full-refresh rebuilds
• SocialListeningPaginationParams is now { pageSize, cursor? }; SocialListeningFeedRequest sends page_size / page_token on the wire; SocialListeningFeedResponse returns an optional page_token (absent = feed exhausted)
• Removed MENTION_MAX_FEED_OFFSET — cursor paging needs no depth cap

apps/lfx-one/src/server/helpers/social-listening-params.helper.ts — Feed param parsing

• parseSocialListeningPagination reads page_size (must be an integer, clamped 1–100; ?page_size= parses to 0 and clamps to 1 deterministically) and decodes page_token into a keyset cursor; malformed or tampered tokens return 400 instead of silently restarting the feed
• base64url page_token codec; the cursor ts is kept verbatim (no timezone reinterpretation) so the bound value round-trips byte-for-byte
• Timestamp validity check generalized into isValidFeedTimestamp (shape regex + Date.parse round-trip), now shared by readBeforeTs and the cursor ts

apps/lfx-one/src/server/helpers/strict-query-param.helper.ts, apps/lfx-one/src/server/helpers/committee-activity-query.helper.ts — Strict query reader

• New shared getStrictStringQueryParam: a repeated query param (?page_token=a&page_token=b, which Express's qs parser turns into an array) now 400s instead of being treated as absent and silently restarting pagination
• Extracted the copy that was inlined in the committee-activity helper into its own module, keeping every consumer's Vitest spec importable without pulling Angular-only runtime code through validation.helper.ts

apps/lfx-one/src/server/services/social-listening.service.ts, apps/lfx-one/src/server/controllers/social-listening.controller.ts — Keyset feed query

• Feed SQL replaces LIMIT/OFFSET with a keyset predicate selecting rows strictly after the cursor under MENTION_TS DESC, _KEY DESC, fetching page_size + 1 rows — the extra row is the hasMore signal and the issued token rides the last kept row
• Explicit NULL-timestamp handling for Snowflake's NULLs-first DESC ordering: a dated cursor excludes the already-paged NULL group, while a NULL-ts cursor continues the NULL group and flows into every dated row
• Controller passes pageSize/cursor through and logs has_cursor instead of raw offsets

apps/lfx-one/src/app/modules/dashboards/social-listening/social-listening.component.ts, apps/lfx-one/src/app/modules/dashboards/social-listening/components/mentions-list/mentions-list.component.ts — Feed UI

• 100-row server windows now chain via the previous window's final page_token (feedChainToken) instead of computed offsets — a window waits while its predecessor fills and treats a tokenless completed window as the feed's end
• The two-phase window fill (fast first page, background remainder) chains phase 2 off phase 1's token, making the phases sequential without offsets
• servableTotal is the uncapped count total; hasMore end-detection reads the missing token; "Data as of" is pinned to window 0's first stamp so a mid-scan rebuild can't contradict rows already on screen
• "Showing X of Y" never prints a total below the rendered count (the session-cached count can lag a rebuild)

apps/lfx-one/src/server/helpers/social-listening-params.helper.spec.ts, apps/lfx-one/src/server/services/social-listening.service.spec.ts, apps/lfx-one/src/app/modules/dashboards/social-listening/social-listening.component.spec.ts — Tests

• Token decode/validation, page_size clamping, and strict-reader 400s
• Keyset SQL predicates and token issuance/absence at the feed's end
• Cursor-chain window behavior in the component pipeline

Breaking changes

The GET /api/social-listening/mentions-feed query contract changed: limit / offset were replaced by page_size / page_token. The endpoint is consumed only by this app's own frontend, which is updated in the same PR, so there is no cross-deploy ordering concern.

[cursor]

PR Summary

Medium Risk
Large, coordinated change to feed SQL, HTTP contract, and pagination state; incorrect cursor or NULL ordering logic could still skip, duplicate, or mislabel rows, though coverage is extensive and the API is app-internal.

Overview
Replaces offset/limit mentions feed paging with keyset cursors (page_size / opaque page_token) so Load More continues after the last shown row and stays stable across hourly Snowflake full refreshes. The old ~100k row depth cap (MENTION_MAX_FEED_OFFSET) is removed; the UI chains 100-row windows off each window’s final token instead of computed offsets.

Backend: Feed SQL uses LIMIT page_size + 1, keyset predicates under MENTION_TS DESC NULLS FIRST, _KEY DESC, and token issuance from the last kept row; param parsing decodes/validates tokens (malformed → 400). getStrictStringQueryParam is shared so repeated query keys 400 instead of silently resetting pagination. Mark-all newest lookup uses allTime + MENTION_TS IS NOT NULL so NULL timestamps are not treated as newest.

Frontend: Two-phase window fills chain phase 2 off phase 1’s token; hasMore / renderCapped lean on missing tokens and failed phase-2 fills; “Data as of” pins to window 0’s first stamp. The mentions list “Showing X of Y” label floors Y at rendered count and shows X of X at feed end when cached totals are stale.

Breaking: GET …/mentions-feed no longer accepts limit/offset (same-PR client only).

^{Reviewed by Cursor Bugbot for commit bb19774. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Essentials

Run ID: 455d5695-873c-4aa4-8b3b-5e121ada5404

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🚀 Deployment Status

Your branch has been deployed to: https://ui-pr-2384.dev.v2.cluster.linuxfound.info

Deployment Details:

• Environment: Development
• Namespace: ui-pr-2384
• ArgoCD App: ui-pr-2384

The deployment will be automatically removed when this PR is closed.

[Copilot]

Pull request overview

Replaces offset-based social-listening pagination with stable Snowflake keyset cursors across hourly rebuilds.

Changes:

• Adds strict cursor token parsing and shared pagination contracts.
• Implements compound-key Snowflake pagination and token chaining.
• Updates Angular feed state, counts, watermark handling, and tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/shared/src/interfaces/social-listening.interface.ts	Defines cursor request and response contracts.
packages/shared/src/constants/social-listening.constants.ts	Removes the server offset ceiling.
apps/lfx-one/src/server/services/social-listening.service.ts	Implements keyset SQL and token issuance.
apps/lfx-one/src/server/services/social-listening.service.spec.ts	Tests cursor predicates and page boundaries.
apps/lfx-one/src/server/helpers/strict-query-param.helper.ts	Adds strict scalar query parsing.
apps/lfx-one/src/server/helpers/social-listening-params.helper.ts	Parses and validates pagination tokens.
apps/lfx-one/src/server/helpers/social-listening-params.helper.spec.ts	Tests token validation and page-size handling.
apps/lfx-one/src/server/helpers/committee-activity-query.helper.ts	Reuses the strict query reader.
apps/lfx-one/src/server/controllers/social-listening.controller.ts	Passes cursor pagination into the service.
apps/lfx-one/src/app/modules/dashboards/social-listening/social-listening.component.ts	Chains feed windows and detects cursor exhaustion.
apps/lfx-one/src/app/modules/dashboards/social-listening/social-listening.component.spec.ts	Tests cursor-chain UI behavior.
apps/lfx-one/src/app/modules/dashboards/social-listening/components/mentions-list/mentions-list.component.ts	Updates total-count presentation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

apps/lfx-one/src/server/helpers/social-listening-params.helper.ts:177

• The token is base64url-encoded JSON with shape validation only, so tampering is not detected: a caller can encode any other valid { ts, key } and it will be accepted. This contradicts both this comment and the PR's claim that tampered tokens return 400. Either authenticate the payload (for example, with a server-side MAC) or narrow the documented contract to malformed/wrong-shaped tokens.

[dealako]

Hey @audigregorie, thanks for the thorough writeup on this one — the cursor-based redesign is a real fix for a genuine bug class (offset drift under hourly rebuilds), and the NULL-timestamp handling in the keyset predicate is the kind of detail that's easy to get subtly wrong and you clearly worked through it carefully. This is a re-review round: the five Copilot findings from earlier rounds (stale-high total at feed end, initHasMore/renderCapped conflating the 500-row DOM cap with true cursor exhaustion, missing explicit NULLS FIRST, a stale ordering assertion, and the all-time mark-all NULL-ts probe bug) all check out fixed and tested against the current head.

Three independent reviewers covered Security/Privacy, Correctness/Performance/Tests, and Style/API/Docs across the full main...HEAD diff.

Issue count

• 🔴 Blocking: 0
• 🟡 Minor: 1: page_token decode has no length cap on the raw token or the decoded key, unlike every other user-supplied string field in the same helper file
• ⚪ Nit: 1: controller success log drops page_size when it logs has_cursor, unlike the service's logging
• ❔ Question: 0

Bot reconciliation

• CodeRabbit: auto-review is disabled on this repo (skipped), nothing to reconcile.
• Cursor Bugbot: summary-only, correctly characterizes the change; no line-level findings to reconcile.
• Copilot (this round, on bb19774): flags that the page_token codec does shape-validation only, not a MAC, so a caller can forge any well-shaped {ts, key} payload and it'll be accepted as "untampered." Disagree that this is a security gap: the cursor only controls sort position within a feed query that still applies the caller's own scope/filters — there's no privilege boundary here to bypass, since a user can already page to any row in their own feed. The PR's "tampered tokens return 400" claim is about malformed/wrong-shape input, which is what's actually guarded. No action needed.

Findings

1. [minor] page_token decode path has no length cap — apps/lfx-one/src/server/helpers/social-listening-params.helper.ts, decodeMentionFeedPageToken. Every other user string field in this file (search, sourceProjectId, platform, language, filter values) is capped at FILTER_VALUE_MAX_LENGTH/SEARCH_MAX_LENGTH before use. The new page_token path has no equivalent: getStrictStringQueryParam accepts an arbitrary-length string, and candidate.key is only checked for typeof === 'string' && !== '', no upper bound. Not an injection vector (it's bound as a query param, not interpolated), but it's an unbounded-input path that skipped this file's own established convention, and it lands in a Snowflake query on every request. Fix: cap the raw token length before base64/JSON decode, and cap candidate.key.length (e.g. to FILTER_VALUE_MAX_LENGTH), rejecting with the same 400 on overflow.
2. [nit] Controller log omits page_size — apps/lfx-one/src/server/controllers/social-listening.controller.ts. The success log now includes has_cursor but not the page_size value, while the service's logs do include it. Minor observability inconsistency; take-it-or-leave-it.

Decision: ✅ Approved with minor comments

[dealako]

Approving. Full write-up: #2384 (comment)

No blocking or privacy findings. Two low-severity items (a page_token length-cap gap, a logging nit) noted in the summary comment — neither blocks merge.