Add filter to GetMinUnfinishedWatermark query

[changliiu]

To improve the performance of the query for GetUnfinishedMinWatermark, we can add a filter to the query.
Note that the query run on a cache to refresh every 1s. We can remember the cached value every time and use the value as a filter in the next time query.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.65%. Comparing base (bea0444) to head (7cc7fcc).
Report is 39 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #34965       +/-   ##
=============================================
+ Coverage     54.50%   68.65%   +14.15%     
  Complexity     1479     1479               
=============================================
  Files          1012      194      -818     
  Lines        160667     8111   -152556     
  Branches       1079     1079               
=============================================
- Hits          87573     5569    -82004     
+ Misses        70991     2158    -68833     
+ Partials       2103      384     -1719     

Flag	Coverage Δ
go	?
java	68.65% <ø> (+0.01%)	⬆️
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Assigning reviewers:

R: @shunping for label python.
R: @m-trieu for label java.
R: @Abacn for label build.
R: @fozzie15 for label kafka.
R: @nielm for label spanner.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions]

Reminder, please take a look at this pr: @shunping @m-trieu @Abacn @fozzie15 @nielm

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label python.
R: @ahmedabu98 for label java.
R: @damccorm for label build.
R: @johnjcasey for label kafka.
R: @nielm for label spanner.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[nielm]

waiting on author

(PR is still in draft state)