fix: Eliminate RwLock in SequenceCounter to avoid read-write deadlock issues

[KKould]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

• fixes: bug: Potential Deadlock in TransformAsyncFunction with Shared RwLock #19392

This issue points out that the current usage of RwLock in SequenceCounter may lead to deadlocks or degraded performance due to blocking. The issue also suggests using tokio::spawn to avoid blocking.

In this PR, instead of introducing a larger refactor(), I aimed to address the root cause of the problem by minimizing the reliance on RwLock. The solution is to initialize the counter using a refill_lock with double-checked locking on the slow path. As a result, locking is only required during the slow-path refill/initialization, while the normal fast-path sequence increment remains lock-free.

This PR also adds two unit test based on the reproduction case from the issue: test_no_stall_when_refill_lock_waiting & test_high_concurrency_fast_path_progress_during_refill_contention.

Special thanks to @YZL0v3ZZ for the careful review and valuable feedback.

Tips:

• As far as I know, the logic between next_val_v0 and next_val_v1 is not fully consistent yet, and the migration is still ongoing. A larger refactor at this stage could further increase the complexity. Therefore, this change tries to preserve the original structure and behavior as much as possible.
• We may advance the cached counter (claim_up_to) before awaiting the remote fetch to avoid duplicates under concurrency. If the fetch fails, the claimed cached values are skipped, which can introduce small sequence gaps.
  • Sequences are not guaranteed to be gapless, and fetch errors are expected to be rare

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[KKould]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c6018ad90c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[KKould]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d06f74c98

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[KKould]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37f0d12ff5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".