Add flag to divert V2 traffic to new backend

[n-h-diaz]

This is to help understand latency (especially website) on dev(/tbd new test instance)

Currently only set to divert Node and Observation, since those are the only ones with feature parity at the moment

Set to always use Spanner for local

[gemini-code-assist]

Summary of Changes

Hello @n-h-diaz, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a mechanism to gradually or fully divert V2 API traffic for specific endpoints (Node and Observation) to a new backend dispatcher. The primary goal is to facilitate latency analysis and performance evaluation of the new backend in development and test environments, ensuring a smooth transition and feature parity before broader deployment. By using a configurable fraction, it allows for controlled experimentation and risk mitigation.

Highlights

• New Feature Flag for V2 Traffic Diversion: Introduced a new feature flag, V2DivertFraction, which is a float64 value between 0.0 and 1.0, to control the percentage of V2 API traffic that should be diverted to a new backend dispatcher.
• Conditional Traffic Diversion for V2 Endpoints: Implemented logic within the V2Node and V2Observation API handlers to conditionally divert requests to a new dispatcher backend based on the V2DivertFraction flag. This uses math/rand to randomly select requests for diversion.
• Local Development Configuration: Configured the local.yaml feature flag file to set V2DivertFraction to 1.0, ensuring that all V2Node and V2Observation requests are diverted to the new backend in local development environments.
• Feature Flag Validation: Added validation rules for V2DivertFraction to ensure its value is within the valid range [0.0, 1.0] and that if it's greater than 0, both UseSpannerGraph and EnableV3 flags must be true, linking V2 diversion to the V3 infrastructure.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a feature flag V2DivertFraction to divert a portion of V2 API traffic to a new dispatcher backend, enabling testing with production traffic. However, the current implementation includes logging statements that record entire incoming request objects, which can lead to the leakage of sensitive information (PII) and poses a Denial of Service (DoS) risk. It is crucial to address these logging practices by sanitizing the logged data. Additionally, the review suggests refactoring duplicated validation logic in internal/featureflags/featureflags.go and simplifying redundant checks in the traffic diversion logic within internal/server/handler_v2.go for improved maintainability.

[keyurva]

Exciting!

[keyurva]

nit: Should we add a log message when traffic is diverted? If we do, add for obs as well.

[n-h-diaz]

Good idea - aded a basic slog message for now. As a follow up I can try to add more detailed metrics for tracking this