Add BulkVariableInfo to dispatcher

[n-h-diaz]

This PR adds BulkVariableInfo to the dispatcher and corresponding datasources

It currently only supports the Spanner implementation, which calls GetProvenanceSummary

This produces a BulkVariableInfoResponse

Note: Not all fields are populated. This only fills a subset of fields required by new frontend design for StatVarExplorer
In particular, this excludes

• PlaceTypeSummary (both top level and per series)
• Provenance release_frequency
• In the case of the remote datasource these fields will be dropped from the merged response. As a follow up I'll deprecate them. (Eventually remote datasource will move to v2 call which will not include these)

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new API endpoint, BulkVariableInfo, to the system. This enhancement allows for retrieving bulk variable information, specifically tailored to support the new StatVarExplorer frontend design. The initial implementation supports the Spanner data source, fetching data using GetProvenanceSummary. The changes also include merging logic to combine responses from multiple data sources, ensuring a unified data structure.

Highlights

• New API Endpoint: Introduces a new BulkVariableInfo endpoint to the dispatcher, enabling retrieval of bulk variable information.
• Data Source Integration: Integrates the new endpoint with Spanner data source, utilizing GetProvenanceSummary to fetch data.
• Data Merging: Implements logic to merge multiple BulkVariableInfoResponse from different data sources.
• Frontend Alignment: Focuses on populating a subset of fields required by the new StatVarExplorer frontend design.

Changelog

• internal/merger/merger.go
  • Added MergeMultiBulkVariableInfo function to merge multiple BulkVariableInfoResponse messages.
• internal/merger/merger_test.go
  • Added TestMergeMultiBulkVariableInfoResponse to test the merging of BulkVariableInfoResponse messages.
• internal/server/datasource/datasource.go
  • Added BulkVariableInfo method to the DataSource interface.
• internal/server/datasources/datasources.go
  • Implemented BulkVariableInfo function to fetch and merge data from multiple sources.
• internal/server/dispatcher/dispatcher.go
  • Added BulkVariableInfo request type and handler function.
• internal/server/redis/processor.go
  • Added BulkVariableInfo to the list of request types that can be cached.
• internal/server/remote/client.go
  • Added BulkVariableInfo method to the RemoteClient.
• internal/server/remote/datasource.go
  • Implemented BulkVariableInfo method for the RemoteDataSource.
• internal/server/spanner/client.go
  • Renamed GetVariableMetadata to GetProvenanceSummary.
• internal/server/spanner/datasource.go
  • Implemented BulkVariableInfo method to fetch data from Spanner.
• internal/server/spanner/dsutil.go
  • Added generateBulkVariableInfoResponse function to generate the BulkVariableInfoResponse.
• internal/server/spanner/golden/datasource_test.go
  • Renamed GetVariableMetadata to GetProvenanceSummary in mockSpannerClient.
• internal/server/spanner/golden/query/get_provenance_summary.json
  • Updated golden file for GetProvenanceSummary query.
• internal/server/spanner/golden/query_builder/get_provenance_summary.sql
  • Updated golden file for GetProvenanceSummary query builder.
• internal/server/spanner/golden/query_builder_test.go
  • Renamed TestGetVariableMetadataQuery to TestGetProvenanceSummaryQuery.
• internal/server/spanner/golden/query_cases_test.go
  • Renamed variableMetadataTestCases to provenanceSummaryTestCases.
• internal/server/spanner/golden/query_test.go
  • Renamed TestGetVariableMetadata to TestGetProvenanceSummary.
• internal/server/spanner/query.go
  • Renamed GetVariableMetadata to GetProvenanceSummary and updated related logic.
• internal/server/spanner/statements.go
  • Updated getCacheData query to include provenance.
• internal/sqldb/datasource.go
  • Added BulkVariableInfo method to the SQLDataSource.

Activity

• The PR introduces a new API endpoint for retrieving bulk variable information.
• The implementation currently supports only the Spanner data source.
• The changes are focused on supporting the new StatVarExplorer frontend design.
• Some fields like PlaceTypeSummary and Provenance release_frequency are excluded and will be deprecated in a follow-up.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for BulkVariableInfo across the dispatcher and various datasources, with an initial implementation for Spanner. The changes are well-structured, introducing the new API endpoint and wiring it through the different layers of the application, including the merger, remote client, and caching processor. I've left a couple of minor suggestions for improving code clarity and consistency.

[keyurva]

Thanks!