Add timeouts for spanner queries

[n-h-diaz]

Currently we were using the default Spanner timeout which is 1hr, so even when our API timed out, we'd still keep trying to process the queries which uses excess resources. (This contributed to the recent Spanner outage.) Instead we'd like to cancel pending requests and free up resources.

This PR adds custom timeouts, including

• Using the API timeout for API requests (

  mixer/esp/endpoints.yaml.tmpl

  Lines 37 to 43 in 9e4bb86

  	rules:
  	# Default timeout for the ESP and mixer GRPC server.
  	- selector: "datacommons.Mixer.*"
  	deadline: 60
  	# Longer timeout for the Sparql endpoint.
  	- selector: "datacommons.Mixer.Query"
  	deadline: 300

  ) with a fallback to 60s which is the timeout for most endpoints
• 10s for Spanner timestamp polling

Also adds some logging for queries that timeout to help us track inefficient queries for performance improvements

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where Spanner queries could run for extended periods, consuming excessive resources even after the initiating API requests had timed out. By introducing explicit timeouts for all Spanner operations, the system can now proactively cancel long-running queries, improving resource efficiency and overall stability. The changes also enhance observability by logging details of timed-out queries, aiding in performance debugging.

Highlights

• Custom Spanner Query Timeouts: Implemented custom timeouts for Spanner queries to prevent resource exhaustion from long-running operations. This includes using the API timeout for API requests with a 60-second fallback, and a dedicated 10-second timeout for Spanner timestamp polling.
• Improved Error Logging: Added detailed logging for Spanner queries that time out, including the SQL statement and the timeout duration. This will help in identifying and optimizing inefficient queries.
• Resource Management: By canceling pending requests when timeouts are exceeded, the system can free up resources that would otherwise be consumed by queries that are no longer relevant or have already been abandoned by the calling API.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively adds timeouts to Spanner queries, which is a crucial improvement for resource management and system stability. The approach of propagating API deadlines to Spanner queries is well-implemented, and the fallback timeout is a good safety measure. The added logging for timed-out queries, including the SQL statement, will be very valuable for performance monitoring and debugging. I have one suggestion to refactor the duplicated timeout-checking logic into a helper function to improve code clarity and maintainability, with a caveat to consider the deprecation status of the Spanner integration.