feat(api-gateway): Introduce cache mode option for /cubesql API (#9972)

* add cache option to the query

* Pass CacheMode from /cubesql to backend

* imject CacheMode into more places

* update normalizeQuery with cache mode

* pass cache mode within graphql

* pass new cache mode in sqlApiLoad in API GW

* fix types imports

* update preAggs to use cache option instead of renewQuery

* code polish

* comments with types

* fix query type

* set default  cacheMode = 'stale-if-slow' in normalize()

* more types and polish

* backbone code for 'stale-if-slow' & 'stale-while-revalidate'

* make query cache aware of queryBody.cache === 'must-revalidate'

* First attempt to implement 'no-cache' scenario

* add cache to open api spec and regenerate rust client

* pass cache mode to cubeScan

* cargo clippy/fmt

* Implement background refresh

* add cache mode descriptions

* remove query cache mode from normalize query

* pass cacheMode to getSqlResponseInternal

* remove obsolete

* add cacheMode as input param in orchestratorApi

* open api spec fix

* fix cubesql after introducing cacheMode

* rename cache → cacheMode

* clean up obsolete

* pass cache_mode from SqlApiLoadPayload

* fix important

* move 'no-cache' variant into queryCache.cachedQueryResult()

* remove cacheMode from client query body types (it's incorrect)

* switch RefreshScheduler to use cacheMode instead of renewQuery

* remove obsolete continueWait flag

* fix refresh scheduler

* add fallback to renewQuery in api gw

* fix tests

* Docs

* Deprecation

* refactor api gw: move copy/paste into this.normalizeCacheMode()

* fix tests snapshots

* return cacheMode into query object

* fix tests

* some cleanup (removed renewQuery)

* update cacheMod in graphql

* return back cache as public prop in query

* fix

* lint fix

# Conflicts:
#	packages/cubejs-api-gateway/package.json

* fix subscribe()

* remove cache from subscribe

* fix CacheMode serialization

* refactor: move normalizeQueryCacheMode to normalize

---------

Co-authored-by: Igor Lukanin <[email protected]>

Author: KSDaemon

DEPRECATION.md

@@ -64,8 +64,9 @@ features:
 | Removed    | [`initApp` hook](#initapp-hook)                                                                                                   | v0.35.0    | v0.35.0   |
 | Removed    | [`/v1/run-scheduled-refresh` REST API endpoint](#v1run-scheduled-refresh-rest-api-endpoint)                                       | v0.35.0    | v0.36.0   |
 | Removed    | [Node.js 18](#nodejs-18)                                                                                                          | v0.36.0    | v1.3.0    |
-| Deprecated | [`CUBEJS_SCHEDULED_REFRESH_CONCURRENCY`](#cubejs_scheduled_refresh_concurrency)                                                   | v1.2.7 |           |
+| Deprecated | [`CUBEJS_SCHEDULED_REFRESH_CONCURRENCY`](#cubejs_scheduled_refresh_concurrency)                                                   | v1.2.7    |           |
 | Deprecated | [Node.js 20](#nodejs-20)                                                                                                          | v1.3.0    |           |
+| Deprecated | [`renewQuery` parameter of the `/v1/load` endpoint](#renewquery-parameter-of-the-v1load-endpoint)                                 | v1.3.73   |           |
 
 ### Node.js 8
 
@@ -412,3 +413,8 @@ This environment variable was renamed to [`CUBEJS_SCHEDULED_REFRESH_QUERIES_PER_
 
 Node.js 20 is in maintenance mode from [November 22, 2024][link-nodejs-eol]. This means
 no more new features, only security updates. Please upgrade to Node.js 22 or higher.
+
+### `renewQuery` parameter of the `/v1/load` endpoint
+
+This parameter is deprecated and will be removed in future releases. See [cache control](https://cube.dev/docs/product/apis-integrations/rest-api#cache-control)
+options and use the new `cache` parameter of the `/v1/load` endpoint instead.

docs/pages/product/apis-integrations/rest-api.mdx

@@ -161,7 +161,7 @@ accessible for everyone.
 | API scope | REST API endpoints | Accessible by default? |
 | --- | --- | --- |
 | `meta`    | [`/v1/meta`][ref-ref-meta] | ✅ Yes |
-| `data`    | [`/v1/load`][ref-ref-load] | ✅ Yes |
+| `data`    | [`/v1/load`][ref-ref-load], [`/v1/cubesql`][ref-ref-cubesql] | ✅ Yes |
 | `graphql` | `/graphql` | ✅ Yes |
 | `sql`     | [`/v1/sql`][ref-ref-sql] | ✅ Yes |
 | `jobs`    | [`/v1/pre-aggregations/jobs`][ref-ref-paj] | ❌ No |
@@ -248,9 +248,20 @@ should be unique for each separate request. `spanId` should define user
 interaction span such us `Continue wait` retry cycle and it's value shouldn't
 change during one single interaction.
 
-## Troubleshooting
+## Cache control
 
-### `Continue wait`
+[`/v1/load`][ref-ref-load] and [`/v1/cubesql`][ref-ref-cubesql] endpoints of the REST API
+allow to control the cache behavior. The following querying strategies with regards to
+the cache are supported:
+
+| Strategy | Description |
+| --- | --- |
+| `stale-if-slow`         | If [refresh keys][ref-refresh-keys] are up-to-date, returns cached value. If expired, tries to return fresh value from the data source. If the data source query is slow (hits [`Continue wait`](#continue-wait)), returns stale value from cache. |
+| `stale-while-revalidate`| If [refresh keys][ref-refresh-keys] are up-to-date, returns cached value. If expired, returns stale data from cache and updates cache in background. |
+| `must-revalidate`       | If [refresh keys][ref-refresh-keys] are up-to-date, returns cached value. If expired, always waits for fresh value from the data source, even if slow (hits one or more [`Continue wait`](#continue-wait) intervals). |
+| `no-cache`              | Skips [refresh key][ref-refresh-keys] checks. Always returns fresh data from the data source, regardless of cache or query performance. |
+
+## `Continue wait`
 
 If the request takes too long to be processed, the REST API responds with
 `{ "error": "Continue wait" }` and the status code 200.
@@ -295,6 +306,7 @@ warehouse][ref-data-warehouses].
 [ref-ref-load]: /product/apis-integrations/rest-api/reference#base_pathv1load
 [ref-ref-meta]: /product/apis-integrations/rest-api/reference#base_pathv1meta
 [ref-ref-sql]: /product/apis-integrations/rest-api/reference#base_pathv1sql
+[ref-ref-cubesql]: /product/apis-integrations/rest-api/reference#base_pathv1cubesql
 [ref-ref-paj]: /product/apis-integrations/rest-api/reference#base_pathv1pre-aggregationsjobs
 [ref-security-context]: /product/auth/context
 [ref-graphql-api]: /product/apis-integrations/graphql-api
@@ -313,4 +325,5 @@ warehouse][ref-data-warehouses].
 [ref-traditional-databases]: /product/configuration/data-sources#transactional-databases
 [ref-pre-aggregations]: /product/caching/using-pre-aggregations
 [ref-javascript-sdk]: /product/apis-integrations/javascript-sdk
-[ref-recipe-real-time-data-fetch]: /product/apis-integrations/recipes/real-time-data-fetch
+[ref-recipe-real-time-data-fetch]: /product/apis-integrations/recipes/real-time-data-fetch
+[ref-refresh-keys]: /product/data-modeling/reference/cube#refresh_key

docs/pages/product/apis-integrations/rest-api/query-format.mdx

@@ -41,17 +41,6 @@ The default value is `false`.
 - `timezone`: A [time zone][ref-time-zone] for your query. You can set the
 desired time zone in the [TZ Database Name](https://en.wikipedia.org/wiki/Tz_database)
 format, e.g., `America/Los_Angeles`.
-- `renewQuery`: If `renewQuery` is set to `true`, Cube will renew all
-  [`refreshKey`][ref-schema-ref-preaggs-refreshkey] for queries and query
-  results in the foreground. However, if the
-  [`refreshKey`][ref-schema-ref-preaggs-refreshkey] (or
-  [`refreshKey.every`][ref-schema-ref-preaggs-refreshkey-every]) doesn't
-  indicate that there's a need for an update this setting has no effect. The
-  default value is `false`.
-  > **NOTE**: Cube provides only eventual consistency guarantee. Using a small
-  > [`refreshKey.every`][ref-schema-ref-preaggs-refreshkey-every] value together
-  > with `renewQuery` to achieve immediate consistency can lead to endless
-  > refresh loops and overall system instability.
 - `ungrouped`: If set to `true`, Cube will run an [ungrouped
 query][ref-ungrouped-query].
 - `joinHints`: Query-time [join hints][ref-join-hints], provided as an array of

docs/pages/product/apis-integrations/rest-api/reference.mdx

@@ -13,10 +13,11 @@ By default, it's `/cubejs-api`.
 
 Run the query to the REST API and get the results. 
 
-| Parameter   | Description                                                                                                           |
-| ----------- | --------------------------------------------------------------------------------------------------------------------- |
-| `query`     | Either a single URL encoded Cube [Query](/product/apis-integrations/rest-api/query-format), or an array of queries    |
-| `queryType` | If multiple queries are passed in `query` for [data blending][ref-recipes-data-blending], this must be set to `multi` |
+| Parameter   | Description                                                                                                           | Required |
+| ----------- | --------------------------------------------------------------------------------------------------------------------- | --- |
+| `query`     | Either a single URL encoded Cube [Query](/product/apis-integrations/rest-api/query-format), or an array of queries    | ✅ Yes |
+| `queryType` | If multiple queries are passed in `query` for [data blending][ref-recipes-data-blending], this must be set to `multi` | ❌ No |
+| `cache`     | See [cache control][ref-cache-control]. `stale-if-slow` by default | ❌ No |
 
 Response
 
@@ -319,9 +320,10 @@ This endpoint is part of the [SQL API][ref-sql-api].
 
 </InfoBox>
 
-| Parameter | Description |
-| --- | --- |
-| `query` | The SQL query to run. |
+| Parameter | Description | Required |
+| --- | --- | --- |
+| `query` | The SQL query to run. | ✅ Yes |
+| `cache` | See [cache control][ref-cache-control]. `stale-if-slow` by default | ❌ No |
 
 Response: a stream of newline-delimited JSON objects. The first object contains
 the `schema` property with column names and types. The following objects contain
@@ -639,4 +641,5 @@ Keep-Alive: timeout=5
 [ref-query-wpd]: /product/apis-integrations/queries#query-with-pushdown
 [ref-sql-api]: /product/apis-integrations/sql-api
 [ref-orchestration-api]: /product/apis-integrations/orchestration-api
-[ref-folders]: /product/data-modeling/reference/view#folders
+[ref-folders]: /product/data-modeling/reference/view#folders
+[ref-cache-control]: /product/apis-integrations/rest-api#cache-control

packages/cubejs-api-gateway/openspec.yml

@@ -484,6 +484,13 @@ components:
       properties:
         queryType:
           type: "string"
+        cache:
+          type: "string"
+          enum:
+            - stale-if-slow
+            - stale-while-revalidate
+            - must-revalidate
+            - no-cache
         query:
           type: "object"
           $ref: "#/components/schemas/V1LoadRequestQuery"

packages/cubejs-api-gateway/package.json

@@ -29,6 +29,7 @@
   "dependencies": {
     "@cubejs-backend/native": "1.3.82",
     "@cubejs-backend/shared": "1.3.82",
+    "@cubejs-backend/query-orchestrator": "1.3.82",
     "@ungap/structured-clone": "^0.3.4",
     "assert-never": "^1.4.0",
     "body-parser": "^1.19.0",

packages/cubejs-api-gateway/src/gateway.ts

@@ -12,6 +12,7 @@ import {
   getRealType,
   parseUtcIntoLocalDate,
   QueryAlias,
+  CacheMode,
 } from '@cubejs-backend/shared';
 import {
   ResultArrayWrapper,
@@ -28,6 +29,7 @@ import type {
 } from 'express';
 import { createProxyMiddleware } from 'http-proxy-middleware';
 
+import { QueryBody } from '@cubejs-backend/query-orchestrator';
 import {
   QueryType,
   ApiScopes,
@@ -50,7 +52,9 @@ import {
   PreAggJob,
   PreAggJobStatusItem,
   PreAggJobStatusResponse,
-  SqlApiRequest, MetaResponseResultFn,
+  SqlApiRequest,
+  MetaResponseResultFn,
+  RequestQuery,
 } from './types/request';
 import {
   CheckAuthInternalOptions,
@@ -177,7 +181,13 @@ class ApiGateway {
 
   public constructor(
     protected readonly apiSecret: string,
+    /**
+     * It actually returns a Promise<CompilerApi>
+     */
     protected readonly compilerApi: (ctx: RequestContext) => Promise<any>,
+    /**
+     * It actually returns a Promise<OrchestratorApi>
+     */
     protected readonly adapterApi: (ctx: RequestContext) => Promise<any>,
     protected readonly logger: any,
     protected readonly options: ApiGatewayOptions,
@@ -311,6 +321,7 @@ class ApiGateway {
         context: req.context,
         res: this.resToResultFn(res),
         queryType: req.query.queryType,
+        cacheMode: req.query.cache,
       });
     }));
 
@@ -320,7 +331,8 @@ class ApiGateway {
         query: req.body.query,
         context: req.context,
         res: this.resToResultFn(res),
-        queryType: req.body.queryType
+        queryType: req.body.queryType,
+        cacheMode: req.body.cache,
       });
     }));
 
@@ -329,7 +341,8 @@ class ApiGateway {
         query: req.query.query,
         context: req.context,
         res: this.resToResultFn(res),
-        queryType: req.query.queryType
+        queryType: req.query.queryType,
+        cacheMode: req.query.cache,
       });
     }));
 
@@ -425,7 +438,7 @@ class ApiGateway {
         try {
           await this.assertApiScope('data', req.context?.securityContext);
 
-          await this.sqlServer.execSql(req.body.query, res, req.context?.securityContext);
+          await this.sqlServer.execSql(req.body.query, res, req.context?.securityContext, req.body.cache);
         } catch (e: any) {
           this.handleError({
             e,
@@ -1200,6 +1213,7 @@ class ApiGateway {
     context: RequestContext,
     persistent = false,
     memberExpressions: boolean = false,
+    cacheMode?: CacheMode,
   ): Promise<[QueryType, NormalizedQuery[], NormalizedQuery[]]> {
     let query = this.parseQueryParam(inputQuery);
 
@@ -1240,7 +1254,7 @@ class ApiGateway {
       }
 
       return {
-        normalizedQuery: (normalizeQuery(currentQuery, persistent)),
+        normalizedQuery: (normalizeQuery(currentQuery, persistent, cacheMode)),
         hasExpressionsInQuery
       };
     });
@@ -1285,13 +1299,13 @@ class ApiGateway {
       type: 'Query Rewrite completed',
       queryRewriteId,
       normalizedQueries,
-      duration: new Date().getTime() - startTime,
+      duration: Date.now() - startTime,
       query
     }, context);
 
     normalizedQueries = normalizedQueries.map(q => remapToQueryAdapterFormat(q));
 
-    if (normalizedQueries.find((currentQuery) => !currentQuery)) {
+    if (normalizedQueries.some((currentQuery) => !currentQuery)) {
       throw new Error('queryTransformer returned null query. Please check your queryTransformer implementation');
     }
 
@@ -1637,12 +1651,11 @@ class ApiGateway {
     normalizedQuery: NormalizedQuery,
     sqlQuery: any,
   ): Promise<ResultWrapper> {
-    const queries = [{
+    const queries: QueryBody[] = [{
       ...sqlQuery,
       query: sqlQuery.sql[0],
       values: sqlQuery.sql[1],
-      continueWait: true,
-      renewQuery: normalizedQuery.renewQuery,
+      cacheMode: normalizedQuery.cacheMode,
       requestId: context.requestId,
       context,
       persistent: false,
@@ -1665,8 +1678,7 @@ class ApiGateway {
         ...totalQuery,
         query: totalQuery.sql[0],
         values: totalQuery.sql[1],
-        continueWait: true,
-        renewQuery: normalizedTotal.renewQuery,
+        cacheMode: normalizedTotal.cacheMode,
         requestId: context.requestId,
         context
       });
@@ -1782,12 +1794,11 @@ class ApiGateway {
       this.log({ type: 'Load Request', query, streaming: true }, context);
       const [, normalizedQueries] = await this.getNormalizedQueries(query, context, true);
       const sqlQuery = (await this.getSqlQueriesInternal(context, normalizedQueries))[0];
-      const q = {
+      const q: QueryBody = {
         ...sqlQuery,
         query: sqlQuery.sql[0],
         values: sqlQuery.sql[1],
-        continueWait: true,
-        renewQuery: false,
+        cacheMode: 'stale-if-slow',
         requestId: context.requestId,
         context,
         persistent: true,
@@ -1826,6 +1837,7 @@ class ApiGateway {
       context,
       res,
       apiType = 'rest',
+      cacheMode,
       ...props
     } = request;
     const requestStarted = new Date();
@@ -1847,7 +1859,7 @@ class ApiGateway {
       }, context);
 
       const [queryType, normalizedQueries] =
-        await this.getNormalizedQueries(query, context);
+        await this.getNormalizedQueries(query, context, false, false, cacheMode);
 
       if (
         queryType !== QueryTypeEnum.REGULAR_QUERY &&
@@ -1941,6 +1953,7 @@ class ApiGateway {
     const {
       context,
       res,
+      cacheMode,
     } = request;
     const requestStarted = new Date();
 
@@ -1955,7 +1968,7 @@ class ApiGateway {
       }
 
       const [queryType, normalizedQueries] =
-        await this.getNormalizedQueries(query, context, request.streaming, request.memberExpressions);
+        await this.getNormalizedQueries(query, context, request.streaming, request.memberExpressions, cacheMode);
 
       const compilerApi = await this.getCompilerApi(context);
       let metaConfigResult = await compilerApi.metaConfig(request.context, {
@@ -1970,17 +1983,16 @@ class ApiGateway {
           normalizedQueries.map(q => ({ ...q, disableExternalPreAggregations: request.sqlQuery }))
         );
 
-      let results;
+      let results: any[];
 
       let slowQuery = false;
 
       const streamResponse = async (sqlQuery) => {
-        const q = {
+        const q: QueryBody = {
           ...sqlQuery,
           query: sqlQuery.query || sqlQuery.sql[0],
           values: sqlQuery.values || sqlQuery.sql[1],
-          continueWait: true,
-          renewQuery: false,
+          cacheMode: 'stale-if-slow',
           requestId: context.requestId,
           context,
           persistent: true,
@@ -1995,11 +2007,10 @@ class ApiGateway {
       };
 
       if (request.sqlQuery) {
-        const finalQuery = {
+        const finalQuery: QueryBody = {
           query: request.sqlQuery[0],
           values: request.sqlQuery[1],
-          continueWait: true,
-          renewQuery: normalizedQueries[0].renewQuery,
+          cacheMode: request.cacheMode,
           requestId: context.requestId,
           context,
           ...sqlQueries[0],
@@ -2133,7 +2144,7 @@ class ApiGateway {
     };
   }
 
-  protected parseQueryParam(query): Query | Query[] {
+  protected parseQueryParam(query: RequestQuery | 'undefined'): Query | Query[] {
     if (!query || query === 'undefined') {
       throw new UserError('Query param is required');
     }